130 lines
4.3 KiB
Markdown
130 lines
4.3 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
library_name: transformers
|
|||
|
|
tags:
|
|||
|
|
- text-generation
|
|||
|
|
- llama
|
|||
|
|
- pretraining
|
|||
|
|
- from-scratch
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Dot-125M
|
|||
|
|
|
|||
|
|
Dot-125M is a 125M-parameter (133.7M actual), Llama-3-style decoder-only transformer,
|
|||
|
|
pretrained **entirely from scratch** — no fine-tuning or continued pretraining from an
|
|||
|
|
existing checkpoint — by **[Perletter](https://perletter.com)**, part of Chirping Waves
|
|||
|
|
Limited (Ireland).
|
|||
|
|
|
|||
|
|
It was trained on 2.0B tokens (4 epochs over a 500M-token filtered, deduplicated sample
|
|||
|
|
of [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)) on a single
|
|||
|
|
consumer laptop GPU.
|
|||
|
|
|
|||
|
|
This is a **base (pretrained) language model** — it has not been instruction-tuned,
|
|||
|
|
RLHF'd, or chat-templated. It completes text; it does not reliably follow instructions
|
|||
|
|
or hold a conversation.
|
|||
|
|
|
|||
|
|
## Model details
|
|||
|
|
|
|||
|
|
| | |
|
|||
|
|
|---|---|
|
|||
|
|
| Parameters | 133.7M |
|
|||
|
|
| Architecture | Llama-3-style decoder-only, GQA, RoPE, RMSNorm, SwiGLU, tied embeddings |
|
|||
|
|
| Layers / heads / KV heads | 12 / 12 / 4 |
|
|||
|
|
| Hidden size | 960 |
|
|||
|
|
| Context length | 512 |
|
|||
|
|
| Vocab size | 16,384 (byte-level BPE, trained from scratch on the training split) |
|
|||
|
|
| Training tokens | 2.0B (4 epochs × 500M-token corpus) |
|
|||
|
|
| Training data | [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (`sample-10BT`), quality-filtered + exact/near-deduplicated, English only |
|
|||
|
|
| License | Apache 2.0 |
|
|||
|
|
|
|||
|
|
## Benchmarks
|
|||
|
|
|
|||
|
|
Compared against GPT-2-small (124M, ~10B training tokens) — see full write-up for
|
|||
|
|
methodology.
|
|||
|
|
|
|||
|
|
**Bits-per-byte (primary metric, tokenizer-fair comparison), on a held-out test split:**
|
|||
|
|
|
|||
|
|
| | bpb | ppl |
|
|||
|
|
|---|---|---|
|
|||
|
|
| **Dot-125M** | **1.0142** | 20.64 |
|
|||
|
|
| GPT-2-small | 1.0281 | 27.19 |
|
|||
|
|
|
|||
|
|
**lm-evaluation-harness:**
|
|||
|
|
|
|||
|
|
| task | Dot-125M | GPT-2-small |
|
|||
|
|
|---|---|---|
|
|||
|
|
| arc_easy (acc) | 48.23% | 43.81% |
|
|||
|
|
| hellaswag (acc_norm) | 30.86% | 31.14% |
|
|||
|
|
| piqa (acc) | 61.43% | 62.89% |
|
|||
|
|
| winogrande (acc) | 49.57% | 51.62% |
|
|||
|
|
| lambada_openai (acc) | 23.02% | 32.56% |
|
|||
|
|
|
|||
|
|
Mixed on the individual benchmark tasks (stronger on arc_easy, weaker on
|
|||
|
|
lambada_openai's long-range prediction — expected given the token/context budget:
|
|||
|
|
Dot-125M saw 500M unique tokens across 4 epochs at 512 context vs. GPT-2's ~10B tokens
|
|||
|
|
single-pass at 1024 context), but wins on the primary bits-per-byte metric.
|
|||
|
|
|
|||
|
|
**Quantization** (GGUF, via llama.cpp): Q8_0 stays within 0.01% bpb of full-precision
|
|||
|
|
f16. Q4_K_M is available but falls back to a different quant scheme for most tensors
|
|||
|
|
(this model's hidden size isn't a multiple of 256, the k-quant block size) — still only
|
|||
|
|
~0.17% bpb degradation vs. f16, but not "true" Q4_K_M. Q8_0/Q4_0/Q5_0/Q5_1 are the
|
|||
|
|
quant types this model size supports natively.
|
|||
|
|
|
|||
|
|
## Intended use
|
|||
|
|
|
|||
|
|
Research, experimentation, and demonstration of from-scratch small-LM pretraining. Not
|
|||
|
|
instruction-tuned — do not expect chat-assistant behavior out of the box. Not suitable
|
|||
|
|
for production use requiring factual reliability, safety filtering, or instruction
|
|||
|
|
following without further fine-tuning.
|
|||
|
|
|
|||
|
|
## How to use
|
|||
|
|
|
|||
|
|
**Transformers (safetensors):**
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
tok = AutoTokenizer.from_pretrained("perletter/Dot-125M")
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained("perletter/Dot-125M")
|
|||
|
|
|
|||
|
|
inputs = tok("The history of the internet", return_tensors="pt")
|
|||
|
|
out = model.generate(**inputs, max_new_tokens=50)
|
|||
|
|
print(tok.decode(out[0], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**llama.cpp (GGUF):**
|
|||
|
|
```bash
|
|||
|
|
llama-cli -m model-Q8_0.gguf -p "The history of the internet" -n 50
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
- Small model, small training budget — general knowledge and reasoning are limited
|
|||
|
|
compared to larger contemporary models.
|
|||
|
|
- English only.
|
|||
|
|
- Base model only — no safety fine-tuning, no RLHF, no instruction-tuning. It will
|
|||
|
|
complete harmful, biased, or false text if prompted toward it, the same as any
|
|||
|
|
unaligned base LM.
|
|||
|
|
- 512-token context window.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Apache 2.0 — the model weights are freely available for any use, including commercial,
|
|||
|
|
with no attribution requirement beyond the license notice. See `LICENSE`.
|
|||
|
|
|
|||
|
|
The training code/pipeline used to produce this model is **not** included in this
|
|||
|
|
release.
|
|||
|
|
|
|||
|
|
## Citation
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
@misc{dot125m2026,
|
|||
|
|
title = {Dot-125M},
|
|||
|
|
author = {Perletter, part of Chirping Waves Limited},
|
|||
|
|
year = {2026},
|
|||
|
|
url = {https://perletter.com}
|
|||
|
|
}
|
|||
|
|
```
|