130 lines
4.3 KiB
Markdown
130 lines
4.3 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
library_name: transformers
|
||
tags:
|
||
- text-generation
|
||
- llama
|
||
- pretraining
|
||
- from-scratch
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# Dot-125M
|
||
|
||
Dot-125M is a 125M-parameter (133.7M actual), Llama-3-style decoder-only transformer,
|
||
pretrained **entirely from scratch** — no fine-tuning or continued pretraining from an
|
||
existing checkpoint — by **[Perletter](https://perletter.com)**, part of Chirping Waves
|
||
Limited (Ireland).
|
||
|
||
It was trained on 2.0B tokens (4 epochs over a 500M-token filtered, deduplicated sample
|
||
of [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)) on a single
|
||
consumer laptop GPU.
|
||
|
||
This is a **base (pretrained) language model** — it has not been instruction-tuned,
|
||
RLHF'd, or chat-templated. It completes text; it does not reliably follow instructions
|
||
or hold a conversation.
|
||
|
||
## Model details
|
||
|
||
| | |
|
||
|---|---|
|
||
| Parameters | 133.7M |
|
||
| Architecture | Llama-3-style decoder-only, GQA, RoPE, RMSNorm, SwiGLU, tied embeddings |
|
||
| Layers / heads / KV heads | 12 / 12 / 4 |
|
||
| Hidden size | 960 |
|
||
| Context length | 512 |
|
||
| Vocab size | 16,384 (byte-level BPE, trained from scratch on the training split) |
|
||
| Training tokens | 2.0B (4 epochs × 500M-token corpus) |
|
||
| Training data | [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (`sample-10BT`), quality-filtered + exact/near-deduplicated, English only |
|
||
| License | Apache 2.0 |
|
||
|
||
## Benchmarks
|
||
|
||
Compared against GPT-2-small (124M, ~10B training tokens) — see full write-up for
|
||
methodology.
|
||
|
||
**Bits-per-byte (primary metric, tokenizer-fair comparison), on a held-out test split:**
|
||
|
||
| | bpb | ppl |
|
||
|---|---|---|
|
||
| **Dot-125M** | **1.0142** | 20.64 |
|
||
| GPT-2-small | 1.0281 | 27.19 |
|
||
|
||
**lm-evaluation-harness:**
|
||
|
||
| task | Dot-125M | GPT-2-small |
|
||
|---|---|---|
|
||
| arc_easy (acc) | 48.23% | 43.81% |
|
||
| hellaswag (acc_norm) | 30.86% | 31.14% |
|
||
| piqa (acc) | 61.43% | 62.89% |
|
||
| winogrande (acc) | 49.57% | 51.62% |
|
||
| lambada_openai (acc) | 23.02% | 32.56% |
|
||
|
||
Mixed on the individual benchmark tasks (stronger on arc_easy, weaker on
|
||
lambada_openai's long-range prediction — expected given the token/context budget:
|
||
Dot-125M saw 500M unique tokens across 4 epochs at 512 context vs. GPT-2's ~10B tokens
|
||
single-pass at 1024 context), but wins on the primary bits-per-byte metric.
|
||
|
||
**Quantization** (GGUF, via llama.cpp): Q8_0 stays within 0.01% bpb of full-precision
|
||
f16. Q4_K_M is available but falls back to a different quant scheme for most tensors
|
||
(this model's hidden size isn't a multiple of 256, the k-quant block size) — still only
|
||
~0.17% bpb degradation vs. f16, but not "true" Q4_K_M. Q8_0/Q4_0/Q5_0/Q5_1 are the
|
||
quant types this model size supports natively.
|
||
|
||
## Intended use
|
||
|
||
Research, experimentation, and demonstration of from-scratch small-LM pretraining. Not
|
||
instruction-tuned — do not expect chat-assistant behavior out of the box. Not suitable
|
||
for production use requiring factual reliability, safety filtering, or instruction
|
||
following without further fine-tuning.
|
||
|
||
## How to use
|
||
|
||
**Transformers (safetensors):**
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
tok = AutoTokenizer.from_pretrained("perletter/Dot-125M")
|
||
model = AutoModelForCausalLM.from_pretrained("perletter/Dot-125M")
|
||
|
||
inputs = tok("The history of the internet", return_tensors="pt")
|
||
out = model.generate(**inputs, max_new_tokens=50)
|
||
print(tok.decode(out[0], skip_special_tokens=True))
|
||
```
|
||
|
||
**llama.cpp (GGUF):**
|
||
```bash
|
||
llama-cli -m model-Q8_0.gguf -p "The history of the internet" -n 50
|
||
```
|
||
|
||
## Limitations
|
||
|
||
- Small model, small training budget — general knowledge and reasoning are limited
|
||
compared to larger contemporary models.
|
||
- English only.
|
||
- Base model only — no safety fine-tuning, no RLHF, no instruction-tuning. It will
|
||
complete harmful, biased, or false text if prompted toward it, the same as any
|
||
unaligned base LM.
|
||
- 512-token context window.
|
||
|
||
## License
|
||
|
||
Apache 2.0 — the model weights are freely available for any use, including commercial,
|
||
with no attribution requirement beyond the license notice. See `LICENSE`.
|
||
|
||
The training code/pipeline used to produce this model is **not** included in this
|
||
release.
|
||
|
||
## Citation
|
||
|
||
```
|
||
@misc{dot125m2026,
|
||
title = {Dot-125M},
|
||
author = {Perletter, part of Chirping Waves Limited},
|
||
year = {2026},
|
||
url = {https://perletter.com}
|
||
}
|
||
```
|