初始化项目，由ModelHub XC社区提供模型
Model: perletter/dot-125m Source: Original Platform
This commit is contained in:
ModelHub XC
129
README.md
Normal file
129
README.md
Normal file
@@ -0,0 +1,129 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
tags:
|
||||
- text-generation
|
||||
- llama
|
||||
- pretraining
|
||||
- from-scratch
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# Dot-125M
|
||||
|
||||
Dot-125M is a 125M-parameter (133.7M actual), Llama-3-style decoder-only transformer,
|
||||
pretrained **entirely from scratch** — no fine-tuning or continued pretraining from an
|
||||
existing checkpoint — by **[Perletter](https://perletter.com)**, part of Chirping Waves
|
||||
Limited (Ireland).
|
||||
|
||||
It was trained on 2.0B tokens (4 epochs over a 500M-token filtered, deduplicated sample
|
||||
of [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)) on a single
|
||||
consumer laptop GPU.
|
||||
|
||||
This is a **base (pretrained) language model** — it has not been instruction-tuned,
|
||||
RLHF'd, or chat-templated. It completes text; it does not reliably follow instructions
|
||||
or hold a conversation.
|
||||
|
||||
## Model details
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Parameters | 133.7M |
|
||||
| Architecture | Llama-3-style decoder-only, GQA, RoPE, RMSNorm, SwiGLU, tied embeddings |
|
||||
| Layers / heads / KV heads | 12 / 12 / 4 |
|
||||
| Hidden size | 960 |
|
||||
| Context length | 512 |
|
||||
| Vocab size | 16,384 (byte-level BPE, trained from scratch on the training split) |
|
||||
| Training tokens | 2.0B (4 epochs × 500M-token corpus) |
|
||||
| Training data | [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (`sample-10BT`), quality-filtered + exact/near-deduplicated, English only |
|
||||
| License | Apache 2.0 |
|
||||
|
||||
## Benchmarks
|
||||
|
||||
Compared against GPT-2-small (124M, ~10B training tokens) — see full write-up for
|
||||
methodology.
|
||||
|
||||
**Bits-per-byte (primary metric, tokenizer-fair comparison), on a held-out test split:**
|
||||
|
||||
| | bpb | ppl |
|
||||
|---|---|---|
|
||||
| **Dot-125M** | **1.0142** | 20.64 |
|
||||
| GPT-2-small | 1.0281 | 27.19 |
|
||||
|
||||
**lm-evaluation-harness:**
|
||||
|
||||
| task | Dot-125M | GPT-2-small |
|
||||
|---|---|---|
|
||||
| arc_easy (acc) | 48.23% | 43.81% |
|
||||
| hellaswag (acc_norm) | 30.86% | 31.14% |
|
||||
| piqa (acc) | 61.43% | 62.89% |
|
||||
| winogrande (acc) | 49.57% | 51.62% |
|
||||
| lambada_openai (acc) | 23.02% | 32.56% |
|
||||
|
||||
Mixed on the individual benchmark tasks (stronger on arc_easy, weaker on
|
||||
lambada_openai's long-range prediction — expected given the token/context budget:
|
||||
Dot-125M saw 500M unique tokens across 4 epochs at 512 context vs. GPT-2's ~10B tokens
|
||||
single-pass at 1024 context), but wins on the primary bits-per-byte metric.
|
||||
|
||||
**Quantization** (GGUF, via llama.cpp): Q8_0 stays within 0.01% bpb of full-precision
|
||||
f16. Q4_K_M is available but falls back to a different quant scheme for most tensors
|
||||
(this model's hidden size isn't a multiple of 256, the k-quant block size) — still only
|
||||
~0.17% bpb degradation vs. f16, but not "true" Q4_K_M. Q8_0/Q4_0/Q5_0/Q5_1 are the
|
||||
quant types this model size supports natively.
|
||||
|
||||
## Intended use
|
||||
|
||||
Research, experimentation, and demonstration of from-scratch small-LM pretraining. Not
|
||||
instruction-tuned — do not expect chat-assistant behavior out of the box. Not suitable
|
||||
for production use requiring factual reliability, safety filtering, or instruction
|
||||
following without further fine-tuning.
|
||||
|
||||
## How to use
|
||||
|
||||
**Transformers (safetensors):**
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
tok = AutoTokenizer.from_pretrained("perletter/Dot-125M")
|
||||
model = AutoModelForCausalLM.from_pretrained("perletter/Dot-125M")
|
||||
|
||||
inputs = tok("The history of the internet", return_tensors="pt")
|
||||
out = model.generate(**inputs, max_new_tokens=50)
|
||||
print(tok.decode(out[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
**llama.cpp (GGUF):**
|
||||
```bash
|
||||
llama-cli -m model-Q8_0.gguf -p "The history of the internet" -n 50
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- Small model, small training budget — general knowledge and reasoning are limited
|
||||
compared to larger contemporary models.
|
||||
- English only.
|
||||
- Base model only — no safety fine-tuning, no RLHF, no instruction-tuning. It will
|
||||
complete harmful, biased, or false text if prompted toward it, the same as any
|
||||
unaligned base LM.
|
||||
- 512-token context window.
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0 — the model weights are freely available for any use, including commercial,
|
||||
with no attribution requirement beyond the license notice. See `LICENSE`.
|
||||
|
||||
The training code/pipeline used to produce this model is **not** included in this
|
||||
release.
|
||||
|
||||
## Citation
|
||||
|
||||
```
|
||||
@misc{dot125m2026,
|
||||
title = {Dot-125M},
|
||||
author = {Perletter, part of Chirping Waves Limited},
|
||||
year = {2026},
|
||||
url = {https://perletter.com}
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user