319 lines
9.9 KiB
Markdown
319 lines
9.9 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- causal-lm
|
|
- pretraining
|
|
- continued-pretraining
|
|
- long-context
|
|
- math
|
|
- code
|
|
- from-scratch
|
|
- fineweb-edu
|
|
- dclm-edu
|
|
- single-gpu
|
|
- l20
|
|
datasets:
|
|
- HuggingFaceFW/fineweb-edu
|
|
- HuggingFaceTB/smollm-corpus
|
|
- HuggingFaceTB/finemath
|
|
- HuggingFaceTB/stack-edu
|
|
- HuggingFaceTB/dclm-edu
|
|
---
|
|
|
|
# l20-edu-135m
|
|
|
|
`l20-edu-135m` is a 134.5M-parameter Llama-style causal language model
|
|
pretrained from scratch on 10B FineWeb-Edu tokens using a single NVIDIA L20 GPU.
|
|
|
|
This is a **base model**, not an instruction-tuned chat model. It is released as
|
|
a small-model pretraining artifact for research, evaluation, reproducibility,
|
|
continued pretraining, and downstream supervised fine-tuning.
|
|
|
|
## Latest Results
|
|
|
|
The latest completed checkpoint is the Stage 2 continued-pretraining checkpoint:
|
|
|
|
`step-001850-stage2-math-code`
|
|
|
|
Stage 2 starts from the 8K long-context checkpoint and continues training on a
|
|
1B-token replay mixture with high-quality educational web data, FineMath,
|
|
Stack-Edu code, Cosmopedia-style textbook data, and replay from the earlier
|
|
educational corpus. The checkpoint and full eval artifacts are included in this
|
|
repository under:
|
|
|
|
- `step-001850-stage2-math-code/`
|
|
- `eval_results/step-001850_sota/`
|
|
- `eval_results/current_vs_smollm_summary.md`
|
|
- `training_artifacts/`
|
|
|
|
### Stage 2 Six-Task Benchmark
|
|
|
|
Zero-shot `lm-eval` results for the Stage 2 checkpoint:
|
|
|
|
| Task | Metric | Score |
|
|
| --- | --- | ---: |
|
|
| ARC-Challenge | acc_norm | 0.2765 |
|
|
| ARC-Easy | acc_norm | 0.5059 |
|
|
| HellaSwag | acc_norm | 0.3272 |
|
|
| LAMBADA OpenAI | acc | 0.2540 |
|
|
| PIQA | acc_norm | 0.6224 |
|
|
| WinoGrande | acc | 0.5099 |
|
|
|
|
### Current SmolLM Target Comparison
|
|
|
|
The table below uses the same target tasks as the current SmolLM comparison
|
|
script in `training_artifacts/scripts/eval_smollm_benchmark.sh`.
|
|
|
|
| Task | Metric | ours-stage2 | SmolLM-135M | SmolLM2-135M |
|
|
| --- | --- | ---: | ---: | ---: |
|
|
| ARC-Challenge | acc_norm | 0.2807 | 0.2875 | 0.2969 |
|
|
| ARC-Easy | acc_norm | 0.5059 | 0.5610 | 0.5854 |
|
|
| HellaSwag | acc_norm | 0.3242 | 0.4265 | 0.4301 |
|
|
| LAMBADA OpenAI | acc | pending rerun | 0.3757 | 0.4289 |
|
|
| PIQA | acc_norm | 0.6137 | 0.6823 | 0.6839 |
|
|
| WinoGrande | acc | pending rerun | 0.5272 | 0.5249 |
|
|
|
|
Mean over available matched metrics:
|
|
|
|
| Model | Mean |
|
|
| --- | ---: |
|
|
| ours-stage2 | 0.4311 |
|
|
| SmolLM-135M | 0.4767 |
|
|
| SmolLM2-135M | 0.4917 |
|
|
|
|
The largest remaining gaps are HellaSwag, PIQA, and ARC-Easy. The next planned
|
|
Stage 3 recipe therefore shifts the data mix toward DCLM-Edu and high-quality
|
|
general educational web data instead of only increasing math/code data.
|
|
|
|
## Model Details
|
|
|
|
| Field | Value |
|
|
| --- | --- |
|
|
| Model type | Decoder-only causal LM |
|
|
| Architecture | Llama-style Transformer |
|
|
| Parameters | 134,515,008 |
|
|
| Layers | 30 |
|
|
| Hidden size | 576 |
|
|
| FFN size | 1536 |
|
|
| Attention | 9 query heads, 3 key/value heads |
|
|
| Context length | 2048 |
|
|
| Tokenizer | `HuggingFaceTB/SmolLM2-135M` tokenizer |
|
|
| Training data | `HuggingFaceFW/fineweb-edu`, `sample-10BT` |
|
|
| Training budget | 10,001,252,352 planned tokens |
|
|
| Tokens / parameter | 74.35 |
|
|
| Final checkpoint | step 18,928 |
|
|
| Hardware | single NVIDIA L20 GPU |
|
|
|
|
## Training Recipe
|
|
|
|
| Field | Value |
|
|
| --- | --- |
|
|
| Sequence length | 2048 tokens |
|
|
| Micro batch size | 6 sequences |
|
|
| Gradient accumulation | 43 steps |
|
|
| Global batch size | 258 sequences |
|
|
| Tokens / optimizer step | 528,384 |
|
|
| Max steps | 18,928 |
|
|
| Optimizer | AdamW |
|
|
| Peak learning rate | `4e-4` |
|
|
| LR schedule | linear warmup + cosine decay to `0.1 * peak_lr` |
|
|
| Warmup | 1,000 steps |
|
|
| Weight decay | 0.1 |
|
|
| Adam beta1 / beta2 | 0.9 / 0.95 |
|
|
| Gradient clipping | 1.0 |
|
|
| Precision | bfloat16 |
|
|
| Gradient checkpointing | enabled |
|
|
| Torch compile | enabled |
|
|
| Eval interval | 500 steps |
|
|
| Checkpoint interval | 1,000 steps, keeping the last 2 regular checkpoints |
|
|
|
|
The training config is included in the repository as
|
|
`configs/l20_135m_deepthin.yaml`. A fuller recipe is available in
|
|
`docs/training_recipe.md`.
|
|
|
|
## Runtime And Cost Notes
|
|
|
|
| Field | Value |
|
|
| --- | --- |
|
|
| GPU | NVIDIA L20 |
|
|
| Reported GPU memory | 46,068 MiB total |
|
|
| Mean logged throughput | 38,541 tokens/s |
|
|
| Mean logged throughput after step 1,000 | 38,587 tokens/s |
|
|
| Estimated training time | about 72 GPU-hours |
|
|
| Final checkpoint mtime | 2026-05-19 05:04:22 +0800 |
|
|
|
|
Exact peak VRAM and billing cost were not logged. A reproducible cost estimate
|
|
is `72 GPU-hours * L20_hourly_rate`, excluding storage, network egress, idle
|
|
time, and engineering time.
|
|
|
|
Known run issues:
|
|
|
|
- Transient dataset mirror read timeouts occurred near the end of training and
|
|
recovered through retry.
|
|
- The final perplexity command printed `loss=2.8731 perplexity=17.69`, then hit
|
|
a Python finalization crash. The metric is reported because it was printed
|
|
before process teardown.
|
|
- The training process printed `terminate called without an active exception`
|
|
after the final checkpoint had been written. The final checkpoint was
|
|
load-tested with `AutoModelForCausalLM`.
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
repo = "AliceYin/l20-edu-135m"
|
|
tokenizer = AutoTokenizer.from_pretrained(repo)
|
|
model = AutoModelForCausalLM.from_pretrained(repo)
|
|
|
|
prompt = "The capital of France is"
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=40,
|
|
do_sample=True,
|
|
temperature=0.8,
|
|
top_p=0.95,
|
|
)
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
Because this is a base model, completion-style prompts work better than
|
|
instruction/chat prompts.
|
|
|
|
## Evaluation
|
|
|
|
Final validation:
|
|
|
|
- Loss: `2.8731`
|
|
- Perplexity: `17.69`
|
|
|
|
Training artifacts:
|
|
|
|
- `training/training_metrics.csv`
|
|
- `training/training_summary.json`
|
|
- `training/loss_curve_zoom.png`
|
|
- `training/training_curves.png`
|
|
|
|
|
|
|
|
|
|
|
|
Zero-shot `lm-eval` results for the final checkpoint:
|
|
|
|
| Task | Metric | Score |
|
|
| --- | --- | ---: |
|
|
| ARC-Challenge | acc_norm | 0.2765 |
|
|
| ARC-Easy | acc_norm | 0.5059 |
|
|
| HellaSwag | acc_norm | 0.3272 |
|
|
| LAMBADA OpenAI | acc | 0.2540 |
|
|
| PIQA | acc_norm | 0.6224 |
|
|
| WinoGrande | acc | 0.5099 |
|
|
|
|
### Benchmark Protocol
|
|
|
|
Candidate and public baseline numbers were run with the same evaluation setup:
|
|
|
|
| Field | Setting |
|
|
| --- | --- |
|
|
| Harness | EleutherAI `lm-evaluation-harness` |
|
|
| Harness version | `0.4.12` |
|
|
| Backend | `--model hf` |
|
|
| Device | `cuda:0` |
|
|
| Dtype | `bfloat16` |
|
|
| Batch size | `auto`, resolved to 64 |
|
|
| Few-shot setting | zero-shot |
|
|
| Dataset limit | none; full task datasets |
|
|
| Samples | `--log_samples` enabled |
|
|
| Seeds | harness defaults: Python 0, NumPy 1234, Torch 1234, few-shot 1234 |
|
|
| Candidate numbers | self-run on the final checkpoint |
|
|
| Baseline numbers | self-run through `scripts/eval_public_baselines.sh`, not copied from leaderboards |
|
|
|
|
The public baselines were evaluated with the same harness version, task list,
|
|
zero-shot setting, dtype, device class, batch policy, and comparison parser. They
|
|
were **not** evaluated with the same tokenizer or model context length; each
|
|
public model used its own released tokenizer and Hugging Face model config. This
|
|
is therefore a public-model benchmark comparison, not a controlled
|
|
same-tokenizer architecture comparison.
|
|
|
|
Public baseline win rates on the same task set:
|
|
|
|
| Baseline | Wins / Tasks | Win Rate |
|
|
| --- | ---: | ---: |
|
|
| GPT-2 small | 5 / 6 | 0.833 |
|
|
| OPT-125M | 4 / 6 | 0.667 |
|
|
| GPT-Neo-125M | 4 / 6 | 0.667 |
|
|
| Cerebras-GPT-111M | 6 / 6 | 1.000 |
|
|
| Pythia-160M | 6 / 6 | 1.000 |
|
|
| SmolLM-135M | 0 / 6 | 0.000 |
|
|
| SmolLM2-135M | 0 / 6 | 0.000 |
|
|
|
|
The full comparison artifacts are included in this repository under:
|
|
|
|
- `eval/comparison.md`
|
|
- `eval/comparison.json`
|
|
- `docs/evaluation_report.md`
|
|
|
|
### Contamination Status
|
|
|
|
No full benchmark contamination pass is claimed for this release. The project
|
|
repository includes `scripts/check_contamination.py` and
|
|
`scripts/sample_training_text.py`, but a separate audit against ARC, HellaSwag,
|
|
PIQA, LAMBADA, and WinoGrande samples was not completed before release. Because
|
|
the model was trained on a public web-scale FineWeb-Edu slice, benchmark overlap
|
|
cannot be ruled out without that audit.
|
|
|
|
## Interpretation
|
|
|
|
This model is competitive with several older 100M-160M public base models on a
|
|
matched `lm-eval` task suite while using only 10B pretraining tokens on one L20.
|
|
It is not SOTA and does not beat modern heavily overtrained compact models such
|
|
as SmolLM-135M or SmolLM2-135M, which use substantially larger token budgets.
|
|
|
|
## Intended Use
|
|
|
|
This checkpoint is suitable for:
|
|
|
|
- base model evaluation
|
|
- continued pretraining experiments
|
|
- supervised fine-tuning experiments
|
|
- small-model training pipeline demonstrations
|
|
- studying single-GPU pretraining tradeoffs
|
|
|
|
It is not suitable as a production assistant without post-training, safety
|
|
evaluation, and domain-specific validation.
|
|
|
|
## Limitations
|
|
|
|
- This is a small base model trained on 10B tokens.
|
|
- It is not instruction-tuned and may not follow user requests reliably.
|
|
- It can produce incorrect facts, repetition, or incomplete generations.
|
|
- It has not been safety aligned.
|
|
- Benchmark results should not be interpreted as general assistant quality.
|
|
- Results should not be described as SOTA without controlled matched-budget
|
|
baselines and contamination checks.
|
|
|
|
## Training-Budget Context
|
|
|
|
For fair interpretation, training data size matters:
|
|
|
|
| Model | Reported Training Budget |
|
|
| --- | --- |
|
|
| `l20-edu-135m` | 10B FineWeb-Edu tokens |
|
|
| GPT-2 small | WebText, about 40GB text; no clean official token count |
|
|
| OPT-125M | 180B tokens |
|
|
| GPT-Neo-125M | The Pile, commonly reported as 300B tokens |
|
|
| Cerebras-GPT-111M | about 2.2B tokens |
|
|
| Pythia-160M | about 300B tokens |
|
|
| SmolLM-135M | 600B tokens |
|
|
| SmolLM2-135M | 2T tokens |
|
|
|
|
## Citation
|
|
|
|
If you use this checkpoint, please cite or link to this repository and include
|
|
the training-token budget when comparing against other compact language models.
|