license, language, library_name, pipeline_tag, tags, datasets

license	language	library_name	pipeline_tag	tags	datasets
apache-2.0	en	transformers	text-generation	causal-lm pretraining continued-pretraining long-context math code from-scratch fineweb-edu dclm-edu single-gpu l20	HuggingFaceFW/fineweb-edu HuggingFaceTB/smollm-corpus HuggingFaceTB/finemath HuggingFaceTB/stack-edu HuggingFaceTB/dclm-edu

l20-edu-135m

l20-edu-135m is a 134.5M-parameter Llama-style causal language model pretrained from scratch on 10B FineWeb-Edu tokens using a single NVIDIA L20 GPU.

This is a base model, not an instruction-tuned chat model. It is released as a small-model pretraining artifact for research, evaluation, reproducibility, continued pretraining, and downstream supervised fine-tuning.

Latest Results

The latest completed checkpoint is the Stage 2 continued-pretraining checkpoint:

step-001850-stage2-math-code

Stage 2 starts from the 8K long-context checkpoint and continues training on a 1B-token replay mixture with high-quality educational web data, FineMath, Stack-Edu code, Cosmopedia-style textbook data, and replay from the earlier educational corpus. The checkpoint and full eval artifacts are included in this repository under:

• step-001850-stage2-math-code/
• eval_results/step-001850_sota/
• eval_results/current_vs_smollm_summary.md
• training_artifacts/

Stage 2 Six-Task Benchmark

Zero-shot lm-eval results for the Stage 2 checkpoint:

Task	Metric	Score
ARC-Challenge	acc_norm	0.2765
ARC-Easy	acc_norm	0.5059
HellaSwag	acc_norm	0.3272
LAMBADA OpenAI	acc	0.2540
PIQA	acc_norm	0.6224
WinoGrande	acc	0.5099

Current SmolLM Target Comparison

The table below uses the same target tasks as the current SmolLM comparison script in training_artifacts/scripts/eval_smollm_benchmark.sh.

Task	Metric	ours-stage2	SmolLM-135M	SmolLM2-135M
ARC-Challenge	acc_norm	0.2807	0.2875	0.2969
ARC-Easy	acc_norm	0.5059	0.5610	0.5854
HellaSwag	acc_norm	0.3242	0.4265	0.4301
LAMBADA OpenAI	acc	pending rerun	0.3757	0.4289
PIQA	acc_norm	0.6137	0.6823	0.6839
WinoGrande	acc	pending rerun	0.5272	0.5249

Mean over available matched metrics:

Model	Mean
ours-stage2	0.4311
SmolLM-135M	0.4767
SmolLM2-135M	0.4917

The largest remaining gaps are HellaSwag, PIQA, and ARC-Easy. The next planned Stage 3 recipe therefore shifts the data mix toward DCLM-Edu and high-quality general educational web data instead of only increasing math/code data.

Model Details

Field	Value
Model type	Decoder-only causal LM
Architecture	Llama-style Transformer
Parameters	134,515,008
Layers	30
Hidden size	576
FFN size	1536
Attention	9 query heads, 3 key/value heads
Context length	2048
Tokenizer	HuggingFaceTB/SmolLM2-135M tokenizer
Training data	HuggingFaceFW/fineweb-edu, sample-10BT
Training budget	10,001,252,352 planned tokens
Tokens / parameter	74.35
Final checkpoint	step 18,928
Hardware	single NVIDIA L20 GPU

Training Recipe

Field	Value
Sequence length	2048 tokens
Micro batch size	6 sequences
Gradient accumulation	43 steps
Global batch size	258 sequences
Tokens / optimizer step	528,384
Max steps	18,928
Optimizer	AdamW
Peak learning rate	4e-4
LR schedule	linear warmup + cosine decay to 0.1 * peak_lr
Warmup	1,000 steps
Weight decay	0.1
Adam beta1 / beta2	0.9 / 0.95
Gradient clipping	1.0
Precision	bfloat16
Gradient checkpointing	enabled
Torch compile	enabled
Eval interval	500 steps
Checkpoint interval	1,000 steps, keeping the last 2 regular checkpoints

The training config is included in the repository as configs/l20_135m_deepthin.yaml. A fuller recipe is available in docs/training_recipe.md.

Runtime And Cost Notes

Field	Value
GPU	NVIDIA L20
Reported GPU memory	46,068 MiB total
Mean logged throughput	38,541 tokens/s
Mean logged throughput after step 1,000	38,587 tokens/s
Estimated training time	about 72 GPU-hours
Final checkpoint mtime	2026-05-19 05:04:22 +0800

Exact peak VRAM and billing cost were not logged. A reproducible cost estimate is 72 GPU-hours * L20_hourly_rate, excluding storage, network egress, idle time, and engineering time.

Known run issues:

• Transient dataset mirror read timeouts occurred near the end of training and recovered through retry.
• The final perplexity command printed loss=2.8731 perplexity=17.69, then hit a Python finalization crash. The metric is reported because it was printed before process teardown.
• The training process printed terminate called without an active exception after the final checkpoint had been written. The final checkpoint was load-tested with AutoModelForCausalLM.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "AliceYin/l20-edu-135m"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=40,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Because this is a base model, completion-style prompts work better than instruction/chat prompts.

Evaluation

Final validation:

• Loss: 2.8731
• Perplexity: 17.69

Training artifacts:

• training/training_metrics.csv
• training/training_summary.json
• training/loss_curve_zoom.png
• training/training_curves.png

Zero-shot lm-eval results for the final checkpoint:

Task	Metric	Score
ARC-Challenge	acc_norm	0.2765
ARC-Easy	acc_norm	0.5059
HellaSwag	acc_norm	0.3272
LAMBADA OpenAI	acc	0.2540
PIQA	acc_norm	0.6224
WinoGrande	acc	0.5099

Benchmark Protocol

Candidate and public baseline numbers were run with the same evaluation setup:

Field	Setting
Harness	EleutherAI lm-evaluation-harness
Harness version	0.4.12
Backend	--model hf
Device	cuda:0
Dtype	bfloat16
Batch size	auto, resolved to 64
Few-shot setting	zero-shot
Dataset limit	none; full task datasets
Samples	--log_samples enabled
Seeds	harness defaults: Python 0, NumPy 1234, Torch 1234, few-shot 1234
Candidate numbers	self-run on the final checkpoint
Baseline numbers	self-run through scripts/eval_public_baselines.sh, not copied from leaderboards

The public baselines were evaluated with the same harness version, task list, zero-shot setting, dtype, device class, batch policy, and comparison parser. They were not evaluated with the same tokenizer or model context length; each public model used its own released tokenizer and Hugging Face model config. This is therefore a public-model benchmark comparison, not a controlled same-tokenizer architecture comparison.

Public baseline win rates on the same task set:

Baseline	Wins / Tasks	Win Rate
GPT-2 small	5 / 6	0.833
OPT-125M	4 / 6	0.667
GPT-Neo-125M	4 / 6	0.667
Cerebras-GPT-111M	6 / 6	1.000
Pythia-160M	6 / 6	1.000
SmolLM-135M	0 / 6	0.000
SmolLM2-135M	0 / 6	0.000

The full comparison artifacts are included in this repository under:

• eval/comparison.md
• eval/comparison.json
• docs/evaluation_report.md

Contamination Status

No full benchmark contamination pass is claimed for this release. The project repository includes scripts/check_contamination.py and scripts/sample_training_text.py, but a separate audit against ARC, HellaSwag, PIQA, LAMBADA, and WinoGrande samples was not completed before release. Because the model was trained on a public web-scale FineWeb-Edu slice, benchmark overlap cannot be ruled out without that audit.

Interpretation

This model is competitive with several older 100M-160M public base models on a matched lm-eval task suite while using only 10B pretraining tokens on one L20. It is not SOTA and does not beat modern heavily overtrained compact models such as SmolLM-135M or SmolLM2-135M, which use substantially larger token budgets.

Intended Use

This checkpoint is suitable for:

• base model evaluation
• continued pretraining experiments
• supervised fine-tuning experiments
• small-model training pipeline demonstrations
• studying single-GPU pretraining tradeoffs

It is not suitable as a production assistant without post-training, safety evaluation, and domain-specific validation.

Limitations

• This is a small base model trained on 10B tokens.
• It is not instruction-tuned and may not follow user requests reliably.
• It can produce incorrect facts, repetition, or incomplete generations.
• It has not been safety aligned.
• Benchmark results should not be interpreted as general assistant quality.
• Results should not be described as SOTA without controlled matched-budget baselines and contamination checks.

Training-Budget Context

For fair interpretation, training data size matters:

Model	Reported Training Budget
l20-edu-135m	10B FineWeb-Edu tokens
GPT-2 small	WebText, about 40GB text; no clean official token count
OPT-125M	180B tokens
GPT-Neo-125M	The Pile, commonly reported as 300B tokens
Cerebras-GPT-111M	about 2.2B tokens
Pythia-160M	about 300B tokens
SmolLM-135M	600B tokens
SmolLM2-135M	2T tokens

Citation

If you use this checkpoint, please cite or link to this repository and include the training-token budget when comparing against other compact language models.