l20-edu-135m/docs/training_recipe.md

Training Recipe

This document records the exact recipe used for the released l20-edu-135m base checkpoint.

Model

Field	Value
Run name	l20-edu-135m-deepthin
Parameters	134,515,008
Architecture	Llama-style decoder-only Transformer
Layers	30
Hidden size	576
FFN size	1536
Attention heads	9 query heads, 3 key/value heads
Context length	2048
Tokenizer	HuggingFaceTB/SmolLM2-135M
Attention implementation	PyTorch SDPA
Tied embeddings	yes

Data

Field	Value
Dataset	HuggingFaceFW/fineweb-edu
Config	sample-10BT
Split	train
Streaming	yes
Text filter	min_chars=300, max_chars=50000
Quality filter	min_score=3.0, min_int_score=3
Packing	EOS-joined documents packed into 2048-token blocks
Planned token budget	10,001,252,352 tokens

Optimization

Field	Value
Optimizer	AdamW
Learning rate	4e-4 peak
LR schedule	linear warmup + cosine decay to 0.1 * peak_lr
Warmup steps	1000
Min LR ratio	0.1
Weight decay	0.1
Adam beta1 / beta2	0.9 / 0.95
Gradient clip	1.0
Precision	bfloat16
Torch compile	enabled
Gradient checkpointing	enabled

Batch And Token Accounting

Field	Value
Micro batch size	6 sequences
Gradient accumulation	43
Global batch size	258 sequences
Sequence length	2048 tokens
Tokens per optimizer step	528,384
Max steps	18,928
Planned tokens	10,001,252,352

Checkpointing And Evaluation

Field	Value
Log interval	10 steps
Eval interval	500 steps
Eval batches	64
Save interval	1000 steps
Checkpoints retained	last 2
Final checkpoint	runs/l20-edu-135m-deepthin/step-018928
Published checkpoint	AliceYin/l20-edu-135m

Each regular checkpoint represents about 528.4M training tokens. The final checkpoint was saved at step 18,928 rather than an even 1000-step boundary.

Runtime And Hardware

Field	Value
GPU	NVIDIA L20
Reported GPU memory	46,068 MiB total
Driver	550.163.01
Mean logged throughput	38,541 tokens/s
Mean logged throughput after step 1000	38,587 tokens/s
Estimated train time from throughput	about 72.0 hours
Final checkpoint mtime	2026-05-19 05:04:22 +0800

The JSON training log did not record an exact wall-clock launch timestamp or peak VRAM. Peak GPU memory should therefore be treated as not measured for this release. Future runs should log nvidia-smi --query-gpu=memory.used during training.

Cost was not available from billing logs. A reproducible estimate is:

estimated_cost = 72 GPU-hours * L20_hourly_rate

Examples:

L20 Hourly Rate	Estimated GPU Cost
$0.60 / hour	$43
$1.00 / hour	$72
$1.50 / hour	$108

This excludes storage, network egress, idle time, and engineering time.

Known Issues During The Run

• The dataset mirror produced transient Read timed out errors near the end of training. The run recovered through retry and continued.
• The final perplexity command printed loss=2.8731 perplexity=17.69, then hit a Python finalization crash. The metric is usable because it was printed before process teardown, but the crash is documented.
• After the final checkpoint was written, the training process printed terminate called without an active exception. The checkpoint was complete and load-tested with AutoModelForCausalLM.
• Peak VRAM and exact cloud cost were not logged.

Reproduction Command

python -m l20_pretrain.train configs/l20_135m_deepthin.yaml

The config file is the source of truth for this recipe: configs/l20_135m_deepthin.yaml.