初始化项目，由ModelHub XC社区提供模型

Model: AliceYin/l20-edu-135m Source: Original Platform
2026-06-10 18:43:03 +08:00
commit 9e2ca78a06
88 changed files with 605541 additions and 0 deletions
--- a/docs/assets/loss_curve_zoom.png
+++ b/docs/assets/loss_curve_zoom.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5aa97aa77c072a884fe4780ca5657b0bae612e71ebd3084387caafb62e695af
+size 182478
--- a/docs/assets/training_curves.png
+++ b/docs/assets/training_curves.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bdf534a23f02067e69006d317571cc503b17136401b0503393f5df9618f0326f
+size 169270
--- a/docs/evaluation_report.md
+++ b/docs/evaluation_report.md
@@ -0,0 +1,185 @@
+# Evaluation Report
+
+This report summarizes the final `l20-edu-135m` base checkpoint and the public
+baseline comparison run.
+
+## Candidate
+
+- Model: `l20-edu-135m-deepthin`
+- Released checkpoint: `runs/l20-edu-135m-deepthin/step-018928`
+- Hugging Face repo: `AliceYin/l20-edu-135m`
+- Parameters: 134,515,008
+- Training tokens: 10,001,252,352 planned tokens
+- Tokens per parameter: 74.35
+- Dataset: `HuggingFaceFW/fineweb-edu`, `sample-10BT`
+- Tokenizer: `HuggingFaceTB/SmolLM2-135M`
+- Final validation: loss `2.8731`, perplexity `17.69`
+
+## Training Curves
+
+The training log contains 1,903 train-loss points and 38 validation-loss points.
+The extracted metrics are committed as `docs/training_metrics.csv`, with a small
+machine-readable summary in `docs/training_summary.json`.
+
+![Loss curve after warmup](assets/loss_curve_zoom.png)
+
+![Training curves](assets/training_curves.png)
+
+## lm-eval Results
+
+The final checkpoint was evaluated with EleutherAI `lm-evaluation-harness` on a
+small-model base-LM suite. All public baselines were run through the same task
+set and comparison script.
+
+## Benchmark Protocol
+
+| Field | Setting |
+| --- | --- |
+| Harness | EleutherAI `lm-evaluation-harness` |
+| Harness version | `0.4.12` |
+| Model backend | `--model hf` |
+| Device | `cuda:0` |
+| Dtype | `bfloat16` |
+| Batch size | `auto`, resolved to 64 |
+| Few-shot setting | `num_fewshot=None` / zero-shot |
+| Limit | none, full task datasets |
+| Samples | `--log_samples` enabled |
+| Random seeds | harness defaults logged as Python 0, NumPy 1234, Torch 1234, few-shot 1234 |
+| Candidate numbers | self-run on final checkpoint |
+| Baseline numbers | self-run locally through `scripts/eval_public_baselines.sh`, not copied from public leaderboards |
+
+Candidate and baseline evaluations used the same harness version, task list,
+dtype, device class, batch policy, and comparison parser. They did **not** use
+the same tokenizer or training context length because public pretrained models
+are evaluated with their own released tokenizers and model configs. That means
+this is a public-model benchmark comparison, not a controlled tokenizer-matched
+architecture comparison.
+
+The eval command template was:
+
+```bash
+lm_eval \
+  --model hf \
+  --model_args "pretrained=<model-or-checkpoint>,dtype=bfloat16" \
+  --tasks lambada_openai,hellaswag,piqa,arc_easy,arc_challenge,winogrande \
+  --device cuda:0 \
+  --batch_size auto \
+  --output_path <output-dir> \
+  --log_samples
+```
+
+Context-length handling was left to the Hugging Face model configuration used by
+`lm-evaluation-harness`. The selected tasks are short enough that this should not
+be the dominant factor, but it is still a difference from a strictly controlled
+same-tokenizer/same-context experiment.
+
+| Task | Metric | l20-edu-135m | gpt2-small | opt-125m | gpt-neo-125m | cerebras-gpt-111m | pythia-160m | smollm-135m | smollm2-135m |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| ARC-Challenge | acc_norm | 0.2765 | 0.2261 | 0.2210 | 0.2321 | 0.2099 | 0.2312 | 0.2875 | 0.2969 |
+| ARC-Easy | acc_norm | 0.5059 | 0.3973 | 0.3990 | 0.3965 | 0.3506 | 0.3641 | 0.5610 | 0.5854 |
+| HellaSwag | acc_norm | 0.3272 | 0.3138 | 0.3160 | 0.3055 | 0.2720 | 0.3030 | 0.4265 | 0.4301 |
+| LAMBADA OpenAI | acc | 0.2540 | 0.3076 | 0.3856 | 0.3765 | 0.1912 | 0.1225 | 0.3757 | 0.4289 |
+| PIQA | acc_norm | 0.6224 | 0.6208 | 0.6202 | 0.6213 | 0.5811 | 0.5979 | 0.6823 | 0.6839 |
+| WinoGrande | acc | 0.5099 | 0.5067 | 0.5178 | 0.5099 | 0.4901 | 0.5075 | 0.5272 | 0.5249 |
+
+Win rates over public baselines:
+
+| Baseline | Wins / Tasks | Win Rate |
+| --- | ---: | ---: |
+| GPT-2 small | 5 / 6 | 0.833 |
+| OPT-125M | 4 / 6 | 0.667 |
+| GPT-Neo-125M | 4 / 6 | 0.667 |
+| Cerebras-GPT-111M | 6 / 6 | 1.000 |
+| Pythia-160M | 6 / 6 | 1.000 |
+| SmolLM-135M | 0 / 6 | 0.000 |
+| SmolLM2-135M | 0 / 6 | 0.000 |
+
+## Training Budget Context
+
+The comparison is useful, but it is not a matched training-budget comparison.
+The public baselines were trained with very different corpora and token budgets.
+
+| Model | Parameters | Reported Training Data / Tokens | Notes |
+| --- | ---: | --- | --- |
+| l20-edu-135m | 134.5M | 10B FineWeb-Edu tokens | This project, single L20 run |
+| GPT-2 small | 124M | WebText, about 40GB text from 8M documents | Official GPT-2 reporting does not give a clean token count |
+| OPT-125M | 125M | 180B tokens | OPT model family training budget |
+| GPT-Neo-125M | 125M | The Pile, commonly reported as 300B tokens | Public GPT-Neo checkpoint |
+| Cerebras-GPT-111M | 111M | About 2.2B tokens | 20 tokens per parameter recipe |
+| Pythia-160M | 160M | About 300B tokens | Pythia suite training budget |
+| SmolLM-135M | 135M | 600B tokens | SmolLM-Corpus |
+| SmolLM2-135M | 135M | 2T tokens | FineWeb-Edu, DCLM, The Stack, and curated data |
+
+The strongest honest reading is:
+
+> `l20-edu-135m` is competitive with several older 100M-160M public base models
+> while using only 10B pretraining tokens on one L20. It is clearly behind modern
+> compact models such as SmolLM and SmolLM2, which use much larger data budgets.
+
+The model should not be described as SOTA. A controlled architecture claim still
+requires training `configs/l20_wide_140m_baseline.yaml` under the same tokenizer,
+data, optimizer, schedule, and token budget.
+
+## Contamination Status
+
+No contamination pass is claimed for this release.
+
+The repository includes a simple n-gram contamination checker
+(`scripts/check_contamination.py`) and a training-text sampler
+(`scripts/sample_training_text.py`), but a full contamination audit against
+ARC, HellaSwag, PIQA, LAMBADA, and WinoGrande was not completed before release.
+Because the model was trained on a public web-scale FineWeb-Edu slice, benchmark
+overlap cannot be ruled out without a separate audit.
+
+For a stricter release, run:
+
+```bash
+python scripts/sample_training_text.py configs/l20_135m_deepthin.yaml \
+  --docs 100000 \
+  --out data/train_sample.txt
+
+python scripts/check_contamination.py \
+  --train data/train_sample.txt \
+  --benchmark eval_results/l20-edu-135m-deepthin-final \
+  --ngram 13 \
+  --out eval_results/contamination_report.json
+```
+
+That still remains a sample-based n-gram check, not a proof of no contamination.
+
+## Interpretation
+
+The model learned usable base-LM behavior: it can continue text, complete simple
+facts sometimes, and score above several older baselines on commonsense and
+reading-style multiple-choice tasks.
+
+The main weaknesses are expected for a 135M base model trained on 10B tokens:
+
+- weak instruction following
+- unstable factual recall
+- repetition during free-form generation
+- lower LAMBADA accuracy than GPT-2/OPT/GPT-Neo and modern SmolLM models
+- no chat alignment or safety tuning
+
+For public presentation, frame this as a training systems and reproducibility
+project rather than as a high-quality assistant model.
+
+## Reproducibility Artifacts
+
+- Training config: `configs/l20_135m_deepthin.yaml`
+- Eval comparison script: `scripts/compare_lm_eval.py`
+- Public baseline runner: `scripts/eval_public_baselines.sh`
+- Raw comparison files on Hugging Face: `eval/comparison.md`,
+  `eval/comparison.json`
+- Local ignored artifacts after a run: `eval_results/`, `logs/`, `runs/`
+
+## Sources For Baseline Budget Notes
+
+- GPT-2 / WebText: https://openai.com/index/better-language-models/
+- OPT model family: https://arxiv.org/abs/2205.01068
+- GPT-Neo-125M: https://huggingface.co/EleutherAI/gpt-neo-125m
+- Cerebras-GPT-111M: https://huggingface.co/cerebras/Cerebras-GPT-111M
+- Pythia suite: https://github.com/EleutherAI/pythia
+- SmolLM: https://huggingface.co/blog/smollm
+- SmolLM2-135M: https://huggingface.co/HuggingFaceTB/SmolLM2-135M
+- FineWeb-Edu: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu
--- a/docs/training_metrics.csv
+++ b/docs/training_metrics.csv
--- a/docs/training_recipe.md
+++ b/docs/training_recipe.md
@@ -0,0 +1,130 @@
+# Training Recipe
+
+This document records the exact recipe used for the released
+`l20-edu-135m` base checkpoint.
+
+## Model
+
+| Field | Value |
+| --- | --- |
+| Run name | `l20-edu-135m-deepthin` |
+| Parameters | 134,515,008 |
+| Architecture | Llama-style decoder-only Transformer |
+| Layers | 30 |
+| Hidden size | 576 |
+| FFN size | 1536 |
+| Attention heads | 9 query heads, 3 key/value heads |
+| Context length | 2048 |
+| Tokenizer | `HuggingFaceTB/SmolLM2-135M` |
+| Attention implementation | PyTorch SDPA |
+| Tied embeddings | yes |
+
+## Data
+
+| Field | Value |
+| --- | --- |
+| Dataset | `HuggingFaceFW/fineweb-edu` |
+| Config | `sample-10BT` |
+| Split | `train` |
+| Streaming | yes |
+| Text filter | `min_chars=300`, `max_chars=50000` |
+| Quality filter | `min_score=3.0`, `min_int_score=3` |
+| Packing | EOS-joined documents packed into 2048-token blocks |
+| Planned token budget | 10,001,252,352 tokens |
+
+## Optimization
+
+| Field | Value |
+| --- | --- |
+| Optimizer | AdamW |
+| Learning rate | `4e-4` peak |
+| LR schedule | linear warmup + cosine decay to `0.1 * peak_lr` |
+| Warmup steps | 1000 |
+| Min LR ratio | 0.1 |
+| Weight decay | 0.1 |
+| Adam beta1 / beta2 | 0.9 / 0.95 |
+| Gradient clip | 1.0 |
+| Precision | bfloat16 |
+| Torch compile | enabled |
+| Gradient checkpointing | enabled |
+
+## Batch And Token Accounting
+
+| Field | Value |
+| --- | ---: |
+| Micro batch size | 6 sequences |
+| Gradient accumulation | 43 |
+| Global batch size | 258 sequences |
+| Sequence length | 2048 tokens |
+| Tokens per optimizer step | 528,384 |
+| Max steps | 18,928 |
+| Planned tokens | 10,001,252,352 |
+
+## Checkpointing And Evaluation
+
+| Field | Value |
+| --- | --- |
+| Log interval | 10 steps |
+| Eval interval | 500 steps |
+| Eval batches | 64 |
+| Save interval | 1000 steps |
+| Checkpoints retained | last 2 |
+| Final checkpoint | `runs/l20-edu-135m-deepthin/step-018928` |
+| Published checkpoint | `AliceYin/l20-edu-135m` |
+
+Each regular checkpoint represents about 528.4M training tokens. The final
+checkpoint was saved at step 18,928 rather than an even 1000-step boundary.
+
+## Runtime And Hardware
+
+| Field | Value |
+| --- | --- |
+| GPU | NVIDIA L20 |
+| Reported GPU memory | 46,068 MiB total |
+| Driver | 550.163.01 |
+| Mean logged throughput | 38,541 tokens/s |
+| Mean logged throughput after step 1000 | 38,587 tokens/s |
+| Estimated train time from throughput | about 72.0 hours |
+| Final checkpoint mtime | 2026-05-19 05:04:22 +0800 |
+
+The JSON training log did not record an exact wall-clock launch timestamp or
+peak VRAM. Peak GPU memory should therefore be treated as **not measured** for
+this release. Future runs should log `nvidia-smi --query-gpu=memory.used` during
+training.
+
+Cost was not available from billing logs. A reproducible estimate is:
+
+```text
+estimated_cost = 72 GPU-hours * L20_hourly_rate
+```
+
+Examples:
+
+| L20 Hourly Rate | Estimated GPU Cost |
+| ---: | ---: |
+| $0.60 / hour | $43 |
+| $1.00 / hour | $72 |
+| $1.50 / hour | $108 |
+
+This excludes storage, network egress, idle time, and engineering time.
+
+## Known Issues During The Run
+
+- The dataset mirror produced transient `Read timed out` errors near the end of
+  training. The run recovered through retry and continued.
+- The final perplexity command printed `loss=2.8731 perplexity=17.69`, then hit
+  a Python finalization crash. The metric is usable because it was printed before
+  process teardown, but the crash is documented.
+- After the final checkpoint was written, the training process printed
+  `terminate called without an active exception`. The checkpoint was complete
+  and load-tested with `AutoModelForCausalLM`.
+- Peak VRAM and exact cloud cost were not logged.
+
+## Reproduction Command
+
+```bash
+python -m l20_pretrain.train configs/l20_135m_deepthin.yaml
+```
+
+The config file is the source of truth for this recipe:
+`configs/l20_135m_deepthin.yaml`.
--- a/docs/training_summary.json
+++ b/docs/training_summary.json
@@ -0,0 +1,37 @@
+{
+  "train_points": 1903,
+  "eval_points": 38,
+  "first_train": {
+    "event": "train",
+    "step": 1,
+    "loss": 10.8943589925766,
+    "lr": 8.000000000000001e-07,
+    "tokens": 528384,
+    "tokens_per_sec_window": 5958.308692440408
+  },
+  "last_train": {
+    "event": "train",
+    "step": 18920,
+    "loss": 2.8648588210344315,
+    "lr": 4.000017687174911e-05,
+    "tokens": 9997025280,
+    "tokens_per_sec_window": 39791.76880639303
+  },
+  "first_eval": {
+    "event": "eval",
+    "step": 500,
+    "loss": 5.337074369192123,
+    "perplexity": 207.90357057348467,
+    "tokens": 264192000
+  },
+  "last_eval": {
+    "event": "eval",
+    "step": 18928,
+    "tokens": 10001252352,
+    "loss": 2.8731,
+    "perplexity": 17.69,
+    "source": "final_eval_pipeline.log"
+  },
+  "mean_tokens_per_sec_window": 38541.19206441507,
+  "mean_tokens_per_sec_after_step_1000": 38587.389181724684
+}