AliceYin/l20-edu-135m
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
9e2ca78a06c2176f76b438b09667c1b0869b6a3e
l20-edu-135m/docs/evaluation_report.md
ModelHub XC 9e2ca78a06 初始化项目，由ModelHub XC社区提供模型 
Model: AliceYin/l20-edu-135m
Source: Original Platform
2026-06-10 18:43:03 +08:00

7.6 KiB
Raw Blame History

Evaluation Report

This report summarizes the final l20-edu-135m base checkpoint and the public baseline comparison run.

Candidate

  • Model: l20-edu-135m-deepthin
  • Released checkpoint: runs/l20-edu-135m-deepthin/step-018928
  • Hugging Face repo: AliceYin/l20-edu-135m
  • Parameters: 134,515,008
  • Training tokens: 10,001,252,352 planned tokens
  • Tokens per parameter: 74.35
  • Dataset: HuggingFaceFW/fineweb-edu, sample-10BT
  • Tokenizer: HuggingFaceTB/SmolLM2-135M
  • Final validation: loss 2.8731, perplexity 17.69

Training Curves

The training log contains 1,903 train-loss points and 38 validation-loss points. The extracted metrics are committed as docs/training_metrics.csv, with a small machine-readable summary in docs/training_summary.json.

Loss curve after warmup

Training curves

lm-eval Results

The final checkpoint was evaluated with EleutherAI lm-evaluation-harness on a small-model base-LM suite. All public baselines were run through the same task set and comparison script.

Benchmark Protocol

Field Setting
Harness EleutherAI lm-evaluation-harness
Harness version 0.4.12
Model backend --model hf
Device cuda:0
Dtype bfloat16
Batch size auto, resolved to 64
Few-shot setting num_fewshot=None / zero-shot
Limit none, full task datasets
Samples --log_samples enabled
Random seeds harness defaults logged as Python 0, NumPy 1234, Torch 1234, few-shot 1234
Candidate numbers self-run on final checkpoint
Baseline numbers self-run locally through scripts/eval_public_baselines.sh, not copied from public leaderboards

Candidate and baseline evaluations used the same harness version, task list, dtype, device class, batch policy, and comparison parser. They did not use the same tokenizer or training context length because public pretrained models are evaluated with their own released tokenizers and model configs. That means this is a public-model benchmark comparison, not a controlled tokenizer-matched architecture comparison.

The eval command template was:

lm_eval \
  --model hf \
  --model_args "pretrained=<model-or-checkpoint>,dtype=bfloat16" \
  --tasks lambada_openai,hellaswag,piqa,arc_easy,arc_challenge,winogrande \
  --device cuda:0 \
  --batch_size auto \
  --output_path <output-dir> \
  --log_samples

Context-length handling was left to the Hugging Face model configuration used by lm-evaluation-harness. The selected tasks are short enough that this should not be the dominant factor, but it is still a difference from a strictly controlled same-tokenizer/same-context experiment.

Task Metric l20-edu-135m gpt2-small opt-125m gpt-neo-125m cerebras-gpt-111m pythia-160m smollm-135m smollm2-135m
ARC-Challenge acc_norm 0.2765 0.2261 0.2210 0.2321 0.2099 0.2312 0.2875 0.2969
ARC-Easy acc_norm 0.5059 0.3973 0.3990 0.3965 0.3506 0.3641 0.5610 0.5854
HellaSwag acc_norm 0.3272 0.3138 0.3160 0.3055 0.2720 0.3030 0.4265 0.4301
LAMBADA OpenAI acc 0.2540 0.3076 0.3856 0.3765 0.1912 0.1225 0.3757 0.4289
PIQA acc_norm 0.6224 0.6208 0.6202 0.6213 0.5811 0.5979 0.6823 0.6839
WinoGrande acc 0.5099 0.5067 0.5178 0.5099 0.4901 0.5075 0.5272 0.5249

Win rates over public baselines:

Baseline Wins / Tasks Win Rate
GPT-2 small 5 / 6 0.833
OPT-125M 4 / 6 0.667
GPT-Neo-125M 4 / 6 0.667
Cerebras-GPT-111M 6 / 6 1.000
Pythia-160M 6 / 6 1.000
SmolLM-135M 0 / 6 0.000
SmolLM2-135M 0 / 6 0.000

Training Budget Context

The comparison is useful, but it is not a matched training-budget comparison. The public baselines were trained with very different corpora and token budgets.

Model Parameters Reported Training Data / Tokens Notes
l20-edu-135m 134.5M 10B FineWeb-Edu tokens This project, single L20 run
GPT-2 small 124M WebText, about 40GB text from 8M documents Official GPT-2 reporting does not give a clean token count
OPT-125M 125M 180B tokens OPT model family training budget
GPT-Neo-125M 125M The Pile, commonly reported as 300B tokens Public GPT-Neo checkpoint
Cerebras-GPT-111M 111M About 2.2B tokens 20 tokens per parameter recipe
Pythia-160M 160M About 300B tokens Pythia suite training budget
SmolLM-135M 135M 600B tokens SmolLM-Corpus
SmolLM2-135M 135M 2T tokens FineWeb-Edu, DCLM, The Stack, and curated data

The strongest honest reading is:

l20-edu-135m is competitive with several older 100M-160M public base models while using only 10B pretraining tokens on one L20. It is clearly behind modern compact models such as SmolLM and SmolLM2, which use much larger data budgets.

The model should not be described as SOTA. A controlled architecture claim still requires training configs/l20_wide_140m_baseline.yaml under the same tokenizer, data, optimizer, schedule, and token budget.

Contamination Status

No contamination pass is claimed for this release.

The repository includes a simple n-gram contamination checker (scripts/check_contamination.py) and a training-text sampler (scripts/sample_training_text.py), but a full contamination audit against ARC, HellaSwag, PIQA, LAMBADA, and WinoGrande was not completed before release. Because the model was trained on a public web-scale FineWeb-Edu slice, benchmark overlap cannot be ruled out without a separate audit.

For a stricter release, run:

python scripts/sample_training_text.py configs/l20_135m_deepthin.yaml \
  --docs 100000 \
  --out data/train_sample.txt

python scripts/check_contamination.py \
  --train data/train_sample.txt \
  --benchmark eval_results/l20-edu-135m-deepthin-final \
  --ngram 13 \
  --out eval_results/contamination_report.json

That still remains a sample-based n-gram check, not a proof of no contamination.

Interpretation

The model learned usable base-LM behavior: it can continue text, complete simple facts sometimes, and score above several older baselines on commonsense and reading-style multiple-choice tasks.

The main weaknesses are expected for a 135M base model trained on 10B tokens:

  • weak instruction following
  • unstable factual recall
  • repetition during free-form generation
  • lower LAMBADA accuracy than GPT-2/OPT/GPT-Neo and modern SmolLM models
  • no chat alignment or safety tuning

For public presentation, frame this as a training systems and reproducibility project rather than as a high-quality assistant model.

Reproducibility Artifacts

  • Training config: configs/l20_135m_deepthin.yaml
  • Eval comparison script: scripts/compare_lm_eval.py
  • Public baseline runner: scripts/eval_public_baselines.sh
  • Raw comparison files on Hugging Face: eval/comparison.md, eval/comparison.json
  • Local ignored artifacts after a run: eval_results/, logs/, runs/

Sources For Baseline Budget Notes