1.7 KiB
license, language, library_name, pipeline_tag, tags
| license | language | library_name | pipeline_tag | tags | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| mit |
|
transformers | text-generation |
|
BabyLM 2026 — MultiLingual track v3 (quality-filter, WSD, 5 epochs)
Iteration v3 of our BabyLM 2026 MultiLingual submission. Same Llama-110M
architecture, same joint BPE 32k tokenizer as v2, with category filtering:
EN drops padding-*; NL drops padding-opensubtitles (= 63% of NL); ZH drops
subtitles (= 66% of ZH = WenetSpeech). Hypothesis: the v1/v2 ceiling was
data-quality driven, not schedule-driven.
Filtered corpora (post-byte-premium reference tokens):
- EN: 78.2 M (was 99 M)
- NL: 38.8 M (was 105 M)
- ZH: 49.7 M (was 147 M)
- Total: 166.7 M ref tokens available
Training
- Schedule: WSD (warmup 200 → constant 6e-4 → linear last 25% to 6e-5)
- Total compute consumed: 500 M effective tokens (5× v2's 100 M)
- Per-language epochs: EN 2.13, NL 4.29, ZH 3.35 — within the ≤10 cap
- 4× NVIDIA A10G, bf16, DDP, eff. batch 131 K tokens/step
- 23,295 steps · 8.6 hours wallclock
Revisions
main is chck_400M (the largest fast-eval checkpoint we saved).
Available revisions: chck_1M, chck_2M, chck_3M, chck_4M, chck_5M, chck_6M, chck_7M, chck_8M, chck_9M, chck_10M, chck_20M, chck_30M, chck_40M, chck_50M, chck_60M, chck_70M, chck_80M, chck_90M, chck_100M, chck_200M, chck_300M, chck_400M.
How to evaluate
git clone https://github.com/babylm-org/babylm-eval
cd babylm-eval/multilingual
bash scripts/zeroshot_model.sh --model_name Shamima/babylm-2026-multilingual-v3-quality-filter
Companion repo (audit, scaffold, ablation configs, iteration log): https://github.com/silvererudite/bb-lm-challenge-sub