license, language, library_name, pipeline_tag, tags

license	language	library_name	pipeline_tag	tags
mit	en nl zh	transformers	text-generation	babylm babylm-2026 multilingual llama pretrained-from-scratch quality-filter

BabyLM 2026 — MultiLingual track v3 (quality-filter, WSD, 5 epochs)

Iteration v3 of our BabyLM 2026 MultiLingual submission. Same Llama-110M architecture, same joint BPE 32k tokenizer as v2, with category filtering: EN drops padding-*; NL drops padding-opensubtitles (= 63% of NL); ZH drops subtitles (= 66% of ZH = WenetSpeech). Hypothesis: the v1/v2 ceiling was data-quality driven, not schedule-driven.

Filtered corpora (post-byte-premium reference tokens):

• EN: 78.2 M (was 99 M)
• NL: 38.8 M (was 105 M)
• ZH: 49.7 M (was 147 M)
• Total: 166.7 M ref tokens available

Training

• Schedule: WSD (warmup 200 → constant 6e-4 → linear last 25% to 6e-5)
• Total compute consumed: 500 M effective tokens (5× v2's 100 M)
• Per-language epochs: EN 2.13, NL 4.29, ZH 3.35 — within the ≤10 cap
• 4× NVIDIA A10G, bf16, DDP, eff. batch 131 K tokens/step
• 23,295 steps · 8.6 hours wallclock

Revisions

main is chck_400M (the largest fast-eval checkpoint we saved). Available revisions: chck_1M, chck_2M, chck_3M, chck_4M, chck_5M, chck_6M, chck_7M, chck_8M, chck_9M, chck_10M, chck_20M, chck_30M, chck_40M, chck_50M, chck_60M, chck_70M, chck_80M, chck_90M, chck_100M, chck_200M, chck_300M, chck_400M.

How to evaluate

git clone https://github.com/babylm-org/babylm-eval
cd babylm-eval/multilingual
bash scripts/zeroshot_model.sh --model_name Shamima/babylm-2026-multilingual-v3-quality-filter

Companion repo (audit, scaffold, ablation configs, iteration log): https://github.com/silvererudite/bb-lm-challenge-sub