---
license: mit
language: [en, nl, zh]
library_name: transformers
pipeline_tag: text-generation
tags: [babylm, babylm-2026, multilingual, llama, pretrained-from-scratch, quality-filter]
---

# BabyLM 2026 — MultiLingual track v3 (quality-filter, WSD, 5 epochs)

Iteration v3 of our BabyLM 2026 MultiLingual submission. Same Llama-110M
architecture, same joint BPE 32k tokenizer as v2, **with category filtering**:
EN drops `padding-*`; NL drops `padding-opensubtitles` (= 63% of NL); ZH drops
`subtitles` (= 66% of ZH = WenetSpeech). Hypothesis: the v1/v2 ceiling was
data-quality driven, not schedule-driven.

Filtered corpora (post-byte-premium reference tokens):
- EN: 78.2 M (was 99 M)
- NL: 38.8 M (was 105 M)
- ZH: 49.7 M (was 147 M)
- **Total: 166.7 M ref tokens available**

## Training

- Schedule: WSD (warmup 200 → constant 6e-4 → linear last 25% to 6e-5)
- Total compute consumed: **500 M effective tokens** (5× v2's 100 M)
- Per-language epochs: EN 2.13, NL 4.29, ZH 3.35 — within the ≤10 cap
- 4× NVIDIA A10G, bf16, DDP, eff. batch 131 K tokens/step
- 23,295 steps · 8.6 hours wallclock

## Revisions

`main` is `chck_400M` (the largest fast-eval checkpoint we saved).
Available revisions: chck_1M, chck_2M, chck_3M, chck_4M, chck_5M, chck_6M, chck_7M, chck_8M, chck_9M, chck_10M, chck_20M, chck_30M, chck_40M, chck_50M, chck_60M, chck_70M, chck_80M, chck_90M, chck_100M, chck_200M, chck_300M, chck_400M.

## How to evaluate

```bash
git clone https://github.com/babylm-org/babylm-eval
cd babylm-eval/multilingual
bash scripts/zeroshot_model.sh --model_name Shamima/babylm-2026-multilingual-v3-quality-filter
```

Companion repo (audit, scaffold, ablation configs, iteration log):
https://github.com/silvererudite/bb-lm-challenge-sub