46 lines
1.7 KiB
Markdown
46 lines
1.7 KiB
Markdown
|
---
|
|||
|
|
license: mit
|
|||
|
|
language: [en, nl, zh]
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags: [babylm, babylm-2026, multilingual, llama, pretrained-from-scratch, quality-filter]
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# BabyLM 2026 — MultiLingual track v3 (quality-filter, WSD, 5 epochs)
|
|||
|
|
|
|||
|
|
Iteration v3 of our BabyLM 2026 MultiLingual submission. Same Llama-110M
|
|||
|
|
architecture, same joint BPE 32k tokenizer as v2, **with category filtering**:
|
|||
|
|
EN drops `padding-*`; NL drops `padding-opensubtitles` (= 63% of NL); ZH drops
|
|||
|
|
`subtitles` (= 66% of ZH = WenetSpeech). Hypothesis: the v1/v2 ceiling was
|
|||
|
|
data-quality driven, not schedule-driven.
|
|||
|
|
|
|||
|
|
Filtered corpora (post-byte-premium reference tokens):
|
|||
|
|
- EN: 78.2 M (was 99 M)
|
|||
|
|
- NL: 38.8 M (was 105 M)
|
|||
|
|
- ZH: 49.7 M (was 147 M)
|
|||
|
|
- **Total: 166.7 M ref tokens available**
|
|||
|
|
|
|||
|
|
## Training
|
|||
|
|
|
|||
|
|
- Schedule: WSD (warmup 200 → constant 6e-4 → linear last 25% to 6e-5)
|
|||
|
|
- Total compute consumed: **500 M effective tokens** (5× v2's 100 M)
|
|||
|
|
- Per-language epochs: EN 2.13, NL 4.29, ZH 3.35 — within the ≤10 cap
|
|||
|
|
- 4× NVIDIA A10G, bf16, DDP, eff. batch 131 K tokens/step
|
|||
|
|
- 23,295 steps · 8.6 hours wallclock
|
|||
|
|
|
|||
|
|
## Revisions
|
|||
|
|
|
|||
|
|
`main` is `chck_400M` (the largest fast-eval checkpoint we saved).
|
|||
|
|
Available revisions: chck_1M, chck_2M, chck_3M, chck_4M, chck_5M, chck_6M, chck_7M, chck_8M, chck_9M, chck_10M, chck_20M, chck_30M, chck_40M, chck_50M, chck_60M, chck_70M, chck_80M, chck_90M, chck_100M, chck_200M, chck_300M, chck_400M.
|
|||
|
|
|
|||
|
|
## How to evaluate
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
git clone https://github.com/babylm-org/babylm-eval
|
|||
|
|
cd babylm-eval/multilingual
|
|||
|
|
bash scripts/zeroshot_model.sh --model_name Shamima/babylm-2026-multilingual-v3-quality-filter
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Companion repo (audit, scaffold, ablation configs, iteration log):
|
|||
|
|
https://github.com/silvererudite/bb-lm-challenge-sub
|