68 lines
2.3 KiB
Markdown
68 lines
2.3 KiB
Markdown
---
|
|
language:
|
|
- en
|
|
- zh
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- pretraining
|
|
- language-ordering
|
|
- curriculum
|
|
- bilingual
|
|
- smollm2
|
|
license: other
|
|
---
|
|
|
|
# pathlang-1p7b-runB-en-first
|
|
|
|
A 1.7B-parameter bilingual (English + Chinese) language model, part of a **controlled
|
|
language-ordering study**. Three models share the *same* architecture, initialization, total data
|
|
diet (50B English + 50B Chinese = 100B tokens), and LR schedule; they differ **only** in the order
|
|
in which the two languages are presented during pretraining. This is the **English-first** run.
|
|
|
|
## Curriculum (this run)
|
|
|
|
- Phase 1 (0-30B): 100% English
|
|
- Phase 2 (30-90B): 75% Chinese / 25% English
|
|
- Phase 3 (90-100B): 50/50
|
|
|
|
The other runs in the study: `Raghav-Singhal/pathlang-1p7b-runA-zh-first`,
|
|
`Raghav-Singhal/pathlang-1p7b-runB-en-first`, `Raghav-Singhal/pathlang-1p7b-runC-5050`.
|
|
|
|
## Architecture
|
|
|
|
- SmolLM2-1.7B backbone: 24 layers, hidden size 2048, FFN 8192, 32 attention heads
|
|
- RoPE (base 10000), RMSNorm, SwiGLU, no biases, sequence length 2048
|
|
- **Tokenizer: Qwen3** (multilingual, vocab 151,936)
|
|
- ~2.1B total parameters (with the Qwen3 embedding)
|
|
|
|
## Training
|
|
|
|
- 100B tokens: 50B English (DCLM-edu) + 50B Chinese (FineWeb-2 `cmn_Hani`)
|
|
- Global batch size 960, sequence length 2048, 50,860 steps
|
|
- WSD LR schedule (peak 2e-4, 2000 warmup, linear decay over the final 10B tokens), bf16, Adam
|
|
- Converted from Megatron-LM to HF `LlamaForCausalLM` format
|
|
|
|
## Held-out validation loss (final checkpoint, 100B tokens)
|
|
|
|
| | English val | Chinese val |
|
|
|---|---|---|
|
|
| this run (English-first) | 2.657 | 2.429 |
|
|
|
|
Cross-entropy in nats/token on held-out blocks of the training corpora. The headline finding of the
|
|
study: each run ends **best at the language that dominated its middle (bulk) phase** — a recency
|
|
effect that also holds on an out-of-distribution corpus (HPLT4.0).
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
m = AutoModelForCausalLM.from_pretrained("Raghav-Singhal/pathlang-1p7b-runB-en-first")
|
|
tok = AutoTokenizer.from_pretrained("Raghav-Singhal/pathlang-1p7b-runB-en-first")
|
|
```
|
|
|
|
## Note
|
|
|
|
License is set to `other` pending confirmation; the training data (DCLM-edu, FineWeb-2) and their
|
|
respective terms apply. This is a base (non-instruction-tuned) research checkpoint.
|