68 lines
2.4 KiB
Markdown
68 lines
2.4 KiB
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
- zh
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags:
|
||
|
|
- pretraining
|
||
|
|
- language-ordering
|
||
|
|
- curriculum
|
||
|
|
- catastrophic-forgetting
|
||
|
|
- bilingual
|
||
|
|
- smollm2
|
||
|
|
license: other
|
||
|
|
---
|
||
|
|
|
||
|
|
# pathlang-1p7b-runE-en30-zh70
|
||
|
|
|
||
|
|
A 1.7B-parameter bilingual (English + Chinese) language model trained with a **purely sequential
|
||
|
|
monolingual curriculum**: all of one language, then all of the other. This is the **English-first sequential (en30->zh70)** run, part
|
||
|
|
of a controlled language-ordering study. Companion blended-curriculum runs (50/50 diet, 3-phase) live at
|
||
|
|
`Raghav-Singhal/pathlang-1p7b-runA-zh-first`, `-runB-en-first`, `-runC-5050`.
|
||
|
|
|
||
|
|
## Curriculum (this run)
|
||
|
|
|
||
|
|
- Phase 1 (0-30B): 100% English
|
||
|
|
- Phase 2 (30-100B): 100% Chinese
|
||
|
|
|
||
|
|
The mirror run is `Raghav-Singhal/pathlang-1p7b-runD-zh30-en70`.
|
||
|
|
|
||
|
|
## Architecture
|
||
|
|
|
||
|
|
- SmolLM2-1.7B backbone: 24 layers, hidden 2048, FFN 8192, 32 heads, RoPE (base 10000), RMSNorm, SwiGLU, seq len 2048
|
||
|
|
- **Tokenizer: Qwen3** (multilingual, vocab 151,936); ~2.1B total params
|
||
|
|
- Converted from Megatron-LM to HF `LlamaForCausalLM`
|
||
|
|
|
||
|
|
## Training
|
||
|
|
|
||
|
|
- 100B tokens total, global batch size 960, 50,860 steps
|
||
|
|
- WSD LR schedule (peak 2e-4, 2000 warmup, linear decay over the final 10B tokens), bf16, Adam
|
||
|
|
- English: DCLM-edu. Chinese: **FineWeb-2 hq-mmbert quality_33** (`cmn_Hani`) — a larger slice than the
|
||
|
|
quality_10 used by the A/B/C runs, needed for the 70B Chinese phase.
|
||
|
|
|
||
|
|
## Held-out validation loss (final checkpoint, 100B tokens)
|
||
|
|
|
||
|
|
| | English val | Chinese val |
|
||
|
|
|---|---|---|
|
||
|
|
| this run (English-first sequential (en30->zh70)) | 3.997 | 2.531 |
|
||
|
|
|
||
|
|
In-loop 128-seq probe, cross-entropy nats/token, on this study's held-out blocks (comparable within the
|
||
|
|
D/E pair). **Key finding — catastrophic forgetting under sequential training:** the model ends excellent
|
||
|
|
at its *second* (70B) language and substantially degraded at the first, abandoned one. E.g. this run's
|
||
|
|
first-phase language rose from ~2.7 loss at the 30B switch to the value above by 100B — much sharper
|
||
|
|
forgetting than the blended A/B runs, which retained 25% of the earlier language throughout.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
m = AutoModelForCausalLM.from_pretrained("Raghav-Singhal/pathlang-1p7b-runE-en30-zh70")
|
||
|
|
tok = AutoTokenizer.from_pretrained("Raghav-Singhal/pathlang-1p7b-runE-en30-zh70")
|
||
|
|
```
|
||
|
|
|
||
|
|
## Note
|
||
|
|
|
||
|
|
License `other` pending confirmation; DCLM-edu and FineWeb-2 terms apply. Base (non-instruction-tuned)
|
||
|
|
research checkpoint.
|