---
language:
- en
- zh
library_name: transformers
pipeline_tag: text-generation
tags:
- pretraining
- language-ordering
- curriculum
- catastrophic-forgetting
- bilingual
- smollm2
license: other
---

# pathlang-1p7b-runE-en30-zh70

A 1.7B-parameter bilingual (English + Chinese) language model trained with a **purely sequential
monolingual curriculum**: all of one language, then all of the other. This is the **English-first sequential (en30->zh70)** run, part
of a controlled language-ordering study. Companion blended-curriculum runs (50/50 diet, 3-phase) live at
`Raghav-Singhal/pathlang-1p7b-runA-zh-first`, `-runB-en-first`, `-runC-5050`.

## Curriculum (this run)

- Phase 1 (0-30B): 100% English
- Phase 2 (30-100B): 100% Chinese

The mirror run is `Raghav-Singhal/pathlang-1p7b-runD-zh30-en70`.

## Architecture

- SmolLM2-1.7B backbone: 24 layers, hidden 2048, FFN 8192, 32 heads, RoPE (base 10000), RMSNorm, SwiGLU, seq len 2048
- **Tokenizer: Qwen3** (multilingual, vocab 151,936); ~2.1B total params
- Converted from Megatron-LM to HF `LlamaForCausalLM`

## Training

- 100B tokens total, global batch size 960, 50,860 steps
- WSD LR schedule (peak 2e-4, 2000 warmup, linear decay over the final 10B tokens), bf16, Adam
- English: DCLM-edu. Chinese: **FineWeb-2 hq-mmbert quality_33** (`cmn_Hani`) — a larger slice than the
  quality_10 used by the A/B/C runs, needed for the 70B Chinese phase.

## Held-out validation loss (final checkpoint, 100B tokens)

| | English val | Chinese val |
|---|---|---|
| this run (English-first sequential (en30->zh70)) | 3.997 | 2.531 |

In-loop 128-seq probe, cross-entropy nats/token, on this study's held-out blocks (comparable within the
D/E pair). **Key finding — catastrophic forgetting under sequential training:** the model ends excellent
at its *second* (70B) language and substantially degraded at the first, abandoned one. E.g. this run's
first-phase language rose from ~2.7 loss at the 30B switch to the value above by 100B — much sharper
forgetting than the blended A/B runs, which retained 25% of the earlier language throughout.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("Raghav-Singhal/pathlang-1p7b-runE-en30-zh70")
tok = AutoTokenizer.from_pretrained("Raghav-Singhal/pathlang-1p7b-runE-en30-zh70")
```

## Note

License `other` pending confirmation; DCLM-edu and FineWeb-2 terms apply. Base (non-instruction-tuned)
research checkpoint.