pathlang-1p7b-runD-zh30-en70/README.md

language, library_name, pipeline_tag, tags, license

language	library_name	pipeline_tag	tags	license
en zh	transformers	text-generation	pretraining language-ordering curriculum catastrophic-forgetting bilingual smollm2	other

pathlang-1p7b-runD-zh30-en70

A 1.7B-parameter bilingual (English + Chinese) language model trained with a purely sequential monolingual curriculum: all of one language, then all of the other. This is the Chinese-first sequential (zh30->en70) run, part of a controlled language-ordering study. Companion blended-curriculum runs (50/50 diet, 3-phase) live at Raghav-Singhal/pathlang-1p7b-runA-zh-first, -runB-en-first, -runC-5050.

Curriculum (this run)

• Phase 1 (0-30B): 100% Chinese
• Phase 2 (30-100B): 100% English

The mirror run is Raghav-Singhal/pathlang-1p7b-runE-en30-zh70.

Architecture

• SmolLM2-1.7B backbone: 24 layers, hidden 2048, FFN 8192, 32 heads, RoPE (base 10000), RMSNorm, SwiGLU, seq len 2048
• Tokenizer: Qwen3 (multilingual, vocab 151,936); ~2.1B total params
• Converted from Megatron-LM to HF LlamaForCausalLM

Training

• 100B tokens total, global batch size 960, 50,860 steps
• WSD LR schedule (peak 2e-4, 2000 warmup, linear decay over the final 10B tokens), bf16, Adam
• English: DCLM-edu. Chinese: FineWeb-2 hq-mmbert quality_33 (cmn_Hani) — a larger slice than the quality_10 used by the A/B/C runs, needed for the 70B Chinese phase.

Held-out validation loss (final checkpoint, 100B tokens)

	English val	Chinese val
this run (Chinese-first sequential (zh30->en70))	2.516	5.133

In-loop 128-seq probe, cross-entropy nats/token, on this study's held-out blocks (comparable within the D/E pair). Key finding — catastrophic forgetting under sequential training: the model ends excellent at its second (70B) language and substantially degraded at the first, abandoned one. E.g. this run's first-phase language rose from ~2.7 loss at the 30B switch to the value above by 100B — much sharper forgetting than the blended A/B runs, which retained 25% of the earlier language throughout.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("Raghav-Singhal/pathlang-1p7b-runD-zh30-en70")
tok = AutoTokenizer.from_pretrained("Raghav-Singhal/pathlang-1p7b-runD-zh30-en70")

Note

License other pending confirmation; DCLM-edu and FineWeb-2 terms apply. Base (non-instruction-tuned) research checkpoint.