84 lines
3.6 KiB
Markdown
84 lines
3.6 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
base_model: RockToken/qwen3-30b-a3b-to-4b-onpolicy-10k
|
|||
|
|
datasets:
|
|||
|
|
- RockToken/openthoughts_prompt_math_5k_src30k-35k
|
|||
|
|
tags:
|
|||
|
|
- knowledge-distillation
|
|||
|
|
- on-policy-distillation
|
|||
|
|
- math
|
|||
|
|
- qwen3
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Qwen3-4B distilled from Qwen3-30B-A3B — On-policy 5k (continual, prompts 30k–35k)
|
|||
|
|
|
|||
|
|
Continual on-policy knowledge distillation of a Qwen3-4B student toward the Qwen3-30B-A3B (MoE) teacher, on 5,000 math prompts from `openthoughts3_math_50k` (positions 30k–35k, disjoint from prior training slices).
|
|||
|
|
|
|||
|
|
The student for this run is [RockToken/qwen3-30b-a3b-to-4b-onpolicy-10k](https://huggingface.co/RockToken/qwen3-30b-a3b-to-4b-onpolicy-10k), so the effective training exposure of this checkpoint is:
|
|||
|
|
|
|||
|
|
1. **Off-policy KD** on 20k teacher responses (positions 0–20k of `openthoughts3_math_50k`)
|
|||
|
|
2. **On-policy KD** on 10k prompts (positions 20k–30k) — produced the 10k model
|
|||
|
|
3. **On-policy KD (this run)** on 5k prompts (positions 30k–35k) — this checkpoint
|
|||
|
|
|
|||
|
|
Stages 2 and 3 use the same on-policy pipeline; only the seed weights, prompt slice, and step count differ.
|
|||
|
|
|
|||
|
|
## Models
|
|||
|
|
|
|||
|
|
| Role | Model |
|
|||
|
|
|---------|--------------------------------------------------------|
|
|||
|
|
| Student | Qwen3-4B-Instruct-2507 (after stages 1–2 above) |
|
|||
|
|
| Teacher | Qwen/Qwen3-30B-A3B-Instruct-2507 (MoE, ~3B active) |
|
|||
|
|
|
|||
|
|
`enable_thinking=False` throughout (both stages).
|
|||
|
|
|
|||
|
|
## Training data
|
|||
|
|
|
|||
|
|
- Source: `openthoughts3_math_50k` (math split of OpenThoughts-3)
|
|||
|
|
- Slice used here: prompts 30k–35k → [RockToken/openthoughts_prompt_math_5k_src30k-35k](https://huggingface.co/datasets/RockToken/openthoughts_prompt_math_5k_src30k-35k)
|
|||
|
|
- 5,000 single-user-turn prompts, mean length 261 chars, p99 845 chars
|
|||
|
|
|
|||
|
|
## Training setup
|
|||
|
|
|
|||
|
|
Framework: [KDFlow](https://github.com/wanng-ide/KDFlow) — FSDP2 + SGLang rollout, Ray-orchestrated GPU co-location with sleep/wakeup.
|
|||
|
|
|
|||
|
|
Hardware: 1× node, 4× H100 (80 GB), ~34 h wall-clock (~136 GPU-hours).
|
|||
|
|
|
|||
|
|
### Key hyperparameters (on-policy stage; same as stage 2)
|
|||
|
|
|
|||
|
|
| Group | Value |
|
|||
|
|
|-----------------|--------------------------------|
|
|||
|
|
| Backend | fsdp2, bf16, gradient ckpt on |
|
|||
|
|
| Epochs | 1 |
|
|||
|
|
| Train batch | 4 (micro 1) |
|
|||
|
|
| Learning rate | 2e-6, cosine, warmup 5% |
|
|||
|
|
| KD ratio | 1.0 |
|
|||
|
|
| KD loss | reverse KL (`rkl`) |
|
|||
|
|
| KD algorithm | `vanilla_kd` |
|
|||
|
|
| Temperature (KD)| 1.0 |
|
|||
|
|
| Rollout engine | SGLang, TP=2, 1 engine |
|
|||
|
|
| Rollout batch | 2 prompts × 4 samples/prompt |
|
|||
|
|
| `generate_max_len` | 8000 |
|
|||
|
|
| `prompt_max_len` | 800 (total `max_len` 8800) |
|
|||
|
|
| Sampling | temperature 1.0, top-p 1.0 |
|
|||
|
|
| Teacher | TP=4, sleep/wakeup enabled |
|
|||
|
|
|
|||
|
|
Off-policy stage 1 hyperparameters (for context): batch 128, LR 2e-5, `kd_ratio` 0.5, `kd_loss` KL, `max_len` 16384, packing_samples on, ring attention.
|
|||
|
|
|
|||
|
|
## Intended use
|
|||
|
|
|
|||
|
|
Research on distillation dynamics. Domain skew is math (OpenThoughts-3 math split).
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
- Trained end-to-end on math prompts only; not tuned for chat, safety, or non-math domains.
|
|||
|
|
- `enable_thinking=False` — this student does not emit `<think>` traces.
|
|||
|
|
- Continual training may drift capabilities relative to the base checkpoint; see the wandb curves for loss dynamics.
|
|||
|
|
|
|||
|
|
## Weights & Biases
|
|||
|
|
|
|||
|
|
`legendary-music-16` (stage 2, 10k) and `lemon-sound-17` / `2gvd60dw` (this run, stage 3) under project `KDFlow`, entity `zhao666`.
|