120 lines
4.5 KiB
Markdown
120 lines
4.5 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model: Qwen/Qwen3-8B
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- grpo
|
|||
|
|
- rlvr
|
|||
|
|
- math
|
|||
|
|
- reasoning
|
|||
|
|
- slime
|
|||
|
|
datasets:
|
|||
|
|
- EleutherAI/hendrycks_math
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Qwen3-8B-GRPO-MATH
|
|||
|
|
|
|||
|
|
Qwen3-8B trained with **GRPO** on the Hendrycks MATH training split, using verifiable
|
|||
|
|
final-answer rewards. Produced as the RL teacher for a study on how much of a teacher's
|
|||
|
|
RL gain survives on-policy distillation into a smaller student.
|
|||
|
|
|
|||
|
|
**This is a short-budget research checkpoint, not a tuned release.** It is 120 optimizer
|
|||
|
|
updates over roughly a quarter of one epoch. See *Honest limitations* below before using it.
|
|||
|
|
|
|||
|
|
## Training
|
|||
|
|
|
|||
|
|
| | |
|
|||
|
|
|---|---|
|
|||
|
|
| Base model | [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) |
|
|||
|
|
| Algorithm | GRPO (group-relative advantage, no critic) |
|
|||
|
|
| Framework | [slime](https://github.com/THUDM/slime) - SGLang rollout + Megatron training + Ray |
|
|||
|
|
| Data | [EleutherAI/hendrycks_math](https://huggingface.co/datasets/EleutherAI/hendrycks_math) train split, 7,496 prompts after dropping 4 with no parseable `oxed{}` answer |
|
|||
|
|
| Reward | Rule-based final-answer equivalence (slime `--rm-type math`) |
|
|||
|
|
| Optimizer updates | 120 |
|
|||
|
|
| Rollout batch | 16 prompts/step x 8 samples = 128 |
|
|||
|
|
| Global batch | 128 (so 1 update per rollout step) |
|
|||
|
|
| Learning rate | 1e-6, constant |
|
|||
|
|
| Max response length | 4,096 tokens |
|
|||
|
|
| Chat mode | non-thinking (`enable_thinking=false`) |
|
|||
|
|
| Precision | bf16, `--use-precision-aware-optimizer --optimizer-cpu-offload` |
|
|||
|
|
| Hardware | 1x H100 NVL (93 GiB), ~2 h |
|
|||
|
|
|
|||
|
|
Prompt coverage is 16 x 120 = 1,920 of 7,496 prompts, i.e. **0.26 epoch**.
|
|||
|
|
|
|||
|
|
## Reward-function caveat worth repeating
|
|||
|
|
|
|||
|
|
slime ships three math scorers and only one is correct for these labels. Verified on 300
|
|||
|
|
real MATH labels by constructing responses that are correct by definition:
|
|||
|
|
|
|||
|
|
| `--rm-type` | true positives | false positives |
|
|||
|
|
|---|---|---|
|
|||
|
|
| `math` | **300/300** | 0/300 |
|
|||
|
|
| `dapo` | 202/300 | 0/300 |
|
|||
|
|
| `deepscaler` | 0/300 | 0/300 |
|
|||
|
|
|
|||
|
|
`deepscaler` splits on `</think>` and returns 0 for every non-thinking response;
|
|||
|
|
`dapo` needs a literal `Answer:` line. Only ~64% of MATH labels are numeric, the rest are
|
|||
|
|
symbolic LaTeX, and `math` handles all 94 symbolic labels in the sample. This model was
|
|||
|
|
trained with `--rm-type math`.
|
|||
|
|
|
|||
|
|
## Evaluation
|
|||
|
|
|
|||
|
|
avg@16 / pass@16, non-thinking chat mode, temperature 1.0, top-p 1.0, no top-k,
|
|||
|
|
scored with the same harness for every row.
|
|||
|
|
|
|||
|
|
AIME 2024 + 2025, at a **16,384-token** budget where truncation is 0-1% and the
|
|||
|
|
headline number is no longer budget-limited:
|
|||
|
|
|
|||
|
|
| | avg@16 | pass@16 | mean length | trunc% |
|
|||
|
|
|---|---|---|---|---|
|
|||
|
|
| Qwen3-8B (base) | 22.09 | 46.7 | 2,325 | 0.0 |
|
|||
|
|
| **this model** | **34.27** | **66.7** | 5,203 | 0.6 / 1.0 |
|
|||
|
|
|
|||
|
|
MATH-500, at a 4,096-token budget:
|
|||
|
|
|
|||
|
|
| | avg@16 | pass@16 |
|
|||
|
|
|---|---|---|
|
|||
|
|
| Qwen3-8B (base) | 80.84 | 92.8 |
|
|||
|
|
| **this model** | **85.50** | 94.6 |
|
|||
|
|
|
|||
|
|
pass@16 rises alongside avg@16 on every benchmark, so this run expanded the set
|
|||
|
|
of solvable problems rather than only sharpening the top-1 answer.
|
|||
|
|
|
|||
|
|
### Read the budget before you read the score
|
|||
|
|
|
|||
|
|
The model roughly doubles its response length (2,325 -> 5,203 tokens), and a
|
|||
|
|
truncated response scores exactly zero. The same checkpoint measured at smaller
|
|||
|
|
budgets:
|
|||
|
|
|
|||
|
|
| AIME 24/25 avg@16 | 4,096 | 8,192 | 16,384 |
|
|||
|
|
|---|---|---|---|
|
|||
|
|
| score | 28.96 | 34.06 | **34.27** |
|
|||
|
|
| truncation | 48.5% / 46.0% | 21.0% / 17.7% | 0.6% / 1.0% |
|
|||
|
|
|
|||
|
|
At 4,096 tokens you are largely measuring how often it finishes, not whether it
|
|||
|
|
can solve the problem. Give it 16k.
|
|||
|
|
|
|||
|
|
## Honest limitations
|
|||
|
|
|
|||
|
|
- **Short budget.** 120 updates, 0.26 epoch of the 7,496 prompts. Not a converged RL run.
|
|||
|
|
- **Weak gradient signal.** MATH is easy for Qwen3-8B: mean rollout reward ran 0.65-0.97,
|
|||
|
|
and 12-13 of every 16 prompt groups had zero reward variance, so ~78% of rollout compute
|
|||
|
|
produced no gradient. A harder prompt set or DAPO-style dynamic sampling would help.
|
|||
|
|
- **Length inflation.** See the budget table above. This is the single easiest way to
|
|||
|
|
mis-measure this model.
|
|||
|
|
- **The training response cap matters enormously.** An otherwise identical run with the
|
|||
|
|
rollout cap at 2,048 instead of 4,096 tokens reached only +2.04 overall and *lost* a
|
|||
|
|
point on MATH-500. The cap, not the step count, was the binding constraint.
|
|||
|
|
- **Single seed.** AIME24/25 are 30 problems each; repeated avg@16 evaluations of the same
|
|||
|
|
model differ by about +/-2 points. Treat any difference under 2 points as noise.
|
|||
|
|
|
|||
|
|
## Intended use
|
|||
|
|
|
|||
|
|
Research on RL-for-reasoning and on-policy distillation. Not intended for deployment.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Apache-2.0, inherited from Qwen3-8B. The MATH dataset is MIT-licensed.
|