120 lines
4.5 KiB
Markdown
120 lines
4.5 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: Qwen/Qwen3-8B
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- grpo
|
||
- rlvr
|
||
- math
|
||
- reasoning
|
||
- slime
|
||
datasets:
|
||
- EleutherAI/hendrycks_math
|
||
language:
|
||
- en
|
||
---
|
||
|
||
# Qwen3-8B-GRPO-MATH
|
||
|
||
Qwen3-8B trained with **GRPO** on the Hendrycks MATH training split, using verifiable
|
||
final-answer rewards. Produced as the RL teacher for a study on how much of a teacher's
|
||
RL gain survives on-policy distillation into a smaller student.
|
||
|
||
**This is a short-budget research checkpoint, not a tuned release.** It is 120 optimizer
|
||
updates over roughly a quarter of one epoch. See *Honest limitations* below before using it.
|
||
|
||
## Training
|
||
|
||
| | |
|
||
|---|---|
|
||
| Base model | [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) |
|
||
| Algorithm | GRPO (group-relative advantage, no critic) |
|
||
| Framework | [slime](https://github.com/THUDM/slime) - SGLang rollout + Megatron training + Ray |
|
||
| Data | [EleutherAI/hendrycks_math](https://huggingface.co/datasets/EleutherAI/hendrycks_math) train split, 7,496 prompts after dropping 4 with no parseable `oxed{}` answer |
|
||
| Reward | Rule-based final-answer equivalence (slime `--rm-type math`) |
|
||
| Optimizer updates | 120 |
|
||
| Rollout batch | 16 prompts/step x 8 samples = 128 |
|
||
| Global batch | 128 (so 1 update per rollout step) |
|
||
| Learning rate | 1e-6, constant |
|
||
| Max response length | 4,096 tokens |
|
||
| Chat mode | non-thinking (`enable_thinking=false`) |
|
||
| Precision | bf16, `--use-precision-aware-optimizer --optimizer-cpu-offload` |
|
||
| Hardware | 1x H100 NVL (93 GiB), ~2 h |
|
||
|
||
Prompt coverage is 16 x 120 = 1,920 of 7,496 prompts, i.e. **0.26 epoch**.
|
||
|
||
## Reward-function caveat worth repeating
|
||
|
||
slime ships three math scorers and only one is correct for these labels. Verified on 300
|
||
real MATH labels by constructing responses that are correct by definition:
|
||
|
||
| `--rm-type` | true positives | false positives |
|
||
|---|---|---|
|
||
| `math` | **300/300** | 0/300 |
|
||
| `dapo` | 202/300 | 0/300 |
|
||
| `deepscaler` | 0/300 | 0/300 |
|
||
|
||
`deepscaler` splits on `</think>` and returns 0 for every non-thinking response;
|
||
`dapo` needs a literal `Answer:` line. Only ~64% of MATH labels are numeric, the rest are
|
||
symbolic LaTeX, and `math` handles all 94 symbolic labels in the sample. This model was
|
||
trained with `--rm-type math`.
|
||
|
||
## Evaluation
|
||
|
||
avg@16 / pass@16, non-thinking chat mode, temperature 1.0, top-p 1.0, no top-k,
|
||
scored with the same harness for every row.
|
||
|
||
AIME 2024 + 2025, at a **16,384-token** budget where truncation is 0-1% and the
|
||
headline number is no longer budget-limited:
|
||
|
||
| | avg@16 | pass@16 | mean length | trunc% |
|
||
|---|---|---|---|---|
|
||
| Qwen3-8B (base) | 22.09 | 46.7 | 2,325 | 0.0 |
|
||
| **this model** | **34.27** | **66.7** | 5,203 | 0.6 / 1.0 |
|
||
|
||
MATH-500, at a 4,096-token budget:
|
||
|
||
| | avg@16 | pass@16 |
|
||
|---|---|---|
|
||
| Qwen3-8B (base) | 80.84 | 92.8 |
|
||
| **this model** | **85.50** | 94.6 |
|
||
|
||
pass@16 rises alongside avg@16 on every benchmark, so this run expanded the set
|
||
of solvable problems rather than only sharpening the top-1 answer.
|
||
|
||
### Read the budget before you read the score
|
||
|
||
The model roughly doubles its response length (2,325 -> 5,203 tokens), and a
|
||
truncated response scores exactly zero. The same checkpoint measured at smaller
|
||
budgets:
|
||
|
||
| AIME 24/25 avg@16 | 4,096 | 8,192 | 16,384 |
|
||
|---|---|---|---|
|
||
| score | 28.96 | 34.06 | **34.27** |
|
||
| truncation | 48.5% / 46.0% | 21.0% / 17.7% | 0.6% / 1.0% |
|
||
|
||
At 4,096 tokens you are largely measuring how often it finishes, not whether it
|
||
can solve the problem. Give it 16k.
|
||
|
||
## Honest limitations
|
||
|
||
- **Short budget.** 120 updates, 0.26 epoch of the 7,496 prompts. Not a converged RL run.
|
||
- **Weak gradient signal.** MATH is easy for Qwen3-8B: mean rollout reward ran 0.65-0.97,
|
||
and 12-13 of every 16 prompt groups had zero reward variance, so ~78% of rollout compute
|
||
produced no gradient. A harder prompt set or DAPO-style dynamic sampling would help.
|
||
- **Length inflation.** See the budget table above. This is the single easiest way to
|
||
mis-measure this model.
|
||
- **The training response cap matters enormously.** An otherwise identical run with the
|
||
rollout cap at 2,048 instead of 4,096 tokens reached only +2.04 overall and *lost* a
|
||
point on MATH-500. The cap, not the step count, was the binding constraint.
|
||
- **Single seed.** AIME24/25 are 30 problems each; repeated avg@16 evaluations of the same
|
||
model differ by about +/-2 points. Treat any difference under 2 points as noise.
|
||
|
||
## Intended use
|
||
|
||
Research on RL-for-reasoning and on-policy distillation. Not intended for deployment.
|
||
|
||
## License
|
||
|
||
Apache-2.0, inherited from Qwen3-8B. The MATH dataset is MIT-licensed.
|