Qwen3-8B-GRPO-MATH/README.md

license, base_model, pipeline_tag, tags, datasets, language

license	base_model	pipeline_tag	tags	datasets	language
apache-2.0	Qwen/Qwen3-8B	text-generation	grpo rlvr math reasoning slime	EleutherAI/hendrycks_math	en

Qwen3-8B-GRPO-MATH

Qwen3-8B trained with GRPO on the Hendrycks MATH training split, using verifiable final-answer rewards. Produced as the RL teacher for a study on how much of a teacher's RL gain survives on-policy distillation into a smaller student.

This is a short-budget research checkpoint, not a tuned release. It is 120 optimizer updates over roughly a quarter of one epoch. See Honest limitations below before using it.

Training

Base model	Qwen/Qwen3-8B
Algorithm	GRPO (group-relative advantage, no critic)
Framework	slime - SGLang rollout + Megatron training + Ray
Data	EleutherAI/hendrycks_math train split, 7,496 prompts after dropping 4 with no parseable oxed{} answer
Reward	Rule-based final-answer equivalence (slime --rm-type math)
Optimizer updates	120
Rollout batch	16 prompts/step x 8 samples = 128
Global batch	128 (so 1 update per rollout step)
Learning rate	1e-6, constant
Max response length	4,096 tokens
Chat mode	non-thinking (enable_thinking=false)
Precision	bf16, --use-precision-aware-optimizer --optimizer-cpu-offload
Hardware	1x H100 NVL (93 GiB), ~2 h

Prompt coverage is 16 x 120 = 1,920 of 7,496 prompts, i.e. 0.26 epoch.

Reward-function caveat worth repeating

slime ships three math scorers and only one is correct for these labels. Verified on 300 real MATH labels by constructing responses that are correct by definition:

--rm-type	true positives	false positives
math	300/300	0/300
dapo	202/300	0/300
deepscaler	0/300	0/300

deepscaler splits on </think> and returns 0 for every non-thinking response; dapo needs a literal Answer: line. Only ~64% of MATH labels are numeric, the rest are symbolic LaTeX, and math handles all 94 symbolic labels in the sample. This model was trained with --rm-type math.

Evaluation

avg@16 / pass@16, non-thinking chat mode, temperature 1.0, top-p 1.0, no top-k, scored with the same harness for every row.

AIME 2024 + 2025, at a 16,384-token budget where truncation is 0-1% and the headline number is no longer budget-limited:

	avg@16	pass@16	mean length	trunc%
Qwen3-8B (base)	22.09	46.7	2,325	0.0
this model	34.27	66.7	5,203	0.6 / 1.0

MATH-500, at a 4,096-token budget:

	avg@16	pass@16
Qwen3-8B (base)	80.84	92.8
this model	85.50	94.6

pass@16 rises alongside avg@16 on every benchmark, so this run expanded the set of solvable problems rather than only sharpening the top-1 answer.

Read the budget before you read the score

The model roughly doubles its response length (2,325 -> 5,203 tokens), and a truncated response scores exactly zero. The same checkpoint measured at smaller budgets:

AIME 24/25 avg@16	4,096	8,192	16,384
score	28.96	34.06	34.27
truncation	48.5% / 46.0%	21.0% / 17.7%	0.6% / 1.0%

At 4,096 tokens you are largely measuring how often it finishes, not whether it can solve the problem. Give it 16k.

Honest limitations

• Short budget. 120 updates, 0.26 epoch of the 7,496 prompts. Not a converged RL run.
• Weak gradient signal. MATH is easy for Qwen3-8B: mean rollout reward ran 0.65-0.97, and 12-13 of every 16 prompt groups had zero reward variance, so ~78% of rollout compute produced no gradient. A harder prompt set or DAPO-style dynamic sampling would help.
• Length inflation. See the budget table above. This is the single easiest way to mis-measure this model.
• The training response cap matters enormously. An otherwise identical run with the rollout cap at 2,048 instead of 4,096 tokens reached only +2.04 overall and lost a point on MATH-500. The cap, not the step count, was the binding constraint.
• Single seed. AIME24/25 are 30 problems each; repeated avg@16 evaluations of the same model differ by about +/-2 points. Treat any difference under 2 points as noise.

Intended use

Research on RL-for-reasoning and on-policy distillation. Not intended for deployment.

License

Apache-2.0, inherited from Qwen3-8B. The MATH dataset is MIT-licensed.