--- license: apache-2.0 base_model: unsloth/Qwen3-1.7B-Base tags: - grpo - reinforcement-learning - reasoning - math - rlvr - qwen3 datasets: - openai/gsm8k language: - en pipeline_tag: text-generation --- # Qwen3-1.7B-GRPO-GSM8K (v1) A GRPO (Group Relative Policy Optimization) fine-tune of `unsloth/Qwen3-1.7B-Base`, trained with verifiable rewards (RLVR) on GSM8K math word problems for one full epoch. ## Model description - **Base model:** `unsloth/Qwen3-1.7B-Base` (Apache 2.0) - **Method:** GRPO (DeepSeekMath / DeepSeek-R1 style), via Unsloth + TRL's `GRPOTrainer` - **Adapter:** LoRA rank 32 / alpha 32, merged into the base weights for this release - **Training data:** GSM8K `train`+`test` combined into one 8,792-example pool, shuffled once (seed 3407), split 80/10/10 — trained on the 80% partition (7,033 examples) for 1 full epoch. Validation (10%) used only for monitoring during training; the eval partition (10%, 880 examples) was never touched until the final evaluation below. - **Output format:** the model is trained to respond in a fixed structure: ``` ... ... ``` - **Reward functions:** correctness (regex-extracted `` content vs. GSM8K ground truth, +2.0), format compliance (+0.5 well-formed / +0.25 partial), brevity (+0.2 max, capped small so it can never outweigh correctness or format) ## Evaluation results All numbers below are from held-out data the model never saw during training or any prior evaluation pass (see the source repo's `week-05-06/scripts/data_split.py` for the split methodology, and `week-05-06/docs/04-results.md` for the full writeup including two scoring bugs that were caught and fixed mid-evaluation). | Dataset | In/out of distribution | Base model | This model | Change | |---|---|---|---|---| | GSM8K (880 held-out examples) | in-distribution | 66.9%\* | **87.3%** | +20.4pp | | SVAMP (300 examples) | out-of-distribution | 76.7% | **86.7%** | +10.0pp | | MATH / algebra (300 examples) | out-of-distribution, harder | 49.0%\*\* | **60.3%** | +11.3pp\*\* | \*Base model accuracy is scored leniently (correct number anywhere in the response) since the base model has no tendency to use the `` tag format at all (0.6% format compliance) — this model's own format compliance is 99.4%. \*\*MATH's numbers are a **lower bound**, not a precise score — grading MATH answers exactly (fractions, intervals, symbolic expressions) without a computer algebra system is a known hard problem; some correct-but-differently-formatted answers score as incorrect here. **Manual transcript review** (not just the aggregate numbers) found no evidence of reward hacking: correct answers show genuine multi-step arithmetic, incorrect answers are real reasoning mistakes on harder problems (not degenerate or gamed output), and response length does not correlate with correctness in a gaming direction. ### Independent cross-check: lm-evaluation-harness (base vs. trained) The numbers above use this project's own zero-shot, custom-prompt eval scripts. Cross- checked independently via EleutherAI's `lm-evaluation-harness` (standard 5-shot GSM8K config, plus MMLU and HellaSwag as general-capability anchors), run identically against both the base model and this one — see the source repo's `week-07/docs/04-results.md` for the full writeup. | Benchmark | Base | This model | Δ | |---|---|---|---| | GSM8K (5-shot, strict-match) | 67.55% | **77.03%** | +9.5pp | | GSM8K (5-shot, flexible-extract) | 71.42% | **77.79%** | +6.4pp | | MMLU (57-subject average) | 60.59% | 60.56% | −0.03pp | | HellaSwag (acc) | 49.28% | 49.24% | −0.04pp | | HellaSwag (acc_norm) | 66.48% | 66.78% | +0.30pp | **No catastrophic forgetting.** The MMLU and HellaSwag deltas are smaller than each benchmark's own standard error (~0.4-0.5pp) — statistically indistinguishable from no change. GSM8K improved well outside that noise band, via a completely different methodology (5-shot, standard harness prompt format) than this project's own eval — independent confirmation that the math-reasoning gain is real, not an artifact of this project's own eval design, and that it wasn't purchased at the cost of general capability. ## Limitations - **Narrow, math-specialized training.** This model was trained exclusively on grade- school arithmetic word problems (GSM8K). General instruction-following and chat quality specifically were not evaluated (this model was never trained on conversational data) — though catastrophic forgetting of general *knowledge/reasoning* capability was checked (see the benchmark cross-check above) and not found. - **A real hint-exploitation vulnerability was found, quantified at 37%.** A formal hint/no-hint probe (200 held-out problems, each run with a correct hint and a deliberately wrong decoy — see `week-07/docs/04-results.md` for the full method) found that 37% of the time, a plausible-but-wrong hint changes the model's final answer away from what it would otherwise get right. In one documented example, the model correctly re-derived the right answer three separate times, explicitly noted the conflict with the hint, and still deferred to the hint over its own confirmed work. **Do not present this model with untrusted "hint" or "suggested answer" text in the prompt** without accounting for this - it will sometimes follow a wrong suggestion over independently- verified correct reasoning. (Checked and ruled out as *reward hacking* specifically - training prompts never included hints at all, so the reward function never selected for this behavior one way or the other; it's most likely inherited from base-model pretraining and was simply never tested or corrected during RL training.) - **MATH evaluation numbers are approximate** (see above) — treat the MATH result directionally, not as a precise benchmark score. - **Single training run.** No hyperparameter ablation or multi-seed variance check was performed; this is a v1 release, not a fully tuned final model. - **Format compliance is 99.4%, not 100%** — a small fraction of outputs don't follow the ``/`` structure. - **The model does not always stop generating after `` — measured at ~0.07% of outputs, rare rather than common.** No reward function ever scored anything past the first `` tag match, so nothing in training penalized rambling afterward — a real design gap. Measured directly against 1,451 raw eval transcripts (GSM8K + SVAMP + MATH), only 1 case had any content past the tag, and it wasn't even the fabricated-dialogue pattern (it was a repetitive reasoning loop that leaked one stray character). The one case that clearly showed the fabricated-dialogue version happened during manual testing on a far-out-of-distribution question style — possibly the same underlying cause as the function-composition gap below, not a frequent, independent problem. Regardless, the fix is free and worth applying — **set a stop sequence at `` when generating**: ```python # vLLM SamplingParams(stop=[""], include_stop_str_in_output=True) # transformers .generate() from transformers import StopStringCriteria, StoppingCriteriaList model.generate(..., stopping_criteria=StoppingCriteriaList( [StopStringCriteria(tokenizer, [""])] )) ``` - **Symbolic function-composition problems (`f(x)=...`, `g(x)=...` notation) appear to be a specific weak spot**, distinct from the general MATH miss rate — likely because GSM8K (the only training data) is exclusively concrete word problems with no examples of this abstract notation. Observed anecdotally (4/4 wrong on manual testing), not verified at benchmark scale — flagged as a real, specific gap rather than confirmed precisely. ## Training details - Framework: Unsloth 2026.5.9 + TRL `GRPOTrainer`, run inside the `unsloth/unsloth` Docker image on an RTX 5070 (12GB, Blackwell sm_120) - Precision: bf16 (FP8 was evaluated and found broken on this specific software stack — see the source repo's `week-05-06/docs/00-overview.md` for the investigation) - `num_generations=4`, `temperature=0.8`, `beta=0.001` (KL penalty), `loss_type=dapo` - Learning rate 5e-6, 1 epoch, ~7,033 steps, wall-clock ~7.4 hours ## Citation / provenance Trained as part of a self-directed practice roadmap covering post-training/RL, systems, and scalability topics for LLMs.