qwen-1.7b-math-reasoner-grpo/README.md

---
license: apache-2.0
base_model: unsloth/Qwen3-1.7B-Base
tags:
  - grpo
  - reinforcement-learning
  - reasoning
  - math
  - rlvr
  - qwen3
datasets:
  - openai/gsm8k
language:
  - en
pipeline_tag: text-generation
---

# Qwen3-1.7B-GRPO-GSM8K (v1)

A GRPO (Group Relative Policy Optimization) fine-tune of `unsloth/Qwen3-1.7B-Base`,
trained with verifiable rewards (RLVR) on GSM8K math word problems for one full epoch.

## Model description

- **Base model:** `unsloth/Qwen3-1.7B-Base` (Apache 2.0)
- **Method:** GRPO (DeepSeekMath / DeepSeek-R1 style), via Unsloth + TRL's `GRPOTrainer`
- **Adapter:** LoRA rank 32 / alpha 32, merged into the base weights for this release
- **Training data:** GSM8K `train`+`test` combined into one 8,792-example pool, shuffled
  once (seed 3407), split 80/10/10 — trained on the 80% partition (7,033 examples) for
  1 full epoch. Validation (10%) used only for monitoring during training; the eval
  partition (10%, 880 examples) was never touched until the final evaluation below.
- **Output format:** the model is trained to respond in a fixed structure:
  ```
  <reasoning>
  ...
  </reasoning>
  <answer>
  ...
  </answer>
  ```
- **Reward functions:** correctness (regex-extracted `<answer>` content vs. GSM8K ground
  truth, +2.0), format compliance (+0.5 well-formed / +0.25 partial), brevity
  (+0.2 max, capped small so it can never outweigh correctness or format)

## Evaluation results

All numbers below are from held-out data the model never saw during training or any
prior evaluation pass (see the source repo's `week-05-06/scripts/data_split.py` for the
split methodology, and `week-05-06/docs/04-results.md` for the full writeup including
two scoring bugs that were caught and fixed mid-evaluation).

| Dataset | In/out of distribution | Base model | This model | Change |
|---|---|---|---|---|
| GSM8K (880 held-out examples) | in-distribution | 66.9%\* | **87.3%** | +20.4pp |
| SVAMP (300 examples) | out-of-distribution | 76.7% | **86.7%** | +10.0pp |
| MATH / algebra (300 examples) | out-of-distribution, harder | 49.0%\*\* | **60.3%** | +11.3pp\*\* |

\*Base model accuracy is scored leniently (correct number anywhere in the response) since
the base model has no tendency to use the `<answer>` tag format at all (0.6% format
compliance) — this model's own format compliance is 99.4%.

\*\*MATH's numbers are a **lower bound**, not a precise score — grading MATH answers
exactly (fractions, intervals, symbolic expressions) without a computer algebra system is
a known hard problem; some correct-but-differently-formatted answers score as incorrect
here.

**Manual transcript review** (not just the aggregate numbers) found no evidence of
reward hacking: correct answers show genuine multi-step arithmetic, incorrect answers
are real reasoning mistakes on harder problems (not degenerate or gamed output), and
response length does not correlate with correctness in a gaming direction.

### Independent cross-check: lm-evaluation-harness (base vs. trained)

The numbers above use this project's own zero-shot, custom-prompt eval scripts. Cross-
checked independently via EleutherAI's `lm-evaluation-harness` (standard 5-shot GSM8K
config, plus MMLU and HellaSwag as general-capability anchors), run identically against
both the base model and this one — see the source repo's `week-07/docs/04-results.md`
for the full writeup.

| Benchmark | Base | This model | Δ |
|---|---|---|---|
| GSM8K (5-shot, strict-match) | 67.55% | **77.03%** | +9.5pp |
| GSM8K (5-shot, flexible-extract) | 71.42% | **77.79%** | +6.4pp |
| MMLU (57-subject average) | 60.59% | 60.56% | −0.03pp |
| HellaSwag (acc) | 49.28% | 49.24% | −0.04pp |
| HellaSwag (acc_norm) | 66.48% | 66.78% | +0.30pp |

**No catastrophic forgetting.** The MMLU and HellaSwag deltas are smaller than each
benchmark's own standard error (~0.4-0.5pp) — statistically indistinguishable from no
change. GSM8K improved well outside that noise band, via a completely different
methodology (5-shot, standard harness prompt format) than this project's own eval —
independent confirmation that the math-reasoning gain is real, not an artifact of this
project's own eval design, and that it wasn't purchased at the cost of general
capability.

## Limitations

- **Narrow, math-specialized training.** This model was trained exclusively on grade-
  school arithmetic word problems (GSM8K). General instruction-following and chat
  quality specifically were not evaluated (this model was never trained on
  conversational data) — though catastrophic forgetting of general *knowledge/reasoning*
  capability was checked (see the benchmark cross-check above) and not found.
- **A real hint-exploitation vulnerability was found, quantified at 37%.** A formal
  hint/no-hint probe (200 held-out problems, each run with a correct hint and a
  deliberately wrong decoy — see `week-07/docs/04-results.md` for the full method) found
  that 37% of the time, a plausible-but-wrong hint changes the model's final answer away
  from what it would otherwise get right. In one documented example, the model correctly
  re-derived the right answer three separate times, explicitly noted the conflict with
  the hint, and still deferred to the hint over its own confirmed work. **Do not present
  this model with untrusted "hint" or "suggested answer" text in the prompt** without
  accounting for this - it will sometimes follow a wrong suggestion over independently-
  verified correct reasoning. (Checked and ruled out as *reward hacking* specifically -
  training prompts never included hints at all, so the reward function never selected
  for this behavior one way or the other; it's most likely inherited from base-model
  pretraining and was simply never tested or corrected during RL training.)
- **MATH evaluation numbers are approximate** (see above) — treat the MATH result
  directionally, not as a precise benchmark score.
- **Single training run.** No hyperparameter ablation or multi-seed variance check was
  performed; this is a v1 release, not a fully tuned final model.
- **Format compliance is 99.4%, not 100%** — a small fraction of outputs don't follow the
  `<reasoning>`/`<answer>` structure.
- **The model does not always stop generating after `</answer>` — measured at ~0.07%
  of outputs, rare rather than common.** No reward function ever scored anything past
  the first `<answer>` tag match, so nothing in training penalized rambling afterward —
  a real design gap. Measured directly against 1,451 raw eval transcripts (GSM8K +
  SVAMP + MATH), only 1 case had any content past the tag, and it wasn't even the
  fabricated-dialogue pattern (it was a repetitive reasoning loop that leaked one stray
  character). The one case that clearly showed the fabricated-dialogue version happened
  during manual testing on a far-out-of-distribution question style — possibly the same
  underlying cause as the function-composition gap below, not a frequent, independent
  problem. Regardless, the fix is free and worth applying — **set a stop sequence at
  `</answer>` when generating**:
  ```python
  # vLLM
  SamplingParams(stop=["</answer>"], include_stop_str_in_output=True)

  # transformers .generate()
  from transformers import StopStringCriteria, StoppingCriteriaList
  model.generate(..., stopping_criteria=StoppingCriteriaList(
      [StopStringCriteria(tokenizer, ["</answer>"])]
  ))
  ```
- **Symbolic function-composition problems (`f(x)=...`, `g(x)=...` notation) appear to be
  a specific weak spot**, distinct from the general MATH miss rate — likely because
  GSM8K (the only training data) is exclusively concrete word problems with no examples
  of this abstract notation. Observed anecdotally (4/4 wrong on manual testing), not
  verified at benchmark scale — flagged as a real, specific gap rather than confirmed
  precisely.

## Training details

- Framework: Unsloth 2026.5.9 + TRL `GRPOTrainer`, run inside the `unsloth/unsloth`
  Docker image on an RTX 5070 (12GB, Blackwell sm_120)
- Precision: bf16 (FP8 was evaluated and found broken on this specific software stack —
  see the source repo's `week-05-06/docs/00-overview.md` for the investigation)
- `num_generations=4`, `temperature=0.8`, `beta=0.001` (KL penalty), `loss_type=dapo`
- Learning rate 5e-6, 1 epoch, ~7,033 steps, wall-clock ~7.4 hours

## Citation / provenance

Trained as part of a self-directed practice roadmap covering post-training/RL, systems,
and scalability topics for LLMs.