163 lines
8.3 KiB
Markdown
163 lines
8.3 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: unsloth/Qwen3-1.7B-Base
|
||
tags:
|
||
- grpo
|
||
- reinforcement-learning
|
||
- reasoning
|
||
- math
|
||
- rlvr
|
||
- qwen3
|
||
datasets:
|
||
- openai/gsm8k
|
||
language:
|
||
- en
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# Qwen3-1.7B-GRPO-GSM8K (v1)
|
||
|
||
A GRPO (Group Relative Policy Optimization) fine-tune of `unsloth/Qwen3-1.7B-Base`,
|
||
trained with verifiable rewards (RLVR) on GSM8K math word problems for one full epoch.
|
||
|
||
## Model description
|
||
|
||
- **Base model:** `unsloth/Qwen3-1.7B-Base` (Apache 2.0)
|
||
- **Method:** GRPO (DeepSeekMath / DeepSeek-R1 style), via Unsloth + TRL's `GRPOTrainer`
|
||
- **Adapter:** LoRA rank 32 / alpha 32, merged into the base weights for this release
|
||
- **Training data:** GSM8K `train`+`test` combined into one 8,792-example pool, shuffled
|
||
once (seed 3407), split 80/10/10 — trained on the 80% partition (7,033 examples) for
|
||
1 full epoch. Validation (10%) used only for monitoring during training; the eval
|
||
partition (10%, 880 examples) was never touched until the final evaluation below.
|
||
- **Output format:** the model is trained to respond in a fixed structure:
|
||
```
|
||
<reasoning>
|
||
...
|
||
</reasoning>
|
||
<answer>
|
||
...
|
||
</answer>
|
||
```
|
||
- **Reward functions:** correctness (regex-extracted `<answer>` content vs. GSM8K ground
|
||
truth, +2.0), format compliance (+0.5 well-formed / +0.25 partial), brevity
|
||
(+0.2 max, capped small so it can never outweigh correctness or format)
|
||
|
||
## Evaluation results
|
||
|
||
All numbers below are from held-out data the model never saw during training or any
|
||
prior evaluation pass (see the source repo's `week-05-06/scripts/data_split.py` for the
|
||
split methodology, and `week-05-06/docs/04-results.md` for the full writeup including
|
||
two scoring bugs that were caught and fixed mid-evaluation).
|
||
|
||
| Dataset | In/out of distribution | Base model | This model | Change |
|
||
|---|---|---|---|---|
|
||
| GSM8K (880 held-out examples) | in-distribution | 66.9%\* | **87.3%** | +20.4pp |
|
||
| SVAMP (300 examples) | out-of-distribution | 76.7% | **86.7%** | +10.0pp |
|
||
| MATH / algebra (300 examples) | out-of-distribution, harder | 49.0%\*\* | **60.3%** | +11.3pp\*\* |
|
||
|
||
\*Base model accuracy is scored leniently (correct number anywhere in the response) since
|
||
the base model has no tendency to use the `<answer>` tag format at all (0.6% format
|
||
compliance) — this model's own format compliance is 99.4%.
|
||
|
||
\*\*MATH's numbers are a **lower bound**, not a precise score — grading MATH answers
|
||
exactly (fractions, intervals, symbolic expressions) without a computer algebra system is
|
||
a known hard problem; some correct-but-differently-formatted answers score as incorrect
|
||
here.
|
||
|
||
**Manual transcript review** (not just the aggregate numbers) found no evidence of
|
||
reward hacking: correct answers show genuine multi-step arithmetic, incorrect answers
|
||
are real reasoning mistakes on harder problems (not degenerate or gamed output), and
|
||
response length does not correlate with correctness in a gaming direction.
|
||
|
||
### Independent cross-check: lm-evaluation-harness (base vs. trained)
|
||
|
||
The numbers above use this project's own zero-shot, custom-prompt eval scripts. Cross-
|
||
checked independently via EleutherAI's `lm-evaluation-harness` (standard 5-shot GSM8K
|
||
config, plus MMLU and HellaSwag as general-capability anchors), run identically against
|
||
both the base model and this one — see the source repo's `week-07/docs/04-results.md`
|
||
for the full writeup.
|
||
|
||
| Benchmark | Base | This model | Δ |
|
||
|---|---|---|---|
|
||
| GSM8K (5-shot, strict-match) | 67.55% | **77.03%** | +9.5pp |
|
||
| GSM8K (5-shot, flexible-extract) | 71.42% | **77.79%** | +6.4pp |
|
||
| MMLU (57-subject average) | 60.59% | 60.56% | −0.03pp |
|
||
| HellaSwag (acc) | 49.28% | 49.24% | −0.04pp |
|
||
| HellaSwag (acc_norm) | 66.48% | 66.78% | +0.30pp |
|
||
|
||
**No catastrophic forgetting.** The MMLU and HellaSwag deltas are smaller than each
|
||
benchmark's own standard error (~0.4-0.5pp) — statistically indistinguishable from no
|
||
change. GSM8K improved well outside that noise band, via a completely different
|
||
methodology (5-shot, standard harness prompt format) than this project's own eval —
|
||
independent confirmation that the math-reasoning gain is real, not an artifact of this
|
||
project's own eval design, and that it wasn't purchased at the cost of general
|
||
capability.
|
||
|
||
## Limitations
|
||
|
||
- **Narrow, math-specialized training.** This model was trained exclusively on grade-
|
||
school arithmetic word problems (GSM8K). General instruction-following and chat
|
||
quality specifically were not evaluated (this model was never trained on
|
||
conversational data) — though catastrophic forgetting of general *knowledge/reasoning*
|
||
capability was checked (see the benchmark cross-check above) and not found.
|
||
- **A real hint-exploitation vulnerability was found, quantified at 37%.** A formal
|
||
hint/no-hint probe (200 held-out problems, each run with a correct hint and a
|
||
deliberately wrong decoy — see `week-07/docs/04-results.md` for the full method) found
|
||
that 37% of the time, a plausible-but-wrong hint changes the model's final answer away
|
||
from what it would otherwise get right. In one documented example, the model correctly
|
||
re-derived the right answer three separate times, explicitly noted the conflict with
|
||
the hint, and still deferred to the hint over its own confirmed work. **Do not present
|
||
this model with untrusted "hint" or "suggested answer" text in the prompt** without
|
||
accounting for this - it will sometimes follow a wrong suggestion over independently-
|
||
verified correct reasoning. (Checked and ruled out as *reward hacking* specifically -
|
||
training prompts never included hints at all, so the reward function never selected
|
||
for this behavior one way or the other; it's most likely inherited from base-model
|
||
pretraining and was simply never tested or corrected during RL training.)
|
||
- **MATH evaluation numbers are approximate** (see above) — treat the MATH result
|
||
directionally, not as a precise benchmark score.
|
||
- **Single training run.** No hyperparameter ablation or multi-seed variance check was
|
||
performed; this is a v1 release, not a fully tuned final model.
|
||
- **Format compliance is 99.4%, not 100%** — a small fraction of outputs don't follow the
|
||
`<reasoning>`/`<answer>` structure.
|
||
- **The model does not always stop generating after `</answer>` — measured at ~0.07%
|
||
of outputs, rare rather than common.** No reward function ever scored anything past
|
||
the first `<answer>` tag match, so nothing in training penalized rambling afterward —
|
||
a real design gap. Measured directly against 1,451 raw eval transcripts (GSM8K +
|
||
SVAMP + MATH), only 1 case had any content past the tag, and it wasn't even the
|
||
fabricated-dialogue pattern (it was a repetitive reasoning loop that leaked one stray
|
||
character). The one case that clearly showed the fabricated-dialogue version happened
|
||
during manual testing on a far-out-of-distribution question style — possibly the same
|
||
underlying cause as the function-composition gap below, not a frequent, independent
|
||
problem. Regardless, the fix is free and worth applying — **set a stop sequence at
|
||
`</answer>` when generating**:
|
||
```python
|
||
# vLLM
|
||
SamplingParams(stop=["</answer>"], include_stop_str_in_output=True)
|
||
|
||
# transformers .generate()
|
||
from transformers import StopStringCriteria, StoppingCriteriaList
|
||
model.generate(..., stopping_criteria=StoppingCriteriaList(
|
||
[StopStringCriteria(tokenizer, ["</answer>"])]
|
||
))
|
||
```
|
||
- **Symbolic function-composition problems (`f(x)=...`, `g(x)=...` notation) appear to be
|
||
a specific weak spot**, distinct from the general MATH miss rate — likely because
|
||
GSM8K (the only training data) is exclusively concrete word problems with no examples
|
||
of this abstract notation. Observed anecdotally (4/4 wrong on manual testing), not
|
||
verified at benchmark scale — flagged as a real, specific gap rather than confirmed
|
||
precisely.
|
||
|
||
## Training details
|
||
|
||
- Framework: Unsloth 2026.5.9 + TRL `GRPOTrainer`, run inside the `unsloth/unsloth`
|
||
Docker image on an RTX 5070 (12GB, Blackwell sm_120)
|
||
- Precision: bf16 (FP8 was evaluated and found broken on this specific software stack —
|
||
see the source repo's `week-05-06/docs/00-overview.md` for the investigation)
|
||
- `num_generations=4`, `temperature=0.8`, `beta=0.001` (KL penalty), `loss_type=dapo`
|
||
- Learning rate 5e-6, 1 epoch, ~7,033 steps, wall-clock ~7.4 hours
|
||
|
||
## Citation / provenance
|
||
|
||
Trained as part of a self-directed practice roadmap covering post-training/RL, systems,
|
||
and scalability topics for LLMs.
|