初始化项目，由ModelHub XC社区提供模型
Model: Lzvick/qwen-1.7b-math-reasoner-grpo Source: Original Platform
This commit is contained in:
ModelHub XC
162
README.md
Normal file
162
README.md
Normal file
@@ -0,0 +1,162 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
base_model: unsloth/Qwen3-1.7B-Base
|
||||
tags:
|
||||
- grpo
|
||||
- reinforcement-learning
|
||||
- reasoning
|
||||
- math
|
||||
- rlvr
|
||||
- qwen3
|
||||
datasets:
|
||||
- openai/gsm8k
|
||||
language:
|
||||
- en
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# Qwen3-1.7B-GRPO-GSM8K (v1)
|
||||
|
||||
A GRPO (Group Relative Policy Optimization) fine-tune of `unsloth/Qwen3-1.7B-Base`,
|
||||
trained with verifiable rewards (RLVR) on GSM8K math word problems for one full epoch.
|
||||
|
||||
## Model description
|
||||
|
||||
- **Base model:** `unsloth/Qwen3-1.7B-Base` (Apache 2.0)
|
||||
- **Method:** GRPO (DeepSeekMath / DeepSeek-R1 style), via Unsloth + TRL's `GRPOTrainer`
|
||||
- **Adapter:** LoRA rank 32 / alpha 32, merged into the base weights for this release
|
||||
- **Training data:** GSM8K `train`+`test` combined into one 8,792-example pool, shuffled
|
||||
once (seed 3407), split 80/10/10 — trained on the 80% partition (7,033 examples) for
|
||||
1 full epoch. Validation (10%) used only for monitoring during training; the eval
|
||||
partition (10%, 880 examples) was never touched until the final evaluation below.
|
||||
- **Output format:** the model is trained to respond in a fixed structure:
|
||||
```
|
||||
<reasoning>
|
||||
...
|
||||
</reasoning>
|
||||
<answer>
|
||||
...
|
||||
</answer>
|
||||
```
|
||||
- **Reward functions:** correctness (regex-extracted `<answer>` content vs. GSM8K ground
|
||||
truth, +2.0), format compliance (+0.5 well-formed / +0.25 partial), brevity
|
||||
(+0.2 max, capped small so it can never outweigh correctness or format)
|
||||
|
||||
## Evaluation results
|
||||
|
||||
All numbers below are from held-out data the model never saw during training or any
|
||||
prior evaluation pass (see the source repo's `week-05-06/scripts/data_split.py` for the
|
||||
split methodology, and `week-05-06/docs/04-results.md` for the full writeup including
|
||||
two scoring bugs that were caught and fixed mid-evaluation).
|
||||
|
||||
| Dataset | In/out of distribution | Base model | This model | Change |
|
||||
|---|---|---|---|---|
|
||||
| GSM8K (880 held-out examples) | in-distribution | 66.9%\* | **87.3%** | +20.4pp |
|
||||
| SVAMP (300 examples) | out-of-distribution | 76.7% | **86.7%** | +10.0pp |
|
||||
| MATH / algebra (300 examples) | out-of-distribution, harder | 49.0%\*\* | **60.3%** | +11.3pp\*\* |
|
||||
|
||||
\*Base model accuracy is scored leniently (correct number anywhere in the response) since
|
||||
the base model has no tendency to use the `<answer>` tag format at all (0.6% format
|
||||
compliance) — this model's own format compliance is 99.4%.
|
||||
|
||||
\*\*MATH's numbers are a **lower bound**, not a precise score — grading MATH answers
|
||||
exactly (fractions, intervals, symbolic expressions) without a computer algebra system is
|
||||
a known hard problem; some correct-but-differently-formatted answers score as incorrect
|
||||
here.
|
||||
|
||||
**Manual transcript review** (not just the aggregate numbers) found no evidence of
|
||||
reward hacking: correct answers show genuine multi-step arithmetic, incorrect answers
|
||||
are real reasoning mistakes on harder problems (not degenerate or gamed output), and
|
||||
response length does not correlate with correctness in a gaming direction.
|
||||
|
||||
### Independent cross-check: lm-evaluation-harness (base vs. trained)
|
||||
|
||||
The numbers above use this project's own zero-shot, custom-prompt eval scripts. Cross-
|
||||
checked independently via EleutherAI's `lm-evaluation-harness` (standard 5-shot GSM8K
|
||||
config, plus MMLU and HellaSwag as general-capability anchors), run identically against
|
||||
both the base model and this one — see the source repo's `week-07/docs/04-results.md`
|
||||
for the full writeup.
|
||||
|
||||
| Benchmark | Base | This model | Δ |
|
||||
|---|---|---|---|
|
||||
| GSM8K (5-shot, strict-match) | 67.55% | **77.03%** | +9.5pp |
|
||||
| GSM8K (5-shot, flexible-extract) | 71.42% | **77.79%** | +6.4pp |
|
||||
| MMLU (57-subject average) | 60.59% | 60.56% | −0.03pp |
|
||||
| HellaSwag (acc) | 49.28% | 49.24% | −0.04pp |
|
||||
| HellaSwag (acc_norm) | 66.48% | 66.78% | +0.30pp |
|
||||
|
||||
**No catastrophic forgetting.** The MMLU and HellaSwag deltas are smaller than each
|
||||
benchmark's own standard error (~0.4-0.5pp) — statistically indistinguishable from no
|
||||
change. GSM8K improved well outside that noise band, via a completely different
|
||||
methodology (5-shot, standard harness prompt format) than this project's own eval —
|
||||
independent confirmation that the math-reasoning gain is real, not an artifact of this
|
||||
project's own eval design, and that it wasn't purchased at the cost of general
|
||||
capability.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Narrow, math-specialized training.** This model was trained exclusively on grade-
|
||||
school arithmetic word problems (GSM8K). General instruction-following and chat
|
||||
quality specifically were not evaluated (this model was never trained on
|
||||
conversational data) — though catastrophic forgetting of general *knowledge/reasoning*
|
||||
capability was checked (see the benchmark cross-check above) and not found.
|
||||
- **A real hint-exploitation vulnerability was found, quantified at 37%.** A formal
|
||||
hint/no-hint probe (200 held-out problems, each run with a correct hint and a
|
||||
deliberately wrong decoy — see `week-07/docs/04-results.md` for the full method) found
|
||||
that 37% of the time, a plausible-but-wrong hint changes the model's final answer away
|
||||
from what it would otherwise get right. In one documented example, the model correctly
|
||||
re-derived the right answer three separate times, explicitly noted the conflict with
|
||||
the hint, and still deferred to the hint over its own confirmed work. **Do not present
|
||||
this model with untrusted "hint" or "suggested answer" text in the prompt** without
|
||||
accounting for this - it will sometimes follow a wrong suggestion over independently-
|
||||
verified correct reasoning. (Checked and ruled out as *reward hacking* specifically -
|
||||
training prompts never included hints at all, so the reward function never selected
|
||||
for this behavior one way or the other; it's most likely inherited from base-model
|
||||
pretraining and was simply never tested or corrected during RL training.)
|
||||
- **MATH evaluation numbers are approximate** (see above) — treat the MATH result
|
||||
directionally, not as a precise benchmark score.
|
||||
- **Single training run.** No hyperparameter ablation or multi-seed variance check was
|
||||
performed; this is a v1 release, not a fully tuned final model.
|
||||
- **Format compliance is 99.4%, not 100%** — a small fraction of outputs don't follow the
|
||||
`<reasoning>`/`<answer>` structure.
|
||||
- **The model does not always stop generating after `</answer>` — measured at ~0.07%
|
||||
of outputs, rare rather than common.** No reward function ever scored anything past
|
||||
the first `<answer>` tag match, so nothing in training penalized rambling afterward —
|
||||
a real design gap. Measured directly against 1,451 raw eval transcripts (GSM8K +
|
||||
SVAMP + MATH), only 1 case had any content past the tag, and it wasn't even the
|
||||
fabricated-dialogue pattern (it was a repetitive reasoning loop that leaked one stray
|
||||
character). The one case that clearly showed the fabricated-dialogue version happened
|
||||
during manual testing on a far-out-of-distribution question style — possibly the same
|
||||
underlying cause as the function-composition gap below, not a frequent, independent
|
||||
problem. Regardless, the fix is free and worth applying — **set a stop sequence at
|
||||
`</answer>` when generating**:
|
||||
```python
|
||||
# vLLM
|
||||
SamplingParams(stop=["</answer>"], include_stop_str_in_output=True)
|
||||
|
||||
# transformers .generate()
|
||||
from transformers import StopStringCriteria, StoppingCriteriaList
|
||||
model.generate(..., stopping_criteria=StoppingCriteriaList(
|
||||
[StopStringCriteria(tokenizer, ["</answer>"])]
|
||||
))
|
||||
```
|
||||
- **Symbolic function-composition problems (`f(x)=...`, `g(x)=...` notation) appear to be
|
||||
a specific weak spot**, distinct from the general MATH miss rate — likely because
|
||||
GSM8K (the only training data) is exclusively concrete word problems with no examples
|
||||
of this abstract notation. Observed anecdotally (4/4 wrong on manual testing), not
|
||||
verified at benchmark scale — flagged as a real, specific gap rather than confirmed
|
||||
precisely.
|
||||
|
||||
## Training details
|
||||
|
||||
- Framework: Unsloth 2026.5.9 + TRL `GRPOTrainer`, run inside the `unsloth/unsloth`
|
||||
Docker image on an RTX 5070 (12GB, Blackwell sm_120)
|
||||
- Precision: bf16 (FP8 was evaluated and found broken on this specific software stack —
|
||||
see the source repo's `week-05-06/docs/00-overview.md` for the investigation)
|
||||
- `num_generations=4`, `temperature=0.8`, `beta=0.001` (KL penalty), `loss_type=dapo`
|
||||
- Learning rate 5e-6, 1 epoch, ~7,033 steps, wall-clock ~7.4 hours
|
||||
|
||||
## Citation / provenance
|
||||
|
||||
Trained as part of a self-directed practice roadmap covering post-training/RL, systems,
|
||||
and scalability topics for LLMs.
|
||||
Reference in New Issue
Block a user