qwen2.5-7b-math-reasoning-grpo/README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-7B-Instruct
language:
- en
tags:
- math
- reasoning
- grpo
- reinforcement-learning
- chain-of-thought
- gsm8k
- qwen
datasets:
- openai/gsm8k
- AI-MO/NuminaMath-CoT
metrics:
- exact_match
---

# Qwen2.5-7B Math Reasoning — GRPO Fine-tuned

A Qwen2.5-7B-Instruct model trained to reason step-by-step through mathematical problems using **Group Relative Policy Optimization (GRPO)** — the reinforcement learning technique introduced in the [DeepSeek-R1 paper](https://arxiv.org/abs/2501.12948).

The model is trained in two stages: a supervised fine-tuning (SFT) cold start to install chain-of-thought reasoning format, followed by GRPO reinforcement learning using verifiable reward signals — no human preference labels required.

**GitHub:** [RohanThawait/qwen2.5-7b-math-reasoning-grpo](https://github.com/RohanThawait/qwen2.5-7b-math-reasoning-grpo)  

---

## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "thawait/qwen2.5-7b-math-reasoning-grpo"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto"
)

problem = "Janet's ducks lay 16 eggs per day. She eats 3 for breakfast and bakes 4 into muffins. She sells the rest for $2 each. How much does she earn per day?"

messages = [{"role": "user", "content": problem}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.1,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

response = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True
)
print(response)
```

**Expected output format:**
```
<think>
Every day she sells 16 - 3 - 4 = 9 eggs.
She makes 9 * 2 = $18 per day at the farmers' market.
</think>

The answer is: 18
```

---

## Training Pipeline

### Stage 1 — SFT Cold Start

The base instruct model is finetuned on a curated math chain-of-thought dataset to install structured reasoning format before RL training begins.

| Detail | Value |
|---|---|
| Dataset | GSM8K train (~7,473 examples) + NuminaMath-CoT (20,000 sampled) |
| Total examples | ~27,000 |
| Epochs | 2 |
| Learning rate | 2e-5 with cosine decay |
| Effective batch size | 32 |
| Hardware | NVIDIA H100 NVL (99.9GB) |
| Training time | ~2 hours |
| Final train loss | 0.3357 |
| Final token accuracy | 92.5% |

### Stage 2 — GRPO Training

GRPO generates multiple candidate responses per problem (rollouts), scores each with reward functions, and updates the policy toward higher-reward responses. The group mean reward replaces the PPO critic network — making GRPO significantly more compute-efficient.

| Detail | Value |
|---|---|
| Dataset | GSM8K train (problems only) |
| Group size G | 4 rollouts per problem |
| GRPO steps | 1,000 |
| Learning rate | 5e-7 |
| KL coefficient | 0.04 |
| Max new tokens | 1,024 |
| Hardware | NVIDIA H100 NVL (99.9GB) |
| Training time | ~2 hours 9 minutes |

**Reward function stack:**

| Reward | Value | Description |
|---|---|---|
| Correctness | 1.0 | Parsed final answer matches ground truth |
| Format | 0.5 | Valid `<think>...</think>` structure present |
| Length penalty | ≤ 0.1 | Soft penalty outside 50–800 token range |

---

## Benchmark Results

Evaluated using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) under identical settings across all three model stages.

| Benchmark | Instruct Baseline | After SFT | After GRPO (this model) |
|---|---|---|---|
| GSM8K 8-shot | 82.64% | 75.51% | **75.66%** |
| MATH/hendrycks_math500 4-shot | 20.60% | 24.20% | **24.20%** |
| ARC-Challenge 25-shot | 67.06% | 62.97% | **62.80%** |

**Delta vs instruct baseline:**

| Benchmark | SFT Δ | GRPO Δ |
|---|---|---|
| GSM8K | -7.13% | -6.98% |
| MATH | **+3.60%** | **+3.60%** |
| ARC-Challenge | -4.09% | -4.26% |

---

## Analysis and Findings

### MATH benchmark improved — the key finding

The model was never trained on MATH benchmark problems. It trained only on GSM8K and NuminaMath-CoT. The improvement from **20.60% → 24.20%** on competition-level math is evidence that SFT successfully installed a **generalizable reasoning format**, not just GSM8K pattern matching.

The slight decline on ARC-Challenge confirms the distribution shift: the model learned to reason more mathematically, at a small cost to general abstract reasoning. This is the expected and honest outcome of math-specific training.

### GSM8K drop after SFT — evaluation artifact, not capability regression

The SFT model now produces structured `<think>...</think>` reasoning chains before answering. The lm-evaluation-harness GSM8K task was calibrated for the original instruct model's direct output style. This format shift suppresses GSM8K scores for the finetuned models even when underlying reasoning capability is unchanged or improved.

This is a known evaluation artifact when finetuning instruct models for chain-of-thought output.

### GRPO showed limited improvement over SFT — reward saturation

The most technically interesting finding of this project is what the GRPO training curves revealed.

The GRPO training encountered **reward saturation** — a documented problem that occurs when the starting model is already too capable on the training distribution. With a strong SFT cold start, the model solved most GSM8K rollouts correctly, meaning all G=4 rollouts within a group frequently received the same reward. When reward variance within a group is zero, advantage signals are zero and the gradient update carries no learning signal.

**Measured directly:** `frac_reward_zero_std` averaged **0.63 throughout training** — meaning 63% of batches produced near-zero gradient signal. KL divergence never exceeded 0.0006 across 1,000 steps, confirming the policy barely moved from the SFT reference model.

This is the same challenge the DeepSeek R1 team addressed through **curriculum filtering** — selecting problems where the model succeeds on roughly 50% of rollouts rather than 80%. With mid-difficulty curriculum selection, reward variance is higher, advantage signals are meaningful, and RL learning is more effective.

### What I would do differently

- **Curriculum filtering before GRPO:** Select problems where the SFT model gets 1–2 out of 4 rollouts correct, maximizing reward variance and learning signal per step
- **Harder dataset for GRPO:** Use NuminaMath competition problems instead of GSM8K — the difficulty distribution better matches what GRPO needs for meaningful learning signal
- **Lower KL coefficient:** Allow the policy more freedom to drift from the SFT reference, trading stability for exploration

---

## Limitations

- GRPO training showed reward saturation on GSM8K — the model did not significantly improve over the SFT checkpoint on in-distribution math problems
- Training was limited to 1,000 GRPO steps due to compute constraints — longer training with curriculum filtering would likely show stronger gains
- The model is specialized for mathematical reasoning — general instruction following capability may be slightly reduced compared to the base instruct model
- Evaluation on GSM8K is affected by the output format change introduced by SFT — raw accuracy numbers underrepresent the model's actual reasoning capability improvement

---

## Training Infrastructure

| Component | Detail |
|---|---|
| GPU | NVIDIA H100 NVL (99.9 GB VRAM) |
| Framework | PyTorch + HuggingFace TRL |
| SFT library | TRL SFTTrainer |
| GRPO library | TRL GRPOTrainer |
| Experiment tracking | Weights & Biases |
| Evaluation | lm-evaluation-harness |

---

## Citation

If you use this model or find this work useful, please cite:

```bibtex
@misc{thawait2026grpo,
  author = {Rohan Thawait},
  title = {Qwen2.5-7B Math Reasoning with GRPO},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/thawait/qwen2.5-7b-math-reasoning-grpo}
}
```

---

## References

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/abs/2501.12948)
- [Qwen2.5 Technical Report](https://arxiv.org/abs/2412.15115)
- [TRL: Transformer Reinforcement Learning](https://github.com/huggingface/trl)
- [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
- [NuminaMath-CoT Dataset](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT)