初始化项目，由ModelHub XC社区提供模型

Model: thawait/qwen2.5-7b-math-reasoning-grpo Source: Original Platform
2026-07-14 07:37:09 +08:00
commit a4d3e83b55
9 changed files with 427 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,224 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-7B-Instruct
+language:
+- en
+tags:
+- math
+- reasoning
+- grpo
+- reinforcement-learning
+- chain-of-thought
+- gsm8k
+- qwen
+datasets:
+- openai/gsm8k
+- AI-MO/NuminaMath-CoT
+metrics:
+- exact_match
+---
+
+# Qwen2.5-7B Math Reasoning — GRPO Fine-tuned
+
+A Qwen2.5-7B-Instruct model trained to reason step-by-step through mathematical problems using **Group Relative Policy Optimization (GRPO)** — the reinforcement learning technique introduced in the [DeepSeek-R1 paper](https://arxiv.org/abs/2501.12948).
+
+The model is trained in two stages: a supervised fine-tuning (SFT) cold start to install chain-of-thought reasoning format, followed by GRPO reinforcement learning using verifiable reward signals — no human preference labels required.
+
+**GitHub:** [RohanThawait/qwen2.5-7b-math-reasoning-grpo](https://github.com/RohanThawait/qwen2.5-7b-math-reasoning-grpo)  
+
+---
+
+## How to Use
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_id = "thawait/qwen2.5-7b-math-reasoning-grpo"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    device_map="auto"
+)
+
+problem = "Janet's ducks lay 16 eggs per day. She eats 3 for breakfast and bakes 4 into muffins. She sells the rest for $2 each. How much does she earn per day?"
+
+messages = [{"role": "user", "content": problem}]
+prompt = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=512,
+        temperature=0.1,
+        do_sample=True,
+        pad_token_id=tokenizer.eos_token_id
+    )
+
+response = tokenizer.decode(
+    outputs[0][inputs["input_ids"].shape[1]:],
+    skip_special_tokens=True
+)
+print(response)
+```
+
+**Expected output format:**
+```
+<think>
+Every day she sells 16 - 3 - 4 = 9 eggs.
+She makes 9 * 2 = $18 per day at the farmers' market.
+</think>
+
+The answer is: 18
+```
+
+---
+
+## Training Pipeline
+
+### Stage 1 — SFT Cold Start
+
+The base instruct model is finetuned on a curated math chain-of-thought dataset to install structured reasoning format before RL training begins.
+
+| Detail | Value |
+|---|---|
+| Dataset | GSM8K train (~7,473 examples) + NuminaMath-CoT (20,000 sampled) |
+| Total examples | ~27,000 |
+| Epochs | 2 |
+| Learning rate | 2e-5 with cosine decay |
+| Effective batch size | 32 |
+| Hardware | NVIDIA H100 NVL (99.9GB) |
+| Training time | ~2 hours |
+| Final train loss | 0.3357 |
+| Final token accuracy | 92.5% |
+
+### Stage 2 — GRPO Training
+
+GRPO generates multiple candidate responses per problem (rollouts), scores each with reward functions, and updates the policy toward higher-reward responses. The group mean reward replaces the PPO critic network — making GRPO significantly more compute-efficient.
+
+| Detail | Value |
+|---|---|
+| Dataset | GSM8K train (problems only) |
+| Group size G | 4 rollouts per problem |
+| GRPO steps | 1,000 |
+| Learning rate | 5e-7 |
+| KL coefficient | 0.04 |
+| Max new tokens | 1,024 |
+| Hardware | NVIDIA H100 NVL (99.9GB) |
+| Training time | ~2 hours 9 minutes |
+
+**Reward function stack:**
+
+| Reward | Value | Description |
+|---|---|---|
+| Correctness | 1.0 | Parsed final answer matches ground truth |
+| Format | 0.5 | Valid `<think>...</think>` structure present |
+| Length penalty | ≤ 0.1 | Soft penalty outside 50–800 token range |
+
+---
+
+## Benchmark Results
+
+Evaluated using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) under identical settings across all three model stages.
+
+| Benchmark | Instruct Baseline | After SFT | After GRPO (this model) |
+|---|---|---|---|
+| GSM8K 8-shot | 82.64% | 75.51% | **75.66%** |
+| MATH/hendrycks_math500 4-shot | 20.60% | 24.20% | **24.20%** |
+| ARC-Challenge 25-shot | 67.06% | 62.97% | **62.80%** |
+
+**Delta vs instruct baseline:**
+
+| Benchmark | SFT Δ | GRPO Δ |
+|---|---|---|
+| GSM8K | -7.13% | -6.98% |
+| MATH | **+3.60%** | **+3.60%** |
+| ARC-Challenge | -4.09% | -4.26% |
+
+---
+
+## Analysis and Findings
+
+### MATH benchmark improved — the key finding
+
+The model was never trained on MATH benchmark problems. It trained only on GSM8K and NuminaMath-CoT. The improvement from **20.60% → 24.20%** on competition-level math is evidence that SFT successfully installed a **generalizable reasoning format**, not just GSM8K pattern matching.
+
+The slight decline on ARC-Challenge confirms the distribution shift: the model learned to reason more mathematically, at a small cost to general abstract reasoning. This is the expected and honest outcome of math-specific training.
+
+### GSM8K drop after SFT — evaluation artifact, not capability regression
+
+The SFT model now produces structured `<think>...</think>` reasoning chains before answering. The lm-evaluation-harness GSM8K task was calibrated for the original instruct model's direct output style. This format shift suppresses GSM8K scores for the finetuned models even when underlying reasoning capability is unchanged or improved.
+
+This is a known evaluation artifact when finetuning instruct models for chain-of-thought output.
+
+### GRPO showed limited improvement over SFT — reward saturation
+
+The most technically interesting finding of this project is what the GRPO training curves revealed.
+
+The GRPO training encountered **reward saturation** — a documented problem that occurs when the starting model is already too capable on the training distribution. With a strong SFT cold start, the model solved most GSM8K rollouts correctly, meaning all G=4 rollouts within a group frequently received the same reward. When reward variance within a group is zero, advantage signals are zero and the gradient update carries no learning signal.
+
+**Measured directly:** `frac_reward_zero_std` averaged **0.63 throughout training** — meaning 63% of batches produced near-zero gradient signal. KL divergence never exceeded 0.0006 across 1,000 steps, confirming the policy barely moved from the SFT reference model.
+
+This is the same challenge the DeepSeek R1 team addressed through **curriculum filtering** — selecting problems where the model succeeds on roughly 50% of rollouts rather than 80%. With mid-difficulty curriculum selection, reward variance is higher, advantage signals are meaningful, and RL learning is more effective.
+
+### What I would do differently
+
+- **Curriculum filtering before GRPO:** Select problems where the SFT model gets 1–2 out of 4 rollouts correct, maximizing reward variance and learning signal per step
+- **Harder dataset for GRPO:** Use NuminaMath competition problems instead of GSM8K — the difficulty distribution better matches what GRPO needs for meaningful learning signal
+- **Lower KL coefficient:** Allow the policy more freedom to drift from the SFT reference, trading stability for exploration
+
+---
+
+## Limitations
+
+- GRPO training showed reward saturation on GSM8K — the model did not significantly improve over the SFT checkpoint on in-distribution math problems
+- Training was limited to 1,000 GRPO steps due to compute constraints — longer training with curriculum filtering would likely show stronger gains
+- The model is specialized for mathematical reasoning — general instruction following capability may be slightly reduced compared to the base instruct model
+- Evaluation on GSM8K is affected by the output format change introduced by SFT — raw accuracy numbers underrepresent the model's actual reasoning capability improvement
+
+---
+
+## Training Infrastructure
+
+| Component | Detail |
+|---|---|
+| GPU | NVIDIA H100 NVL (99.9 GB VRAM) |
+| Framework | PyTorch + HuggingFace TRL |
+| SFT library | TRL SFTTrainer |
+| GRPO library | TRL GRPOTrainer |
+| Experiment tracking | Weights & Biases |
+| Evaluation | lm-evaluation-harness |
+
+---
+
+## Citation
+
+If you use this model or find this work useful, please cite:
+
+```bibtex
+@misc{thawait2026grpo,
+  author = {Rohan Thawait},
+  title = {Qwen2.5-7B Math Reasoning with GRPO},
+  year = {2026},
+  publisher = {HuggingFace},
+  url = {https://huggingface.co/thawait/qwen2.5-7b-math-reasoning-grpo}
+}
+```
+
+---
+
+## References
+
+- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/abs/2501.12948)
+- [Qwen2.5 Technical Report](https://arxiv.org/abs/2412.15115)
+- [TRL: Transformer Reinforcement Learning](https://github.com/huggingface/trl)
+- [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
+- [NuminaMath-CoT Dataset](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT)
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,61 @@
+{
+  "architectures": [
+    "Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "hidden_act": "silu",
+  "hidden_size": 3584,
+  "initializer_range": 0.02,
+  "intermediate_size": 18944,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen2",
+  "num_attention_heads": 28,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 4,
+  "pad_token_id": 151643,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.7.0",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "vocab_size": 152064
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,13 @@
+{
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "repetition_penalty": 1.05,
+  "temperature": 0.7,
+  "top_k": 20,
+  "top_p": 0.8,
+  "transformers_version": "5.7.0"
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ca10b6cc7bc25bcb18e98e95d6e56ab8a29fae57e6d629d1063e07e35fd1781
+size 15231272152
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8
+size 11421892
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}
--- a/training_args.bin
+++ b/training_args.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d596bb0b44b912d1a1ebbb76be7577d028e0876af53715d790a734659e4113be
+size 7249