83 lines
3.9 KiB
Markdown
83 lines
3.9 KiB
Markdown
---
|
||
library_name: transformers
|
||
tags:
|
||
- trl
|
||
- grpo
|
||
- qwen
|
||
- gsm8k
|
||
---
|
||
|
||
# Qwen-0.5B-GRPO: A Fine-Tuned Math Reasoner
|
||
|
||
This model is a fine-tuned version of the Qwen 0.5B model (based on [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)) using GRPO (Generative Reward Policy Optimization). It has been trained on the GSM8K math dataset to improve its ability to generate step-by-step reasoning for math problems, following a structured output format with explicit `<reasoning>` and `<answer>` sections.
|
||
|
||
## Model Details
|
||
|
||
### Model Description
|
||
|
||
Qwen-0.5B-GRPO is designed to serve as a lightweight math reasoning assistant. By fine-tuning with reinforcement learning using GRPO, the model learns to produce responses that include both intermediate reasoning and final answers. Key adaptations include:
|
||
- **Base Model:** Qwen/Qwen2.5-0.5B-Instruct
|
||
- **Fine-Tuning Method:** GRPO (reinforcement learning with custom reward functions)
|
||
- **Dataset:** GSM8K – a collection of challenging grade-school math problems
|
||
- **Generation Engine:** Utilizes vLLM for faster inference on a single GPU setup
|
||
- **Precision:** BF16 training for efficiency on Colab GPUs
|
||
|
||
- **Developed by:** Davut Emre Taşar
|
||
- **License:** Please refer to the license of the base model on its Hugging Face Hub page
|
||
|
||
### Model Sources
|
||
|
||
- **Repository (this model):** [https://huggingface.co/emre/Qwen-0.5B-GRPO](https://huggingface.co/emre/Qwen-0.5B-GRPO)
|
||
- **Base Model Repository:** [https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
|
||
- **Dataset:** [https://huggingface.co/datasets/openai/gsm8k](https://huggingface.co/datasets/openai/gsm8k)
|
||
|
||
## Uses
|
||
|
||
### Intended Use
|
||
|
||
This model is intended for educational and research purposes, particularly to demonstrate and support math problem solving with clear, step-by-step reasoning. It is well-suited for:
|
||
- Generating structured explanations for math problems.
|
||
- Serving as a lightweight assistant in educational applications focused on math reasoning.
|
||
|
||
### Out-of-Scope Use
|
||
|
||
- **High-Stakes Decision Making:** This model is not designed for critical decision making.
|
||
- **Non-Math Domains:** Its performance is tailored to math problems; performance on other domains may be limited.
|
||
- **Over-Reliance on Automated Reasoning:** The reward functions used during fine-tuning (e.g., exact string matching) may not capture all nuances, so human oversight is recommended.
|
||
|
||
## Bias, Risks, and Limitations
|
||
|
||
- **Model Size:** With only 0.5B parameters, it may not perform as robustly as larger models.
|
||
- **Training Duration:** Fine-tuning was performed for a single epoch; further training might be needed for more challenging tasks.
|
||
- **Reward Function Limitations:** The custom reward functions (checking for correct formatting and numerical correctness) are heuristic and may occasionally miss subtleties in reasoning.
|
||
- **Generalization:** The structured format (with `<reasoning>` and `<answer>` tags) is enforced during training and may require adaptation for other use cases.
|
||
|
||
### Recommendations
|
||
|
||
Users should:
|
||
- Validate model outputs on a case-by-case basis.
|
||
- Consider further fine-tuning for domain-specific applications.
|
||
- Use the model as a supplementary tool rather than the sole resource for critical math reasoning tasks.
|
||
|
||
## How to Get Started with the Model
|
||
|
||
Below is an example code snippet to load and use the model:
|
||
|
||
```python
|
||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
import torch
|
||
|
||
model_name = "emre/Qwen-0.5B-GRPO"
|
||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16).to("cuda")
|
||
|
||
# Example prompt: structured with <reasoning> and <answer> tags.
|
||
prompt = """<reasoning>
|
||
Step-by-step reasoning:
|
||
</reasoning>
|
||
<answer>
|
||
"""
|
||
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
|
||
outputs = model.generate(**inputs, max_length=300)
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|