141 lines
4.5 KiB
Markdown
141 lines
4.5 KiB
Markdown
---
|
|
library_name: transformers
|
|
tags:
|
|
- unsloth
|
|
- trl
|
|
- grpo
|
|
- reasoning
|
|
- gsm8k
|
|
datasets:
|
|
- openai/gsm8k
|
|
language:
|
|
- en
|
|
base_model:
|
|
- Qwen/Qwen2.5-0.5B-Instruct
|
|
pipeline_tag: question-answering
|
|
license: apache-2.0
|
|
---
|
|
|
|
# Model Card for Qwen2.5-0.5B-Instruct-GSM8K-Reasoning
|
|
|
|
<!-- Provide a quick summary of what the model is/does. -->
|
|
|
|
This model is a fine-tuned version of the **Qwen2.5-0.5B-Instruct** model, specifically adapted for **mathematical reasoning tasks** using the **GSM8K dataset**. It leverages **GPRO (Generalized Policy Optimization for Reasoning)** methods, as described in the *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models* paper, to enhance its reasoning capabilities. The fine-tuning process was performed using **Unsloth** for efficiency and **TRL (Transformer Reinforcement Learning)** for reinforcement learning-based training.
|
|
|
|
## Model Details
|
|
|
|
## How to Get Started with the Model
|
|
|
|
Use the code below to load and use the model with vLLM & Unsloth:
|
|
|
|
```python
|
|
from unsloth import FastLanguageModel
|
|
from vllm import SamplingParams
|
|
import torch
|
|
|
|
# Load the Model & Tokenizer
|
|
model, tokenizer = FastLanguageModel.from_pretrained(
|
|
model_name = "AdamLucek/Qwen2.5-3B-Instruct-GRPO-2K-GSM8K",
|
|
max_seq_length = 2048,
|
|
load_in_4bit = True,
|
|
fast_inference = True,
|
|
gpu_memory_utilization = 0.7,
|
|
)
|
|
|
|
# Prep the Message
|
|
PROMPT = "How many r's are in the word strawberry?"
|
|
|
|
SYSTEM_PROMPT = """
|
|
A conversation between User and Assistant. The user asks a question,
|
|
and the Assistant solves it. The assistant first thinks about the
|
|
reasoning process in the mind and then provides the user with the answer.
|
|
Respond in the following format:
|
|
<reasoning>
|
|
...
|
|
</reasoning>
|
|
<answer>
|
|
...
|
|
</answer>
|
|
"""
|
|
|
|
text = tokenizer.apply_chat_template([
|
|
{"role" : "system", "content" : SYSTEM_PROMPT},
|
|
{"role" : "user", "content" : PROMPT},
|
|
], tokenize = False, add_generation_prompt = True)
|
|
|
|
# Generate a response
|
|
sampling_params = SamplingParams(
|
|
temperature = 0.8,
|
|
top_p = 0.95,
|
|
max_tokens = 1024,
|
|
)
|
|
output = model.fast_generate(
|
|
text,
|
|
sampling_params = sampling_params,
|
|
)[0].outputs[0].text
|
|
```
|
|
|
|
### Model Description
|
|
|
|
- **Model type:** Transformer-based language model fine-tuned for mathematical reasoning.
|
|
- **Language(s) (NLP):** English
|
|
- **License:** Apache 2.0
|
|
- **Finetuned from model:** [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
|
|
|
|
## Uses
|
|
|
|
### Direct Use
|
|
|
|
This model is intended for **mathematical reasoning tasks**, particularly for solving grade-school-level math problems as found in the GSM8K dataset. It can be used directly for question-answering tasks involving arithmetic and reasoning.
|
|
|
|
### Downstream Use [optional]
|
|
|
|
The model can be fine-tuned further for specific applications, such as tutoring systems, automated problem-solving tools, or other educational technologies.
|
|
|
|
### Out-of-Scope Use
|
|
|
|
This model is not designed for:
|
|
- High-level mathematical research or advanced problem-solving.
|
|
- Non-mathematical reasoning tasks without additional fine-tuning.
|
|
- Applications requiring high precision in domains outside its training data.
|
|
|
|
## Bias, Risks, and Limitations
|
|
|
|
- **Bias:** The model may inherit biases present in the GSM8K dataset or the base model.
|
|
- **Risks:** Incorrect reasoning or answers in critical applications (e.g., education or finance) could lead to misinformation.
|
|
- **Limitations:** The model's performance is constrained by the quality and scope of the GSM8K dataset and the base model's capabilities.
|
|
|
|
### Recommendations
|
|
|
|
Users should:
|
|
- Validate the model's outputs for critical applications.
|
|
- Fine-tune the model further for domain-specific tasks.
|
|
- Be aware of potential biases and limitations in reasoning capabilities.
|
|
|
|
## Citations
|
|
|
|
Cite GRPO as:
|
|
|
|
```bibtex
|
|
@article{zhihong2024deepseekmath,
|
|
title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
|
|
author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
|
|
year = 2024,
|
|
eprint = {arXiv:2402.03300},
|
|
}
|
|
|
|
```
|
|
|
|
Cite TRL as:
|
|
|
|
```bibtex
|
|
@misc{vonwerra2022trl,
|
|
title = {{TRL: Transformer Reinforcement Learning}},
|
|
author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
|
|
year = 2020,
|
|
journal = {GitHub repository},
|
|
publisher = {GitHub},
|
|
howpublished = {\url{https://github.com/huggingface/trl}}
|
|
}
|
|
```
|