49 lines
1.5 KiB
Markdown
49 lines
1.5 KiB
Markdown
|
---
|
||
|
|
library_name: transformers
|
||
|
|
license: mit
|
||
|
|
datasets:
|
||
|
|
- openai/gsm8k
|
||
|
|
base_model:
|
||
|
|
- Qwen/Qwen2.5-Math-1.5B
|
||
|
|
---
|
||
|
|
|
||
|
|
used GRPO on top of Qwen/Qwen2.5-Math-1.5B model on the gsm8k training set
|
||
|
|
|
||
|
|
- used REINFORCE loss with baseline
|
||
|
|
- used [per-sample length normalization for loss aggregation](https://rlhfbook.com/c/11-policy-gradients#loss-aggregation)
|
||
|
|
|
||
|
|
[wandb training run metrics](https://api.wandb.ai/links/teammapo-mapo-labs/l6awailg)
|
||
|
|
|
||
|
|
hyperparameters:
|
||
|
|
```
|
||
|
|
lr = 3e-5
|
||
|
|
n_grpo_steps = 100
|
||
|
|
advantage_eps = 1e-6
|
||
|
|
rollout_batch_size = 256
|
||
|
|
group_size = 8
|
||
|
|
gradient_accumulation_steps = 128
|
||
|
|
epochs_per_rollout_batch = 1
|
||
|
|
train_batch_size = 256 # on policy since same as rollout_batch_size
|
||
|
|
use_std_normalization = True
|
||
|
|
|
||
|
|
optim = torch.optim.AdamW(
|
||
|
|
hf_model.parameters(), lr=lr, weight_decay=0.0, betas=(0.9, 0.95)
|
||
|
|
)
|
||
|
|
scheduler = LinearLR(optim, start_factor=1.0, end_factor=0.1, total_iters=n_grpo_steps)
|
||
|
|
|
||
|
|
```
|
||
|
|
|
||
|
|
prompt template:
|
||
|
|
```
|
||
|
|
A conversation between User and Assistant. The User asks a question, and the Assistant solves it. The Assistant first thinks about the reasoning process in the mind and then provides the User with the answer. The reasoning process is enclosed within <think> </think> and answer is enclosed within <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>.
|
||
|
|
User: {question}
|
||
|
|
Assistant: <think>
|
||
|
|
```
|
||
|
|
|
||
|
|
reward format checks for proper use of `<think></think>` and `<answer></answer>` tags
|
||
|
|
|
||
|
|
performance on GSM8K test set
|
||
|
|
```
|
||
|
|
correct format: 1172/1319
|
||
|
|
correct reward: 966/1319
|
||
|
|
```
|