84 lines
3.0 KiB
Markdown
84 lines
3.0 KiB
Markdown
---
|
||
base_model: Qwen/Qwen2.5-0.5B-Instruct
|
||
library_name: transformers
|
||
model_name: DeepMath-GRPO_Qwen2.5-0.5B-Instruct
|
||
tags:
|
||
- generated_from_trainer
|
||
- grpo
|
||
- trl
|
||
licence: license
|
||
---
|
||
|
||
# GRPO 微调 Qwen2.5-0.5B-Instruct
|
||
- 训练环境：
|
||
- 显卡：2 * NVIDIA 4080 32GB
|
||
- 加速框架：DeepSpeed，采用 bf16 混合精度训练
|
||
|
||
- 训练集：[zwhe99/DeepMath-103K](https://huggingface.co/datasets/zwhe99/DeepMath-103K)
|
||
- 训练数据量：97870
|
||
- 测试数据量：5152
|
||
- 验证集选取：从测试集随机抽取 100 条
|
||
- 最大迭代步数限制： 5000
|
||
- 训练参数：
|
||
```python
|
||
grpo_config = GRPOConfig(
|
||
# ---- 基础配置 ----
|
||
output_dir="./deepmath_grpo_output",
|
||
save_strategy='best',
|
||
save_total_limit=5,
|
||
#save_steps=100,
|
||
# ---- 批次大小 ----
|
||
per_device_train_batch_size=4, # 每设备批次大小
|
||
per_device_eval_batch_size=4,
|
||
gradient_accumulation_steps=4, # 梯度累积步数
|
||
# 有效批次大小 = 4 * 2 * 8 GPUs = 64（与论文 512 有差距，可根据硬件调整）
|
||
|
||
# ---- 训练步数 ----
|
||
max_steps=1000, # 论文中 DeepMath-Zero 训练 500 步
|
||
#num_train_epochs=1,
|
||
# ---- 推理框架配置
|
||
use_vllm=True,
|
||
vllm_gpu_memory_utilization=0.3,
|
||
|
||
# 评估策略
|
||
eval_strategy='steps',
|
||
eval_steps=50,
|
||
metric_for_best_model="eval_reward",
|
||
greater_is_better=True,
|
||
logging_strategy='epoch',
|
||
logging_dir="train_logs/",
|
||
load_best_model_at_end=True,
|
||
|
||
# ---- 学习率 ----
|
||
learning_rate=1e-6, # 论文 Table 5: lr=1e-6
|
||
|
||
# ---- GRPO 特有参数 ----
|
||
num_generations=settings.GROUP_SIZE_TRAIN, # 4
|
||
num_generations_eval=settings.GROUP_SIZE_EVAL, # 4
|
||
generation_batch_size=4, # 生成批次大小
|
||
max_completion_length=2048, # 最大生成长度（论文推理时为 32768，训练时 2048）
|
||
loss_type='grpo', # 使用标准 GRPO 算法训练
|
||
|
||
# ---- KL 散度控制 ----
|
||
beta=0.001, # 论文 Table 5: kl_coef=1e-3
|
||
|
||
# ---- 裁剪参数 ----
|
||
epsilon=0.2, # 论文 Table 5: clip_ratio_low=0.2
|
||
epsilon_high=0.28, # 论文 Table 5: clip_ratio_high=0.28
|
||
|
||
# ---- 温度参数 ----
|
||
temperature=1.0, # 论文 Table 5: temperature=1.0 (Zero RL)
|
||
|
||
# ---- 内存优化 ----
|
||
bf16=torch.cuda.is_bf16_supported(),
|
||
fp16=not torch.cuda.is_bf16_supported(),
|
||
gradient_checkpointing=True,
|
||
|
||
# ---- 报告与日志 ----
|
||
report_to="tensorboard", # 可改为 "wandb" 启用 wandb 日志
|
||
run_name="deepmath-grpo-qwen-0.5b-instruct",
|
||
|
||
# 随机种子
|
||
seed=settings.SEED
|
||
)
|
||
``` |