127 lines
3.5 KiB
Markdown
127 lines
3.5 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: Qwen/Qwen3-4B
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- qwen3
|
|
- chemistry
|
|
- reinforcement-learning
|
|
- verl
|
|
- sciknoweval
|
|
- sdpo
|
|
- self-distillation
|
|
---
|
|
|
|
# Qwen3-4B Chemistry SDPO
|
|
|
|
This repository contains Chemistry fine-tuned Qwen3-4B checkpoints from the local SciKnowEval-style generalization setup.
|
|
|
|
- Root checkpoint: final `global_step_100` merged to Hugging Face safetensors.
|
|
- `best_avg16/`: checkpoint with the highest validation `avg@16` during training, merged to Hugging Face safetensors.
|
|
|
|
## Checkpoints
|
|
|
|
| Checkpoint | Source step | Validation avg@16 | best@16 | maj@16 |
|
|
|---|---:|---:|---:|---:|
|
|
| Root final | 100 | 0.720536 | 0.731567 | 0.721367 |
|
|
| `best_avg16/` | 20 | 0.766369 | 0.822871 | 0.779486 |
|
|
|
|
## Training Run
|
|
|
|
`qwen3gen-chemistry-SDPO-Qwen-Qwen3-4B-mbs32-ema0.05-train256-rollout8-lr1e-5-vllm0.8`
|
|
|
|
W&B run: `https://wandb.ai/seongryongjung-chung-ang-university/qwen3-generalization-batch256/runs/1qazekk3`
|
|
|
|
## Base Model
|
|
|
|
- Base model: `Qwen/Qwen3-4B`
|
|
- Fine-tuning type: full-parameter FSDP RL training
|
|
- Dataset: `datasets/sciknoweval/chemistry`
|
|
- Train split: 1,890 examples
|
|
- Validation split: 210 examples
|
|
|
|
## Method
|
|
|
|
- Method: SDPO
|
|
- Config: `sdpo`
|
|
- Policy loss mode: `sdpo`
|
|
- Reward: local SciKnowEval multiple-choice reward checker
|
|
- Rollout correction: token-level importance sampling, threshold 2.0
|
|
|
|
## Hyperparameters
|
|
|
|
| Field | Value |
|
|
|---|---:|
|
|
| Base model | `Qwen/Qwen3-4B` |
|
|
| Training steps | 100 |
|
|
| Train batch size | 256 |
|
|
| Rollouts per prompt | 8 |
|
|
| Generations per step | 2048 |
|
|
| PPO mini batch size | 32 |
|
|
| Learning rate | `1e-5` |
|
|
| LR warmup steps | 10 |
|
|
| Weight decay | 0.01 |
|
|
| Grad clip | 1.0 |
|
|
| Max prompt length | 2048 |
|
|
| Max response length | 8192 |
|
|
| Max model length | 10240 |
|
|
| Train temperature | 1.0 |
|
|
| Train top_p | 1.0 |
|
|
| Validation generations | 16 |
|
|
| Validation temperature | 0.6 |
|
|
| Validation top_p | 0.95 |
|
|
| vLLM GPU memory utilization | 0.8 |
|
|
| GPUs | 8 x NVIDIA H200 |
|
|
| Save frequency | every 10 steps |
|
|
| Validation frequency | every 10 steps |
|
|
| Distillation top-k | 100 |
|
|
| SDPO alpha | 0.5 |
|
|
| Teacher update rate | 0.05 |
|
|
| Distillation IS clip | 2.0 |
|
|
| Max reprompt length | 10240 |
|
|
|
|
## Metrics
|
|
|
|
|
|
|
|
CSV files:
|
|
|
|
- [`training_score.csv`](training_score.csv)
|
|
- [`validation_metrics.csv`](validation_metrics.csv)
|
|
|
|
| Metric | Value |
|
|
|---|---:|
|
|
| Final training step | 100 |
|
|
| Final `critic/score/mean` | 0.845215 |
|
|
| Final `critic/rewards/mean` | 0.845215 |
|
|
| Final validation `avg@16` | 0.720536 |
|
|
| Peak validation `avg@16` | 0.766369 |
|
|
| Peak validation step | 20 |
|
|
|
|
## Loading
|
|
|
|
Root final checkpoint:
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("SeongryongJung/Qwen3-4B-Chemistry-SDPO")
|
|
tokenizer = AutoTokenizer.from_pretrained("SeongryongJung/Qwen3-4B-Chemistry-SDPO")
|
|
```
|
|
|
|
Best avg@16 checkpoint:
|
|
|
|
```python
|
|
model = AutoModelForCausalLM.from_pretrained("SeongryongJung/Qwen3-4B-Chemistry-SDPO", subfolder="best_avg16")
|
|
tokenizer = AutoTokenizer.from_pretrained("SeongryongJung/Qwen3-4B-Chemistry-SDPO", subfolder="best_avg16")
|
|
```
|
|
|
|
## Intended Use
|
|
|
|
This model is intended for research on RL fine-tuning and self-distillation behavior on science/generalization tasks. It has not been broadly safety evaluated for production use.
|
|
|
|
## Limitations
|
|
|
|
The reported scores are training-time and validation-time metrics from the local experimental setup. They should not be interpreted as broad benchmark results without independent evaluation.
|