120 lines
5.5 KiB
Markdown
120 lines
5.5 KiB
Markdown
---
|
|
license: apache-2.0
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- qwen3
|
|
- reinforcement-learning
|
|
- rlsd-tr
|
|
- text-generation
|
|
- trust-region
|
|
base_model: Qwen/Qwen3-4B
|
|
---
|
|
|
|
# Qwen3-4B-Material-RLSD-TR
|
|
|
|
This repository contains the Qwen3-4B `material` `RLSD_TR` batch-size-32 run.
|
|
|
|
The repository root contains the best validation checkpoint, selected by validation `mean@16`. `checkpoints/last/` contains the final checkpoint.
|
|
|
|
## Performance
|
|
|
|
| Dataset | Method | Base model | Train batch size | Best val mean@16 | Best checkpoint | Final val mean@16 | Final checkpoint |
|
|
|---|---|---|---:|---:|---:|---:|---:|
|
|
| Material / SciKnowEval material | RLSD_TR | Qwen3-4B | 32 | 76.86% | 100 | 76.86% | 100 |
|
|
|
|
|
|
|
|
## Validation Mean@16
|
|
|
|
| step | val_mean16 | percent |
|
|
|---:|---:|---:|
|
|
| 10 | 0.652925531915 | 65.29% |
|
|
| 20 | 0.701462765957 | 70.15% |
|
|
| 30 | 0.725398936170 | 72.54% |
|
|
| 40 | 0.742021276596 | 74.20% |
|
|
| 50 | 0.747340425532 | 74.73% |
|
|
| 60 | 0.760638297872 | 76.06% |
|
|
| 70 | 0.766622340426 | 76.66% |
|
|
| 80 | 0.757978723404 | 75.80% |
|
|
| 90 | 0.763962765957 | 76.40% |
|
|
| 100 | 0.768617021277 | 76.86% |
|
|
|
|
## Detailed Training Hyperparameters
|
|
|
|
| Section | Parameter | Value | Source |
|
|
|---|---|---:|---|
|
|
| Run identity | `Base model` | `Qwen/Qwen3-4B` | queue/script override |
|
|
| Run identity | `Dataset` | `Material / SciKnowEval material` | run_qwen3_generalization.sh |
|
|
| Run identity | `Method` | `RLSD_TR` | run_qwen3_generalization.sh |
|
|
| Run identity | `Config` | `rlsd` | run_qwen3_generalization.sh |
|
|
| Run identity | `Experiment` | `qwen3gen-material-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8` | run_qwen3_generalization.sh |
|
|
| Run identity | `W&B run` | `run-20260702_135403-jd6eugui` | wandb |
|
|
| Data | `Train file` | `datasets/sciknoweval/material/train.parquet` | script override |
|
|
| Data | `Validation file` | `datasets/sciknoweval/material/test.parquet` | script override |
|
|
| Data | `Train batch size` | `32` | queue/script override |
|
|
| Data | `Train max samples` | `3200` | queue/script override |
|
|
| Schedule | `Total training steps` | `100` | queue/script override |
|
|
| Schedule | `Validation before train` | `False` | queue/script override |
|
|
| Schedule | `Save frequency` | `10` | queue/script override |
|
|
| Schedule | `Validation frequency` | `10` | queue/script override |
|
|
| Sequence | `Max prompt length` | `2048` | queue/script override |
|
|
| Sequence | `Max response length` | `8192` | queue/script override |
|
|
| Sequence | `Max model length` | `10240` | queue/script override |
|
|
| Rollout | `Train rollout n` | `8` | queue/script override |
|
|
| Rollout | `Validation rollout n` | `16` | queue/script override |
|
|
| Rollout | `vLLM GPU memory utilization` | `0.8` | queue/script override |
|
|
| Optimization | `Learning rate` | `1e-6` | RLSD_TR method override |
|
|
| Optimization | `Weight decay` | `0.01` | script override |
|
|
| PPO/GRPO | `PPO mini batch size` | `8` | queue/script override |
|
|
| PPO/GRPO | `Normalize GRPO advantages by std` | `False` | baseline_grpo.yaml / script override |
|
|
| Rollout correction | `Importance sampling mode` | `token` | script override |
|
|
| Rollout correction | `IS threshold` | `2.0` | script override |
|
|
| Checkpoint/Logging | `Checkpoint root` | `checkpoints/datasets/sciknoweval/material/qwen3gen-material-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8` | script override |
|
|
| Checkpoint/Logging | `Latest checkpointed iteration` | `100` | latest_checkpointed_iteration.txt |
|
|
| Checkpoint/Logging | `External actor archive` | `checkpoints/datasets/sciknoweval/material/qwen3gen-material-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/_actor_archive` | preserve_actor_checkpoints.py |
|
|
| Checkpoint/Logging | `Logger` | `console, wandb` | ppo_trainer.yaml |
|
|
| RLSD_TR | `Policy loss mode` | `rlsd` | method override |
|
|
| RLSD_TR | `Teacher regularization` | `trust-region` | method override |
|
|
| RLSD_TR | `Trust-region mix / teacher update rate` | `0.1` | queue/script override |
|
|
| RLSD_TR | `Token reweight lambda` | `0.5` | queue/script override |
|
|
| RLSD_TR | `Token reweight eps_w` | `0.2` | queue/script override |
|
|
| RLSD_TR | `Token reweight decay steps` | `0` | queue/script override |
|
|
| RLSD_TR | `Fused kernels` | `False` | method override |
|
|
|
|
Raw result and artifact files:
|
|
|
|
- `results/validation_mean16.csv`
|
|
- `results/training_scores.csv`
|
|
- `results/hyperparameters.csv`
|
|
- `results/training_score.png`
|
|
- `results/training_score.svg`
|
|
- `artifacts/config.yaml`
|
|
- `artifacts/wandb-summary.json`
|
|
- `artifacts/wandb-metadata.json`
|
|
- `artifacts/output.log`
|
|
- `artifacts/queue.log`
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
repo_id = "SeongryongJung/Qwen3-4B-Material-RLSD-TR"
|
|
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
repo_id,
|
|
torch_dtype="auto",
|
|
device_map="auto",
|
|
trust_remote_code=True,
|
|
)
|
|
```
|
|
|
|
## Source
|
|
|
|
- Checkpoint: `checkpoints/datasets/sciknoweval/material/qwen3gen-material-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8`
|
|
- Root actor checkpoint: `checkpoints/datasets/sciknoweval/material/qwen3gen-material-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_100/actor`
|
|
- Last actor checkpoint: `checkpoints/datasets/sciknoweval/material/qwen3gen-material-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_100/actor`
|
|
- W&B run: `run-20260702_135403-jd6eugui`
|
|
- Queue log: `artifacts/queue.log`
|