---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen3
- reinforcement-learning
- rlsd
- trust-region
- text-generation
base_model: Qwen/Qwen3-4B
---

# Qwen3-4B-Chemical-RLSD-TR

This repository contains the Qwen3-4B `Chemical` `RLSD_TR` batch-size-32 run.

The repository root contains the best validation checkpoint, selected by validation `mean@16`. `checkpoints/last/` contains the final checkpoint. For this run, best and final are both `global_step_100`.

## Performance

| Dataset | Method | Base model | Train batch size | Best val mean@16 | Best checkpoint | Final val mean@16 | Final checkpoint |
|---|---|---|---:|---:|---:|---:|---:|
| Chemical | RLSD_TR | Qwen3-4B | 32 | 68.96% | 100 | 68.96% | 100 |

![Training and validation scores](results/training_score.png)

## Validation Mean@16

| step | val_mean16 | percent |
|---:|---:|---:|
| 10 | 0.439880952381 | 43.99% |
| 20 | 0.510119047619 | 51.01% |
| 30 | 0.594345238095 | 59.43% |
| 40 | 0.635714285714 | 63.57% |
| 50 | 0.658630952381 | 65.86% |
| 60 | 0.673511904762 | 67.35% |
| 70 | 0.685119047619 | 68.51% |
| 80 | 0.678869047619 | 67.89% |
| 90 | 0.682142857143 | 68.21% |
| 100 | 0.689583333333 | 68.96% |

## Detailed Training Hyperparameters

| Section | Parameter | Value | Source |
|---|---|---:|---|
| Run identity | `Base model` | `Qwen/Qwen3-4B` | queue/script override |
| Run identity | `Dataset` | `Chemical / SciKnowEval chemistry` | run_qwen3_generalization.sh |
| Run identity | `Method` | `RLSD_TR` | run_qwen3_generalization.sh |
| Run identity | `Config` | `rlsd` | run_qwen3_generalization.sh |
| Run identity | `Experiment` | `qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8` | run_qwen3_generalization.sh |
| Run identity | `W&B run` | `run-20260702_062054-bz6p2yxy` | wandb |
| Data | `Train file` | `datasets/sciknoweval/chemistry/train.parquet` | script override |
| Data | `Validation file` | `datasets/sciknoweval/chemistry/test.parquet` | script override |
| Data | `Train batch size` | `32` | queue/script override |
| Data | `Train max samples` | `3200` | queue/script override |
| Data | `Prompt key` | `prompt` | legacy_data.yaml default |
| Data | `Reward key` | `data_source` | legacy_data.yaml default |
| Data | `Shuffle train data` | `True` | user.yaml / legacy_data.yaml |
| Data | `Validation shuffle` | `False` | legacy_data.yaml default |
| Data | `Filter overlong prompts` | `True` | user.yaml |
| Data | `Prompt truncation` | `error` | legacy_data.yaml default |
| Data | `enable_thinking` | `false` | script override |
| Schedule | `Total training steps` | `100` | queue/script override |
| Schedule | `Total epochs` | `30` | ppo_trainer/user.yaml default |
| Schedule | `Validation before train` | `False` | queue/script override |
| Schedule | `Save frequency` | `10` | queue/script override |
| Schedule | `Validation frequency` | `10` | queue/script override |
| Sequence | `Max prompt length` | `2048` | queue/script override |
| Sequence | `Max response length` | `8192` | queue/script override |
| Sequence | `Max model length` | `10240` | queue/script override |
| Sequence | `Actor max token length per GPU` | `10240` | queue/script override |
| Rollout | `Rollout engine` | `vllm` | user.yaml |
| Rollout | `Rollout dtype` | `bfloat16` | rollout.yaml default |
| Rollout | `Train rollout n` | `8` | queue/script override |
| Rollout | `Train rollout temperature` | `1.0` | script override |
| Rollout | `Train rollout top_p` | `1.0` | script override |
| Rollout | `Train rollout do_sample` | `True` | rollout.yaml default |
| Rollout | `Calculate rollout log probs` | `True` | rlsd.yaml / script override |
| Rollout | `Max num batched tokens` | `10240` | queue/script override |
| Rollout | `vLLM GPU memory utilization` | `0.8` | queue/script override |
| Rollout | `Tensor model parallel size` | `2` | rollout.yaml default |
| Rollout | `Free cache engine` | `True` | rollout.yaml default |
| Validation | `Validation rollout n` | `16` | queue/script override |
| Validation | `Validation temperature` | `0.6` | queue/script override |
| Validation | `Validation top_p` | `0.95` | queue/script override |
| Validation | `Validation do_sample` | `True` | queue/script override |
| Optimization | `Optimizer` | `AdamW` | fsdp optimizer config |
| Optimization | `Learning rate` | `1e-6` | RLSD_TR method override |
| Optimization | `LR scheduler` | `constant` | W&B config |
| Optimization | `LR warmup steps` | `10` | script override |
| Optimization | `Weight decay` | `0.01` | script override |
| Optimization | `Betas` | `(0.9, 0.999)` | W&B config |
| Optimization | `Gradient clip` | `1.0` | script override |
| PPO/GRPO | `Advantage estimator` | `grpo` | rlsd.yaml |
| PPO/GRPO | `Normalize GRPO advantages by std` | `False` | script override |
| PPO/GRPO | `PPO epochs` | `1` | W&B config |
| PPO/GRPO | `PPO mini batch size` | `8` | queue/script override |
| PPO/GRPO | `PPO micro batch size per GPU` | `1` | user.yaml |
| PPO/GRPO | `Clip ratio low` | `0.2` | script override |
| PPO/GRPO | `Clip ratio high` | `0.28` | script override |
| PPO/GRPO | `Gamma` | `1.0` | ppo_trainer.yaml default |
| PPO/GRPO | `Lambda` | `1.0` | ppo_trainer.yaml default |
| PPO/GRPO | `Use KL in reward` | `False` | ppo_trainer/user.yaml |
| PPO/GRPO | `Actor KL loss observed` | `0.0` | output.log |
| Rollout correction | `Importance sampling mode` | `token` | script override |
| Rollout correction | `IS threshold` | `2.0` | script override |
| RLSD_TR | `Policy loss mode` | `rlsd` | method override |
| RLSD_TR | `Teacher regularization` | `trust-region` | method override |
| RLSD_TR | `Trust-region mix / teacher update rate` | `0.1` | queue/script override |
| RLSD_TR | `Token reweight lambda` | `0.5` | queue/script override |
| RLSD_TR | `Token reweight eps_w` | `0.2` | queue/script override |
| RLSD_TR | `Token reweight decay steps` | `0` | queue/script override |
| RLSD_TR | `Max reprompt length` | `10240` | method override |
| RLSD_TR | `Fused kernels` | `False` | method override |
| FSDP/System | `Actor strategy` | `fsdp` | dp_actor.yaml |
| FSDP/System | `FSDP dtype` | `bfloat16` | W&B config |
| FSDP/System | `FSDP model dtype` | `fp32` | W&B config |
| FSDP/System | `Use torch compile` | `True` | W&B config |
| FSDP/System | `GPUs per node` | `8` | queue/script override |
| FSDP/System | `Nodes` | `1` | user.yaml |
| FSDP/System | `GPU type` | `NVIDIA H200` | wandb-metadata |
| Checkpoint/Logging | `Checkpoint root` | `checkpoints/datasets/sciknoweval/chemistry` | script override |
| Checkpoint/Logging | `Latest checkpointed iteration` | `100` | latest_checkpointed_iteration.txt |
| Checkpoint/Logging | `Max actor checkpoints to keep` | `1` | user.yaml |
| Checkpoint/Logging | `Logger` | `console, wandb` | ppo_trainer.yaml |
| Checkpoint/Logging | `W&B entity` | `seongryongjung-chung-ang-university` | environment |
| Checkpoint/Logging | `W&B project` | `SDPO-root` | user.yaml project_name |
| Checkpoint/Logging | `W&B group` | `QWEN3-RLSD-TR-GRPO-matched-generalization` | method override |

Raw result and artifact files:

- `results/validation_mean16.csv`
- `results/training_scores.csv`
- `results/hyperparameters.csv`
- `results/training_score.png`
- `results/training_score.svg`
- `artifacts/config.yaml`
- `artifacts/wandb-summary.json`
- `artifacts/wandb-metadata.json`
- `artifacts/output.log`
- `artifacts/queue.log`

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "SeongryongJung/Qwen3-4B-Chemical-RLSD-TR"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
```

## Source

- Checkpoint: `checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8`
- W&B run: `run-20260702_062054-bz6p2yxy`
- Queue log: `artifacts/queue.log`