license, library_name, pipeline_tag, tags, base_model

license	library_name	pipeline_tag	tags	base_model
apache-2.0	transformers	text-generation	qwen3 reinforcement-learning rlsd trust-region text-generation	Qwen/Qwen3-4B

Qwen3-4B-Biology-RLSD-TR

This repository contains the Qwen3-4B Biology RLSD_TR batch-size-32 run.

The repository root contains the best validation checkpoint, selected by validation mean@16. checkpoints/last/ contains the final checkpoint.

Performance

Dataset	Method	Base model	Train batch size	Best val mean@16	Best checkpoint	Final val mean@16	Final checkpoint
Biology	RLSD_TR	Qwen3-4B	32	48.50%	90	42.88%	100

Validation Mean@16

step	val_mean16	percent
10	0.325000000000	32.50%
20	0.328750000000	32.88%
30	0.362500000000	36.25%
40	0.371250000000	37.12%
50	0.411250000000	41.12%
60	0.406250000000	40.62%
70	0.420000000000	42.00%
80	0.418750000000	41.88%
90	0.485000000000	48.50%
100	0.428750000000	42.88%

Detailed Training Hyperparameters

Section	Parameter	Value	Source
Run identity	Base model	Qwen/Qwen3-4B	queue/script override
Run identity	Dataset	Biology / SciKnowEval biology	run_qwen3_generalization.sh
Run identity	Method	RLSD_TR	run_qwen3_generalization.sh
Run identity	Config	rlsd	run_qwen3_generalization.sh
Run identity	Experiment	qwen3gen-biology-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8	run_qwen3_generalization.sh
Run identity	W&B run	run-20260702_113907-xauqhwk1	wandb
Data	Train file	datasets/sciknoweval/biology/train.parquet	script override
Data	Validation file	datasets/sciknoweval/biology/test.parquet	script override
Data	Train batch size	32	queue/script override
Data	Train max samples	3200	queue/script override
Data	Prompt key	prompt	legacy_data.yaml default
Data	Reward key	data_source	legacy_data.yaml default
Data	Shuffle train data	True	user.yaml / legacy_data.yaml
Data	Validation shuffle	False	legacy_data.yaml default
Data	Filter overlong prompts	True	user.yaml
Data	Prompt truncation	error	legacy_data.yaml default
Data	enable_thinking	false	script override
Schedule	Total training steps	100	queue/script override
Schedule	Total epochs	30	ppo_trainer/user.yaml default
Schedule	Validation before train	False	queue/script override
Schedule	Save frequency	10	queue/script override
Schedule	Validation frequency	10	queue/script override
Sequence	Max prompt length	2048	queue/script override
Sequence	Max response length	8192	queue/script override
Sequence	Max model length	10240	queue/script override
Sequence	Actor max token length per GPU	10240	queue/script override
Rollout	Rollout engine	vllm	user.yaml
Rollout	Rollout dtype	bfloat16	rollout.yaml default
Rollout	Train rollout n	8	queue/script override
Rollout	Train rollout temperature	1.0	script override
Rollout	Train rollout top_p	1.0	script override
Rollout	Train rollout do_sample	True	rollout.yaml default
Rollout	Calculate rollout log probs	True	rlsd.yaml / script override
Rollout	Max num batched tokens	10240	queue/script override
Rollout	vLLM GPU memory utilization	0.8	queue/script override
Rollout	Tensor model parallel size	2	rollout.yaml default
Rollout	Free cache engine	True	rollout.yaml default
Validation	Validation rollout n	16	queue/script override
Validation	Validation temperature	0.6	queue/script override
Validation	Validation top_p	0.95	queue/script override
Validation	Validation do_sample	True	queue/script override
Optimization	Optimizer	AdamW	fsdp optimizer config
Optimization	Learning rate	1e-6	RLSD_TR method override
Optimization	LR scheduler	constant	W&B config
Optimization	LR warmup steps	10	script override
Optimization	Weight decay	0.01	script override
Optimization	Betas	(0.9, 0.999)	W&B config
Optimization	Gradient clip	1.0	script override
PPO/GRPO	Advantage estimator	grpo	rlsd.yaml
PPO/GRPO	Normalize GRPO advantages by std	False	script override
PPO/GRPO	PPO epochs	1	W&B config
PPO/GRPO	PPO mini batch size	8	queue/script override
PPO/GRPO	PPO micro batch size per GPU	1	user.yaml
PPO/GRPO	Clip ratio low	0.2	script override
PPO/GRPO	Clip ratio high	0.28	script override
PPO/GRPO	Gamma	1.0	ppo_trainer.yaml default
PPO/GRPO	Lambda	1.0	ppo_trainer.yaml default
PPO/GRPO	Use KL in reward	False	ppo_trainer/user.yaml
PPO/GRPO	Actor KL loss observed	0.0	output.log
Rollout correction	Importance sampling mode	token	script override
Rollout correction	IS threshold	2.0	script override
RLSD_TR	Policy loss mode	rlsd	method override
RLSD_TR	Teacher regularization	trust-region	method override
RLSD_TR	Trust-region mix / teacher update rate	0.1	queue/script override
RLSD_TR	Token reweight lambda	0.5	queue/script override
RLSD_TR	Token reweight eps_w	0.2	queue/script override
RLSD_TR	Token reweight decay steps	0	queue/script override
RLSD_TR	Max reprompt length	10240	method override
RLSD_TR	Fused kernels	False	method override
FSDP/System	Actor strategy	fsdp	dp_actor.yaml
FSDP/System	FSDP dtype	bfloat16	W&B config
FSDP/System	FSDP model dtype	fp32	W&B config
FSDP/System	Use torch compile	True	W&B config
FSDP/System	GPUs per node	8	queue/script override
FSDP/System	Nodes	1	user.yaml
FSDP/System	GPU type	NVIDIA H200	wandb-metadata
Checkpoint/Logging	Checkpoint root	checkpoints/datasets/sciknoweval/biology	script override
Checkpoint/Logging	Latest checkpointed iteration	100	latest_checkpointed_iteration.txt
Checkpoint/Logging	Max actor checkpoints to keep	1 + external _actor_archive preservation	user.yaml
Checkpoint/Logging	Logger	console, wandb	ppo_trainer.yaml
Checkpoint/Logging	W&B entity	seongryongjung-chung-ang-university	environment
Checkpoint/Logging	W&B project	SDPO-root	user.yaml project_name
Checkpoint/Logging	W&B group	QWEN3-RLSD-TR-GRPO-matched-generalization	method override
Checkpoint/Logging	External actor archive	checkpoints/datasets/sciknoweval/biology/qwen3gen-biology-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/_actor_archive	preserve_actor_checkpoints.py

Raw result and artifact files:

• results/validation_mean16.csv
• results/training_scores.csv
• results/hyperparameters.csv
• results/export_note.csv
• results/training_score.png
• results/training_score.svg
• artifacts/config.yaml
• artifacts/wandb-summary.json
• artifacts/wandb-metadata.json
• artifacts/output.log
• artifacts/queue.log

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "SeongryongJung/Qwen3-4B-Biology-RLSD-TR"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

Source

• Checkpoint: checkpoints/datasets/sciknoweval/biology/qwen3gen-biology-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8
• Root actor checkpoint: checkpoints/datasets/sciknoweval/biology/qwen3gen-biology-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/_actor_archive/global_step_90/actor
• Last actor checkpoint: checkpoints/datasets/sciknoweval/biology/qwen3gen-biology-RLSD_TR-Qwen-Qwen3-4B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_100/actor
• W&B run: run-20260702_113907-xauqhwk1
• Queue log: artifacts/queue.log