SeongryongJung/qwen3-4b-tooluse-srpo-ema005
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: SeongryongJung/qwen3-4b-tooluse-srpo-ema005
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-09-20 16:14:29 +08:00
commit 9aabae373d
19 changed files with 153324 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

37
.gitattributes vendored Normal file
View File

@@ -0,0 +1,37 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text
results/training_score.png filter=lfs diff=lfs merge=lfs -text

65
README.md Normal file
View File

@@ -0,0 +1,65 @@
---
license: apache-2.0
base_model: Qwen/Qwen3-4B
library_name: transformers
tags:
- qwen3
- reinforcement-learning
- sdpo
- srpo
- ema
---
# qwen3-4b-tooluse-srpo-ema005
This repository contains the **last checkpoint** (`global_step_100`) for `qwen3gen-tooluse-SRPO-Qwen-Qwen3-4B-mbs32-ema0.05-dwtrue-train64-rollout8-lr5e-6-vllm0.8`,
converted to Hugging Face Transformers format.
## Evaluation
The reported headline score is the **best validation `mean@16` observed during training**.
It is not necessarily the score of the uploaded last checkpoint.
| Dataset | Method | Model | Uploaded checkpoint | Best val mean@16 | Best step | Final val mean@16 |
|---|---|---|---|---:|---:|---:|
| tooluse | SRPO | Qwen3-4B | global_step_100 | 62.59% | 40 | 57.26% |
![Training and validation scores](results/training_score.png)
Raw result files:
- `results/validation_mean16.csv`
- `results/training_scores.csv`
- `artifacts/config.yaml`
- `artifacts/wandb-summary.json`
## Training Setup
- Base model: `Qwen/Qwen3-4B`
- Dataset: `tooluse`
- Method: `SRPO`
- EMA teacher update rate: `0.05`
- Uploaded weights: last checkpoint, `global_step_100`
- Validation metric used for headline score: `val-aux/*/mean@16`
- Validation sampling: `n=16`
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "SeongryongJung/qwen3-4b-tooluse-srpo-ema005"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
torch_dtype="auto",
device_map="auto",
trust_remote_code=True,
)
```
## Notes
Most best-validation intermediate checkpoints were not retained as full actor checkpoints because
training kept only the latest actor checkpoint. Therefore, this repository publishes the last
checkpoint and records the best validation score separately.

28
added_tokens.json Normal file
View File

@@ -0,0 +1,28 @@
{
"</think>": 151668,
"</tool_call>": 151658,
"</tool_response>": 151666,
"<think>": 151667,
"<tool_call>": 151657,
"<tool_response>": 151665,
"<|box_end|>": 151649,
"<|box_start|>": 151648,
"<|endoftext|>": 151643,
"<|file_sep|>": 151664,
"<|fim_middle|>": 151660,
"<|fim_pad|>": 151662,
"<|fim_prefix|>": 151659,
"<|fim_suffix|>": 151661,
"<|im_end|>": 151645,
"<|im_start|>": 151644,
"<|image_pad|>": 151655,
"<|object_ref_end|>": 151647,
"<|object_ref_start|>": 151646,
"<|quad_end|>": 151651,
"<|quad_start|>": 151650,
"<|repo_name|>": 151663,
"<|video_pad|>": 151656,
"<|vision_end|>": 151653,
"<|vision_pad|>": 151654,
"<|vision_start|>": 151652
}

843
artifacts/config.yaml Normal file
View File

@@ -0,0 +1,843 @@
_wandb:
value:
cli_version: 0.23.1
e:
p3inywz7efk1hpyldgk0t4fc257rbpgl:
args:
- --node-ip-address=198.19.35.151
- --node-manager-port=33715
- --object-store-name=/tmp/ray_q3g_tooluse_srpo_Qwen3_4B/session_2026-06-30_06-12-46_221885_752733/sockets/plasma_store
- --raylet-name=/tmp/ray_q3g_tooluse_srpo_Qwen3_4B/session_2026-06-30_06-12-46_221885_752733/sockets/raylet
- --redis-address=None
- --metrics-agent-port=57754
- --logging-rotate-bytes=536870912
- --logging-rotate-backup-count=5
- --runtime-env-agent-port=38395
- --gcs-address=198.19.35.151:62485
- --session-name=session_2026-06-30_06-12-46_221885_752733
- --temp-dir=/tmp/ray_q3g_tooluse_srpo_Qwen3_4B
- --webui=
- --cluster-id=65c41a6cecd205891f2c8aa4b78ab56248af70fef2d21e6a2819db6f
- --startup-token=32
- --worker-launch-time-ms=1782799969624
- --node-id=9f4bd0a1963471a22b7027466c8990b5b7250c7e0574e41ec90cf7d0
- --runtime-env-hash=-1580796864
cpu_count: 64
cpu_count_logical: 128
cudaVersion: "13.0"
disk:
/:
total: "46086056050688"
used: "5140013613056"
email: jungsr1116@cau.ac.kr
executable: /workspace/SIPO/.venv/bin/python
git:
commit: c237ec221cb7566d5f85232708425314a6ffc926
remote: https://github.com/jungseongryong/L2T
gpu: NVIDIA H200
gpu_count: 8
gpu_nvidia:
- architecture: Hopper
cudaCores: 16896
memoryTotal: "150754820096"
name: NVIDIA H200
uuid: GPU-b897a6c4-7996-e2f1-0cc0-7b02d6a38587
- architecture: Hopper
cudaCores: 16896
memoryTotal: "150754820096"
name: NVIDIA H200
uuid: GPU-f7cf7fff-d108-62cb-498a-225bffd6d151
- architecture: Hopper
cudaCores: 16896
memoryTotal: "150754820096"
name: NVIDIA H200
uuid: GPU-5a9f5a22-0fdb-36cc-c516-0b618776d8ca
- architecture: Hopper
cudaCores: 16896
memoryTotal: "150754820096"
name: NVIDIA H200
uuid: GPU-d547a11f-7fca-c74b-76fb-73993c30f136
- architecture: Hopper
cudaCores: 16896
memoryTotal: "150754820096"
name: NVIDIA H200
uuid: GPU-ac3cb18b-d562-1d58-725e-10738ff5694c
- architecture: Hopper
cudaCores: 16896
memoryTotal: "150754820096"
name: NVIDIA H200
uuid: GPU-e8484200-8103-548e-6421-89d2884c6d32
- architecture: Hopper
cudaCores: 16896
memoryTotal: "150754820096"
name: NVIDIA H200
uuid: GPU-83471bd2-f389-d36d-63de-d4759c4435d3
- architecture: Hopper
cudaCores: 16896
memoryTotal: "150754820096"
name: NVIDIA H200
uuid: GPU-885aa58c-6341-2956-85d4-ca01897dd4a2
host: mole-gpu-2
memory:
total: "2163980390400"
os: Linux-6.8.0-71-generic-x86_64-with-glibc2.36
program: /workspace/SIPO/.venv/lib/python3.12/site-packages/ray/_private/workers/default_worker.py
python: CPython 3.12.13
root: /workspace/L2T
startedAt: "2026-06-30T06:14:42.554183Z"
writerId: p3inywz7efk1hpyldgk0t4fc257rbpgl
m: []
python_version: 3.12.13
t:
"1":
- 1
- 11
- 30
- 41
- 49
- 50
- 51
- 71
- 95
- 98
- 105
"2":
- 1
- 11
- 30
- 41
- 49
- 50
- 51
- 71
- 95
- 98
- 105
"3":
- 2
- 13
- 16
- 61
"4": 3.12.13
"5": 0.23.1
"6": 4.57.1
"12": 0.23.1
"13": linux-x86_64
actor_rollout_ref:
value:
actor:
_target_: verl.workers.config.FSDPActorConfig
calculate_entropy: false
calculate_sum_pi_squared: false
checkpoint:
_target_: verl.trainer.config.CheckpointConfig
async_save: false
load_contents:
- model
- optimizer
- extra
save_contents:
- model
- optimizer
- extra
clip_ratio: 0.2
clip_ratio_c: 3
clip_ratio_high: 0.28
clip_ratio_low: 0.2
data_loader_seed: 42
entropy_checkpointing: false
entropy_coeff: 0
entropy_from_logits_with_chunking: false
freeze_vision_tower: false
fsdp_config:
_target_: verl.workers.config.FSDPEngineConfig
dtype: bfloat16
entropy_checkpointing: false
entropy_from_logits_with_chunking: false
forward_only: false
forward_prefetch: false
fsdp_size: -1
full_determinism: false
model_dtype: fp32
offload_policy: false
optimizer_offload: false
param_offload: false
reshard_after_forward: true
seed: 42
strategy: fsdp
ulysses_sequence_parallel_size: 1
use_orig_params: false
use_torch_compile: true
wrap_policy:
min_num_params: 0
grad_clip: 1
kl_loss_coef: 0.001
kl_loss_type: low_var_kl
loss_agg_mode: token-mean
loss_scale_factor: null
optim:
_target_: verl.workers.config.FSDPOptimizerConfig
betas:
- 0.9
- 0.999
clip_grad: 1
lr: 5e-06
lr_scheduler_type: constant
lr_warmup_steps: 10
lr_warmup_steps_ratio: 0
min_lr_ratio: 0
num_cycles: 0.5
optimizer: AdamW
optimizer_impl: torch.optim
override_optimizer_config: null
total_training_steps: 100
warmup_style: null
weight_decay: 0.01
policy_loss:
_target_: verl.workers.config.PolicyLossConfig
clip_cov_lb: 1
clip_cov_ratio: 0.0002
clip_cov_ub: 5
kl_cov_ratio: 0.0002
loss_mode: srpo
ppo_kl_coef: 0.1
ppo_epochs: 1
ppo_max_token_len_per_gpu: 10240
ppo_micro_batch_size: null
ppo_micro_batch_size_per_gpu: 1
ppo_mini_batch_size: 32
profiler:
_target_: verl.utils.profiler.ProfilerConfig
all_ranks: false
enable: false
ranks: []
save_path: outputs/profile
tool: null
tool_config:
npu:
_target_: verl.utils.profiler.config.NPUToolConfig
analysis: true
contents: []
discrete: false
level: level0
nsys:
_target_: verl.utils.profiler.config.NsightToolConfig
discrete: false
torch:
_target_: verl.utils.profiler.config.TorchProfilerToolConfig
step_end: null
step_start: 0
torch_memory:
_target_: verl.utils.profiler.config.TorchMemoryToolConfig
stack_depth: 32
trace_alloc_max_entries: 100000
rollout_n: 8
router_replay:
_target_: verl.workers.config.RouterReplayConfig
mode: disabled
record_file: null
replay_file: null
self_distillation:
_target_: verl.workers.config.SelfDistillationConfig
alpha: 0.5
distillation_add_tail: true
distillation_topk: 100
dont_reprompt_on_self_success: true
environment_feedback_only_without_solution: true
feedback_template: |4-
The following is feedback from your unsuccessful earlier attempt:
{feedback_raw}
full_logit_distillation: true
include_environment_feedback: false
is_clip: 2
max_reprompt_len: 10240
remove_thinking_from_demonstration: false
reprompt_template: |-
{prompt}{solution}{feedback}
Correctly solve the original question.
reprompt_truncation: right
solution_template: |4-
Correct solution:
{successful_previous_attempt}
srpo_dynamic_weighting: true
srpo_dynamic_weighting_temperature: 1
success_reward_threshold: 0.5
teacher_regularization: ema
teacher_update_rate: 0.05
token_reweight_decay_steps: null
token_reweight_eps_w: 0.2
token_reweight_lambda: 0.5
shuffle: false
strategy: fsdp
sum_pi_squared_checkpointing: false
tau_neg: 1.05
tau_pos: 1
ulysses_sequence_parallel_size: 1
use_dynamic_bsz: false
use_fused_kernels: false
use_kl_loss: false
use_prefix_grouper: false
use_remove_padding: true
use_torch_compile: true
hybrid_engine: true
model:
_target_: verl.workers.config.HFModelConfig
custom_chat_template: null
enable_activation_offload: false
enable_gradient_checkpointing: true
exclude_modules: null
external_lib: null
fused_kernel_options:
impl_backend: torch
hf_config_path: null
lora_adapter_path: null
lora_alpha: 16
lora_rank: 0
path: Qwen/Qwen3-4B
target_modules: all-linear
tiled_mlp:
enabled: false
num_shards: 4
tokenizer_path: null
trust_remote_code: true
use_fused_kernels: false
use_liger: false
use_remove_padding: true
use_shm: false
nccl_timeout: 600
ref:
_target_: verl.workers.config.FSDPActorConfig
entropy_checkpointing: false
entropy_from_logits_with_chunking: false
fsdp_config:
_target_: verl.workers.config.FSDPEngineConfig
dtype: bfloat16
entropy_checkpointing: false
entropy_from_logits_with_chunking: false
forward_only: true
forward_prefetch: false
fsdp_size: -1
full_determinism: false
model_dtype: fp32
offload_policy: false
optimizer_offload: false
param_offload: false
reshard_after_forward: true
seed: 42
strategy: fsdp
ulysses_sequence_parallel_size: 1
use_orig_params: false
use_torch_compile: true
wrap_policy:
min_num_params: 0
log_prob_max_token_len_per_gpu: 10240
log_prob_micro_batch_size: null
log_prob_micro_batch_size_per_gpu: 1
log_prob_use_dynamic_bsz: false
profiler:
_target_: verl.utils.profiler.ProfilerConfig
all_ranks: false
enable: false
ranks: []
save_path: outputs/profile
tool: null
tool_config:
npu:
_target_: verl.utils.profiler.config.NPUToolConfig
analysis: true
contents: []
discrete: false
level: level0
nsys:
_target_: verl.utils.profiler.config.NsightToolConfig
discrete: false
torch:
_target_: verl.utils.profiler.config.TorchProfilerToolConfig
step_end: null
step_start: 0
torch_memory:
_target_: verl.utils.profiler.config.TorchMemoryToolConfig
stack_depth: 32
trace_alloc_max_entries: 100000
rollout_n: 8
router_replay:
_target_: verl.workers.config.RouterReplayConfig
mode: disabled
record_file: null
replay_file: null
strategy: fsdp
ulysses_sequence_parallel_size: 1
use_torch_compile: true
rollout:
_target_: verl.workers.config.RolloutConfig
agent:
_target_: verl.workers.config.AgentLoopConfig
agent_loop_config_path: null
custom_async_server:
_target_: verl.workers.config.CustomAsyncServerConfig
name: null
path: null
default_agent_loop: single_turn_agent
num_workers: 8
calculate_log_probs: true
cudagraph_capture_sizes: null
data_parallel_size: 1
disable_log_stats: true
do_sample: true
dtype: bfloat16
enable_chunked_prefill: true
enable_prefix_caching: true
enable_rollout_routing_replay: false
enforce_eager: false
expert_parallel_size: 1
free_cache_engine: true
gpu_memory_utilization: 0.8
ignore_eos: false
layered_summon: false
load_format: dummy
log_prob_max_token_len_per_gpu: 10240
log_prob_micro_batch_size: null
log_prob_micro_batch_size_per_gpu: 1
log_prob_use_dynamic_bsz: false
logprobs_mode: processed_logprobs
max_model_len: 10240
max_num_batched_tokens: 10240
max_num_seqs: 1024
mode: async
multi_stage_wake_up: false
multi_turn:
_target_: verl.workers.config.MultiTurnConfig
enable: false
format: hermes
interaction_config_path: null
max_assistant_turns: null
max_parallel_calls: 1
max_tool_response_length: 256
max_user_turns: null
num_repeat_rollouts: null
tokenization_sanity_check_mode: strict
tool_config_path: null
tool_response_truncate_side: middle
use_inference_chat_template: false
"n": 8
name: vllm
over_sample_rate: 0
pipeline_model_parallel_size: 1
profiler:
_target_: verl.utils.profiler.ProfilerConfig
all_ranks: false
enable: false
ranks: []
save_path: outputs/profile
tool: null
tool_config:
npu:
_target_: verl.utils.profiler.config.NPUToolConfig
analysis: true
contents: []
discrete: false
level: level0
nsys:
_target_: verl.utils.profiler.config.NsightToolConfig
discrete: false
torch:
_target_: verl.utils.profiler.config.TorchProfilerToolConfig
step_end: null
step_start: 0
torch_memory:
_target_: verl.utils.profiler.config.TorchMemoryToolConfig
stack_depth: 32
trace_alloc_max_entries: 100000
prometheus:
_target_: verl.workers.config.PrometheusConfig
enable: false
file: /tmp/ray/session_latest/metrics/prometheus/prometheus.yml
port: 9090
served_model_name: Qwen/Qwen3-4B
prompt_length: 2048
quantization: null
quantization_config_file: null
response_length: 8192
scheduling_policy: fcfs
skip_dump_dir: /tmp/rollout_dump
skip_rollout: false
skip_tokenizer_init: true
temperature: 1
tensor_model_parallel_size: 2
top_k: -1
top_p: 1
trace:
_target_: verl.workers.config.TraceConfig
backend: null
max_samples_per_step_per_worker: null
token2text: false
update_weights_bucket_megabytes: 512
val_kwargs:
_target_: verl.workers.config.SamplingConfig
do_sample: true
"n": 16
temperature: 0.6
top_k: -1
top_p: 0.95
algorithm:
value:
_target_: verl.trainer.config.AlgoConfig
adv_estimator: grpo
gamma: 1
kl_ctrl:
_target_: verl.trainer.config.KLControlConfig
horizon: 10000
kl_coef: 0.001
target_kl: 0.1
type: fixed
kl_penalty: kl
lam: 1
norm_adv_by_std_in_grpo: false
pf_ppo:
reweight_method: pow
weight_pow: 2
rollout_correction:
bypass_mode: false
loss_type: ppo_clip
rollout_is: token
rollout_is_batch_normalize: false
rollout_is_threshold: 2
rollout_rs: null
rollout_rs_threshold: null
use_kl_in_reward: false
use_pf_ppo: false
critic:
value:
_target_: verl.workers.config.FSDPCriticConfig
checkpoint:
_target_: verl.trainer.config.CheckpointConfig
async_save: false
load_contents:
- model
- optimizer
- extra
save_contents:
- model
- optimizer
- extra
cliprange_value: 0.5
data_loader_seed: 42
enable: null
forward_max_token_len_per_gpu: 32768
forward_micro_batch_size: null
forward_micro_batch_size_per_gpu: null
grad_clip: 1
loss_agg_mode: token-mean
model:
_target_: verl.workers.config.FSDPCriticModelCfg
enable_activation_offload: false
enable_gradient_checkpointing: true
external_lib: null
fsdp_config:
_target_: verl.workers.config.FSDPEngineConfig
dtype: bfloat16
entropy_checkpointing: false
entropy_from_logits_with_chunking: false
forward_only: false
forward_prefetch: false
fsdp_size: -1
full_determinism: false
model_dtype: fp32
offload_policy: false
optimizer_offload: false
param_offload: false
reshard_after_forward: true
seed: 42
strategy: fsdp
ulysses_sequence_parallel_size: 1
use_orig_params: false
use_torch_compile: true
wrap_policy:
min_num_params: 0
lora_alpha: 16
lora_rank: 0
path: Qwen/Qwen3-8B
target_modules: all-linear
tiled_mlp:
enabled: false
num_shards: 4
tokenizer_path: Qwen/Qwen3-4B
trust_remote_code: true
use_remove_padding: false
use_shm: false
optim:
_target_: verl.workers.config.FSDPOptimizerConfig
betas:
- 0.9
- 0.999
clip_grad: 1
lr: 1e-05
lr_scheduler_type: constant
lr_warmup_steps: -1
lr_warmup_steps_ratio: 0
min_lr_ratio: 0
num_cycles: 0.5
optimizer: AdamW
optimizer_impl: torch.optim
override_optimizer_config: null
total_training_steps: 100
warmup_style: null
weight_decay: 0.01
ppo_epochs: 1
ppo_max_token_len_per_gpu: 32768
ppo_micro_batch_size: null
ppo_micro_batch_size_per_gpu: null
ppo_mini_batch_size: 32
profiler:
_target_: verl.utils.profiler.ProfilerConfig
all_ranks: false
enable: false
ranks: []
save_path: outputs/profile
tool: null
tool_config:
npu:
_target_: verl.utils.profiler.config.NPUToolConfig
analysis: true
contents: []
discrete: false
level: level0
nsys:
_target_: verl.utils.profiler.config.NsightToolConfig
discrete: false
torch:
_target_: verl.utils.profiler.config.TorchProfilerToolConfig
step_end: null
step_start: 0
torch_memory:
_target_: verl.utils.profiler.config.TorchMemoryToolConfig
stack_depth: 32
trace_alloc_max_entries: 100000
rollout_n: 8
shuffle: false
strategy: fsdp
ulysses_sequence_parallel_size: 1
use_dynamic_bsz: false
custom_reward_function:
value:
name: compute_score
path: /workspace/SIPO/verl/utils/reward_score/feedback/__init__.py
data:
value:
apply_chat_template_kwargs:
enable_thinking: false
custom_cls:
name: null
path: null
datagen:
name: null
path: null
dataloader_num_workers: 8
filter_overlong_prompts: true
filter_overlong_prompts_workers: 1
image_key: images
image_patch_size: 14
max_prompt_length: 2048
max_response_length: 8192
prompt_key: prompt
return_full_prompt: false
return_multi_modal_inputs: true
return_raw_chat: true
return_raw_input_ids: false
reward_fn_key: data_source
sampler:
class_name: null
class_path: null
seed: null
shuffle: true
tokenizer: null
tool_config_path: null
train_batch_size: 64
train_files:
- /workspace/SIPO/datasets/tooluse/train.parquet
train_max_samples: 6400
truncation: error
trust_remote_code: true
use_shm: false
val_batch_size: null
val_files:
- /workspace/SIPO/datasets/tooluse/test.parquet
val_max_samples: -1
validation_shuffle: false
video_key: videos
global_profiler:
value:
_target_: verl.utils.profiler.ProfilerConfig
global_tool_config:
nsys:
_target_: verl.utils.profiler.config.NsightToolConfig
controller_nsight_options:
cuda-graph-trace: graph
cuda-memory-usage: "true"
trace: cuda,nvtx,cublas,ucx
discrete: false
worker_nsight_options:
capture-range: cudaProfilerApi
capture-range-end: null
cuda-graph-trace: graph
cuda-memory-usage: "true"
kill: none
trace: cuda,nvtx,cublas,ucx
torch_memory:
context: all
stack_depth: 32
stacks: all
trace_alloc_max_entries: 100000
profile_continuous_steps: false
save_path: outputs/profile
steps: null
tool: null
max_model_len:
value: 10240
ray_kwargs:
value:
ray_init:
_temp_dir: /tmp/ray_q3g_tooluse_srpo_Qwen3_4B
include_dashboard: false
num_cpus: null
timeline_json_file: null
reward_manager:
value:
_target_: verl.trainer.config.config.RewardManagerConfig
module:
_target_: verl.trainer.config.config.ModuleConfig
name: custom_reward_manager
path: null
name: naive
source: register
reward_model:
value:
enable: false
enable_resource_pool: false
forward_max_token_len_per_gpu: 32768
launch_reward_fn_async: false
max_length: null
micro_batch_size: null
micro_batch_size_per_gpu: null
model:
external_lib: null
fsdp_config:
_target_: verl.workers.config.FSDPEngineConfig
forward_prefetch: false
fsdp_size: -1
param_offload: false
reshard_after_forward: true
wrap_policy:
min_num_params: 0
input_tokenizer: Qwen/Qwen3-4B
path: ~/models/FsfairX-LLaMA3-RM-v0.1
trust_remote_code: false
use_fused_kernels: false
use_remove_padding: false
use_shm: false
n_gpus_per_node: 8
nnodes: 0
num_workers: 1
profiler:
_target_: verl.utils.profiler.ProfilerConfig
all_ranks: false
enable: false
ranks: []
save_path: outputs/profile
tool: null
tool_config:
npu:
_target_: verl.utils.profiler.config.NPUToolConfig
analysis: true
contents: []
discrete: false
level: level0
nsys:
_target_: verl.utils.profiler.config.NsightToolConfig
discrete: false
torch:
_target_: verl.utils.profiler.config.TorchProfilerToolConfig
step_end: null
step_start: 0
torch_memory:
_target_: verl.utils.profiler.config.TorchMemoryToolConfig
stack_depth: 32
trace_alloc_max_entries: 100000
reward_loop_class_name: null
reward_loop_module_path: null
reward_loop_source: register
reward_manager: naive
rollout:
_target_: verl.workers.config.RolloutConfig
cudagraph_capture_sizes: null
data_parallel_size: 1
disable_log_stats: true
dtype: bfloat16
enable_chunked_prefill: true
enable_prefix_caching: true
enforce_eager: true
expert_parallel_size: 1
free_cache_engine: true
gpu_memory_utilization: 0.5
limit_images: null
load_format: auto
max_model_len: null
max_num_batched_tokens: 8192
max_num_seqs: 1024
name: ???
prompt_length: 2048
response_length: 2048
skip_tokenizer_init: false
tensor_model_parallel_size: 2
sandbox_fusion:
max_concurrent: 64
memory_limit_mb: 1024
url: null
strategy: fsdp
ulysses_sequence_parallel_size: 1
use_dynamic_bsz: false
use_reward_loop: false
trainer:
value:
balance_batch: true
critic_warmup: 0
default_hdfs_dir: null
default_local_dir: /workspace/L2T/checkpoints/datasets/tooluse/qwen3gen-tooluse-SRPO-Qwen-Qwen3-4B-mbs32-ema0.05-dwtrue-train64-rollout8-lr5e-6-vllm0.8
del_local_ckpt_after_load: false
device: cuda
esi_redundant_time: 0
experiment_name: qwen3gen-tooluse-SRPO-Qwen-Qwen3-4B-mbs32-ema0.05-dwtrue-train64-rollout8-lr5e-6-vllm0.8
group_name: QWEN3-SRPO-GRPO-matched-generalization
log_val_generations: 0
logger:
- console
- wandb
max_actor_ckpt_to_keep: 1
max_critic_ckpt_to_keep: null
n_gpus_per_node: 8
nnodes: 1
project_name: SDPO-root
ray_wait_register_center_timeout: 300
resume_from_path: null
resume_mode: auto
rollout_data_dir: null
save_freq: 10
test_freq: 10
total_epochs: 30
total_training_steps: 100
use_legacy_worker_impl: auto
val_before_train: false
val_only: false
validation_data_dir: null
transfer_queue:
value:
enable: false
vars:
value:
ckpt_dir: /capstor/scratch/cscs/root/ttrl_runs/datasets/tooluse
dir: /users/root/SDPO
log_dir: /users/root/output
task: datasets/tooluse

1
artifacts/wandb-summary.json Normal file
View File

File diff suppressed because one or more lines are too long

89
chat_template.jinja Normal file
View File

@@ -0,0 +1,89 @@
{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0].role == 'system' %}
{{- messages[0].content + '\n\n' }}
{%- endif %}
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0].role == 'system' %}
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
{%- set index = (messages|length - 1) - loop.index0 %}
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
{%- set ns.multi_step_tool = false %}
{%- set ns.last_query_index = index %}
{%- endif %}
{%- endfor %}
{%- for message in messages %}
{%- if message.content is string %}
{%- set content = message.content %}
{%- else %}
{%- set content = '' %}
{%- endif %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{%- set reasoning_content = '' %}
{%- if message.reasoning_content is string %}
{%- set reasoning_content = message.reasoning_content %}
{%- else %}
{%- if '</think>' in content %}
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
{%- endif %}
{%- endif %}
{%- if loop.index0 > ns.last_query_index %}
{%- if loop.last or (not loop.last and reasoning_content) %}
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- if message.tool_calls %}
{%- for tool_call in message.tool_calls %}
{%- if (loop.first and content) or (not loop.first) %}
{{- '\n' }}
{%- endif %}
{%- if tool_call.function %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{%- if tool_call.arguments is string %}
{{- tool_call.arguments }}
{%- else %}
{{- tool_call.arguments | tojson }}
{%- endif %}
{{- '}\n</tool_call>' }}
{%- endfor %}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- if enable_thinking is defined and enable_thinking is false %}
{{- '<think>\n\n</think>\n\n' }}
{%- endif %}
{%- endif %}

68
config.json Normal file
View File

@@ -0,0 +1,68 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"dtype": "bfloat16",
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 2560,
"initializer_range": 0.02,
"intermediate_size": 9728,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 40960,
"max_window_layers": 36,
"model_type": "qwen3",
"num_attention_heads": 32,
"num_hidden_layers": 36,
"num_key_value_heads": 8,
"pad_token_id": 151643,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "4.57.1",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

13
generation_config.json Normal file
View File

@@ -0,0 +1,13 @@
{
"bos_token_id": 151643,
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151643,
"temperature": 0.6,
"top_k": 20,
"top_p": 0.95,
"transformers_version": "4.57.1"
}

151388
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model-00001-of-00002.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:575c2d0ea25f6df107cfe207b39ce5701aab116b87fd09af89d4d4eb91e39d69
size 4971786144

3
model-00002-of-00002.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:de7381c25ad04f10a221bb654a3be1700771ba17442d32a2f89d03dcb01d2645
size 3851108296

407
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,407 @@
{
"metadata": {
"total_parameters": 4411424256,
"total_size": 8822848512
},
"weight_map": {
"lm_head.weight": "model-00001-of-00002.safetensors",
"model.embed_tokens.weight": "model-00001-of-00002.safetensors",
"model.layers.0.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.0.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.0.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.1.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.1.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.10.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.11.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.13.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.13.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.15.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.16.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.18.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.19.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.2.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.2.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.2.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.21.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.21.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.22.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.22.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.22.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.22.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.23.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.23.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.24.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.24.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.24.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.25.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.25.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.25.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.25.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.25.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.26.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.26.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.26.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.26.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.27.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.27.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.27.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.27.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.28.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.28.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.28.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.28.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.28.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.28.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.28.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.28.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.28.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.28.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.28.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.29.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.29.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.29.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.29.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.29.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.29.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.29.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.29.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.29.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.29.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.29.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.3.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.30.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.30.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.30.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.30.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.30.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.30.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.30.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.30.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.30.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.30.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.30.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.31.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.31.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.31.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.31.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.31.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.31.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.31.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.31.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.31.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.31.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.31.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.32.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.32.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.32.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.32.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.32.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.32.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.32.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.32.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.32.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.32.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.32.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.33.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.33.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.33.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.33.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.33.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.33.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.33.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.33.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.33.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.33.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.33.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.34.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.34.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.34.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.34.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.34.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.34.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.34.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.34.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.34.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.34.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.34.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.35.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.35.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.35.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.35.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.35.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.35.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.35.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.35.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.35.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.35.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.35.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.4.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.4.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.4.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.5.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.6.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.7.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.8.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.8.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.9.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.9.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.norm.weight": "model-00002-of-00002.safetensors"
}
}

3
results/training_score.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d56b97d485bd24953861da024015c467e9b6b7db7262dc91ccabdded453db12b
size 114330

91
results/training_scores.csv Normal file
View File

@@ -0,0 +1,91 @@
step,critic_score_mean
1,0.259765625
2,0.328125
3,0.421875
4,0.326171875
5,0.31640625
6,0.37109375
7,0.318359375
8,0.4453125
9,0.4140625
11,0.353515625
12,0.357421875
13,0.330078125
14,0.3828125
15,0.296875
16,0.279296875
17,0.384765625
18,0.4140625
19,0.2578125
21,0.353515625
22,0.314453125
23,0.380859375
24,0.359375
25,0.265625
26,0.41015625
27,0.400390625
28,0.34765625
29,0.369140625
31,0.400390625
32,0.302734375
33,0.203125
34,0.318359375
35,0.384765625
36,0.359375
37,0.271484375
38,0.337890625
39,0.38671875
41,0.40625
42,0.23828125
43,0.31640625
44,0.263671875
45,0.390625
46,0.392578125
47,0.435546875
48,0.326171875
49,0.30078125
51,0.5859375
52,0.337890625
53,0.384765625
54,0.30859375
55,0.474609375
56,0.318359375
57,0.361328125
58,0.33984375
59,0.27734375
61,0.384765625
62,0.33203125
63,0.373046875
64,0.416015625
65,0.474609375
66,0.47265625
67,0.373046875
68,0.228515625
69,0.501953125
71,0.328125
72,0.3828125
73,0.30859375
74,0.310546875
75,0.353515625
76,0.416015625
77,0.470703125
78,0.3984375
79,0.3828125
81,0.31640625
82,0.341796875
83,0.419921875
84,0.369140625
85,0.3046875
86,0.3046875
87,0.375
88,0.455078125
89,0.318359375
91,0.361328125
92,0.330078125
93,0.4375
94,0.34765625
95,0.314453125
96,0.328125
97,0.298828125
98,0.421875
99,0.431640625
1 step critic_score_mean
2 1 0.259765625
3 2 0.328125
4 3 0.421875
5 4 0.326171875
6 5 0.31640625
7 6 0.37109375
8 7 0.318359375
9 8 0.4453125
10 9 0.4140625
11 11 0.353515625
12 12 0.357421875
13 13 0.330078125
14 14 0.3828125
15 15 0.296875
16 16 0.279296875
17 17 0.384765625
18 18 0.4140625
19 19 0.2578125
20 21 0.353515625
21 22 0.314453125
22 23 0.380859375
23 24 0.359375
24 25 0.265625
25 26 0.41015625
26 27 0.400390625
27 28 0.34765625
28 29 0.369140625
29 31 0.400390625
30 32 0.302734375
31 33 0.203125
32 34 0.318359375
33 35 0.384765625
34 36 0.359375
35 37 0.271484375
36 38 0.337890625
37 39 0.38671875
38 41 0.40625
39 42 0.23828125
40 43 0.31640625
41 44 0.263671875
42 45 0.390625
43 46 0.392578125
44 47 0.435546875
45 48 0.326171875
46 49 0.30078125
47 51 0.5859375
48 52 0.337890625
49 53 0.384765625
50 54 0.30859375
51 55 0.474609375
52 56 0.318359375
53 57 0.361328125
54 58 0.33984375
55 59 0.27734375
56 61 0.384765625
57 62 0.33203125
58 63 0.373046875
59 64 0.416015625
60 65 0.474609375
61 66 0.47265625
62 67 0.373046875
63 68 0.228515625
64 69 0.501953125
65 71 0.328125
66 72 0.3828125
67 73 0.30859375
68 74 0.310546875
69 75 0.353515625
70 76 0.416015625
71 77 0.470703125
72 78 0.3984375
73 79 0.3828125
74 81 0.31640625
75 82 0.341796875
76 83 0.419921875
77 84 0.369140625
78 85 0.3046875
79 86 0.3046875
80 87 0.375
81 88 0.455078125
82 89 0.318359375
83 91 0.361328125
84 92 0.330078125
85 93 0.4375
86 94 0.34765625
87 95 0.314453125
88 96 0.328125
89 97 0.298828125
90 98 0.421875
91 99 0.431640625

11
results/validation_mean16.csv Normal file
View File

@@ -0,0 +1,11 @@
step,val_mean16
10,0.597426470588
20,0.567095588235
30,0.566176470588
40,0.625919117647
50,0.611213235294
60,0.606617647059
70,0.605698529412
80,0.577205882353
90,0.576286764706
100,0.572610294118
1 step val_mean16
2 10 0.597426470588
3 20 0.567095588235
4 30 0.566176470588
5 40 0.625919117647
6 50 0.611213235294
7 60 0.606617647059
8 70 0.605698529412
9 80 0.577205882353
10 90 0.576286764706
11 100 0.572610294118

31
special_tokens_map.json Normal file
View File

@@ -0,0 +1,31 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"eos_token": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

BIN
tokenizer.json (Stored with Git LFS) Normal file
View File

Binary file not shown.

239
tokenizer_config.json Normal file
View File

@@ -0,0 +1,239 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "<|object_ref_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<|object_ref_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151648": {
"content": "<|box_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151649": {
"content": "<|box_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151650": {
"content": "<|quad_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151651": {
"content": "<|quad_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151652": {
"content": "<|vision_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151653": {
"content": "<|vision_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151654": {
"content": "<|vision_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151655": {
"content": "<|image_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151656": {
"content": "<|video_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151659": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151660": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151661": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151662": {
"content": "<|fim_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151663": {
"content": "<|repo_name|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151664": {
"content": "<|file_sep|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151665": {
"content": "<tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151666": {
"content": "</tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151667": {
"content": "<think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151668": {
"content": "</think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"bos_token": null,
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"extra_special_tokens": {},
"model_max_length": 131072,
"pad_token": "<|endoftext|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long