初始化项目，由ModelHub XC社区提供模型
Model: SeongryongJung/Qwen3-8B-Physics-GRPO-TR Source: Original Platform
This commit is contained in:
ModelHub XC
38
.gitattributes
vendored
Normal file
38
.gitattributes
vendored
Normal file
@@ -0,0 +1,38 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
results/training_score.png filter=lfs diff=lfs merge=lfs -text
|
||||||
|
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
|
checkpoints/last/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
110
README.md
Normal file
110
README.md
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
---
|
||||||
|
license: apache-2.0
|
||||||
|
library_name: transformers
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
tags:
|
||||||
|
- qwen3
|
||||||
|
- reinforcement-learning
|
||||||
|
- grpo
|
||||||
|
- text-generation
|
||||||
|
base_model: Qwen/Qwen3-8B
|
||||||
|
---
|
||||||
|
|
||||||
|
# Qwen3-8B-Physics-GRPO-TR
|
||||||
|
|
||||||
|
This repository contains the Qwen3-8B `physics` `GRPO` batch-size-32 run. The repository name uses the project `GRPO-TR` naming convention, but the actual training method for this checkpoint is GRPO.
|
||||||
|
|
||||||
|
The repository root contains the best validation checkpoint, selected by validation `mean@16`. `checkpoints/last/` contains the final checkpoint.
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
| Dataset | Method | Base model | Train batch size | Best val mean@16 | Best checkpoint | Final val mean@16 | Final checkpoint |
|
||||||
|
|---|---|---|---:|---:|---:|---:|---:|
|
||||||
|
| Physics / SciKnowEval physics | GRPO | Qwen3-8B | 32 | 72.97% | 100 | 72.97% | 100 |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Validation Mean@16
|
||||||
|
|
||||||
|
| step | val_mean16 | percent |
|
||||||
|
|---:|---:|---:|
|
||||||
|
| 10 | 0.583593750000 | 58.36% |
|
||||||
|
| 20 | 0.599218750000 | 59.92% |
|
||||||
|
| 30 | 0.604687500000 | 60.47% |
|
||||||
|
| 40 | 0.621093750000 | 62.11% |
|
||||||
|
| 50 | 0.653906250000 | 65.39% |
|
||||||
|
| 60 | 0.671875000000 | 67.19% |
|
||||||
|
| 70 | 0.681250000000 | 68.12% |
|
||||||
|
| 80 | 0.712500000000 | 71.25% |
|
||||||
|
| 90 | 0.722656250000 | 72.27% |
|
||||||
|
| 100 | 0.729687500000 | 72.97% |
|
||||||
|
|
||||||
|
## Detailed Training Hyperparameters
|
||||||
|
|
||||||
|
| Section | Parameter | Value | Source |
|
||||||
|
|---|---|---:|---|
|
||||||
|
| Run identity | `Base model` | `Qwen/Qwen3-8B` | queue/script override |
|
||||||
|
| Run identity | `Dataset` | `Physics / SciKnowEval physics` | run_qwen3_generalization.sh |
|
||||||
|
| Run identity | `Method` | `GRPO` | run_qwen3_generalization.sh |
|
||||||
|
| Run identity | `Config` | `baseline_grpo` | run_qwen3_generalization.sh |
|
||||||
|
| Run identity | `Experiment` | `qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8` | run_qwen3_generalization.sh |
|
||||||
|
| Run identity | `W&B run` | `run-20260703_032040-6ig3l55l` | wandb |
|
||||||
|
| Data | `Train file` | `datasets/sciknoweval/physics/train.parquet` | script override |
|
||||||
|
| Data | `Validation file` | `datasets/sciknoweval/physics/test.parquet` | script override |
|
||||||
|
| Data | `Train batch size` | `32` | queue/script override |
|
||||||
|
| Data | `Train max samples` | `3200` | queue/script override |
|
||||||
|
| Schedule | `Total training steps` | `100` | queue/script override |
|
||||||
|
| Schedule | `Validation before train` | `False` | queue/script override |
|
||||||
|
| Schedule | `Save frequency` | `10` | queue/script override |
|
||||||
|
| Schedule | `Validation frequency` | `10` | queue/script override |
|
||||||
|
| Sequence | `Max prompt length` | `2048` | queue/script override |
|
||||||
|
| Sequence | `Max response length` | `8192` | queue/script override |
|
||||||
|
| Sequence | `Max model length` | `10240` | queue/script override |
|
||||||
|
| Rollout | `Train rollout n` | `8` | queue/script override |
|
||||||
|
| Rollout | `Validation rollout n` | `16` | queue/script override |
|
||||||
|
| Rollout | `vLLM GPU memory utilization` | `0.8` | queue/script override |
|
||||||
|
| Optimization | `Learning rate` | `1e-6` | GRPO method override |
|
||||||
|
| Optimization | `Weight decay` | `0.01` | script override |
|
||||||
|
| PPO/GRPO | `PPO mini batch size` | `8` | queue/script override |
|
||||||
|
| PPO/GRPO | `Normalize GRPO advantages by std` | `False` | baseline_grpo.yaml / script override |
|
||||||
|
| Rollout correction | `Importance sampling mode` | `token` | script override |
|
||||||
|
| Rollout correction | `IS threshold` | `2.0` | script override |
|
||||||
|
| Checkpoint/Logging | `Checkpoint root` | `checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8` | script override |
|
||||||
|
| Checkpoint/Logging | `Latest checkpointed iteration` | `100` | latest_checkpointed_iteration.txt |
|
||||||
|
| Checkpoint/Logging | `External actor archive` | `checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/_actor_archive` | preserve_actor_checkpoints.py |
|
||||||
|
| Checkpoint/Logging | `Logger` | `console, wandb` | ppo_trainer.yaml |
|
||||||
|
| PPO/GRPO | `Policy loss mode` | `vanilla` | method override |
|
||||||
|
| PPO/GRPO | `Actor KL loss coef` | `0.0` | method override |
|
||||||
|
|
||||||
|
Raw result and artifact files:
|
||||||
|
|
||||||
|
- `results/validation_mean16.csv`
|
||||||
|
- `results/training_scores.csv`
|
||||||
|
- `results/hyperparameters.csv`
|
||||||
|
- `results/training_score.png`
|
||||||
|
- `results/training_score.svg`
|
||||||
|
- `artifacts/output.log`
|
||||||
|
- `artifacts/queue.log`
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||||
|
|
||||||
|
repo_id = "SeongryongJung/Qwen3-8B-Physics-GRPO-TR"
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
repo_id,
|
||||||
|
torch_dtype="auto",
|
||||||
|
device_map="auto",
|
||||||
|
trust_remote_code=True,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Source
|
||||||
|
|
||||||
|
- Checkpoint: `checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8`
|
||||||
|
- Root actor checkpoint: `checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_100/actor`
|
||||||
|
- Last actor checkpoint: `checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_100/actor`
|
||||||
|
- W&B run: `run-20260703_032040-6ig3l55l`
|
||||||
|
- Queue log: `artifacts/queue.log`
|
||||||
28
added_tokens.json
Normal file
28
added_tokens.json
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
{
|
||||||
|
"</think>": 151668,
|
||||||
|
"</tool_call>": 151658,
|
||||||
|
"</tool_response>": 151666,
|
||||||
|
"<think>": 151667,
|
||||||
|
"<tool_call>": 151657,
|
||||||
|
"<tool_response>": 151665,
|
||||||
|
"<|box_end|>": 151649,
|
||||||
|
"<|box_start|>": 151648,
|
||||||
|
"<|endoftext|>": 151643,
|
||||||
|
"<|file_sep|>": 151664,
|
||||||
|
"<|fim_middle|>": 151660,
|
||||||
|
"<|fim_pad|>": 151662,
|
||||||
|
"<|fim_prefix|>": 151659,
|
||||||
|
"<|fim_suffix|>": 151661,
|
||||||
|
"<|im_end|>": 151645,
|
||||||
|
"<|im_start|>": 151644,
|
||||||
|
"<|image_pad|>": 151655,
|
||||||
|
"<|object_ref_end|>": 151647,
|
||||||
|
"<|object_ref_start|>": 151646,
|
||||||
|
"<|quad_end|>": 151651,
|
||||||
|
"<|quad_start|>": 151650,
|
||||||
|
"<|repo_name|>": 151663,
|
||||||
|
"<|video_pad|>": 151656,
|
||||||
|
"<|vision_end|>": 151653,
|
||||||
|
"<|vision_pad|>": 151654,
|
||||||
|
"<|vision_start|>": 151652
|
||||||
|
}
|
||||||
855
artifacts/config.yaml
Normal file
855
artifacts/config.yaml
Normal file
@@ -0,0 +1,855 @@
|
|||||||
|
_wandb:
|
||||||
|
value:
|
||||||
|
cli_version: 0.23.1
|
||||||
|
e:
|
||||||
|
1n1ucg63toheyodgdd2dcwkutcbcxhn8:
|
||||||
|
args:
|
||||||
|
- --node-ip-address=198.19.44.212
|
||||||
|
- --node-manager-port=36871
|
||||||
|
- --object-store-name=/tmp/ray_q3g_physics_grpo_Qwen3_8B/session_2026-07-03_03-18-52_070071_2279047/sockets/plasma_store
|
||||||
|
- --raylet-name=/tmp/ray_q3g_physics_grpo_Qwen3_8B/session_2026-07-03_03-18-52_070071_2279047/sockets/raylet
|
||||||
|
- --redis-address=None
|
||||||
|
- --metrics-agent-port=56405
|
||||||
|
- --logging-rotate-bytes=536870912
|
||||||
|
- --logging-rotate-backup-count=5
|
||||||
|
- --runtime-env-agent-port=64277
|
||||||
|
- --gcs-address=198.19.44.212:64574
|
||||||
|
- --session-name=session_2026-07-03_03-18-52_070071_2279047
|
||||||
|
- --temp-dir=/tmp/ray_q3g_physics_grpo_Qwen3_8B
|
||||||
|
- --webui=
|
||||||
|
- --cluster-id=d7576956015edcd024cf2ca706acce8be3fb6519bb7b0c87bfccae89
|
||||||
|
- --startup-token=32
|
||||||
|
- --worker-launch-time-ms=1783048735496
|
||||||
|
- --node-id=e88e44745be3f8c4449808648428760387c2aa53671532e80c9f13b3
|
||||||
|
- --runtime-env-hash=1652680156
|
||||||
|
codePath: .venv/lib/python3.12/site-packages/ray/_private/workers/default_worker.py
|
||||||
|
codePathLocal: .venv/lib/python3.12/site-packages/ray/_private/workers/default_worker.py
|
||||||
|
cpu_count: 64
|
||||||
|
cpu_count_logical: 128
|
||||||
|
cudaVersion: "13.0"
|
||||||
|
disk:
|
||||||
|
/:
|
||||||
|
total: "46086056050688"
|
||||||
|
used: "25787742543872"
|
||||||
|
email: jungsr1116@cau.ac.kr
|
||||||
|
executable: /mnt/mole/SDPO/L2T/.venv/bin/python
|
||||||
|
git:
|
||||||
|
commit: ee6a667700697069b93db636aa937f970c1fb57a
|
||||||
|
remote: https://github.com/jungseongryong/L2T.git
|
||||||
|
gpu: NVIDIA H200
|
||||||
|
gpu_count: 8
|
||||||
|
gpu_nvidia:
|
||||||
|
- architecture: Hopper
|
||||||
|
cudaCores: 16896
|
||||||
|
memoryTotal: "150754820096"
|
||||||
|
name: NVIDIA H200
|
||||||
|
uuid: GPU-c0451621-891c-7169-976d-71d81958db34
|
||||||
|
- architecture: Hopper
|
||||||
|
cudaCores: 16896
|
||||||
|
memoryTotal: "150754820096"
|
||||||
|
name: NVIDIA H200
|
||||||
|
uuid: GPU-e2a29dfa-7dad-6cd7-fb05-79d0060f27f0
|
||||||
|
- architecture: Hopper
|
||||||
|
cudaCores: 16896
|
||||||
|
memoryTotal: "150754820096"
|
||||||
|
name: NVIDIA H200
|
||||||
|
uuid: GPU-d69bb741-f2a7-a375-7f56-5c0d09d797b2
|
||||||
|
- architecture: Hopper
|
||||||
|
cudaCores: 16896
|
||||||
|
memoryTotal: "150754820096"
|
||||||
|
name: NVIDIA H200
|
||||||
|
uuid: GPU-1bbbea1e-7541-3126-1bfc-51b43a67577e
|
||||||
|
- architecture: Hopper
|
||||||
|
cudaCores: 16896
|
||||||
|
memoryTotal: "150754820096"
|
||||||
|
name: NVIDIA H200
|
||||||
|
uuid: GPU-6e0055af-452d-3f46-ce21-0c71e347a608
|
||||||
|
- architecture: Hopper
|
||||||
|
cudaCores: 16896
|
||||||
|
memoryTotal: "150754820096"
|
||||||
|
name: NVIDIA H200
|
||||||
|
uuid: GPU-317fdb5b-b280-71d8-398e-a843ad897437
|
||||||
|
- architecture: Hopper
|
||||||
|
cudaCores: 16896
|
||||||
|
memoryTotal: "150754820096"
|
||||||
|
name: NVIDIA H200
|
||||||
|
uuid: GPU-f19d8161-d2dd-49a1-dcb8-5c341b5e4514
|
||||||
|
- architecture: Hopper
|
||||||
|
cudaCores: 16896
|
||||||
|
memoryTotal: "150754820096"
|
||||||
|
name: NVIDIA H200
|
||||||
|
uuid: GPU-961f5ebf-4bf8-6855-828b-c065af57b87c
|
||||||
|
host: mole-workspace-rw
|
||||||
|
memory:
|
||||||
|
total: "2163980390400"
|
||||||
|
os: Linux-6.8.0-71-generic-x86_64-with-glibc2.36
|
||||||
|
program: /mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/ray/_private/workers/default_worker.py
|
||||||
|
python: CPython 3.12.13
|
||||||
|
root: /mnt/mole/SDPO/L2T
|
||||||
|
startedAt: "2026-07-03T03:20:40.327312Z"
|
||||||
|
writerId: 1n1ucg63toheyodgdd2dcwkutcbcxhn8
|
||||||
|
m: []
|
||||||
|
python_version: 3.12.13
|
||||||
|
t:
|
||||||
|
"1":
|
||||||
|
- 1
|
||||||
|
- 11
|
||||||
|
- 30
|
||||||
|
- 41
|
||||||
|
- 49
|
||||||
|
- 50
|
||||||
|
- 51
|
||||||
|
- 71
|
||||||
|
- 95
|
||||||
|
- 98
|
||||||
|
- 105
|
||||||
|
"2":
|
||||||
|
- 1
|
||||||
|
- 11
|
||||||
|
- 30
|
||||||
|
- 41
|
||||||
|
- 49
|
||||||
|
- 50
|
||||||
|
- 51
|
||||||
|
- 71
|
||||||
|
- 95
|
||||||
|
- 98
|
||||||
|
- 105
|
||||||
|
"3":
|
||||||
|
- 2
|
||||||
|
- 13
|
||||||
|
- 16
|
||||||
|
- 61
|
||||||
|
"4": 3.12.13
|
||||||
|
"5": 0.23.1
|
||||||
|
"6": 4.57.1
|
||||||
|
"12": 0.23.1
|
||||||
|
"13": linux-x86_64
|
||||||
|
actor_rollout_ref:
|
||||||
|
value:
|
||||||
|
actor:
|
||||||
|
_target_: verl.workers.config.FSDPActorConfig
|
||||||
|
calculate_entropy: false
|
||||||
|
calculate_sum_pi_squared: false
|
||||||
|
checkpoint:
|
||||||
|
_target_: verl.trainer.config.CheckpointConfig
|
||||||
|
async_save: false
|
||||||
|
load_contents:
|
||||||
|
- model
|
||||||
|
- optimizer
|
||||||
|
- extra
|
||||||
|
save_contents:
|
||||||
|
- model
|
||||||
|
- optimizer
|
||||||
|
- extra
|
||||||
|
clip_ratio: 0.2
|
||||||
|
clip_ratio_c: 3
|
||||||
|
clip_ratio_high: 0.28
|
||||||
|
clip_ratio_low: 0.2
|
||||||
|
data_loader_seed: 42
|
||||||
|
entropy_checkpointing: false
|
||||||
|
entropy_coeff: 0
|
||||||
|
entropy_from_logits_with_chunking: false
|
||||||
|
freeze_vision_tower: false
|
||||||
|
fsdp_config:
|
||||||
|
_target_: verl.workers.config.FSDPEngineConfig
|
||||||
|
dtype: bfloat16
|
||||||
|
entropy_checkpointing: false
|
||||||
|
entropy_from_logits_with_chunking: false
|
||||||
|
forward_only: false
|
||||||
|
forward_prefetch: false
|
||||||
|
fsdp_size: -1
|
||||||
|
full_determinism: false
|
||||||
|
model_dtype: fp32
|
||||||
|
offload_policy: false
|
||||||
|
optimizer_offload: false
|
||||||
|
param_offload: false
|
||||||
|
reshard_after_forward: true
|
||||||
|
seed: 42
|
||||||
|
strategy: fsdp
|
||||||
|
ulysses_sequence_parallel_size: 1
|
||||||
|
use_orig_params: false
|
||||||
|
use_torch_compile: true
|
||||||
|
wrap_policy:
|
||||||
|
min_num_params: 0
|
||||||
|
grad_clip: 1
|
||||||
|
kl_loss_coef: 0
|
||||||
|
kl_loss_type: low_var_kl
|
||||||
|
loss_agg_mode: token-mean
|
||||||
|
loss_scale_factor: null
|
||||||
|
optim:
|
||||||
|
_target_: verl.workers.config.FSDPOptimizerConfig
|
||||||
|
betas:
|
||||||
|
- 0.9
|
||||||
|
- 0.999
|
||||||
|
clip_grad: 1
|
||||||
|
lr: 1e-06
|
||||||
|
lr_scheduler_type: constant
|
||||||
|
lr_warmup_steps: 10
|
||||||
|
lr_warmup_steps_ratio: 0
|
||||||
|
min_lr_ratio: 0
|
||||||
|
num_cycles: 0.5
|
||||||
|
optimizer: AdamW
|
||||||
|
optimizer_impl: torch.optim
|
||||||
|
override_optimizer_config: null
|
||||||
|
total_training_steps: 100
|
||||||
|
warmup_style: null
|
||||||
|
weight_decay: 0.01
|
||||||
|
policy_loss:
|
||||||
|
_target_: verl.workers.config.PolicyLossConfig
|
||||||
|
clip_cov_lb: 1
|
||||||
|
clip_cov_ratio: 0.0002
|
||||||
|
clip_cov_ub: 5
|
||||||
|
kl_cov_ratio: 0.0002
|
||||||
|
loss_mode: vanilla
|
||||||
|
ppo_kl_coef: 0.1
|
||||||
|
ppo_epochs: 1
|
||||||
|
ppo_max_token_len_per_gpu: 10240
|
||||||
|
ppo_micro_batch_size: null
|
||||||
|
ppo_micro_batch_size_per_gpu: 1
|
||||||
|
ppo_mini_batch_size: 8
|
||||||
|
profiler:
|
||||||
|
_target_: verl.utils.profiler.ProfilerConfig
|
||||||
|
all_ranks: false
|
||||||
|
enable: false
|
||||||
|
ranks: []
|
||||||
|
save_path: outputs/profile
|
||||||
|
tool: null
|
||||||
|
tool_config:
|
||||||
|
npu:
|
||||||
|
_target_: verl.utils.profiler.config.NPUToolConfig
|
||||||
|
analysis: true
|
||||||
|
contents: []
|
||||||
|
discrete: false
|
||||||
|
level: level0
|
||||||
|
nsys:
|
||||||
|
_target_: verl.utils.profiler.config.NsightToolConfig
|
||||||
|
discrete: false
|
||||||
|
torch:
|
||||||
|
_target_: verl.utils.profiler.config.TorchProfilerToolConfig
|
||||||
|
step_end: null
|
||||||
|
step_start: 0
|
||||||
|
torch_memory:
|
||||||
|
_target_: verl.utils.profiler.config.TorchMemoryToolConfig
|
||||||
|
stack_depth: 32
|
||||||
|
trace_alloc_max_entries: 100000
|
||||||
|
rollout_n: 8
|
||||||
|
router_replay:
|
||||||
|
_target_: verl.workers.config.RouterReplayConfig
|
||||||
|
mode: disabled
|
||||||
|
record_file: null
|
||||||
|
replay_file: null
|
||||||
|
self_distillation:
|
||||||
|
_target_: verl.workers.config.SelfDistillationConfig
|
||||||
|
alpha: 0.5
|
||||||
|
distillation_add_tail: true
|
||||||
|
distillation_topk: 100
|
||||||
|
dont_reprompt_on_self_success: true
|
||||||
|
environment_feedback_only_without_solution: true
|
||||||
|
evolving_teacher:
|
||||||
|
_target_: verl.workers.config.EvolvingTeacherConfig
|
||||||
|
enable: false
|
||||||
|
loss_weight: 0
|
||||||
|
mask: all
|
||||||
|
feedback_template: |4-
|
||||||
|
The following is feedback from your unsuccessful earlier attempt:
|
||||||
|
|
||||||
|
{feedback_raw}
|
||||||
|
full_logit_distillation: true
|
||||||
|
include_environment_feedback: true
|
||||||
|
is_clip: 2
|
||||||
|
max_reprompt_len: 22528
|
||||||
|
remove_thinking_from_demonstration: false
|
||||||
|
reprompt_template: |-
|
||||||
|
{prompt}{solution}{feedback}
|
||||||
|
|
||||||
|
Correctly solve the original question.
|
||||||
|
reprompt_truncation: right
|
||||||
|
solution_template: |4-
|
||||||
|
Correct solution:
|
||||||
|
|
||||||
|
{successful_previous_attempt}
|
||||||
|
srpo_dynamic_weighting: false
|
||||||
|
srpo_dynamic_weighting_temperature: 1
|
||||||
|
success_reward_threshold: 0.5
|
||||||
|
teacher_regularization: ema
|
||||||
|
teacher_update_rate: 0
|
||||||
|
token_reweight_decay_steps: null
|
||||||
|
token_reweight_eps_w: 0.2
|
||||||
|
token_reweight_lambda: 0.5
|
||||||
|
shuffle: false
|
||||||
|
strategy: fsdp
|
||||||
|
sum_pi_squared_checkpointing: false
|
||||||
|
tau_neg: 1.05
|
||||||
|
tau_pos: 1
|
||||||
|
ulysses_sequence_parallel_size: 1
|
||||||
|
use_dynamic_bsz: false
|
||||||
|
use_fused_kernels: false
|
||||||
|
use_kl_loss: false
|
||||||
|
use_prefix_grouper: false
|
||||||
|
use_remove_padding: true
|
||||||
|
use_torch_compile: true
|
||||||
|
hybrid_engine: true
|
||||||
|
model:
|
||||||
|
_target_: verl.workers.config.HFModelConfig
|
||||||
|
custom_chat_template: null
|
||||||
|
enable_activation_offload: false
|
||||||
|
enable_gradient_checkpointing: true
|
||||||
|
exclude_modules: null
|
||||||
|
external_lib: null
|
||||||
|
fused_kernel_options:
|
||||||
|
impl_backend: torch
|
||||||
|
hf_config_path: null
|
||||||
|
lora_adapter_path: null
|
||||||
|
lora_alpha: 16
|
||||||
|
lora_rank: 0
|
||||||
|
path: Qwen/Qwen3-8B
|
||||||
|
target_modules: all-linear
|
||||||
|
tiled_mlp:
|
||||||
|
enabled: false
|
||||||
|
num_shards: 4
|
||||||
|
tokenizer_path: null
|
||||||
|
trust_remote_code: true
|
||||||
|
use_fused_kernels: false
|
||||||
|
use_liger: false
|
||||||
|
use_remove_padding: true
|
||||||
|
use_shm: false
|
||||||
|
nccl_timeout: 600
|
||||||
|
ref:
|
||||||
|
_target_: verl.workers.config.FSDPActorConfig
|
||||||
|
entropy_checkpointing: false
|
||||||
|
entropy_from_logits_with_chunking: false
|
||||||
|
fsdp_config:
|
||||||
|
_target_: verl.workers.config.FSDPEngineConfig
|
||||||
|
dtype: bfloat16
|
||||||
|
entropy_checkpointing: false
|
||||||
|
entropy_from_logits_with_chunking: false
|
||||||
|
forward_only: true
|
||||||
|
forward_prefetch: false
|
||||||
|
fsdp_size: -1
|
||||||
|
full_determinism: false
|
||||||
|
model_dtype: fp32
|
||||||
|
offload_policy: false
|
||||||
|
optimizer_offload: false
|
||||||
|
param_offload: false
|
||||||
|
reshard_after_forward: true
|
||||||
|
seed: 42
|
||||||
|
strategy: fsdp
|
||||||
|
ulysses_sequence_parallel_size: 1
|
||||||
|
use_orig_params: false
|
||||||
|
use_torch_compile: true
|
||||||
|
wrap_policy:
|
||||||
|
min_num_params: 0
|
||||||
|
log_prob_max_token_len_per_gpu: 10240
|
||||||
|
log_prob_micro_batch_size: null
|
||||||
|
log_prob_micro_batch_size_per_gpu: 1
|
||||||
|
log_prob_use_dynamic_bsz: false
|
||||||
|
profiler:
|
||||||
|
_target_: verl.utils.profiler.ProfilerConfig
|
||||||
|
all_ranks: false
|
||||||
|
enable: false
|
||||||
|
ranks: []
|
||||||
|
save_path: outputs/profile
|
||||||
|
tool: null
|
||||||
|
tool_config:
|
||||||
|
npu:
|
||||||
|
_target_: verl.utils.profiler.config.NPUToolConfig
|
||||||
|
analysis: true
|
||||||
|
contents: []
|
||||||
|
discrete: false
|
||||||
|
level: level0
|
||||||
|
nsys:
|
||||||
|
_target_: verl.utils.profiler.config.NsightToolConfig
|
||||||
|
discrete: false
|
||||||
|
torch:
|
||||||
|
_target_: verl.utils.profiler.config.TorchProfilerToolConfig
|
||||||
|
step_end: null
|
||||||
|
step_start: 0
|
||||||
|
torch_memory:
|
||||||
|
_target_: verl.utils.profiler.config.TorchMemoryToolConfig
|
||||||
|
stack_depth: 32
|
||||||
|
trace_alloc_max_entries: 100000
|
||||||
|
rollout_n: 8
|
||||||
|
router_replay:
|
||||||
|
_target_: verl.workers.config.RouterReplayConfig
|
||||||
|
mode: disabled
|
||||||
|
record_file: null
|
||||||
|
replay_file: null
|
||||||
|
strategy: fsdp
|
||||||
|
ulysses_sequence_parallel_size: 1
|
||||||
|
use_torch_compile: true
|
||||||
|
rollout:
|
||||||
|
_target_: verl.workers.config.RolloutConfig
|
||||||
|
agent:
|
||||||
|
_target_: verl.workers.config.AgentLoopConfig
|
||||||
|
agent_loop_config_path: null
|
||||||
|
custom_async_server:
|
||||||
|
_target_: verl.workers.config.CustomAsyncServerConfig
|
||||||
|
name: null
|
||||||
|
path: null
|
||||||
|
default_agent_loop: single_turn_agent
|
||||||
|
num_workers: 8
|
||||||
|
calculate_log_probs: true
|
||||||
|
cudagraph_capture_sizes: null
|
||||||
|
data_parallel_size: 1
|
||||||
|
disable_log_stats: true
|
||||||
|
do_sample: true
|
||||||
|
dtype: bfloat16
|
||||||
|
enable_chunked_prefill: true
|
||||||
|
enable_prefix_caching: true
|
||||||
|
enable_rollout_routing_replay: false
|
||||||
|
enforce_eager: false
|
||||||
|
expert_parallel_size: 1
|
||||||
|
free_cache_engine: true
|
||||||
|
gpu_memory_utilization: 0.8
|
||||||
|
ignore_eos: false
|
||||||
|
layered_summon: false
|
||||||
|
load_format: dummy
|
||||||
|
log_prob_max_token_len_per_gpu: 10240
|
||||||
|
log_prob_micro_batch_size: null
|
||||||
|
log_prob_micro_batch_size_per_gpu: 1
|
||||||
|
log_prob_use_dynamic_bsz: false
|
||||||
|
logprobs_mode: processed_logprobs
|
||||||
|
max_model_len: 10240
|
||||||
|
max_num_batched_tokens: 10240
|
||||||
|
max_num_seqs: 1024
|
||||||
|
mode: async
|
||||||
|
multi_stage_wake_up: false
|
||||||
|
multi_turn:
|
||||||
|
_target_: verl.workers.config.MultiTurnConfig
|
||||||
|
enable: false
|
||||||
|
format: hermes
|
||||||
|
interaction_config_path: null
|
||||||
|
max_assistant_turns: null
|
||||||
|
max_parallel_calls: 1
|
||||||
|
max_tool_response_length: 256
|
||||||
|
max_user_turns: null
|
||||||
|
num_repeat_rollouts: null
|
||||||
|
tokenization_sanity_check_mode: strict
|
||||||
|
tool_config_path: null
|
||||||
|
tool_response_truncate_side: middle
|
||||||
|
use_inference_chat_template: false
|
||||||
|
"n": 8
|
||||||
|
name: vllm
|
||||||
|
over_sample_rate: 0
|
||||||
|
pipeline_model_parallel_size: 1
|
||||||
|
profiler:
|
||||||
|
_target_: verl.utils.profiler.ProfilerConfig
|
||||||
|
all_ranks: false
|
||||||
|
enable: false
|
||||||
|
ranks: []
|
||||||
|
save_path: outputs/profile
|
||||||
|
tool: null
|
||||||
|
tool_config:
|
||||||
|
npu:
|
||||||
|
_target_: verl.utils.profiler.config.NPUToolConfig
|
||||||
|
analysis: true
|
||||||
|
contents: []
|
||||||
|
discrete: false
|
||||||
|
level: level0
|
||||||
|
nsys:
|
||||||
|
_target_: verl.utils.profiler.config.NsightToolConfig
|
||||||
|
discrete: false
|
||||||
|
torch:
|
||||||
|
_target_: verl.utils.profiler.config.TorchProfilerToolConfig
|
||||||
|
step_end: null
|
||||||
|
step_start: 0
|
||||||
|
torch_memory:
|
||||||
|
_target_: verl.utils.profiler.config.TorchMemoryToolConfig
|
||||||
|
stack_depth: 32
|
||||||
|
trace_alloc_max_entries: 100000
|
||||||
|
prometheus:
|
||||||
|
_target_: verl.workers.config.PrometheusConfig
|
||||||
|
enable: false
|
||||||
|
file: /tmp/ray/session_latest/metrics/prometheus/prometheus.yml
|
||||||
|
port: 9090
|
||||||
|
served_model_name: Qwen/Qwen3-8B
|
||||||
|
prompt_length: 2048
|
||||||
|
quantization: null
|
||||||
|
quantization_config_file: null
|
||||||
|
response_length: 8192
|
||||||
|
scheduling_policy: fcfs
|
||||||
|
skip_dump_dir: /tmp/rollout_dump
|
||||||
|
skip_rollout: false
|
||||||
|
skip_tokenizer_init: true
|
||||||
|
temperature: 1
|
||||||
|
tensor_model_parallel_size: 2
|
||||||
|
top_k: -1
|
||||||
|
top_p: 1
|
||||||
|
trace:
|
||||||
|
_target_: verl.workers.config.TraceConfig
|
||||||
|
backend: null
|
||||||
|
max_samples_per_step_per_worker: null
|
||||||
|
token2text: false
|
||||||
|
update_weights_bucket_megabytes: 512
|
||||||
|
val_kwargs:
|
||||||
|
_target_: verl.workers.config.SamplingConfig
|
||||||
|
do_sample: true
|
||||||
|
"n": 16
|
||||||
|
temperature: 0.6
|
||||||
|
top_k: -1
|
||||||
|
top_p: 0.95
|
||||||
|
algorithm:
|
||||||
|
value:
|
||||||
|
_target_: verl.trainer.config.AlgoConfig
|
||||||
|
adv_estimator: grpo
|
||||||
|
gamma: 1
|
||||||
|
kl_ctrl:
|
||||||
|
_target_: verl.trainer.config.KLControlConfig
|
||||||
|
horizon: 10000
|
||||||
|
kl_coef: 0.001
|
||||||
|
target_kl: 0.1
|
||||||
|
type: fixed
|
||||||
|
kl_penalty: kl
|
||||||
|
lam: 1
|
||||||
|
norm_adv_by_std_in_grpo: false
|
||||||
|
pf_ppo:
|
||||||
|
reweight_method: pow
|
||||||
|
weight_pow: 2
|
||||||
|
rollout_correction:
|
||||||
|
bypass_mode: false
|
||||||
|
loss_type: ppo_clip
|
||||||
|
rollout_is: token
|
||||||
|
rollout_is_batch_normalize: false
|
||||||
|
rollout_is_threshold: 2
|
||||||
|
rollout_rs: null
|
||||||
|
rollout_rs_threshold: null
|
||||||
|
use_kl_in_reward: false
|
||||||
|
use_pf_ppo: false
|
||||||
|
critic:
|
||||||
|
value:
|
||||||
|
_target_: verl.workers.config.FSDPCriticConfig
|
||||||
|
checkpoint:
|
||||||
|
_target_: verl.trainer.config.CheckpointConfig
|
||||||
|
async_save: false
|
||||||
|
load_contents:
|
||||||
|
- model
|
||||||
|
- optimizer
|
||||||
|
- extra
|
||||||
|
save_contents:
|
||||||
|
- model
|
||||||
|
- optimizer
|
||||||
|
- extra
|
||||||
|
cliprange_value: 0.5
|
||||||
|
data_loader_seed: 42
|
||||||
|
enable: null
|
||||||
|
forward_max_token_len_per_gpu: 32768
|
||||||
|
forward_micro_batch_size: null
|
||||||
|
forward_micro_batch_size_per_gpu: null
|
||||||
|
grad_clip: 1
|
||||||
|
loss_agg_mode: token-mean
|
||||||
|
model:
|
||||||
|
_target_: verl.workers.config.FSDPCriticModelCfg
|
||||||
|
enable_activation_offload: false
|
||||||
|
enable_gradient_checkpointing: true
|
||||||
|
external_lib: null
|
||||||
|
fsdp_config:
|
||||||
|
_target_: verl.workers.config.FSDPEngineConfig
|
||||||
|
dtype: bfloat16
|
||||||
|
entropy_checkpointing: false
|
||||||
|
entropy_from_logits_with_chunking: false
|
||||||
|
forward_only: false
|
||||||
|
forward_prefetch: false
|
||||||
|
fsdp_size: -1
|
||||||
|
full_determinism: false
|
||||||
|
model_dtype: fp32
|
||||||
|
offload_policy: false
|
||||||
|
optimizer_offload: false
|
||||||
|
param_offload: false
|
||||||
|
reshard_after_forward: true
|
||||||
|
seed: 42
|
||||||
|
strategy: fsdp
|
||||||
|
ulysses_sequence_parallel_size: 1
|
||||||
|
use_orig_params: false
|
||||||
|
use_torch_compile: true
|
||||||
|
wrap_policy:
|
||||||
|
min_num_params: 0
|
||||||
|
lora_alpha: 16
|
||||||
|
lora_rank: 0
|
||||||
|
path: Qwen/Qwen3-8B
|
||||||
|
target_modules: all-linear
|
||||||
|
tiled_mlp:
|
||||||
|
enabled: false
|
||||||
|
num_shards: 4
|
||||||
|
tokenizer_path: Qwen/Qwen3-8B
|
||||||
|
trust_remote_code: true
|
||||||
|
use_remove_padding: false
|
||||||
|
use_shm: false
|
||||||
|
optim:
|
||||||
|
_target_: verl.workers.config.FSDPOptimizerConfig
|
||||||
|
betas:
|
||||||
|
- 0.9
|
||||||
|
- 0.999
|
||||||
|
clip_grad: 1
|
||||||
|
lr: 1e-05
|
||||||
|
lr_scheduler_type: constant
|
||||||
|
lr_warmup_steps: -1
|
||||||
|
lr_warmup_steps_ratio: 0
|
||||||
|
min_lr_ratio: 0
|
||||||
|
num_cycles: 0.5
|
||||||
|
optimizer: AdamW
|
||||||
|
optimizer_impl: torch.optim
|
||||||
|
override_optimizer_config: null
|
||||||
|
total_training_steps: 100
|
||||||
|
warmup_style: null
|
||||||
|
weight_decay: 0.01
|
||||||
|
ppo_epochs: 1
|
||||||
|
ppo_max_token_len_per_gpu: 32768
|
||||||
|
ppo_micro_batch_size: null
|
||||||
|
ppo_micro_batch_size_per_gpu: null
|
||||||
|
ppo_mini_batch_size: 8
|
||||||
|
profiler:
|
||||||
|
_target_: verl.utils.profiler.ProfilerConfig
|
||||||
|
all_ranks: false
|
||||||
|
enable: false
|
||||||
|
ranks: []
|
||||||
|
save_path: outputs/profile
|
||||||
|
tool: null
|
||||||
|
tool_config:
|
||||||
|
npu:
|
||||||
|
_target_: verl.utils.profiler.config.NPUToolConfig
|
||||||
|
analysis: true
|
||||||
|
contents: []
|
||||||
|
discrete: false
|
||||||
|
level: level0
|
||||||
|
nsys:
|
||||||
|
_target_: verl.utils.profiler.config.NsightToolConfig
|
||||||
|
discrete: false
|
||||||
|
torch:
|
||||||
|
_target_: verl.utils.profiler.config.TorchProfilerToolConfig
|
||||||
|
step_end: null
|
||||||
|
step_start: 0
|
||||||
|
torch_memory:
|
||||||
|
_target_: verl.utils.profiler.config.TorchMemoryToolConfig
|
||||||
|
stack_depth: 32
|
||||||
|
trace_alloc_max_entries: 100000
|
||||||
|
rollout_n: 8
|
||||||
|
shuffle: false
|
||||||
|
strategy: fsdp
|
||||||
|
ulysses_sequence_parallel_size: 1
|
||||||
|
use_dynamic_bsz: false
|
||||||
|
custom_reward_function:
|
||||||
|
value:
|
||||||
|
name: compute_score
|
||||||
|
path: /mnt/mole/SDPO/L2T/verl/utils/reward_score/feedback/__init__.py
|
||||||
|
data:
|
||||||
|
value:
|
||||||
|
apply_chat_template_kwargs:
|
||||||
|
enable_thinking: false
|
||||||
|
custom_cls:
|
||||||
|
name: null
|
||||||
|
path: null
|
||||||
|
datagen:
|
||||||
|
name: null
|
||||||
|
path: null
|
||||||
|
dataloader_num_workers: 8
|
||||||
|
filter_overlong_prompts: true
|
||||||
|
filter_overlong_prompts_workers: 1
|
||||||
|
image_key: images
|
||||||
|
image_patch_size: 14
|
||||||
|
max_prompt_length: 2048
|
||||||
|
max_response_length: 8192
|
||||||
|
prompt_key: prompt
|
||||||
|
return_full_prompt: false
|
||||||
|
return_multi_modal_inputs: true
|
||||||
|
return_raw_chat: true
|
||||||
|
return_raw_input_ids: false
|
||||||
|
reward_fn_key: data_source
|
||||||
|
sampler:
|
||||||
|
class_name: null
|
||||||
|
class_path: null
|
||||||
|
seed: null
|
||||||
|
shuffle: true
|
||||||
|
tokenizer: null
|
||||||
|
tool_config_path: null
|
||||||
|
train_batch_size: 32
|
||||||
|
train_files:
|
||||||
|
- /mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/train.parquet
|
||||||
|
train_max_samples: 3200
|
||||||
|
truncation: error
|
||||||
|
trust_remote_code: true
|
||||||
|
use_shm: false
|
||||||
|
val_batch_size: null
|
||||||
|
val_files:
|
||||||
|
- /mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/test.parquet
|
||||||
|
val_max_samples: -1
|
||||||
|
validation_shuffle: false
|
||||||
|
video_key: videos
|
||||||
|
global_profiler:
|
||||||
|
value:
|
||||||
|
_target_: verl.utils.profiler.ProfilerConfig
|
||||||
|
global_tool_config:
|
||||||
|
nsys:
|
||||||
|
_target_: verl.utils.profiler.config.NsightToolConfig
|
||||||
|
controller_nsight_options:
|
||||||
|
cuda-graph-trace: graph
|
||||||
|
cuda-memory-usage: "true"
|
||||||
|
trace: cuda,nvtx,cublas,ucx
|
||||||
|
discrete: false
|
||||||
|
worker_nsight_options:
|
||||||
|
capture-range: cudaProfilerApi
|
||||||
|
capture-range-end: null
|
||||||
|
cuda-graph-trace: graph
|
||||||
|
cuda-memory-usage: "true"
|
||||||
|
kill: none
|
||||||
|
trace: cuda,nvtx,cublas,ucx
|
||||||
|
torch_memory:
|
||||||
|
context: all
|
||||||
|
stack_depth: 32
|
||||||
|
stacks: all
|
||||||
|
trace_alloc_max_entries: 100000
|
||||||
|
profile_continuous_steps: false
|
||||||
|
save_path: outputs/profile
|
||||||
|
steps: null
|
||||||
|
tool: null
|
||||||
|
max_model_len:
|
||||||
|
value: 10240
|
||||||
|
ray_kwargs:
|
||||||
|
value:
|
||||||
|
ray_init:
|
||||||
|
_temp_dir: /tmp/ray_q3g_physics_grpo_Qwen3_8B
|
||||||
|
include_dashboard: false
|
||||||
|
num_cpus: null
|
||||||
|
runtime_env:
|
||||||
|
env_vars:
|
||||||
|
EXPERIMENT: qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8
|
||||||
|
TASK: datasets/sciknoweval/physics
|
||||||
|
USER: root
|
||||||
|
timeline_json_file: null
|
||||||
|
reward_manager:
|
||||||
|
value:
|
||||||
|
_target_: verl.trainer.config.config.RewardManagerConfig
|
||||||
|
module:
|
||||||
|
_target_: verl.trainer.config.config.ModuleConfig
|
||||||
|
name: custom_reward_manager
|
||||||
|
path: null
|
||||||
|
name: naive
|
||||||
|
source: register
|
||||||
|
reward_model:
|
||||||
|
value:
|
||||||
|
enable: false
|
||||||
|
enable_resource_pool: false
|
||||||
|
forward_max_token_len_per_gpu: 32768
|
||||||
|
launch_reward_fn_async: false
|
||||||
|
max_length: null
|
||||||
|
micro_batch_size: null
|
||||||
|
micro_batch_size_per_gpu: null
|
||||||
|
model:
|
||||||
|
external_lib: null
|
||||||
|
fsdp_config:
|
||||||
|
_target_: verl.workers.config.FSDPEngineConfig
|
||||||
|
forward_prefetch: false
|
||||||
|
fsdp_size: -1
|
||||||
|
param_offload: false
|
||||||
|
reshard_after_forward: true
|
||||||
|
wrap_policy:
|
||||||
|
min_num_params: 0
|
||||||
|
input_tokenizer: Qwen/Qwen3-8B
|
||||||
|
path: ~/models/FsfairX-LLaMA3-RM-v0.1
|
||||||
|
trust_remote_code: false
|
||||||
|
use_fused_kernels: false
|
||||||
|
use_remove_padding: false
|
||||||
|
use_shm: false
|
||||||
|
n_gpus_per_node: 8
|
||||||
|
nnodes: 0
|
||||||
|
num_workers: 1
|
||||||
|
profiler:
|
||||||
|
_target_: verl.utils.profiler.ProfilerConfig
|
||||||
|
all_ranks: false
|
||||||
|
enable: false
|
||||||
|
ranks: []
|
||||||
|
save_path: outputs/profile
|
||||||
|
tool: null
|
||||||
|
tool_config:
|
||||||
|
npu:
|
||||||
|
_target_: verl.utils.profiler.config.NPUToolConfig
|
||||||
|
analysis: true
|
||||||
|
contents: []
|
||||||
|
discrete: false
|
||||||
|
level: level0
|
||||||
|
nsys:
|
||||||
|
_target_: verl.utils.profiler.config.NsightToolConfig
|
||||||
|
discrete: false
|
||||||
|
torch:
|
||||||
|
_target_: verl.utils.profiler.config.TorchProfilerToolConfig
|
||||||
|
step_end: null
|
||||||
|
step_start: 0
|
||||||
|
torch_memory:
|
||||||
|
_target_: verl.utils.profiler.config.TorchMemoryToolConfig
|
||||||
|
stack_depth: 32
|
||||||
|
trace_alloc_max_entries: 100000
|
||||||
|
reward_loop_class_name: null
|
||||||
|
reward_loop_module_path: null
|
||||||
|
reward_loop_source: register
|
||||||
|
reward_manager: naive
|
||||||
|
rollout:
|
||||||
|
_target_: verl.workers.config.RolloutConfig
|
||||||
|
cudagraph_capture_sizes: null
|
||||||
|
data_parallel_size: 1
|
||||||
|
disable_log_stats: true
|
||||||
|
dtype: bfloat16
|
||||||
|
enable_chunked_prefill: true
|
||||||
|
enable_prefix_caching: true
|
||||||
|
enforce_eager: true
|
||||||
|
expert_parallel_size: 1
|
||||||
|
free_cache_engine: true
|
||||||
|
gpu_memory_utilization: 0.5
|
||||||
|
limit_images: null
|
||||||
|
load_format: auto
|
||||||
|
max_model_len: null
|
||||||
|
max_num_batched_tokens: 8192
|
||||||
|
max_num_seqs: 1024
|
||||||
|
name: ???
|
||||||
|
prompt_length: 2048
|
||||||
|
response_length: 2048
|
||||||
|
skip_tokenizer_init: false
|
||||||
|
tensor_model_parallel_size: 2
|
||||||
|
sandbox_fusion:
|
||||||
|
max_concurrent: 64
|
||||||
|
memory_limit_mb: 1024
|
||||||
|
url: null
|
||||||
|
strategy: fsdp
|
||||||
|
ulysses_sequence_parallel_size: 1
|
||||||
|
use_dynamic_bsz: false
|
||||||
|
use_reward_loop: false
|
||||||
|
trainer:
|
||||||
|
value:
|
||||||
|
balance_batch: true
|
||||||
|
critic_warmup: 0
|
||||||
|
default_hdfs_dir: null
|
||||||
|
default_local_dir: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8
|
||||||
|
del_local_ckpt_after_load: false
|
||||||
|
device: cuda
|
||||||
|
esi_redundant_time: 0
|
||||||
|
experiment_name: qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8
|
||||||
|
group_name: QWEN3-GRPO-generalization
|
||||||
|
log_val_generations: 0
|
||||||
|
logger:
|
||||||
|
- console
|
||||||
|
- wandb
|
||||||
|
max_actor_ckpt_to_keep: 100
|
||||||
|
max_critic_ckpt_to_keep: null
|
||||||
|
n_gpus_per_node: 8
|
||||||
|
nnodes: 1
|
||||||
|
project_name: SDPO-root
|
||||||
|
ray_wait_register_center_timeout: 300
|
||||||
|
resume_from_path: null
|
||||||
|
resume_mode: auto
|
||||||
|
rollout_data_dir: null
|
||||||
|
save_freq: 10
|
||||||
|
test_freq: 10
|
||||||
|
total_epochs: 30
|
||||||
|
total_training_steps: 100
|
||||||
|
use_legacy_worker_impl: auto
|
||||||
|
val_before_train: false
|
||||||
|
val_only: false
|
||||||
|
validation_data_dir: null
|
||||||
|
transfer_queue:
|
||||||
|
value:
|
||||||
|
enable: false
|
||||||
|
vars:
|
||||||
|
value:
|
||||||
|
ckpt_dir: /capstor/scratch/cscs/root/ttrl_runs/datasets/sciknoweval/physics
|
||||||
|
dir: /users/root/SDPO
|
||||||
|
log_dir: /users/root/output
|
||||||
|
task: datasets/sciknoweval/physics
|
||||||
1189
artifacts/output.log
Normal file
1189
artifacts/output.log
Normal file
File diff suppressed because it is too large
Load Diff
7823
artifacts/queue.log
Normal file
7823
artifacts/queue.log
Normal file
File diff suppressed because one or more lines are too long
286
artifacts/requirements.txt
Normal file
286
artifacts/requirements.txt
Normal file
@@ -0,0 +1,286 @@
|
|||||||
|
colorama==0.4.6
|
||||||
|
psutil==7.1.3
|
||||||
|
wheel==0.47.0
|
||||||
|
pip==26.1.2
|
||||||
|
mpmath==1.3.0
|
||||||
|
typing_extensions==4.15.0
|
||||||
|
pillow==12.2.0
|
||||||
|
nvidia-cuda-runtime-cu12==12.8.90
|
||||||
|
nvidia-cuda-nvrtc-cu12==12.8.93
|
||||||
|
nvidia-cusparse-cu12==12.5.8.93
|
||||||
|
nvidia-cufft-cu12==11.3.3.83
|
||||||
|
nvidia-cuda-cupti-cu12==12.8.90
|
||||||
|
nvidia-cublas-cu12==12.8.4.1
|
||||||
|
llguidance==1.3.0
|
||||||
|
anthropic==0.71.0
|
||||||
|
nodeenv==1.10.0
|
||||||
|
networkx==3.6.1
|
||||||
|
MarkupSafe==3.0.3
|
||||||
|
filelock==3.29.0
|
||||||
|
nvidia-cudnn-cu12==9.10.2.21
|
||||||
|
transformers==4.57.1
|
||||||
|
Jinja2==3.1.6
|
||||||
|
nvidia-cusolver-cu12==11.7.3.90
|
||||||
|
torch==2.9.0
|
||||||
|
rich-toolkit==0.20.1
|
||||||
|
z3-solver==4.15.4.0
|
||||||
|
word2number==1.1
|
||||||
|
pytz==2026.2
|
||||||
|
pylatexenc==2.10
|
||||||
|
py-spy==0.4.2
|
||||||
|
opencensus-context==0.1.3
|
||||||
|
distlib==0.4.3
|
||||||
|
colorful==0.5.8
|
||||||
|
antlr4-python3-runtime==4.9.3
|
||||||
|
zipp==4.1.0
|
||||||
|
xxhash==3.7.0
|
||||||
|
wrapt==2.2.1
|
||||||
|
Werkzeug==3.1.8
|
||||||
|
urllib3==2.7.0
|
||||||
|
tzdata==2026.2
|
||||||
|
typing-inspection==0.4.2
|
||||||
|
tqdm==4.68.2
|
||||||
|
tensorboard-data-server==0.7.2
|
||||||
|
smmap==5.0.3
|
||||||
|
six==1.17.0
|
||||||
|
safetensors==0.8.0
|
||||||
|
rpds-py==2026.5.1
|
||||||
|
regex==2026.5.9
|
||||||
|
pyzmq==27.1.0
|
||||||
|
PyYAML==6.0.3
|
||||||
|
Pygments==2.20.0
|
||||||
|
pydantic_core==2.46.4
|
||||||
|
pycparser==3.0
|
||||||
|
pybind11==3.0.1
|
||||||
|
pyasn1==0.6.3
|
||||||
|
pyarrow==22.0.0
|
||||||
|
psutil==7.1.3
|
||||||
|
protobuf==6.33.6
|
||||||
|
propcache==0.5.2
|
||||||
|
prometheus_client==0.25.0
|
||||||
|
pluggy==1.6.0
|
||||||
|
platformdirs==4.10.0
|
||||||
|
httpcore==1.0.9
|
||||||
|
packaging==25.0
|
||||||
|
orjson==3.11.9
|
||||||
|
opentelemetry-api==1.42.1
|
||||||
|
numpy==2.1.0
|
||||||
|
multidict==6.7.1
|
||||||
|
msgpack==1.2.0
|
||||||
|
Markdown==3.10.2
|
||||||
|
iniconfig==2.3.0
|
||||||
|
idna==3.18
|
||||||
|
identify==2.6.19
|
||||||
|
hf-xet==1.5.1
|
||||||
|
h11==0.16.0
|
||||||
|
grpcio==1.81.1
|
||||||
|
gitdb==4.0.12
|
||||||
|
fsspec==2025.10.0
|
||||||
|
frozenlist==1.8.0
|
||||||
|
dill==0.4.0
|
||||||
|
codetiming==1.4.0
|
||||||
|
cloudpickle==3.1.2
|
||||||
|
click==8.4.1
|
||||||
|
charset-normalizer==3.4.7
|
||||||
|
cfgv==3.5.0
|
||||||
|
certifi==2026.5.20
|
||||||
|
attrs==26.1.0
|
||||||
|
annotated-types==0.7.0
|
||||||
|
annotated-doc==0.0.4
|
||||||
|
aiohappyeyeballs==2.6.2
|
||||||
|
absl-py==2.4.0
|
||||||
|
yarl==1.24.2
|
||||||
|
uvicorn==0.40.0
|
||||||
|
tensorboard==2.20.0
|
||||||
|
smart_open==7.6.1
|
||||||
|
sentry-sdk==2.62.0
|
||||||
|
requests==2.34.2
|
||||||
|
referencing==0.37.0
|
||||||
|
pyvers==0.1.0
|
||||||
|
python-discovery==1.4.2
|
||||||
|
python-dateutil==2.9.0.post0
|
||||||
|
pytest==9.1.0
|
||||||
|
pydantic==2.13.4
|
||||||
|
pyasn1_modules==0.4.2
|
||||||
|
proto-plus==1.28.0
|
||||||
|
nvidia-nvjitlink-cu12==12.8.93
|
||||||
|
nvidia-curand-cu12==10.3.9.90
|
||||||
|
omegaconf==2.3.1
|
||||||
|
multiprocess==0.70.18
|
||||||
|
latex2sympy2_extended==1.10.2
|
||||||
|
latex2sympy2==1.5.4
|
||||||
|
googleapis-common-protos==1.75.0
|
||||||
|
cffi==2.0.0
|
||||||
|
anyio==4.13.0
|
||||||
|
aiosignal==1.4.0
|
||||||
|
virtualenv==21.5.0
|
||||||
|
starlette==0.50.0
|
||||||
|
pytest-rerunfailures==16.3
|
||||||
|
pytest-asyncio==1.4.0
|
||||||
|
pandas==2.3.3
|
||||||
|
nvidia-nccl-cu12==2.27.5
|
||||||
|
math-verify==0.8.0
|
||||||
|
jsonschema-specifications==2025.9.1
|
||||||
|
hydra-core==1.3.2
|
||||||
|
huggingface_hub==0.36.2
|
||||||
|
httpx==0.28.1
|
||||||
|
GitPython==3.1.50
|
||||||
|
cryptography==49.0.0
|
||||||
|
aiohttp==3.14.1
|
||||||
|
wandb==0.23.1
|
||||||
|
torchdata==0.11.0
|
||||||
|
numba==0.61.2
|
||||||
|
tensordict==0.10.0
|
||||||
|
pre_commit==4.6.0
|
||||||
|
liger_kernel==0.8.0
|
||||||
|
jsonschema==4.26.0
|
||||||
|
google-auth==2.54.0
|
||||||
|
fastapi==0.127.0
|
||||||
|
aiohttp-cors==0.8.1
|
||||||
|
accelerate==1.12.0
|
||||||
|
llvmlite==0.44.0
|
||||||
|
google-api-core==2.29.0
|
||||||
|
datasets==4.4.2
|
||||||
|
peft==0.18.0
|
||||||
|
opencensus==0.11.4
|
||||||
|
verl==0.7.0.dev0
|
||||||
|
einops==0.8.2
|
||||||
|
py-cpuinfo==9.0.0
|
||||||
|
nvidia-cusparselt-cu12==0.7.1
|
||||||
|
websockets==16.0
|
||||||
|
uvloop==0.22.1
|
||||||
|
sniffio==1.3.1
|
||||||
|
shellingham==1.5.4
|
||||||
|
sentencepiece==0.2.1
|
||||||
|
scipy==1.17.1
|
||||||
|
rignore==0.7.6
|
||||||
|
python-multipart==0.0.32
|
||||||
|
python-json-logger==4.1.0
|
||||||
|
python-dotenv==1.2.2
|
||||||
|
pycountry==26.2.16
|
||||||
|
partial-json-parser==0.2.1.1.post7
|
||||||
|
opentelemetry-semantic-conventions-ai==0.4.13
|
||||||
|
opencv-python-headless==4.13.0.92
|
||||||
|
ninja==1.13.0
|
||||||
|
nest-asyncio==1.6.0
|
||||||
|
msgspec==0.21.1
|
||||||
|
mdurl==0.1.2
|
||||||
|
lark==1.2.2
|
||||||
|
jiter==0.15.0
|
||||||
|
interegular==0.3.3
|
||||||
|
importlib_metadata==8.0.0
|
||||||
|
httptools==0.8.0
|
||||||
|
fastar==0.11.0
|
||||||
|
dnspython==2.8.0
|
||||||
|
distro==1.9.0
|
||||||
|
diskcache==5.6.3
|
||||||
|
detect-installer==0.1.0
|
||||||
|
Deprecated==1.3.1
|
||||||
|
cuda-pathfinder==1.5.5
|
||||||
|
cachetools==7.1.4
|
||||||
|
blake3==1.0.8
|
||||||
|
astor==0.8.1
|
||||||
|
airportsdata==20260315
|
||||||
|
watchfiles==1.2.0
|
||||||
|
tiktoken==0.13.0
|
||||||
|
tilelang==0.1.9
|
||||||
|
markdown-it-py==4.2.0
|
||||||
|
gguf==0.19.0
|
||||||
|
email-validator==2.3.0
|
||||||
|
cuda-tile==1.3.0
|
||||||
|
cupy-cuda12x==14.1.1
|
||||||
|
nvidia-ml-py==13.610.43
|
||||||
|
rich==15.0.0
|
||||||
|
pydantic-settings==2.14.1
|
||||||
|
pydantic-extra-types==2.11.1
|
||||||
|
prometheus-fastapi-instrumentator==7.1.0
|
||||||
|
supervisor==4.3.0
|
||||||
|
tokenspeed-mla==0.1.2
|
||||||
|
outlines_core==0.2.11
|
||||||
|
typer==0.26.7
|
||||||
|
quack-kernels==0.5.0
|
||||||
|
nvidia-nvvm==13.2.78
|
||||||
|
torch_c_dlpack_ext==0.1.5
|
||||||
|
mistral_common==1.11.3
|
||||||
|
fastapi-cloud-cli==0.20.0
|
||||||
|
fastapi-cli==0.0.24
|
||||||
|
cuda-toolkit==13.0.2
|
||||||
|
tabulate==0.10.0
|
||||||
|
sympy==1.14.0
|
||||||
|
setuptools==79.0.1
|
||||||
|
pybase64==1.4.3
|
||||||
|
nvidia-cufile-cu12==1.13.1.3
|
||||||
|
humming-kernels==0.1.4
|
||||||
|
setproctitle==1.3.7
|
||||||
|
triton==3.5.0
|
||||||
|
nvidia-nvtx-cu12==12.8.90
|
||||||
|
nvidia-cusparselt-cu13==0.8.0
|
||||||
|
tokenspeed-triton==3.7.10.post20260531
|
||||||
|
PyJWT==2.13.0
|
||||||
|
pyelftools==0.33
|
||||||
|
nvidia-nvshmem-cu12==3.3.20
|
||||||
|
nvidia-nvtx==13.0.85
|
||||||
|
nvidia-nvshmem-cu13==3.4.5
|
||||||
|
nvidia-nvjitlink==13.0.88
|
||||||
|
nvidia-nccl-cu13==2.28.9
|
||||||
|
nvidia-curand==10.4.0.35
|
||||||
|
nvidia-cufile==1.15.1.6
|
||||||
|
nvidia-cudnn-frontend==1.25.0
|
||||||
|
nvidia-cuda-runtime==13.0.96
|
||||||
|
nvidia-cuda-nvrtc==13.0.88
|
||||||
|
nvidia-cuda-cupti==13.0.85
|
||||||
|
nvidia-cuda-crt==13.3.33
|
||||||
|
nvidia-cuda-cccl==13.3.3.3.1
|
||||||
|
nvidia-cublas==13.1.0.3
|
||||||
|
ml_dtypes==0.5.4
|
||||||
|
loguru==0.7.3
|
||||||
|
jmespath==1.1.0
|
||||||
|
ijson==3.5.0
|
||||||
|
httpx-sse==0.4.3
|
||||||
|
docstring_parser==0.18.0
|
||||||
|
depyf==0.20.0
|
||||||
|
cuda-core==1.0.1
|
||||||
|
cuda-bindings==13.3.1
|
||||||
|
cbor2==6.1.2
|
||||||
|
apache-tvm-ffi==0.1.9
|
||||||
|
mcp==1.27.2
|
||||||
|
opentelemetry-semantic-conventions==0.63b1
|
||||||
|
opentelemetry-proto==1.42.1
|
||||||
|
nvidia-cusparse==12.6.3.3
|
||||||
|
nvidia-cufft==12.0.0.61
|
||||||
|
nvidia-cudnn-cu13==9.19.0.56
|
||||||
|
nvidia-cuda-nvcc==13.2.78
|
||||||
|
cuda-python==13.3.1
|
||||||
|
fastsafetensors==0.3.2
|
||||||
|
tokenizers==0.22.2
|
||||||
|
sse-starlette==3.4.4
|
||||||
|
opentelemetry-sdk==1.42.1
|
||||||
|
nvidia-cutlass-dsl==4.5.2
|
||||||
|
opentelemetry-exporter-otlp-proto-common==1.42.1
|
||||||
|
openai-harmony==0.0.8
|
||||||
|
openai==2.41.1
|
||||||
|
nvidia-cutlass-dsl-libs-base==4.5.2
|
||||||
|
nvidia-cusolver==12.0.4.66
|
||||||
|
nvidia-cuda-tileiras==13.2.78
|
||||||
|
nvidia-cutlass-dsl-libs-cu13==4.5.2
|
||||||
|
lm-format-enforcer==0.11.3
|
||||||
|
ray==2.53.0
|
||||||
|
model-hosting-container-standards==0.1.15
|
||||||
|
opentelemetry-exporter-otlp-proto-http==1.42.1
|
||||||
|
opentelemetry-exporter-otlp-proto-grpc==1.42.1
|
||||||
|
opentelemetry-exporter-otlp==1.42.1
|
||||||
|
opentelemetry-exporter-prometheus==0.63b1
|
||||||
|
xgrammar==0.1.27
|
||||||
|
torchvision==0.24.0
|
||||||
|
torchaudio==2.9.0
|
||||||
|
flashinfer-python==0.5.3
|
||||||
|
compressed-tensors==0.12.2
|
||||||
|
vllm==0.12.0
|
||||||
|
flash_attn==2.8.3
|
||||||
|
flashinfer-cubin==0.5.3
|
||||||
|
pyparsing==3.3.2
|
||||||
|
kiwisolver==1.5.0
|
||||||
|
fonttools==4.63.0
|
||||||
|
cycler==0.12.1
|
||||||
|
contourpy==1.3.3
|
||||||
|
matplotlib==3.11.0
|
||||||
109
artifacts/wandb-metadata.json
Normal file
109
artifacts/wandb-metadata.json
Normal file
@@ -0,0 +1,109 @@
|
|||||||
|
{
|
||||||
|
"os": "Linux-6.8.0-71-generic-x86_64-with-glibc2.36",
|
||||||
|
"python": "CPython 3.12.13",
|
||||||
|
"startedAt": "2026-07-03T03:20:40.327312Z",
|
||||||
|
"args": [
|
||||||
|
"--node-ip-address=198.19.44.212",
|
||||||
|
"--node-manager-port=36871",
|
||||||
|
"--object-store-name=/tmp/ray_q3g_physics_grpo_Qwen3_8B/session_2026-07-03_03-18-52_070071_2279047/sockets/plasma_store",
|
||||||
|
"--raylet-name=/tmp/ray_q3g_physics_grpo_Qwen3_8B/session_2026-07-03_03-18-52_070071_2279047/sockets/raylet",
|
||||||
|
"--redis-address=None",
|
||||||
|
"--metrics-agent-port=56405",
|
||||||
|
"--logging-rotate-bytes=536870912",
|
||||||
|
"--logging-rotate-backup-count=5",
|
||||||
|
"--runtime-env-agent-port=64277",
|
||||||
|
"--gcs-address=198.19.44.212:64574",
|
||||||
|
"--session-name=session_2026-07-03_03-18-52_070071_2279047",
|
||||||
|
"--temp-dir=/tmp/ray_q3g_physics_grpo_Qwen3_8B",
|
||||||
|
"--webui=",
|
||||||
|
"--cluster-id=d7576956015edcd024cf2ca706acce8be3fb6519bb7b0c87bfccae89",
|
||||||
|
"--startup-token=32",
|
||||||
|
"--worker-launch-time-ms=1783048735496",
|
||||||
|
"--node-id=e88e44745be3f8c4449808648428760387c2aa53671532e80c9f13b3",
|
||||||
|
"--runtime-env-hash=1652680156"
|
||||||
|
],
|
||||||
|
"program": "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/ray/_private/workers/default_worker.py",
|
||||||
|
"codePath": ".venv/lib/python3.12/site-packages/ray/_private/workers/default_worker.py",
|
||||||
|
"codePathLocal": ".venv/lib/python3.12/site-packages/ray/_private/workers/default_worker.py",
|
||||||
|
"git": {
|
||||||
|
"remote": "https://github.com/jungseongryong/L2T.git",
|
||||||
|
"commit": "ee6a667700697069b93db636aa937f970c1fb57a"
|
||||||
|
},
|
||||||
|
"email": "jungsr1116@cau.ac.kr",
|
||||||
|
"root": "/mnt/mole/SDPO/L2T",
|
||||||
|
"host": "mole-workspace-rw",
|
||||||
|
"executable": "/mnt/mole/SDPO/L2T/.venv/bin/python",
|
||||||
|
"cpu_count": 64,
|
||||||
|
"cpu_count_logical": 128,
|
||||||
|
"gpu": "NVIDIA H200",
|
||||||
|
"gpu_count": 8,
|
||||||
|
"disk": {
|
||||||
|
"/": {
|
||||||
|
"total": "46086056050688",
|
||||||
|
"used": "25787742543872"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"memory": {
|
||||||
|
"total": "2163980390400"
|
||||||
|
},
|
||||||
|
"gpu_nvidia": [
|
||||||
|
{
|
||||||
|
"name": "NVIDIA H200",
|
||||||
|
"memoryTotal": "150754820096",
|
||||||
|
"cudaCores": 16896,
|
||||||
|
"architecture": "Hopper",
|
||||||
|
"uuid": "GPU-c0451621-891c-7169-976d-71d81958db34"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "NVIDIA H200",
|
||||||
|
"memoryTotal": "150754820096",
|
||||||
|
"cudaCores": 16896,
|
||||||
|
"architecture": "Hopper",
|
||||||
|
"uuid": "GPU-e2a29dfa-7dad-6cd7-fb05-79d0060f27f0"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "NVIDIA H200",
|
||||||
|
"memoryTotal": "150754820096",
|
||||||
|
"cudaCores": 16896,
|
||||||
|
"architecture": "Hopper",
|
||||||
|
"uuid": "GPU-d69bb741-f2a7-a375-7f56-5c0d09d797b2"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "NVIDIA H200",
|
||||||
|
"memoryTotal": "150754820096",
|
||||||
|
"cudaCores": 16896,
|
||||||
|
"architecture": "Hopper",
|
||||||
|
"uuid": "GPU-1bbbea1e-7541-3126-1bfc-51b43a67577e"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "NVIDIA H200",
|
||||||
|
"memoryTotal": "150754820096",
|
||||||
|
"cudaCores": 16896,
|
||||||
|
"architecture": "Hopper",
|
||||||
|
"uuid": "GPU-6e0055af-452d-3f46-ce21-0c71e347a608"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "NVIDIA H200",
|
||||||
|
"memoryTotal": "150754820096",
|
||||||
|
"cudaCores": 16896,
|
||||||
|
"architecture": "Hopper",
|
||||||
|
"uuid": "GPU-317fdb5b-b280-71d8-398e-a843ad897437"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "NVIDIA H200",
|
||||||
|
"memoryTotal": "150754820096",
|
||||||
|
"cudaCores": 16896,
|
||||||
|
"architecture": "Hopper",
|
||||||
|
"uuid": "GPU-f19d8161-d2dd-49a1-dcb8-5c341b5e4514"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "NVIDIA H200",
|
||||||
|
"memoryTotal": "150754820096",
|
||||||
|
"cudaCores": 16896,
|
||||||
|
"architecture": "Hopper",
|
||||||
|
"uuid": "GPU-961f5ebf-4bf8-6855-828b-c065af57b87c"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"cudaVersion": "13.0",
|
||||||
|
"writerId": "1n1ucg63toheyodgdd2dcwkutcbcxhn8"
|
||||||
|
}
|
||||||
1
artifacts/wandb-summary.json
Normal file
1
artifacts/wandb-summary.json
Normal file
File diff suppressed because one or more lines are too long
89
chat_template.jinja
Normal file
89
chat_template.jinja
Normal file
@@ -0,0 +1,89 @@
|
|||||||
|
{%- if tools %}
|
||||||
|
{{- '<|im_start|>system\n' }}
|
||||||
|
{%- if messages[0].role == 'system' %}
|
||||||
|
{{- messages[0].content + '\n\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||||
|
{%- for tool in tools %}
|
||||||
|
{{- "\n" }}
|
||||||
|
{{- tool | tojson }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||||
|
{%- else %}
|
||||||
|
{%- if messages[0].role == 'system' %}
|
||||||
|
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||||
|
{%- for message in messages[::-1] %}
|
||||||
|
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||||
|
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
|
||||||
|
{%- set ns.multi_step_tool = false %}
|
||||||
|
{%- set ns.last_query_index = index %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- for message in messages %}
|
||||||
|
{%- if message.content is string %}
|
||||||
|
{%- set content = message.content %}
|
||||||
|
{%- else %}
|
||||||
|
{%- set content = '' %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||||
|
{%- elif message.role == "assistant" %}
|
||||||
|
{%- set reasoning_content = '' %}
|
||||||
|
{%- if message.reasoning_content is string %}
|
||||||
|
{%- set reasoning_content = message.reasoning_content %}
|
||||||
|
{%- else %}
|
||||||
|
{%- if '</think>' in content %}
|
||||||
|
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
|
||||||
|
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if loop.index0 > ns.last_query_index %}
|
||||||
|
{%- if loop.last or (not loop.last and reasoning_content) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if message.tool_calls %}
|
||||||
|
{%- for tool_call in message.tool_calls %}
|
||||||
|
{%- if (loop.first and content) or (not loop.first) %}
|
||||||
|
{{- '\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if tool_call.function %}
|
||||||
|
{%- set tool_call = tool_call.function %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '<tool_call>\n{"name": "' }}
|
||||||
|
{{- tool_call.name }}
|
||||||
|
{{- '", "arguments": ' }}
|
||||||
|
{%- if tool_call.arguments is string %}
|
||||||
|
{{- tool_call.arguments }}
|
||||||
|
{%- else %}
|
||||||
|
{{- tool_call.arguments | tojson }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '}\n</tool_call>' }}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- elif message.role == "tool" %}
|
||||||
|
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
|
||||||
|
{{- '<|im_start|>user' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_response>\n' }}
|
||||||
|
{{- content }}
|
||||||
|
{{- '\n</tool_response>' }}
|
||||||
|
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- if add_generation_prompt %}
|
||||||
|
{{- '<|im_start|>assistant\n' }}
|
||||||
|
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||||
|
{{- '<think>\n\n</think>\n\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
28
checkpoints/last/added_tokens.json
Normal file
28
checkpoints/last/added_tokens.json
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
{
|
||||||
|
"</think>": 151668,
|
||||||
|
"</tool_call>": 151658,
|
||||||
|
"</tool_response>": 151666,
|
||||||
|
"<think>": 151667,
|
||||||
|
"<tool_call>": 151657,
|
||||||
|
"<tool_response>": 151665,
|
||||||
|
"<|box_end|>": 151649,
|
||||||
|
"<|box_start|>": 151648,
|
||||||
|
"<|endoftext|>": 151643,
|
||||||
|
"<|file_sep|>": 151664,
|
||||||
|
"<|fim_middle|>": 151660,
|
||||||
|
"<|fim_pad|>": 151662,
|
||||||
|
"<|fim_prefix|>": 151659,
|
||||||
|
"<|fim_suffix|>": 151661,
|
||||||
|
"<|im_end|>": 151645,
|
||||||
|
"<|im_start|>": 151644,
|
||||||
|
"<|image_pad|>": 151655,
|
||||||
|
"<|object_ref_end|>": 151647,
|
||||||
|
"<|object_ref_start|>": 151646,
|
||||||
|
"<|quad_end|>": 151651,
|
||||||
|
"<|quad_start|>": 151650,
|
||||||
|
"<|repo_name|>": 151663,
|
||||||
|
"<|video_pad|>": 151656,
|
||||||
|
"<|vision_end|>": 151653,
|
||||||
|
"<|vision_pad|>": 151654,
|
||||||
|
"<|vision_start|>": 151652
|
||||||
|
}
|
||||||
89
checkpoints/last/chat_template.jinja
Normal file
89
checkpoints/last/chat_template.jinja
Normal file
@@ -0,0 +1,89 @@
|
|||||||
|
{%- if tools %}
|
||||||
|
{{- '<|im_start|>system\n' }}
|
||||||
|
{%- if messages[0].role == 'system' %}
|
||||||
|
{{- messages[0].content + '\n\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||||
|
{%- for tool in tools %}
|
||||||
|
{{- "\n" }}
|
||||||
|
{{- tool | tojson }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||||
|
{%- else %}
|
||||||
|
{%- if messages[0].role == 'system' %}
|
||||||
|
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||||
|
{%- for message in messages[::-1] %}
|
||||||
|
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||||
|
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
|
||||||
|
{%- set ns.multi_step_tool = false %}
|
||||||
|
{%- set ns.last_query_index = index %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- for message in messages %}
|
||||||
|
{%- if message.content is string %}
|
||||||
|
{%- set content = message.content %}
|
||||||
|
{%- else %}
|
||||||
|
{%- set content = '' %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||||
|
{%- elif message.role == "assistant" %}
|
||||||
|
{%- set reasoning_content = '' %}
|
||||||
|
{%- if message.reasoning_content is string %}
|
||||||
|
{%- set reasoning_content = message.reasoning_content %}
|
||||||
|
{%- else %}
|
||||||
|
{%- if '</think>' in content %}
|
||||||
|
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
|
||||||
|
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if loop.index0 > ns.last_query_index %}
|
||||||
|
{%- if loop.last or (not loop.last and reasoning_content) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if message.tool_calls %}
|
||||||
|
{%- for tool_call in message.tool_calls %}
|
||||||
|
{%- if (loop.first and content) or (not loop.first) %}
|
||||||
|
{{- '\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if tool_call.function %}
|
||||||
|
{%- set tool_call = tool_call.function %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '<tool_call>\n{"name": "' }}
|
||||||
|
{{- tool_call.name }}
|
||||||
|
{{- '", "arguments": ' }}
|
||||||
|
{%- if tool_call.arguments is string %}
|
||||||
|
{{- tool_call.arguments }}
|
||||||
|
{%- else %}
|
||||||
|
{{- tool_call.arguments | tojson }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '}\n</tool_call>' }}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- elif message.role == "tool" %}
|
||||||
|
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
|
||||||
|
{{- '<|im_start|>user' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_response>\n' }}
|
||||||
|
{{- content }}
|
||||||
|
{{- '\n</tool_response>' }}
|
||||||
|
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- if add_generation_prompt %}
|
||||||
|
{{- '<|im_start|>assistant\n' }}
|
||||||
|
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||||
|
{{- '<think>\n\n</think>\n\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
68
checkpoints/last/config.json
Normal file
68
checkpoints/last/config.json
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"Qwen3ForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"eos_token_id": 151645,
|
||||||
|
"head_dim": 128,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 4096,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 12288,
|
||||||
|
"layer_types": [
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention"
|
||||||
|
],
|
||||||
|
"max_position_embeddings": 40960,
|
||||||
|
"max_window_layers": 36,
|
||||||
|
"model_type": "qwen3",
|
||||||
|
"num_attention_heads": 32,
|
||||||
|
"num_hidden_layers": 36,
|
||||||
|
"num_key_value_heads": 8,
|
||||||
|
"pad_token_id": 151643,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_scaling": null,
|
||||||
|
"rope_theta": 1000000,
|
||||||
|
"sliding_window": null,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"transformers_version": "4.57.1",
|
||||||
|
"use_cache": true,
|
||||||
|
"use_sliding_window": false,
|
||||||
|
"vocab_size": 151936
|
||||||
|
}
|
||||||
13
checkpoints/last/generation_config.json
Normal file
13
checkpoints/last/generation_config.json
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
"bos_token_id": 151643,
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": [
|
||||||
|
151645,
|
||||||
|
151643
|
||||||
|
],
|
||||||
|
"pad_token_id": 151643,
|
||||||
|
"temperature": 0.6,
|
||||||
|
"top_k": 20,
|
||||||
|
"top_p": 0.95,
|
||||||
|
"transformers_version": "4.57.1"
|
||||||
|
}
|
||||||
151388
checkpoints/last/merges.txt
Normal file
151388
checkpoints/last/merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
checkpoints/last/model-00001-of-00004.safetensors
Normal file
3
checkpoints/last/model-00001-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:18be607d3bb302aa3ad580d51f9f91c47386d9d5f00777c7eb04d77b2c2422d4
|
||||||
|
size 4986142864
|
||||||
3
checkpoints/last/model-00002-of-00004.safetensors
Normal file
3
checkpoints/last/model-00002-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:523c254543d31bac458eae00ea9de986bd6b7d68920172dffd296672afee9e61
|
||||||
|
size 4924319976
|
||||||
3
checkpoints/last/model-00003-of-00004.safetensors
Normal file
3
checkpoints/last/model-00003-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:4fc7e01b21f81cd924b5bfd87fa4e193ed84da064a6ec7c2a398590e4d1af2e5
|
||||||
|
size 4127416336
|
||||||
3
checkpoints/last/model-00004-of-00004.safetensors
Normal file
3
checkpoints/last/model-00004-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:28f6b19bcd3e40893217e972b125ee1644439fc54866132d77ad0cef3133e3c2
|
||||||
|
size 2343637648
|
||||||
407
checkpoints/last/model.safetensors.index.json
Normal file
407
checkpoints/last/model.safetensors.index.json
Normal file
@@ -0,0 +1,407 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"total_parameters": 8190735360,
|
||||||
|
"total_size": 16381470720
|
||||||
|
},
|
||||||
|
"weight_map": {
|
||||||
|
"lm_head.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.embed_tokens.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.0.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.1.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.1.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.10.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.10.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.11.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.12.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.14.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.14.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.15.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.16.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.17.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.18.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.18.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.19.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.20.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.23.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.24.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.26.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.26.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.3.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.30.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.31.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.31.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.32.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.32.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.33.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.34.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.35.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.35.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.6.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.6.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.7.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.9.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.9.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.norm.weight": "model-00002-of-00004.safetensors"
|
||||||
|
}
|
||||||
|
}
|
||||||
31
checkpoints/last/special_tokens_map.json
Normal file
31
checkpoints/last/special_tokens_map.json
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
{
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>"
|
||||||
|
],
|
||||||
|
"eos_token": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
BIN
checkpoints/last/tokenizer.json
(Stored with Git LFS)
Normal file
BIN
checkpoints/last/tokenizer.json
(Stored with Git LFS)
Normal file
Binary file not shown.
239
checkpoints/last/tokenizer_config.json
Normal file
239
checkpoints/last/tokenizer_config.json
Normal file
@@ -0,0 +1,239 @@
|
|||||||
|
{
|
||||||
|
"add_bos_token": false,
|
||||||
|
"add_prefix_space": false,
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"151643": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151644": {
|
||||||
|
"content": "<|im_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151645": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151646": {
|
||||||
|
"content": "<|object_ref_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151647": {
|
||||||
|
"content": "<|object_ref_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151648": {
|
||||||
|
"content": "<|box_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151649": {
|
||||||
|
"content": "<|box_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151650": {
|
||||||
|
"content": "<|quad_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151651": {
|
||||||
|
"content": "<|quad_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151652": {
|
||||||
|
"content": "<|vision_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151653": {
|
||||||
|
"content": "<|vision_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151654": {
|
||||||
|
"content": "<|vision_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151655": {
|
||||||
|
"content": "<|image_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151656": {
|
||||||
|
"content": "<|video_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151657": {
|
||||||
|
"content": "<tool_call>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151658": {
|
||||||
|
"content": "</tool_call>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151659": {
|
||||||
|
"content": "<|fim_prefix|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151660": {
|
||||||
|
"content": "<|fim_middle|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151661": {
|
||||||
|
"content": "<|fim_suffix|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151662": {
|
||||||
|
"content": "<|fim_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151663": {
|
||||||
|
"content": "<|repo_name|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151664": {
|
||||||
|
"content": "<|file_sep|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151665": {
|
||||||
|
"content": "<tool_response>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151666": {
|
||||||
|
"content": "</tool_response>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151667": {
|
||||||
|
"content": "<think>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151668": {
|
||||||
|
"content": "</think>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>"
|
||||||
|
],
|
||||||
|
"bos_token": null,
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|im_end|>",
|
||||||
|
"errors": "replace",
|
||||||
|
"extra_special_tokens": {},
|
||||||
|
"model_max_length": 131072,
|
||||||
|
"pad_token": "<|endoftext|>",
|
||||||
|
"split_special_tokens": false,
|
||||||
|
"tokenizer_class": "Qwen2Tokenizer",
|
||||||
|
"unk_token": null
|
||||||
|
}
|
||||||
1
checkpoints/last/vocab.json
Normal file
1
checkpoints/last/vocab.json
Normal file
File diff suppressed because one or more lines are too long
68
config.json
Normal file
68
config.json
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"Qwen3ForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"eos_token_id": 151645,
|
||||||
|
"head_dim": 128,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 4096,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 12288,
|
||||||
|
"layer_types": [
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention"
|
||||||
|
],
|
||||||
|
"max_position_embeddings": 40960,
|
||||||
|
"max_window_layers": 36,
|
||||||
|
"model_type": "qwen3",
|
||||||
|
"num_attention_heads": 32,
|
||||||
|
"num_hidden_layers": 36,
|
||||||
|
"num_key_value_heads": 8,
|
||||||
|
"pad_token_id": 151643,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_scaling": null,
|
||||||
|
"rope_theta": 1000000,
|
||||||
|
"sliding_window": null,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"transformers_version": "4.57.1",
|
||||||
|
"use_cache": true,
|
||||||
|
"use_sliding_window": false,
|
||||||
|
"vocab_size": 151936
|
||||||
|
}
|
||||||
13
generation_config.json
Normal file
13
generation_config.json
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
"bos_token_id": 151643,
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": [
|
||||||
|
151645,
|
||||||
|
151643
|
||||||
|
],
|
||||||
|
"pad_token_id": 151643,
|
||||||
|
"temperature": 0.6,
|
||||||
|
"top_k": 20,
|
||||||
|
"top_p": 0.95,
|
||||||
|
"transformers_version": "4.57.1"
|
||||||
|
}
|
||||||
151388
merges.txt
Normal file
151388
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model-00001-of-00004.safetensors
Normal file
3
model-00001-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:18be607d3bb302aa3ad580d51f9f91c47386d9d5f00777c7eb04d77b2c2422d4
|
||||||
|
size 4986142864
|
||||||
3
model-00002-of-00004.safetensors
Normal file
3
model-00002-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:523c254543d31bac458eae00ea9de986bd6b7d68920172dffd296672afee9e61
|
||||||
|
size 4924319976
|
||||||
3
model-00003-of-00004.safetensors
Normal file
3
model-00003-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:4fc7e01b21f81cd924b5bfd87fa4e193ed84da064a6ec7c2a398590e4d1af2e5
|
||||||
|
size 4127416336
|
||||||
3
model-00004-of-00004.safetensors
Normal file
3
model-00004-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:28f6b19bcd3e40893217e972b125ee1644439fc54866132d77ad0cef3133e3c2
|
||||||
|
size 2343637648
|
||||||
407
model.safetensors.index.json
Normal file
407
model.safetensors.index.json
Normal file
@@ -0,0 +1,407 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"total_parameters": 8190735360,
|
||||||
|
"total_size": 16381470720
|
||||||
|
},
|
||||||
|
"weight_map": {
|
||||||
|
"lm_head.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.embed_tokens.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.0.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.1.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.1.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.10.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.10.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.11.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.12.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.14.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.14.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.15.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.16.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.17.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.18.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.18.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.19.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.20.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.23.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.24.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.26.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.26.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.3.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.30.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.31.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.31.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.32.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.32.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.33.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.34.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.35.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.35.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.6.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.6.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.7.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.9.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.9.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.norm.weight": "model-00002-of-00004.safetensors"
|
||||||
|
}
|
||||||
|
}
|
||||||
33
results/hyperparameters.csv
Normal file
33
results/hyperparameters.csv
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
section,parameter,value,source
|
||||||
|
Run identity,Base model,Qwen/Qwen3-8B,queue/script override
|
||||||
|
Run identity,Dataset,Physics / SciKnowEval physics,run_qwen3_generalization.sh
|
||||||
|
Run identity,Method,GRPO,run_qwen3_generalization.sh
|
||||||
|
Run identity,Config,baseline_grpo,run_qwen3_generalization.sh
|
||||||
|
Run identity,Experiment,qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8,run_qwen3_generalization.sh
|
||||||
|
Run identity,W&B run,run-20260703_032040-6ig3l55l,wandb
|
||||||
|
Data,Train file,datasets/sciknoweval/physics/train.parquet,script override
|
||||||
|
Data,Validation file,datasets/sciknoweval/physics/test.parquet,script override
|
||||||
|
Data,Train batch size,32,queue/script override
|
||||||
|
Data,Train max samples,3200,queue/script override
|
||||||
|
Schedule,Total training steps,100,queue/script override
|
||||||
|
Schedule,Validation before train,False,queue/script override
|
||||||
|
Schedule,Save frequency,10,queue/script override
|
||||||
|
Schedule,Validation frequency,10,queue/script override
|
||||||
|
Sequence,Max prompt length,2048,queue/script override
|
||||||
|
Sequence,Max response length,8192,queue/script override
|
||||||
|
Sequence,Max model length,10240,queue/script override
|
||||||
|
Rollout,Train rollout n,8,queue/script override
|
||||||
|
Rollout,Validation rollout n,16,queue/script override
|
||||||
|
Rollout,vLLM GPU memory utilization,0.8,queue/script override
|
||||||
|
Optimization,Learning rate,1e-6,GRPO method override
|
||||||
|
Optimization,Weight decay,0.01,script override
|
||||||
|
PPO/GRPO,PPO mini batch size,8,queue/script override
|
||||||
|
PPO/GRPO,Normalize GRPO advantages by std,False,baseline_grpo.yaml / script override
|
||||||
|
Rollout correction,Importance sampling mode,token,script override
|
||||||
|
Rollout correction,IS threshold,2.0,script override
|
||||||
|
Checkpoint/Logging,Checkpoint root,checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8,script override
|
||||||
|
Checkpoint/Logging,Latest checkpointed iteration,100,latest_checkpointed_iteration.txt
|
||||||
|
Checkpoint/Logging,External actor archive,checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/_actor_archive,preserve_actor_checkpoints.py
|
||||||
|
Checkpoint/Logging,Logger,"console, wandb",ppo_trainer.yaml
|
||||||
|
PPO/GRPO,Policy loss mode,vanilla,method override
|
||||||
|
PPO/GRPO,Actor KL loss coef,0.0,method override
|
||||||
|
29
results/summary.json
Normal file
29
results/summary.json
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
{
|
||||||
|
"repo_id": "SeongryongJung/Qwen3-8B-Physics-GRPO-TR",
|
||||||
|
"output_dir": "/mnt/mole/SDPO/L2T/hf_upload_tr/Qwen3-8B-Physics-GRPO-TR",
|
||||||
|
"experiment": "qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8",
|
||||||
|
"best_step": 100,
|
||||||
|
"best_val_mean16": 0.7296875,
|
||||||
|
"best_actor_dir": "/mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_100/actor",
|
||||||
|
"final_step": 100,
|
||||||
|
"final_val_mean16": 0.7296875,
|
||||||
|
"final_actor_dir": "/mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_100/actor",
|
||||||
|
"train_rows": 90,
|
||||||
|
"val_rows": 10,
|
||||||
|
"hf_model_files": [
|
||||||
|
"added_tokens.json",
|
||||||
|
"chat_template.jinja",
|
||||||
|
"config.json",
|
||||||
|
"generation_config.json",
|
||||||
|
"merges.txt",
|
||||||
|
"model.safetensors.index.json",
|
||||||
|
"special_tokens_map.json",
|
||||||
|
"tokenizer.json",
|
||||||
|
"tokenizer_config.json",
|
||||||
|
"vocab.json",
|
||||||
|
"model-00001-of-00004.safetensors",
|
||||||
|
"model-00002-of-00004.safetensors",
|
||||||
|
"model-00003-of-00004.safetensors",
|
||||||
|
"model-00004-of-00004.safetensors"
|
||||||
|
]
|
||||||
|
}
|
||||||
3
results/training_score.png
Normal file
3
results/training_score.png
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:453f8ccb73768a3b3ae4b970f7f0b84df4b8d760e9ac177c35f17270c51db265
|
||||||
|
size 129239
|
||||||
1591
results/training_score.svg
Normal file
1591
results/training_score.svg
Normal file
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 43 KiB |
91
results/training_scores.csv
Normal file
91
results/training_scores.csv
Normal file
@@ -0,0 +1,91 @@
|
|||||||
|
step,critic_score_mean
|
||||||
|
1,0.6015625
|
||||||
|
2,0.5859375
|
||||||
|
3,0.52734375
|
||||||
|
4,0.63671875
|
||||||
|
5,0.73046875
|
||||||
|
6,0.65625
|
||||||
|
7,0.75390625
|
||||||
|
8,0.7265625
|
||||||
|
9,0.58984375
|
||||||
|
11,0.62890625
|
||||||
|
12,0.5234375
|
||||||
|
13,0.5703125
|
||||||
|
14,0.64453125
|
||||||
|
15,0.69140625
|
||||||
|
16,0.765625
|
||||||
|
17,0.70703125
|
||||||
|
18,0.41015625
|
||||||
|
19,0.71875
|
||||||
|
21,0.6640625
|
||||||
|
22,0.71484375
|
||||||
|
23,0.5
|
||||||
|
24,0.74609375
|
||||||
|
25,0.71875
|
||||||
|
26,0.7109375
|
||||||
|
27,0.796875
|
||||||
|
28,0.6171875
|
||||||
|
29,0.70703125
|
||||||
|
31,0.62890625
|
||||||
|
32,0.73828125
|
||||||
|
33,0.70703125
|
||||||
|
34,0.6796875
|
||||||
|
35,0.75390625
|
||||||
|
36,0.578125
|
||||||
|
37,0.6640625
|
||||||
|
38,0.66015625
|
||||||
|
39,0.87890625
|
||||||
|
41,0.70703125
|
||||||
|
42,0.6171875
|
||||||
|
43,0.74609375
|
||||||
|
44,0.66796875
|
||||||
|
45,0.66796875
|
||||||
|
46,0.7890625
|
||||||
|
47,0.75
|
||||||
|
48,0.765625
|
||||||
|
49,0.859375
|
||||||
|
51,0.7109375
|
||||||
|
52,0.71484375
|
||||||
|
53,0.73828125
|
||||||
|
54,0.76171875
|
||||||
|
55,0.84765625
|
||||||
|
56,0.6484375
|
||||||
|
57,0.734375
|
||||||
|
58,0.64453125
|
||||||
|
59,0.71875
|
||||||
|
61,0.72265625
|
||||||
|
62,0.65625
|
||||||
|
63,0.7109375
|
||||||
|
64,0.6875
|
||||||
|
65,0.8671875
|
||||||
|
66,0.765625
|
||||||
|
67,0.8046875
|
||||||
|
68,0.734375
|
||||||
|
69,0.80859375
|
||||||
|
71,0.82421875
|
||||||
|
72,0.76953125
|
||||||
|
73,0.66015625
|
||||||
|
74,0.82421875
|
||||||
|
75,0.85546875
|
||||||
|
76,0.6953125
|
||||||
|
77,0.72265625
|
||||||
|
78,0.80859375
|
||||||
|
79,0.82421875
|
||||||
|
81,0.734375
|
||||||
|
82,0.76953125
|
||||||
|
83,0.6953125
|
||||||
|
84,0.68359375
|
||||||
|
85,0.73828125
|
||||||
|
86,0.77734375
|
||||||
|
87,0.76953125
|
||||||
|
88,0.72265625
|
||||||
|
89,0.7421875
|
||||||
|
91,0.81640625
|
||||||
|
92,0.73046875
|
||||||
|
93,0.75
|
||||||
|
94,0.8203125
|
||||||
|
95,0.6328125
|
||||||
|
96,0.7421875
|
||||||
|
97,0.734375
|
||||||
|
98,0.79296875
|
||||||
|
99,0.84765625
|
||||||
|
11
results/validation_mean16.csv
Normal file
11
results/validation_mean16.csv
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
step,val_mean16
|
||||||
|
10,0.58359375
|
||||||
|
20,0.59921875
|
||||||
|
30,0.6046875
|
||||||
|
40,0.62109375
|
||||||
|
50,0.65390625
|
||||||
|
60,0.671875
|
||||||
|
70,0.68125
|
||||||
|
80,0.7125
|
||||||
|
90,0.72265625
|
||||||
|
100,0.7296875
|
||||||
|
31
special_tokens_map.json
Normal file
31
special_tokens_map.json
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
{
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>"
|
||||||
|
],
|
||||||
|
"eos_token": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
Binary file not shown.
239
tokenizer_config.json
Normal file
239
tokenizer_config.json
Normal file
@@ -0,0 +1,239 @@
|
|||||||
|
{
|
||||||
|
"add_bos_token": false,
|
||||||
|
"add_prefix_space": false,
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"151643": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151644": {
|
||||||
|
"content": "<|im_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151645": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151646": {
|
||||||
|
"content": "<|object_ref_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151647": {
|
||||||
|
"content": "<|object_ref_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151648": {
|
||||||
|
"content": "<|box_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151649": {
|
||||||
|
"content": "<|box_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151650": {
|
||||||
|
"content": "<|quad_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151651": {
|
||||||
|
"content": "<|quad_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151652": {
|
||||||
|
"content": "<|vision_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151653": {
|
||||||
|
"content": "<|vision_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151654": {
|
||||||
|
"content": "<|vision_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151655": {
|
||||||
|
"content": "<|image_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151656": {
|
||||||
|
"content": "<|video_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151657": {
|
||||||
|
"content": "<tool_call>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151658": {
|
||||||
|
"content": "</tool_call>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151659": {
|
||||||
|
"content": "<|fim_prefix|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151660": {
|
||||||
|
"content": "<|fim_middle|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151661": {
|
||||||
|
"content": "<|fim_suffix|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151662": {
|
||||||
|
"content": "<|fim_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151663": {
|
||||||
|
"content": "<|repo_name|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151664": {
|
||||||
|
"content": "<|file_sep|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151665": {
|
||||||
|
"content": "<tool_response>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151666": {
|
||||||
|
"content": "</tool_response>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151667": {
|
||||||
|
"content": "<think>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151668": {
|
||||||
|
"content": "</think>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>"
|
||||||
|
],
|
||||||
|
"bos_token": null,
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|im_end|>",
|
||||||
|
"errors": "replace",
|
||||||
|
"extra_special_tokens": {},
|
||||||
|
"model_max_length": 131072,
|
||||||
|
"pad_token": "<|endoftext|>",
|
||||||
|
"split_special_tokens": false,
|
||||||
|
"tokenizer_class": "Qwen2Tokenizer",
|
||||||
|
"unk_token": null
|
||||||
|
}
|
||||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user