jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200-batch-128-20260428-035521
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200-batch-128-20260428-035521
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-05-17 12:31:53 +08:00
commit 2da6de0b35
20 changed files with 4864 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

78
README.md Normal file
View File

@@ -0,0 +1,78 @@
---
library_name: transformers
base_model: W-61/llama-3-8b-base-sft-ultrachat-8xh200
tags:
- alignment-handbook
- r-dpo
- generated_from_trainer
datasets:
- HuggingFaceH4/ultrafeedback_binarized
model-index:
- name: llama-3-8b-base-r-dpo-ultrafeedback-4xh200-batch-128-20260428-035521
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# llama-3-8b-base-r-dpo-ultrafeedback-4xh200-batch-128-20260428-035521
This model is a fine-tuned version of [W-61/llama-3-8b-base-sft-ultrachat-8xh200](https://huggingface.co/W-61/llama-3-8b-base-sft-ultrachat-8xh200) on the HuggingFaceH4/ultrafeedback_binarized dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5327
- R Dpo/chosen Len: 286.9760
- R Dpo/rejected Len: 246.0880
- R Dpo/length Delta: 40.8880
- R Dpo/regularization Term: 0.0
- Logps/chosen: -414.4475
- Logps/rejected: -451.4492
- Logps/ref Chosen: -288.6415
- Logps/ref Rejected: -265.9616
- Logits/chosen: -0.8584
- Logits/rejected: -0.8411
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 4
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 8
- total_train_batch_size: 128
- total_eval_batch_size: 8
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1
### Training results
| Training Loss | Epoch | Step | Validation Loss | R Dpo/chosen Len | R Dpo/rejected Len | R Dpo/length Delta | R Dpo/regularization Term | Logps/chosen | Logps/rejected | Logps/ref Chosen | Logps/ref Rejected | Logits/chosen | Logits/rejected |
|:-------------:|:------:|:----:|:---------------:|:----------------:|:------------------:|:------------------:|:-------------------------:|:------------:|:--------------:|:----------------:|:------------------:|:-------------:|:---------------:|
| 4.4576 | 0.4188 | 200 | 0.5649 | 286.9760 | 246.0880 | 40.8880 | 0.0 | -391.9658 | -416.9744 | -288.6415 | -265.9616 | -0.8860 | -0.8662 |
| 4.2579 | 0.8377 | 400 | 0.5327 | 286.9760 | 246.0880 | 40.8880 | 0.0 | -414.4475 | -451.4492 | -288.6415 | -265.9616 | -0.8584 | -0.8411 |
### Framework versions
- Transformers 4.51.0
- Pytorch 2.3.1+cu121
- Datasets 2.21.0
- Tokenizers 0.21.4

24
all_results.json Normal file
View File

@@ -0,0 +1,24 @@
{
"epoch": 0.9989528795811519,
"eval_logits/chosen": -0.8675644993782043,
"eval_logits/rejected": -0.8504053950309753,
"eval_logps/chosen": -423.36651611328125,
"eval_logps/ref_chosen": -288.6414794921875,
"eval_logps/ref_rejected": -265.96160888671875,
"eval_logps/rejected": -462.2294616699219,
"eval_loss": 0.5316001772880554,
"eval_r_dpo/chosen_len": 286.97601318359375,
"eval_r_dpo/length_delta": 40.88800048828125,
"eval_r_dpo/regularization_term": 0.0,
"eval_r_dpo/rejected_len": 246.08799743652344,
"eval_runtime": 78.0724,
"eval_samples": 2000,
"eval_samples_per_second": 25.617,
"eval_steps_per_second": 3.202,
"total_flos": 0.0,
"train_loss": 4.583878276233153,
"train_runtime": 6810.0393,
"train_samples": 61135,
"train_samples_per_second": 8.977,
"train_steps_per_second": 0.07
}

29
config.json Normal file
View File

@@ -0,0 +1,29 @@
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 128000,
"eos_token_id": 128001,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"max_position_embeddings": 8192,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 500000.0,
"tie_word_embeddings": false,
"torch_dtype": "float32",
"transformers_version": "4.51.0",
"use_cache": true,
"vocab_size": 128256
}

18
eval_results.json Normal file
View File

@@ -0,0 +1,18 @@
{
"epoch": 0.9989528795811519,
"eval_logits/chosen": -0.8675644993782043,
"eval_logits/rejected": -0.8504053950309753,
"eval_logps/chosen": -423.36651611328125,
"eval_logps/ref_chosen": -288.6414794921875,
"eval_logps/ref_rejected": -265.96160888671875,
"eval_logps/rejected": -462.2294616699219,
"eval_loss": 0.5316001772880554,
"eval_r_dpo/chosen_len": 286.97601318359375,
"eval_r_dpo/length_delta": 40.88800048828125,
"eval_r_dpo/regularization_term": 0.0,
"eval_r_dpo/rejected_len": 246.08799743652344,
"eval_runtime": 78.0724,
"eval_samples": 2000,
"eval_samples_per_second": 25.617,
"eval_steps_per_second": 3.202
}

9
generation_config.json Normal file
View File

@@ -0,0 +1,9 @@
{
"bos_token_id": 128000,
"do_sample": true,
"eos_token_id": 128001,
"max_length": 4096,
"temperature": 0.6,
"top_p": 0.9,
"transformers_version": "4.51.0"
}

3
model-00001-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:671065aee4a9d698bd3a35561fe34160c97e7ba1dba3ed9c7a9a3ae3f65712c4
size 4886466168

3
model-00002-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:da4814d706f1d2a69484fbcd4bc6597567a89e9d498949095c0424fd1451899c
size 4832007448

3
model-00003-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f5059dfaf3a1efee7a4362afa72035aa3cd9084f8799111c7ad9ee2de80dfe49
size 4999813112

3
model-00004-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e46db811f06e3b690fc164696b019b19af44d777d25dd0bf8d99267ea9f4d30d
size 4999813128

3
model-00005-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:22b010d61479c6d78c09c1ef5a51ca761b13250f75bab188f71040d5f31404f3
size 4832007496

3
model-00006-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2612435cfa2775ec8b09a5bde3d26b337120e0f7c269e829848534d024623954
size 4999813120

3
model-00007-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:80540f5adacd539e613e53c8274692bd0842612e9355005dd830b740f68b03bb
size 2571158184

298
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,298 @@
{
"metadata": {
"total_size": 32121044992
},
"weight_map": {
"lm_head.weight": "model-00007-of-00007.safetensors",
"model.embed_tokens.weight": "model-00001-of-00007.safetensors",
"model.layers.0.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.1.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.10.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.10.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.10.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.10.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.10.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.11.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.11.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.11.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.11.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.11.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.12.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.12.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.12.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.12.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.12.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.13.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.13.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.13.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.13.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.13.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.14.input_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.14.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.14.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.14.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.14.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.15.input_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.15.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.15.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.15.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.15.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.16.input_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.16.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.16.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.16.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.16.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.17.input_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.17.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.17.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.17.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.17.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.18.input_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.18.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.18.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.19.input_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.19.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.19.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.19.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.19.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.2.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.20.input_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.20.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.20.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.20.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.20.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.21.input_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.21.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.21.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.21.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.21.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.22.input_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.22.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.22.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.22.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.22.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.23.input_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.23.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.23.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.23.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.23.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.24.input_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.24.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.24.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.24.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.24.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.24.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.24.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.24.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.24.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.25.input_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.25.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.25.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.25.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.25.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.25.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.25.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.25.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.25.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.26.input_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.26.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.26.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.26.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.26.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.26.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.26.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.26.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.26.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.27.input_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.27.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.27.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.27.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.27.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.27.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.27.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.27.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.27.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.28.input_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.28.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.28.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.28.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.28.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.28.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.28.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.28.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.28.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.29.input_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.29.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.29.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.29.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.29.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.29.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.29.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.29.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.29.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.3.input_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.3.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.3.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.30.input_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.30.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.30.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.30.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.30.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.30.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.30.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.30.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.30.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.31.input_layernorm.weight": "model-00007-of-00007.safetensors",
"model.layers.31.mlp.down_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.31.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.31.mlp.up_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.31.post_attention_layernorm.weight": "model-00007-of-00007.safetensors",
"model.layers.31.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.31.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.31.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.31.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.4.input_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.4.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.4.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.4.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.4.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.5.input_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.5.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.5.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.5.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.5.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.6.input_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.6.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.6.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.6.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.6.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.7.input_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.7.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.7.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.7.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.7.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.8.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.8.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.8.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.8.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.8.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.9.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.9.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.9.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.norm.weight": "model-00007-of-00007.safetensors"
}
}

23
special_tokens_map.json Normal file
View File

@@ -0,0 +1,23 @@
{
"bos_token": {
"content": "<|begin_of_text|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|end_of_text|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|end_of_text|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3c5cf44023714fb39b05e71e425f8d7b92805ff73f7988b083b8c87f0bf87393
size 17209961

2064
tokenizer_config.json Normal file
View File

File diff suppressed because it is too large Load Diff

1357
train.log Normal file
View File

File diff suppressed because one or more lines are too long

9
train_results.json Normal file
View File

@@ -0,0 +1,9 @@
{
"epoch": 0.9989528795811519,
"total_flos": 0.0,
"train_loss": 4.583878276233153,
"train_runtime": 6810.0393,
"train_samples": 61135,
"train_samples_per_second": 8.977,
"train_steps_per_second": 0.07
}

895
trainer_state.json Normal file
View File

@@ -0,0 +1,895 @@
{
"best_global_step": null,
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 0.9989528795811519,
"eval_steps": 200,
"global_step": 477,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.0020942408376963353,
"grad_norm": 28.589035034179688,
"learning_rate": 0.0,
"logits/chosen": -0.5995081663131714,
"logits/rejected": -0.6144353747367859,
"logps/chosen": -267.5272216796875,
"logps/ref_chosen": -267.5935363769531,
"logps/ref_rejected": -204.2306671142578,
"logps/rejected": -204.23907470703125,
"loss": 5.5463,
"r_dpo/chosen_len": 257.75,
"r_dpo/length_delta": 47.875,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 209.875,
"step": 1
},
{
"epoch": 0.020942408376963352,
"grad_norm": 26.56291389465332,
"learning_rate": 9.375e-08,
"logits/chosen": -0.6324527263641357,
"logits/rejected": -0.6372823119163513,
"logps/chosen": -296.603759765625,
"logps/ref_chosen": -296.63226318359375,
"logps/ref_rejected": -258.9539489746094,
"logps/rejected": -259.0047302246094,
"loss": 5.5445,
"r_dpo/chosen_len": 291.8680419921875,
"r_dpo/length_delta": 49.76388931274414,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 242.1041717529297,
"step": 10
},
{
"epoch": 0.041884816753926704,
"grad_norm": 29.713520050048828,
"learning_rate": 1.9791666666666664e-07,
"logits/chosen": -0.5963870286941528,
"logits/rejected": -0.6269619464874268,
"logps/chosen": -297.92315673828125,
"logps/ref_chosen": -297.9349365234375,
"logps/ref_rejected": -256.9902648925781,
"logps/rejected": -256.97802734375,
"loss": 5.5435,
"r_dpo/chosen_len": 291.29998779296875,
"r_dpo/length_delta": 52.89374923706055,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 238.40625,
"step": 20
},
{
"epoch": 0.06282722513089005,
"grad_norm": 28.98917007446289,
"learning_rate": 3.020833333333333e-07,
"logits/chosen": -0.6142657995223999,
"logits/rejected": -0.6058592796325684,
"logps/chosen": -278.4171142578125,
"logps/ref_chosen": -278.64752197265625,
"logps/ref_rejected": -249.309814453125,
"logps/rejected": -249.23779296875,
"loss": 5.5396,
"r_dpo/chosen_len": 270.8812561035156,
"r_dpo/length_delta": 25.228124618530273,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 245.6531219482422,
"step": 30
},
{
"epoch": 0.08376963350785341,
"grad_norm": 27.35612678527832,
"learning_rate": 4.0625e-07,
"logits/chosen": -0.6192952394485474,
"logits/rejected": -0.644347071647644,
"logps/chosen": -282.6344299316406,
"logps/ref_chosen": -283.49981689453125,
"logps/ref_rejected": -265.32733154296875,
"logps/rejected": -265.03369140625,
"loss": 5.521,
"r_dpo/chosen_len": 281.43438720703125,
"r_dpo/length_delta": 33.34375,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 248.0906219482422,
"step": 40
},
{
"epoch": 0.10471204188481675,
"grad_norm": 27.939252853393555,
"learning_rate": 4.999932966293553e-07,
"logits/chosen": -0.6302677392959595,
"logits/rejected": -0.6705285310745239,
"logps/chosen": -278.43548583984375,
"logps/ref_chosen": -280.224365234375,
"logps/ref_rejected": -274.3541259765625,
"logps/rejected": -273.73004150390625,
"loss": 5.4954,
"r_dpo/chosen_len": 290.32501220703125,
"r_dpo/length_delta": 35.11249923706055,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 255.21249389648438,
"step": 50
},
{
"epoch": 0.1256544502617801,
"grad_norm": 27.91963005065918,
"learning_rate": 4.991893270335525e-07,
"logits/chosen": -0.6450083255767822,
"logits/rejected": -0.6583200693130493,
"logps/chosen": -278.49346923828125,
"logps/ref_chosen": -281.12664794921875,
"logps/ref_rejected": -259.86456298828125,
"logps/rejected": -259.6600646972656,
"loss": 5.4458,
"r_dpo/chosen_len": 273.953125,
"r_dpo/length_delta": 29.084375381469727,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 244.86874389648438,
"step": 60
},
{
"epoch": 0.14659685863874344,
"grad_norm": 28.88052749633789,
"learning_rate": 4.970496218214204e-07,
"logits/chosen": -0.7053675055503845,
"logits/rejected": -0.7107682228088379,
"logps/chosen": -283.94683837890625,
"logps/ref_chosen": -287.71063232421875,
"logps/ref_rejected": -276.839599609375,
"logps/rejected": -277.0175476074219,
"loss": 5.3873,
"r_dpo/chosen_len": 267.4937438964844,
"r_dpo/length_delta": 14.484375,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 253.00936889648438,
"step": 70
},
{
"epoch": 0.16753926701570682,
"grad_norm": 28.927474975585938,
"learning_rate": 4.935856505068998e-07,
"logits/chosen": -0.6918989419937134,
"logits/rejected": -0.6877058148384094,
"logps/chosen": -276.62353515625,
"logps/ref_chosen": -280.123046875,
"logps/ref_rejected": -258.8989562988281,
"logps/rejected": -260.3608093261719,
"loss": 5.3156,
"r_dpo/chosen_len": 267.4781188964844,
"r_dpo/length_delta": 32.46562576293945,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 235.0124969482422,
"step": 80
},
{
"epoch": 0.18848167539267016,
"grad_norm": 29.801456451416016,
"learning_rate": 4.8881598109976e-07,
"logits/chosen": -0.715398907661438,
"logits/rejected": -0.7198300361633301,
"logps/chosen": -277.6268615722656,
"logps/ref_chosen": -278.02545166015625,
"logps/ref_rejected": -251.0922393798828,
"logps/rejected": -258.9493713378906,
"loss": 5.2562,
"r_dpo/chosen_len": 274.20623779296875,
"r_dpo/length_delta": 44.97187423706055,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 229.234375,
"step": 90
},
{
"epoch": 0.2094240837696335,
"grad_norm": 35.680721282958984,
"learning_rate": 4.827661805750437e-07,
"logits/chosen": -0.7235929369926453,
"logits/rejected": -0.7395325303077698,
"logps/chosen": -277.54632568359375,
"logps/ref_chosen": -274.0089416503906,
"logps/ref_rejected": -274.14447021484375,
"logps/rejected": -288.9579162597656,
"loss": 5.1804,
"r_dpo/chosen_len": 275.3343811035156,
"r_dpo/length_delta": 21.912500381469727,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 253.421875,
"step": 100
},
{
"epoch": 0.23036649214659685,
"grad_norm": 34.81735610961914,
"learning_rate": 4.75468677825789e-07,
"logits/chosen": -0.7712054252624512,
"logits/rejected": -0.7870631814002991,
"logps/chosen": -280.66912841796875,
"logps/ref_chosen": -273.23333740234375,
"logps/ref_rejected": -263.88787841796875,
"logps/rejected": -287.0477600097656,
"loss": 5.0027,
"r_dpo/chosen_len": 283.43438720703125,
"r_dpo/length_delta": 50.34375,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 233.0906219482422,
"step": 110
},
{
"epoch": 0.2513089005235602,
"grad_norm": 41.90164566040039,
"learning_rate": 4.669625898336438e-07,
"logits/chosen": -0.8202114105224609,
"logits/rejected": -0.8147541284561157,
"logps/chosen": -291.4042663574219,
"logps/ref_chosen": -269.77142333984375,
"logps/ref_rejected": -272.7685546875,
"logps/rejected": -311.06072998046875,
"loss": 4.9989,
"r_dpo/chosen_len": 264.7593688964844,
"r_dpo/length_delta": 13.840624809265137,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 250.9187469482422,
"step": 120
},
{
"epoch": 0.27225130890052357,
"grad_norm": 57.423763275146484,
"learning_rate": 4.5729351198915705e-07,
"logits/chosen": -0.8498390316963196,
"logits/rejected": -0.8324364423751831,
"logps/chosen": -301.84613037109375,
"logps/ref_chosen": -275.03448486328125,
"logps/ref_rejected": -276.39862060546875,
"logps/rejected": -325.33062744140625,
"loss": 4.8776,
"r_dpo/chosen_len": 266.625,
"r_dpo/length_delta": 18.668750762939453,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 247.9562530517578,
"step": 130
},
{
"epoch": 0.2931937172774869,
"grad_norm": 60.88969039916992,
"learning_rate": 4.4651327368569684e-07,
"logits/chosen": -0.8470001220703125,
"logits/rejected": -0.8457162976264954,
"logps/chosen": -308.84027099609375,
"logps/ref_chosen": -276.0029602050781,
"logps/ref_rejected": -255.9320526123047,
"logps/rejected": -314.761962890625,
"loss": 4.8439,
"r_dpo/chosen_len": 261.46875,
"r_dpo/length_delta": 22.375,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 239.09375,
"step": 140
},
{
"epoch": 0.31413612565445026,
"grad_norm": 59.7264518737793,
"learning_rate": 4.346796604970912e-07,
"logits/chosen": -0.8876619338989258,
"logits/rejected": -0.8721216320991516,
"logps/chosen": -330.7905578613281,
"logps/ref_chosen": -298.2093505859375,
"logps/ref_rejected": -254.8907012939453,
"logps/rejected": -320.9139709472656,
"loss": 4.7236,
"r_dpo/chosen_len": 283.84375,
"r_dpo/length_delta": 48.359375,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 235.484375,
"step": 150
},
{
"epoch": 0.33507853403141363,
"grad_norm": 58.573604583740234,
"learning_rate": 4.218561044282098e-07,
"logits/chosen": -0.8934988975524902,
"logits/rejected": -0.8782498240470886,
"logps/chosen": -337.43865966796875,
"logps/ref_chosen": -281.94189453125,
"logps/ref_rejected": -255.5653533935547,
"logps/rejected": -353.12567138671875,
"loss": 4.4456,
"r_dpo/chosen_len": 267.828125,
"r_dpo/length_delta": 41.368751525878906,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 226.45938110351562,
"step": 160
},
{
"epoch": 0.35602094240837695,
"grad_norm": 92.63309478759766,
"learning_rate": 4.081113438988443e-07,
"logits/chosen": -0.851898193359375,
"logits/rejected": -0.8330786824226379,
"logps/chosen": -346.3147888183594,
"logps/ref_chosen": -288.2863464355469,
"logps/ref_rejected": -239.758056640625,
"logps/rejected": -337.37396240234375,
"loss": 4.4733,
"r_dpo/chosen_len": 285.203125,
"r_dpo/length_delta": 46.396873474121094,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 238.80624389648438,
"step": 170
},
{
"epoch": 0.3769633507853403,
"grad_norm": 93.2479019165039,
"learning_rate": 3.935190552834828e-07,
"logits/chosen": -0.8184630274772644,
"logits/rejected": -0.8205466270446777,
"logps/chosen": -341.13372802734375,
"logps/ref_chosen": -286.17889404296875,
"logps/ref_rejected": -249.9820098876953,
"logps/rejected": -348.2437438964844,
"loss": 4.512,
"r_dpo/chosen_len": 266.09063720703125,
"r_dpo/length_delta": 40.12812423706055,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 225.96249389648438,
"step": 180
},
{
"epoch": 0.39790575916230364,
"grad_norm": 80.11067962646484,
"learning_rate": 3.781574579820464e-07,
"logits/chosen": -0.859279453754425,
"logits/rejected": -0.8603144884109497,
"logps/chosen": -355.4273376464844,
"logps/ref_chosen": -280.9278259277344,
"logps/ref_rejected": -254.3533477783203,
"logps/rejected": -383.27703857421875,
"loss": 4.3425,
"r_dpo/chosen_len": 276.33123779296875,
"r_dpo/length_delta": 41.993751525878906,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 234.33749389648438,
"step": 190
},
{
"epoch": 0.418848167539267,
"grad_norm": 117.87115478515625,
"learning_rate": 3.621088951385353e-07,
"logits/chosen": -0.8809002041816711,
"logits/rejected": -0.8806599378585815,
"logps/chosen": -334.10260009765625,
"logps/ref_chosen": -253.1712188720703,
"logps/ref_rejected": -241.90478515625,
"logps/rejected": -369.25811767578125,
"loss": 4.4576,
"r_dpo/chosen_len": 248.0749969482422,
"r_dpo/length_delta": 28.131250381469727,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 219.94375610351562,
"step": 200
},
{
"epoch": 0.418848167539267,
"eval_logits/chosen": -0.8859605193138123,
"eval_logits/rejected": -0.8661972880363464,
"eval_logps/chosen": -391.96575927734375,
"eval_logps/ref_chosen": -288.6414794921875,
"eval_logps/ref_rejected": -265.96160888671875,
"eval_logps/rejected": -416.974365234375,
"eval_loss": 0.5648660659790039,
"eval_r_dpo/chosen_len": 286.97601318359375,
"eval_r_dpo/length_delta": 40.88800048828125,
"eval_r_dpo/regularization_term": 0.0,
"eval_r_dpo/rejected_len": 246.08799743652344,
"eval_runtime": 78.8271,
"eval_samples_per_second": 25.372,
"eval_steps_per_second": 3.171,
"step": 200
},
{
"epoch": 0.4397905759162304,
"grad_norm": 102.8453140258789,
"learning_rate": 3.454593922550693e-07,
"logits/chosen": -0.8247052431106567,
"logits/rejected": -0.8323475122451782,
"logps/chosen": -390.46563720703125,
"logps/ref_chosen": -287.9228210449219,
"logps/ref_rejected": -263.35595703125,
"logps/rejected": -411.89306640625,
"loss": 4.5528,
"r_dpo/chosen_len": 280.3125,
"r_dpo/length_delta": 36.68437576293945,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 243.6281280517578,
"step": 210
},
{
"epoch": 0.4607329842931937,
"grad_norm": 84.93110656738281,
"learning_rate": 3.2829819606729477e-07,
"logits/chosen": -0.8513854742050171,
"logits/rejected": -0.8432670831680298,
"logps/chosen": -361.94427490234375,
"logps/ref_chosen": -282.3331604003906,
"logps/ref_rejected": -272.5645446777344,
"logps/rejected": -407.8734436035156,
"loss": 4.3287,
"r_dpo/chosen_len": 261.359375,
"r_dpo/length_delta": 17.865625381469727,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 243.49374389648438,
"step": 220
},
{
"epoch": 0.4816753926701571,
"grad_norm": 88.449951171875,
"learning_rate": 3.1071729615293424e-07,
"logits/chosen": -0.8409557342529297,
"logits/rejected": -0.8231566548347473,
"logps/chosen": -375.97259521484375,
"logps/ref_chosen": -276.1485595703125,
"logps/ref_rejected": -252.81198120117188,
"logps/rejected": -408.45989990234375,
"loss": 4.2955,
"r_dpo/chosen_len": 264.43438720703125,
"r_dpo/length_delta": 31.256250381469727,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 233.17813110351562,
"step": 230
},
{
"epoch": 0.5026178010471204,
"grad_norm": 87.3523941040039,
"learning_rate": 2.9281093183781403e-07,
"logits/chosen": -0.8152298927307129,
"logits/rejected": -0.8264015316963196,
"logps/chosen": -361.360595703125,
"logps/ref_chosen": -270.52520751953125,
"logps/ref_rejected": -254.83334350585938,
"logps/rejected": -398.59173583984375,
"loss": 4.3402,
"r_dpo/chosen_len": 271.81561279296875,
"r_dpo/length_delta": 37.099998474121094,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 234.7156219482422,
"step": 240
},
{
"epoch": 0.5235602094240838,
"grad_norm": 88.13154602050781,
"learning_rate": 2.7467508704251135e-07,
"logits/chosen": -0.845689594745636,
"logits/rejected": -0.8341258764266968,
"logps/chosen": -376.0411682128906,
"logps/ref_chosen": -289.6054992675781,
"logps/ref_rejected": -265.0482482910156,
"logps/rejected": -409.2091369628906,
"loss": 4.3706,
"r_dpo/chosen_len": 277.50311279296875,
"r_dpo/length_delta": 41.103126525878906,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 236.39999389648438,
"step": 250
},
{
"epoch": 0.5445026178010471,
"grad_norm": 99.26053619384766,
"learning_rate": 2.5640697577740815e-07,
"logits/chosen": -0.8479117155075073,
"logits/rejected": -0.8312094807624817,
"logps/chosen": -401.3951110839844,
"logps/ref_chosen": -288.6393737792969,
"logps/ref_rejected": -265.315673828125,
"logps/rejected": -437.06854248046875,
"loss": 4.3553,
"r_dpo/chosen_len": 271.48126220703125,
"r_dpo/length_delta": 24.390625,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 247.0906219482422,
"step": 260
},
{
"epoch": 0.5654450261780105,
"grad_norm": 88.68135070800781,
"learning_rate": 2.381045210440644e-07,
"logits/chosen": -0.8226224184036255,
"logits/rejected": -0.8202828168869019,
"logps/chosen": -395.2716064453125,
"logps/ref_chosen": -280.1373596191406,
"logps/ref_rejected": -264.84295654296875,
"logps/rejected": -442.5419921875,
"loss": 4.228,
"r_dpo/chosen_len": 272.2875061035156,
"r_dpo/length_delta": 19.956249237060547,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 252.3312530517578,
"step": 270
},
{
"epoch": 0.5863874345549738,
"grad_norm": 84.24311828613281,
"learning_rate": 2.1986582993616925e-07,
"logits/chosen": -0.8553133010864258,
"logits/rejected": -0.8398975133895874,
"logps/chosen": -408.2679748535156,
"logps/ref_chosen": -301.7547912597656,
"logps/ref_rejected": -254.6543731689453,
"logps/rejected": -426.813720703125,
"loss": 4.2273,
"r_dpo/chosen_len": 285.44061279296875,
"r_dpo/length_delta": 52.962501525878906,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 232.47811889648438,
"step": 280
},
{
"epoch": 0.6073298429319371,
"grad_norm": 103.96916198730469,
"learning_rate": 2.0178866775369774e-07,
"logits/chosen": -0.8476747274398804,
"logits/rejected": -0.8177559971809387,
"logps/chosen": -426.84906005859375,
"logps/ref_chosen": -302.79217529296875,
"logps/ref_rejected": -292.9220275878906,
"logps/rejected": -473.33697509765625,
"loss": 4.4579,
"r_dpo/chosen_len": 294.90625,
"r_dpo/length_delta": 20.774999618530273,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 274.1312561035156,
"step": 290
},
{
"epoch": 0.6282722513089005,
"grad_norm": 112.53483581542969,
"learning_rate": 1.839699339491937e-07,
"logits/chosen": -0.8564668893814087,
"logits/rejected": -0.8317262530326843,
"logps/chosen": -385.36322021484375,
"logps/ref_chosen": -275.8238220214844,
"logps/ref_rejected": -264.05743408203125,
"logps/rejected": -432.60552978515625,
"loss": 4.251,
"r_dpo/chosen_len": 266.859375,
"r_dpo/length_delta": 20.734375,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 246.125,
"step": 300
},
{
"epoch": 0.6492146596858639,
"grad_norm": 88.61668395996094,
"learning_rate": 1.6650514271527465e-07,
"logits/chosen": -0.8322170376777649,
"logits/rejected": -0.8294069170951843,
"logps/chosen": -419.35638427734375,
"logps/ref_chosen": -296.6716003417969,
"logps/ref_rejected": -278.68426513671875,
"logps/rejected": -460.2979431152344,
"loss": 4.1383,
"r_dpo/chosen_len": 292.91876220703125,
"r_dpo/length_delta": 32.55937576293945,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 260.359375,
"step": 310
},
{
"epoch": 0.6701570680628273,
"grad_norm": 88.22819519042969,
"learning_rate": 1.4948791099758052e-07,
"logits/chosen": -0.8486505746841431,
"logits/rejected": -0.8500319719314575,
"logps/chosen": -415.5774841308594,
"logps/ref_chosen": -284.1717529296875,
"logps/ref_rejected": -261.2606506347656,
"logps/rejected": -457.5267639160156,
"loss": 4.095,
"r_dpo/chosen_len": 279.90313720703125,
"r_dpo/length_delta": 44.537498474121094,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 235.36563110351562,
"step": 320
},
{
"epoch": 0.6910994764397905,
"grad_norm": 103.7956771850586,
"learning_rate": 1.3300945667758012e-07,
"logits/chosen": -0.8600236773490906,
"logits/rejected": -0.8557920455932617,
"logps/chosen": -416.3182678222656,
"logps/ref_chosen": -283.40338134765625,
"logps/ref_rejected": -271.27569580078125,
"logps/rejected": -467.2439880371094,
"loss": 4.2369,
"r_dpo/chosen_len": 267.67498779296875,
"r_dpo/length_delta": 13.015625,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 254.6593780517578,
"step": 330
},
{
"epoch": 0.7120418848167539,
"grad_norm": 103.91631317138672,
"learning_rate": 1.1715810961514072e-07,
"logits/chosen": -0.8616652488708496,
"logits/rejected": -0.845625102519989,
"logps/chosen": -396.5005187988281,
"logps/ref_chosen": -259.7261962890625,
"logps/ref_rejected": -243.4088897705078,
"logps/rejected": -445.0213317871094,
"loss": 4.2243,
"r_dpo/chosen_len": 256.11248779296875,
"r_dpo/length_delta": 32.546875,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 223.5656280517578,
"step": 340
},
{
"epoch": 0.7329842931937173,
"grad_norm": 87.64006805419922,
"learning_rate": 1.0201883817182949e-07,
"logits/chosen": -0.8792071342468262,
"logits/rejected": -0.869489312171936,
"logps/chosen": -430.8462829589844,
"logps/ref_chosen": -298.24725341796875,
"logps/ref_rejected": -272.657958984375,
"logps/rejected": -468.8990173339844,
"loss": 4.3118,
"r_dpo/chosen_len": 281.4624938964844,
"r_dpo/length_delta": 45.275001525878906,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 236.1875,
"step": 350
},
{
"epoch": 0.7539267015706806,
"grad_norm": 109.85051727294922,
"learning_rate": 8.76727937529367e-08,
"logits/chosen": -0.8390272855758667,
"logits/rejected": -0.83982914686203,
"logps/chosen": -407.44158935546875,
"logps/ref_chosen": -281.881103515625,
"logps/ref_rejected": -265.4746398925781,
"logps/rejected": -459.4778747558594,
"loss": 4.3315,
"r_dpo/chosen_len": 272.64373779296875,
"r_dpo/length_delta": 30.071874618530273,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 242.57186889648438,
"step": 360
},
{
"epoch": 0.774869109947644,
"grad_norm": 75.16207885742188,
"learning_rate": 7.419687580962222e-08,
"logits/chosen": -0.8644768595695496,
"logits/rejected": -0.8538848161697388,
"logps/chosen": -422.214111328125,
"logps/ref_chosen": -302.17822265625,
"logps/ref_rejected": -265.92877197265625,
"logps/rejected": -455.0023498535156,
"loss": 4.0813,
"r_dpo/chosen_len": 273.88751220703125,
"r_dpo/length_delta": 33.759376525878906,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 240.1281280517578,
"step": 370
},
{
"epoch": 0.7958115183246073,
"grad_norm": 131.6292266845703,
"learning_rate": 6.166331963291519e-08,
"logits/chosen": -0.8290479779243469,
"logits/rejected": -0.8196717500686646,
"logps/chosen": -419.83135986328125,
"logps/ref_chosen": -301.2120361328125,
"logps/ref_rejected": -266.4872741699219,
"logps/rejected": -442.3086853027344,
"loss": 4.2961,
"r_dpo/chosen_len": 286.75311279296875,
"r_dpo/length_delta": 33.53125,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 253.2218780517578,
"step": 380
},
{
"epoch": 0.8167539267015707,
"grad_norm": 74.49371337890625,
"learning_rate": 5.013930914912476e-08,
"logits/chosen": -0.84967440366745,
"logits/rejected": -0.8341225385665894,
"logps/chosen": -414.8948669433594,
"logps/ref_chosen": -296.6472473144531,
"logps/ref_rejected": -278.953857421875,
"logps/rejected": -463.76885986328125,
"loss": 4.1709,
"r_dpo/chosen_len": 287.91876220703125,
"r_dpo/length_delta": 30.081249237060547,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 257.8374938964844,
"step": 390
},
{
"epoch": 0.837696335078534,
"grad_norm": 97.45182037353516,
"learning_rate": 3.968661679220467e-08,
"logits/chosen": -0.8492805361747742,
"logits/rejected": -0.8483866453170776,
"logps/chosen": -420.40008544921875,
"logps/ref_chosen": -296.6556091308594,
"logps/ref_rejected": -256.9266662597656,
"logps/rejected": -440.20458984375,
"loss": 4.2579,
"r_dpo/chosen_len": 278.96875,
"r_dpo/length_delta": 39.60625076293945,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 239.3625030517578,
"step": 400
},
{
"epoch": 0.837696335078534,
"eval_logits/chosen": -0.8584261536598206,
"eval_logits/rejected": -0.8411309719085693,
"eval_logps/chosen": -414.447509765625,
"eval_logps/ref_chosen": -288.6414794921875,
"eval_logps/ref_rejected": -265.96160888671875,
"eval_logps/rejected": -451.4491882324219,
"eval_loss": 0.5327035188674927,
"eval_r_dpo/chosen_len": 286.97601318359375,
"eval_r_dpo/length_delta": 40.88800048828125,
"eval_r_dpo/regularization_term": 0.0,
"eval_r_dpo/rejected_len": 246.08799743652344,
"eval_runtime": 78.3571,
"eval_samples_per_second": 25.524,
"eval_steps_per_second": 3.191,
"step": 400
},
{
"epoch": 0.8586387434554974,
"grad_norm": 85.77227020263672,
"learning_rate": 3.036127238347164e-08,
"logits/chosen": -0.830724835395813,
"logits/rejected": -0.8160354495048523,
"logps/chosen": -421.79754638671875,
"logps/ref_chosen": -289.9568786621094,
"logps/ref_rejected": -272.4674377441406,
"logps/rejected": -459.7171936035156,
"loss": 4.1428,
"r_dpo/chosen_len": 282.40625,
"r_dpo/length_delta": 26.265625,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 256.140625,
"step": 410
},
{
"epoch": 0.8795811518324608,
"grad_norm": 128.62335205078125,
"learning_rate": 2.2213262793589482e-08,
"logits/chosen": -0.8531728982925415,
"logits/rejected": -0.8363476991653442,
"logps/chosen": -435.88006591796875,
"logps/ref_chosen": -307.40240478515625,
"logps/ref_rejected": -279.85760498046875,
"logps/rejected": -469.5859375,
"loss": 4.1314,
"r_dpo/chosen_len": 296.8343811035156,
"r_dpo/length_delta": 37.165626525878906,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 259.66876220703125,
"step": 420
},
{
"epoch": 0.900523560209424,
"grad_norm": 97.44547271728516,
"learning_rate": 1.5286263996730026e-08,
"logits/chosen": -0.842852771282196,
"logits/rejected": -0.8340854644775391,
"logps/chosen": -426.78375244140625,
"logps/ref_chosen": -297.7133483886719,
"logps/ref_rejected": -266.862060546875,
"logps/rejected": -465.3374938964844,
"loss": 4.0648,
"r_dpo/chosen_len": 290.703125,
"r_dpo/length_delta": 48.868751525878906,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 241.83438110351562,
"step": 430
},
{
"epoch": 0.9214659685863874,
"grad_norm": 95.01736450195312,
"learning_rate": 9.617406953185136e-09,
"logits/chosen": -0.8415233492851257,
"logits/rejected": -0.8436342477798462,
"logps/chosen": -426.42633056640625,
"logps/ref_chosen": -293.67095947265625,
"logps/ref_rejected": -289.4698791503906,
"logps/rejected": -479.0205993652344,
"loss": 4.2677,
"r_dpo/chosen_len": 285.3656311035156,
"r_dpo/length_delta": 12.868749618530273,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 272.49688720703125,
"step": 440
},
{
"epoch": 0.9424083769633508,
"grad_norm": 91.81388092041016,
"learning_rate": 5.2370785753763356e-09,
"logits/chosen": -0.8524943590164185,
"logits/rejected": -0.8484461903572083,
"logps/chosen": -422.54278564453125,
"logps/ref_chosen": -296.9415283203125,
"logps/ref_rejected": -262.6710510253906,
"logps/rejected": -453.1717834472656,
"loss": 4.0939,
"r_dpo/chosen_len": 282.81561279296875,
"r_dpo/length_delta": 40.631248474121094,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 242.1843719482422,
"step": 450
},
{
"epoch": 0.9633507853403142,
"grad_norm": 85.33897399902344,
"learning_rate": 2.168758844148272e-09,
"logits/chosen": -0.866470992565155,
"logits/rejected": -0.8637819290161133,
"logps/chosen": -444.849609375,
"logps/ref_chosen": -312.42291259765625,
"logps/ref_rejected": -278.7356262207031,
"logps/rejected": -475.77838134765625,
"loss": 4.2438,
"r_dpo/chosen_len": 288.9125061035156,
"r_dpo/length_delta": 43.23125076293945,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 245.68124389648438,
"step": 460
},
{
"epoch": 0.9842931937172775,
"grad_norm": 81.34379577636719,
"learning_rate": 4.288949484559934e-10,
"logits/chosen": -0.832243800163269,
"logits/rejected": -0.8189598321914673,
"logps/chosen": -404.83941650390625,
"logps/ref_chosen": -278.0654602050781,
"logps/ref_rejected": -256.5596618652344,
"logps/rejected": -447.73919677734375,
"loss": 4.1491,
"r_dpo/chosen_len": 268.8687438964844,
"r_dpo/length_delta": 26.262500762939453,
"r_dpo/regularization_term": 0.0,
"r_dpo/rejected_len": 242.6062469482422,
"step": 470
},
{
"epoch": 0.9989528795811519,
"step": 477,
"total_flos": 0.0,
"train_loss": 4.583878276233153,
"train_runtime": 6810.0393,
"train_samples_per_second": 8.977,
"train_steps_per_second": 0.07
}
],
"logging_steps": 10,
"max_steps": 477,
"num_input_tokens_seen": 0,
"num_train_epochs": 1,
"save_steps": 200,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": true
},
"attributes": {}
}
},
"total_flos": 0.0,
"train_batch_size": 4,
"trial_name": null,
"trial_params": null
}