swift/MS-LongWriter-Qwen2-7B-Instruct
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: swift/MS-LongWriter-Qwen2-7B-Instruct
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-11 12:19:13 +08:00
commit 89c3260676
22 changed files with 456268 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
.gitattributes vendored Normal file
View File

@@ -0,0 +1,38 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.gguf* filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text

76
README.md Normal file
View File

@@ -0,0 +1,76 @@
---
language:
- en
- zh
library_name: transformers
tags:
- Long Context
- qwen2
---
# MS-LongWriter-Qwen2-7B-Instruct
<p align="center">
🤖 <a href="https://modelscope.cn/datasets/swift/longwriter-6k-filtered" target="_blank">[LongWriter Dataset] </a> • 💻 <a href="https://github.com/THUDM/LongWriter" target="_blank">[Github Repo]</a> • 📃 <a href="https://arxiv.org/abs/2408.07055" target="_blank">[LongWriter Paper]</a>
</p>
MS-LongWriter-Qwen2-7B-Instruct is trained based on [https://modelscope.cn/models/qwen/Qwen2-7B-Instruct](https://modelscope.cn/models/qwen/Qwen2-7B-Instruct), and is capable of generating 10,000+ words at once.
MS-LongWriter-Qwen2-7B-Instruct begins training directly from the Qwen2-7B-Instruct, while performing significant distillation on the [LongWriter-6k](https://modelscope.cn/datasets/ZhipuAI/LongWriter-6k) to obtain 666 high-quality samples.
## Datasets
1. [LongWriter-6k-filtered](https://modelscope.cn/datasets/swift/longwriter-6k-filtered), which is based on the [LongWriter-6k](https://modelscope.cn/datasets/ZhipuAI/LongWriter-6k)
2. [Magpie-Qwen2-Pro-200K-Chinese](https://modelscope.cn/datasets/AI-ModelScope/Magpie-Qwen2-Pro-200K-Chinese) , random sampling 6k examples.
3. [Magpie-Qwen2-Pro-200K-English](https://modelscope.cn/datasets/AI-ModelScope/Magpie-Qwen2-Pro-200K-English) , random sampling 6k examples.
## Model
We use [ms-swift](https://github.com/modelscope/swift) to fine-tune the Qwen2-7B-Instruct model.
1. Installation
```python
pip install ms-swift[llm]
```
2. Fine-tuning
Envs:
```text
Nvidia A100(80G) x 4
```
Run:
```shell
swift sft \
--model_type qwen2-7b-instruct \
--dataset longwriter-6k-filtered#666 qwen2-pro-zh#6660 qwen2-pro-en#6660 \
--max_length 28672 \
--num_train_epochs 2 \
--eval_steps 200 \
--batch_size 1 \
--gradient_accumulation_steps 64 \
--gradient_checkpointing true \
--warmup_ratio 0.1 \
--learning_rate 1e-5 \
--sft_type full \
--loss_name long-ce \
--check_dataset_strategy warning \
--save_only_model false \
--save_total_limit -1 \
--lazy_tokenize true \
--dataloader_num_workers 1 \
--resume_only_model true \
--neftune_noise_alpha 5 \
--use_flash_attn true
```
## Evaluation
Refer to [LongWriter Evaluation](https://github.com/modelscope/evalscope/tree/main/evalscope/third_party/longbench_write) from the [EvalScope](https://github.com/modelscope/evalscope).

5
added_tokens.json Normal file
View File

@@ -0,0 +1,5 @@
{
"<|endoftext|>": 151643,
"<|im_end|>": 151645,
"<|im_start|>": 151644
}

28
config.json Normal file
View File

@@ -0,0 +1,28 @@
{
"_name_or_path": "/mnt/nas2/huangjintao.hjt/.cache/modelscope/hub/qwen/Qwen2-7B-Instruct",
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151645,
"hidden_act": "silu",
"hidden_size": 3584,
"initializer_range": 0.02,
"intermediate_size": 18944,
"max_position_embeddings": 32768,
"max_window_layers": 28,
"model_type": "qwen2",
"num_attention_heads": 28,
"num_hidden_layers": 28,
"num_key_value_heads": 4,
"rms_norm_eps": 1e-06,
"rope_theta": 1000000.0,
"sliding_window": 131072,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.42.4",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 152064
}

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework":"Pytorch","task":"text-generation"}

12
generation_config.json Normal file
View File

@@ -0,0 +1,12 @@
{
"bos_token_id": 151643,
"do_sample": true,
"eos_token_id": 151645,
"max_new_tokens": 2048,
"pad_token_id": 151643,
"repetition_penalty": 1.05,
"temperature": 0.7,
"top_k": 20,
"top_p": 0.8,
"transformers_version": "4.42.4"
}

151388
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model-00001-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ab0de5a00337c91e54b81ba4a681dd8c31d6dd6cdbfaae70ceda57a195207c96
size 4877660776

3
model-00002-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5582aac209a09cfc25e4195c5f4a2175c1b0149c6e2312b30a9b7b25fa4eeaf7
size 4932751008

3
model-00003-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5614bd6b55243f709f698b3cb4bf043840a62697348875b44ed864b7d31bd008
size 4330865200

3
model-00004-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2fff7a607bf2a49d174a90c0dc7b1a1e143352eeb4b2dcdddaa69a58d4295689
size 1089994880

346
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,346 @@
{
"metadata": {
"total_size": 15231233024
},
"weight_map": {
"lm_head.weight": "model-00004-of-00004.safetensors",
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.0.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.1.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.10.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.11.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.12.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.13.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.14.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.15.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.16.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.17.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.17.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.17.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.18.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.18.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.18.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.18.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.18.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.19.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.19.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.19.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.2.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.20.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.21.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.22.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.23.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.24.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.25.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.26.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.26.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.26.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.27.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.27.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.27.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.3.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.4.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.5.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.6.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.7.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.8.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.8.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.8.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.norm.weight": "model-00003-of-00004.safetensors"
}
}

3
optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc0802fabfbb8c7ba1a1bd45edb754a297fb91fe1cb0ec25743b7c909c2cfc30
size 30462761583

3
rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6416e992797ca06b75b0d21587f32db29da0ffa33ff8f87fd50b62dce973affd
size 14244

3
scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:61c64d81104619ca3adfa8f5d94c94c806b06b20a9577221cd2113ae44f70927
size 1064

250
sft_args.json Normal file
View File

@@ -0,0 +1,250 @@
{
"model_type": "qwen2-7b-instruct",
"model_id_or_path": "qwen/Qwen2-7B-Instruct",
"model_revision": "master",
"full_determinism": false,
"sft_type": "full",
"freeze_parameters": [],
"freeze_vit": false,
"freeze_parameters_ratio": 0.0,
"additional_trainable_parameters": [],
"tuner_backend": "peft",
"template_type": "qwen",
"output_dir": "/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455",
"add_output_dir_suffix": true,
"ddp_backend": null,
"ddp_find_unused_parameters": null,
"ddp_broadcast_buffers": null,
"ddp_timeout": 1800,
"seed": 42,
"resume_from_checkpoint": null,
"resume_only_model": true,
"ignore_data_skip": false,
"dtype": "bf16",
"packing": false,
"train_backend": "transformers",
"tp": 1,
"pp": 1,
"min_lr": null,
"sequence_parallel": false,
"model_kwargs": null,
"loss_name": "long-ce",
"dataset": [
"longwriter-6k-filtered#666",
"qwen2-pro-zh#6660",
"qwen2-pro-en#6660"
],
"val_dataset": [],
"dataset_seed": 42,
"dataset_test_ratio": 0.01,
"use_loss_scale": false,
"loss_scale_config_path": "/mnt/workspace/xingjun.wxj/swift_work/swift-by/swift/llm/agent/default_loss_scale_config.json",
"system": "You are a helpful assistant.",
"tools_prompt": "react_en",
"max_length": 28672,
"truncation_strategy": "delete",
"check_dataset_strategy": "warning",
"streaming": false,
"streaming_val_size": 0,
"streaming_buffer_size": 16384,
"model_name": [
null,
null
],
"model_author": [
null,
null
],
"quant_method": null,
"quantization_bit": 0,
"hqq_axis": 0,
"hqq_dynamic_config_path": null,
"bnb_4bit_comp_dtype": "bf16",
"bnb_4bit_quant_type": "nf4",
"bnb_4bit_use_double_quant": true,
"bnb_4bit_quant_storage": null,
"rescale_image": -1,
"target_modules": [
"q_proj",
"k_proj",
"v_proj"
],
"target_regex": null,
"modules_to_save": [],
"lora_rank": 8,
"lora_alpha": 32,
"lora_dropout": 0.05,
"lora_bias_trainable": "none",
"lora_dtype": "AUTO",
"lora_lr_ratio": null,
"use_rslora": false,
"use_dora": false,
"init_lora_weights": "true",
"fourier_n_frequency": 2000,
"fourier_scaling": 300.0,
"rope_scaling": null,
"boft_block_size": 4,
"boft_block_num": 0,
"boft_n_butterfly_factor": 1,
"boft_dropout": 0.0,
"vera_rank": 256,
"vera_projection_prng_key": 0,
"vera_dropout": 0.0,
"vera_d_initial": 0.1,
"adapter_act": "gelu",
"adapter_length": 128,
"use_galore": false,
"galore_target_modules": null,
"galore_rank": 128,
"galore_update_proj_gap": 50,
"galore_scale": 1.0,
"galore_proj_type": "std",
"galore_optim_per_parameter": false,
"galore_with_embedding": false,
"galore_quantization": false,
"galore_proj_quant": false,
"galore_proj_bits": 4,
"galore_proj_group_size": 256,
"galore_cos_threshold": 0.4,
"galore_gamma_proj": 2,
"galore_queue_size": 5,
"adalora_target_r": 8,
"adalora_init_r": 12,
"adalora_tinit": 0,
"adalora_tfinal": 0,
"adalora_deltaT": 1,
"adalora_beta1": 0.85,
"adalora_beta2": 0.85,
"adalora_orth_reg_weight": 0.5,
"ia3_feedforward_modules": [],
"llamapro_num_new_blocks": 4,
"llamapro_num_groups": null,
"neftune_noise_alpha": 5.0,
"neftune_backend": "transformers",
"lisa_activated_layers": 0,
"lisa_step_interval": 20,
"reft_layer_key": null,
"reft_layers": null,
"reft_rank": 4,
"reft_intervention_type": "LoreftIntervention",
"reft_args": null,
"use_liger": false,
"gradient_checkpointing": true,
"deepspeed": null,
"batch_size": 1,
"eval_batch_size": 1,
"auto_find_batch_size": false,
"num_train_epochs": 2,
"max_steps": -1,
"optim": "adamw_torch",
"adam_beta1": 0.9,
"adam_beta2": 0.95,
"adam_epsilon": 1e-08,
"learning_rate": 1e-05,
"weight_decay": 0.1,
"gradient_accumulation_steps": 64,
"max_grad_norm": 1,
"predict_with_generate": false,
"lr_scheduler_type": "cosine",
"lr_scheduler_kwargs": {},
"warmup_ratio": 0.1,
"warmup_steps": 0,
"eval_steps": 200,
"save_steps": 200,
"save_only_model": false,
"save_total_limit": null,
"logging_steps": 5,
"acc_steps": 1,
"dataloader_num_workers": 1,
"dataloader_pin_memory": true,
"dataloader_drop_last": false,
"push_to_hub": false,
"hub_model_id": null,
"hub_token": null,
"hub_private_repo": false,
"hub_strategy": "every_save",
"test_oom_error": false,
"disable_tqdm": false,
"lazy_tokenize": true,
"preprocess_num_proc": 1,
"use_flash_attn": true,
"ignore_args_error": false,
"check_model_is_latest": true,
"logging_dir": "/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455/runs",
"report_to": [
"tensorboard"
],
"acc_strategy": "token",
"save_on_each_node": false,
"evaluation_strategy": "steps",
"save_strategy": "steps",
"save_safetensors": true,
"gpu_memory_fraction": null,
"include_num_input_tokens_seen": false,
"local_repo_path": null,
"custom_register_path": null,
"custom_dataset_info": null,
"device_map_config": null,
"device_max_memory": [],
"max_new_tokens": 2048,
"do_sample": null,
"temperature": null,
"top_k": null,
"top_p": null,
"repetition_penalty": null,
"num_beams": 1,
"fsdp": "",
"fsdp_config": null,
"sequence_parallel_size": 1,
"model_layer_cls_name": null,
"metric_warmup_step": 0,
"fsdp_num": 1,
"per_device_train_batch_size": null,
"per_device_eval_batch_size": null,
"eval_strategy": null,
"self_cognition_sample": 0,
"train_dataset_mix_ratio": 0.0,
"train_dataset_mix_ds": [
"ms-bench"
],
"train_dataset_sample": -1,
"val_dataset_sample": null,
"safe_serialization": null,
"only_save_model": null,
"neftune_alpha": null,
"deepspeed_config_path": null,
"model_cache_dir": null,
"lora_dropout_p": null,
"lora_target_modules": [],
"lora_target_regex": null,
"lora_modules_to_save": [],
"boft_target_modules": [],
"boft_modules_to_save": [],
"vera_target_modules": [],
"vera_modules_to_save": [],
"ia3_target_modules": [],
"ia3_modules_to_save": [],
"custom_train_dataset_path": [],
"custom_val_dataset_path": [],
"device_map_config_path": null,
"push_hub_strategy": null,
"use_self_cognition": false,
"is_multimodal": false,
"is_vision": false,
"lora_use_embedding": false,
"lora_use_all": false,
"lora_m2s_use_embedding": false,
"lora_m2s_use_ln": false,
"torch_dtype": "torch.bfloat16",
"fp16": false,
"bf16": true,
"rank": -1,
"local_rank": -1,
"world_size": 1,
"local_world_size": 1,
"bnb_4bit_compute_dtype": "torch.bfloat16",
"load_in_4bit": false,
"load_in_8bit": false,
"train_sampler_random": true,
"training_args": "Seq2SeqTrainingArguments(output_dir='/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455', overwrite_output_dir=False, do_train=False, do_eval=True, do_predict=False, eval_strategy=<IntervalStrategy.STEPS: 'steps'>, prediction_loss_only=False, per_device_train_batch_size=1, per_device_eval_batch_size=1, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=64, eval_accumulation_steps=None, eval_delay=0, learning_rate=1e-05, weight_decay=0.1, adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1e-08, max_grad_norm=1, num_train_epochs=2, max_steps=-1, lr_scheduler_type=<SchedulerType.COSINE: 'cosine'>, lr_scheduler_kwargs={}, warmup_ratio=0.1, warmup_steps=0, log_level='passive', log_level_replica='warning', log_on_each_node=True, logging_dir='/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455/runs', logging_strategy=<IntervalStrategy.STEPS: 'steps'>, logging_first_step=True, logging_steps=5, logging_nan_inf_filter=True, save_strategy=<IntervalStrategy.STEPS: 'steps'>, save_steps=200, save_total_limit=None, save_safetensors=True, save_on_each_node=False, save_only_model=False, restore_callback_states_from_checkpoint=False, no_cuda=False, use_cpu=False, use_mps_device=False, seed=42, data_seed=None, jit_mode_eval=False, use_ipex=False, bf16=True, fp16=False, fp16_opt_level='O1', half_precision_backend='auto', bf16_full_eval=False, fp16_full_eval=False, tf32=None, local_rank=0, ddp_backend=None, tpu_num_cores=None, tpu_metrics_debug=False, debug=[], dataloader_drop_last=False, eval_steps=200, dataloader_num_workers=1, dataloader_prefetch_factor=None, past_index=-1, run_name='/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455', disable_tqdm=False, remove_unused_columns=False, label_names=None, load_best_model_at_end=False, metric_for_best_model='loss', greater_is_better=False, ignore_data_skip=False, fsdp=[], fsdp_min_num_params=0, fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, fsdp_transformer_layer_cls_to_wrap=None, accelerator_config=AcceleratorConfig(split_batches=False, dispatch_batches=False, even_batches=True, use_seedable_sampler=True, non_blocking=False, gradient_accumulation_kwargs=None, use_configured_state=False), deepspeed=None, label_smoothing_factor=0.0, optim=<OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, optim_args=None, adafactor=False, group_by_length=False, length_column_name='length', report_to=['tensorboard'], ddp_find_unused_parameters=None, ddp_bucket_cap_mb=None, ddp_broadcast_buffers=None, dataloader_pin_memory=True, dataloader_persistent_workers=False, skip_memory_metrics=True, use_legacy_prediction_loop=False, push_to_hub=False, resume_from_checkpoint=None, hub_model_id=None, hub_strategy=<HubStrategy.EVERY_SAVE: 'every_save'>, hub_token=None, hub_private_repo=False, hub_always_push=False, gradient_checkpointing=True, gradient_checkpointing_kwargs=None, include_inputs_for_metrics=False, eval_do_concat_batches=True, fp16_backend='auto', evaluation_strategy=None, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=None, mp_parameters='', auto_find_batch_size=False, full_determinism=False, torchdynamo=None, ray_scope='last', ddp_timeout=1800, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, dispatch_batches=None, split_batches=None, include_tokens_per_second=False, include_num_input_tokens_seen=False, neftune_noise_alpha=5.0, optim_target_modules=None, batch_eval_metrics=False, eval_on_start=False, sortish_sampler=True, predict_with_generate=False, generation_max_length=None, generation_num_beams=None, generation_config=GenerationConfig {\n \"bos_token_id\": 151643,\n \"do_sample\": true,\n \"eos_token_id\": 151645,\n \"max_new_tokens\": 2048,\n \"pad_token_id\": 151643,\n \"repetition_penalty\": 1.05,\n \"temperature\": 0.7,\n \"top_k\": 20,\n \"top_p\": 0.8\n}\n, train_sampler_random=True, acc_strategy='token', loss_name='long-ce', additional_saved_files=[], metric_warmup_step=0, train_dataset_sample=-1)"
}

14
special_tokens_map.json Normal file
View File

@@ -0,0 +1,14 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"eos_token": "<|im_end|>",
"pad_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

303112
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

43
tokenizer_config.json Normal file
View File

@@ -0,0 +1,43 @@
{
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"bos_token": null,
"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"model_max_length": 131072,
"pad_token": "<|endoftext|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

930
trainer_state.json Normal file
View File

@@ -0,0 +1,930 @@
{
"best_metric": 0.59626758,
"best_model_checkpoint": "/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455/checkpoint-432",
"epoch": 1.9965337954939342,
"eval_steps": 200,
"global_step": 432,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"acc": 0.81453747,
"epoch": 0.00462160600808781,
"grad_norm": 2.1875,
"learning_rate": 2.2727272727272729e-07,
"loss": 0.76972306,
"memory(GiB)": 94.61,
"step": 1,
"train_speed(iter/s)": 0.038737
},
{
"acc": 0.8124181,
"epoch": 0.023108030040439053,
"grad_norm": 2.6875,
"learning_rate": 1.1363636363636364e-06,
"loss": 0.64217675,
"memory(GiB)": 100.77,
"step": 5,
"train_speed(iter/s)": 0.049551
},
{
"acc": 0.80393114,
"epoch": 0.046216060080878106,
"grad_norm": 2.40625,
"learning_rate": 2.2727272727272728e-06,
"loss": 0.72643595,
"memory(GiB)": 141.03,
"step": 10,
"train_speed(iter/s)": 0.049763
},
{
"acc": 0.79872313,
"epoch": 0.06932409012131716,
"grad_norm": 2.09375,
"learning_rate": 3.409090909090909e-06,
"loss": 0.73066745,
"memory(GiB)": 141.03,
"step": 15,
"train_speed(iter/s)": 0.049428
},
{
"acc": 0.8086524,
"epoch": 0.09243212016175621,
"grad_norm": 2.078125,
"learning_rate": 4.5454545454545455e-06,
"loss": 0.76757164,
"memory(GiB)": 184.34,
"step": 20,
"train_speed(iter/s)": 0.047908
},
{
"acc": 0.80337076,
"epoch": 0.11554015020219527,
"grad_norm": 1.625,
"learning_rate": 5.681818181818183e-06,
"loss": 0.81441174,
"memory(GiB)": 150.97,
"step": 25,
"train_speed(iter/s)": 0.046696
},
{
"acc": 0.8120677,
"epoch": 0.1386481802426343,
"grad_norm": 1.453125,
"learning_rate": 6.818181818181818e-06,
"loss": 0.7133667,
"memory(GiB)": 150.97,
"step": 30,
"train_speed(iter/s)": 0.046543
},
{
"acc": 0.82139053,
"epoch": 0.16175621028307338,
"grad_norm": 1.4765625,
"learning_rate": 7.954545454545455e-06,
"loss": 0.59637189,
"memory(GiB)": 150.98,
"step": 35,
"train_speed(iter/s)": 0.047037
},
{
"acc": 0.81568699,
"epoch": 0.18486424032351242,
"grad_norm": 1.1796875,
"learning_rate": 9.090909090909091e-06,
"loss": 0.70513887,
"memory(GiB)": 150.98,
"step": 40,
"train_speed(iter/s)": 0.046892
},
{
"acc": 0.820224,
"epoch": 0.20797227036395147,
"grad_norm": 1.171875,
"learning_rate": 9.999836101895782e-06,
"loss": 0.69312973,
"memory(GiB)": 150.98,
"step": 45,
"train_speed(iter/s)": 0.046848
},
{
"acc": 0.81471777,
"epoch": 0.23108030040439054,
"grad_norm": 1.234375,
"learning_rate": 9.994100796397954e-06,
"loss": 0.71612887,
"memory(GiB)": 150.98,
"step": 50,
"train_speed(iter/s)": 0.04669
},
{
"acc": 0.81478605,
"epoch": 0.2541883304448296,
"grad_norm": 1.2578125,
"learning_rate": 9.98018132755861e-06,
"loss": 0.64565916,
"memory(GiB)": 150.98,
"step": 55,
"train_speed(iter/s)": 0.046851
},
{
"acc": 0.81756649,
"epoch": 0.2772963604852686,
"grad_norm": 1.28125,
"learning_rate": 9.958100506132127e-06,
"loss": 0.66640944,
"memory(GiB)": 150.98,
"step": 60,
"train_speed(iter/s)": 0.047121
},
{
"acc": 0.81747818,
"epoch": 0.30040439052570767,
"grad_norm": 1.4375,
"learning_rate": 9.927894517421252e-06,
"loss": 0.63872623,
"memory(GiB)": 150.98,
"step": 65,
"train_speed(iter/s)": 0.047364
},
{
"acc": 0.81161108,
"epoch": 0.32351242056614676,
"grad_norm": 1.28125,
"learning_rate": 9.889612861977855e-06,
"loss": 0.7101274,
"memory(GiB)": 150.98,
"step": 70,
"train_speed(iter/s)": 0.047269
},
{
"acc": 0.81774912,
"epoch": 0.3466204506065858,
"grad_norm": 1.3828125,
"learning_rate": 9.843318274483089e-06,
"loss": 0.64770298,
"memory(GiB)": 150.98,
"step": 75,
"train_speed(iter/s)": 0.047298
},
{
"acc": 0.81703777,
"epoch": 0.36972848064702485,
"grad_norm": 1.1328125,
"learning_rate": 9.789086620939936e-06,
"loss": 0.64908137,
"memory(GiB)": 150.98,
"step": 80,
"train_speed(iter/s)": 0.047241
},
{
"acc": 0.8161418,
"epoch": 0.3928365106874639,
"grad_norm": 1.3671875,
"learning_rate": 9.727006774346551e-06,
"loss": 0.66009436,
"memory(GiB)": 150.98,
"step": 85,
"train_speed(iter/s)": 0.047297
},
{
"acc": 0.82398653,
"epoch": 0.41594454072790293,
"grad_norm": 1.3203125,
"learning_rate": 9.657180469054213e-06,
"loss": 0.64000554,
"memory(GiB)": 150.98,
"step": 90,
"train_speed(iter/s)": 0.047467
},
{
"acc": 0.82110977,
"epoch": 0.439052570768342,
"grad_norm": 1.28125,
"learning_rate": 9.579722134048506e-06,
"loss": 0.67605906,
"memory(GiB)": 150.98,
"step": 95,
"train_speed(iter/s)": 0.047599
},
{
"acc": 0.82549,
"epoch": 0.4621606008087811,
"grad_norm": 1.4140625,
"learning_rate": 9.494758705426978e-06,
"loss": 0.58404155,
"memory(GiB)": 150.98,
"step": 100,
"train_speed(iter/s)": 0.047822
},
{
"acc": 0.82027588,
"epoch": 0.4852686308492201,
"grad_norm": 1.28125,
"learning_rate": 9.402429418380555e-06,
"loss": 0.69437795,
"memory(GiB)": 150.98,
"step": 105,
"train_speed(iter/s)": 0.047539
},
{
"acc": 0.82060165,
"epoch": 0.5083766608896592,
"grad_norm": 1.4296875,
"learning_rate": 9.302885579019626e-06,
"loss": 0.60217834,
"memory(GiB)": 150.98,
"step": 110,
"train_speed(iter/s)": 0.047767
},
{
"acc": 0.81884346,
"epoch": 0.5314846909300982,
"grad_norm": 1.40625,
"learning_rate": 9.196290316418712e-06,
"loss": 0.62260752,
"memory(GiB)": 150.98,
"step": 115,
"train_speed(iter/s)": 0.047925
},
{
"acc": 0.81530762,
"epoch": 0.5545927209705372,
"grad_norm": 1.1796875,
"learning_rate": 9.082818315286054e-06,
"loss": 0.67854958,
"memory(GiB)": 150.98,
"step": 120,
"train_speed(iter/s)": 0.048032
},
{
"acc": 0.8216691,
"epoch": 0.5777007510109763,
"grad_norm": 1.28125,
"learning_rate": 8.962655529696236e-06,
"loss": 0.61422577,
"memory(GiB)": 150.98,
"step": 125,
"train_speed(iter/s)": 0.048043
},
{
"acc": 0.82413073,
"epoch": 0.6008087810514153,
"grad_norm": 1.1796875,
"learning_rate": 8.83599887835493e-06,
"loss": 0.59601698,
"memory(GiB)": 150.98,
"step": 130,
"train_speed(iter/s)": 0.048165
},
{
"acc": 0.82421112,
"epoch": 0.6239168110918544,
"grad_norm": 1.3515625,
"learning_rate": 8.7030559218952e-06,
"loss": 0.62850995,
"memory(GiB)": 150.98,
"step": 135,
"train_speed(iter/s)": 0.048186
},
{
"acc": 0.81220446,
"epoch": 0.6470248411322935,
"grad_norm": 1.40625,
"learning_rate": 8.564044522734147e-06,
"loss": 0.65466747,
"memory(GiB)": 150.98,
"step": 140,
"train_speed(iter/s)": 0.048296
},
{
"acc": 0.8225975,
"epoch": 0.6701328711727326,
"grad_norm": 1.3515625,
"learning_rate": 8.419192488047369e-06,
"loss": 0.68851438,
"memory(GiB)": 150.98,
"step": 145,
"train_speed(iter/s)": 0.048275
},
{
"acc": 0.82113705,
"epoch": 0.6932409012131716,
"grad_norm": 1.3203125,
"learning_rate": 8.268737196446264e-06,
"loss": 0.66175132,
"memory(GiB)": 150.98,
"step": 150,
"train_speed(iter/s)": 0.048236
},
{
"acc": 0.82547045,
"epoch": 0.7163489312536107,
"grad_norm": 1.1875,
"learning_rate": 8.112925208969996e-06,
"loss": 0.58082228,
"memory(GiB)": 150.98,
"step": 155,
"train_speed(iter/s)": 0.048304
},
{
"acc": 0.81520081,
"epoch": 0.7394569612940497,
"grad_norm": 1.15625,
"learning_rate": 7.952011865029614e-06,
"loss": 0.7566462,
"memory(GiB)": 150.98,
"step": 160,
"train_speed(iter/s)": 0.04806
},
{
"acc": 0.82244606,
"epoch": 0.7625649913344887,
"grad_norm": 1.265625,
"learning_rate": 7.786260863966468e-06,
"loss": 0.64169059,
"memory(GiB)": 150.98,
"step": 165,
"train_speed(iter/s)": 0.048128
},
{
"acc": 0.82460566,
"epoch": 0.7856730213749278,
"grad_norm": 1.203125,
"learning_rate": 7.61594383291065e-06,
"loss": 0.64750757,
"memory(GiB)": 150.98,
"step": 170,
"train_speed(iter/s)": 0.048108
},
{
"acc": 0.82297211,
"epoch": 0.8087810514153668,
"grad_norm": 1.1484375,
"learning_rate": 7.441339881647654e-06,
"loss": 0.62678857,
"memory(GiB)": 150.98,
"step": 175,
"train_speed(iter/s)": 0.048123
},
{
"acc": 0.82202625,
"epoch": 0.8318890814558059,
"grad_norm": 1.2109375,
"learning_rate": 7.262735145222696e-06,
"loss": 0.63359532,
"memory(GiB)": 150.98,
"step": 180,
"train_speed(iter/s)": 0.04815
},
{
"acc": 0.81521053,
"epoch": 0.8549971114962449,
"grad_norm": 1.3984375,
"learning_rate": 7.080422315032298e-06,
"loss": 0.66546955,
"memory(GiB)": 150.98,
"step": 185,
"train_speed(iter/s)": 0.048203
},
{
"acc": 0.83204832,
"epoch": 0.878105141536684,
"grad_norm": 1.3203125,
"learning_rate": 6.894700159171535e-06,
"loss": 0.60190206,
"memory(GiB)": 150.98,
"step": 190,
"train_speed(iter/s)": 0.04822
},
{
"acc": 0.81509237,
"epoch": 0.901213171577123,
"grad_norm": 1.15625,
"learning_rate": 6.705873032822981e-06,
"loss": 0.71670413,
"memory(GiB)": 150.98,
"step": 195,
"train_speed(iter/s)": 0.048127
},
{
"acc": 0.82601328,
"epoch": 0.9243212016175621,
"grad_norm": 1.3984375,
"learning_rate": 6.514250379489754e-06,
"loss": 0.55743012,
"memory(GiB)": 150.98,
"step": 200,
"train_speed(iter/s)": 0.048221
},
{
"epoch": 0.9243212016175621,
"eval_acc": 0.7485949746717407,
"eval_loss": 0.6565380096435547,
"eval_runtime": 11.0891,
"eval_samples_per_second": 12.445,
"eval_steps_per_second": 12.445,
"step": 200
},
{
"acc": 0.82308855,
"epoch": 0.9474292316580012,
"grad_norm": 1.4375,
"learning_rate": 6.320146223889966e-06,
"loss": 0.63252063,
"memory(GiB)": 150.98,
"step": 205,
"train_speed(iter/s)": 0.046847
},
{
"acc": 0.82361374,
"epoch": 0.9705372616984402,
"grad_norm": 1.421875,
"learning_rate": 6.123878657343648e-06,
"loss": 0.59042826,
"memory(GiB)": 150.98,
"step": 210,
"train_speed(iter/s)": 0.046942
},
{
"acc": 0.82532425,
"epoch": 0.9936452917388793,
"grad_norm": 0.98828125,
"learning_rate": 5.925769316495461e-06,
"loss": 0.60934105,
"memory(GiB)": 150.98,
"step": 215,
"train_speed(iter/s)": 0.046992
},
{
"acc": 0.83245983,
"epoch": 1.0167533217793183,
"grad_norm": 1.1953125,
"learning_rate": 5.726142856227453e-06,
"loss": 0.65204225,
"memory(GiB)": 150.98,
"step": 220,
"train_speed(iter/s)": 0.046872
},
{
"acc": 0.83649168,
"epoch": 1.0398613518197575,
"grad_norm": 1.2109375,
"learning_rate": 5.52532641762562e-06,
"loss": 0.66422782,
"memory(GiB)": 150.98,
"step": 225,
"train_speed(iter/s)": 0.046806
},
{
"acc": 0.83163223,
"epoch": 1.0629693818601964,
"grad_norm": 1.140625,
"learning_rate": 5.323649091872179e-06,
"loss": 0.60072503,
"memory(GiB)": 150.98,
"step": 230,
"train_speed(iter/s)": 0.046824
},
{
"acc": 0.83812866,
"epoch": 1.0860774119006356,
"grad_norm": 1.25,
"learning_rate": 5.121441380942066e-06,
"loss": 0.58486128,
"memory(GiB)": 150.98,
"step": 235,
"train_speed(iter/s)": 0.046865
},
{
"acc": 0.82630444,
"epoch": 1.1091854419410745,
"grad_norm": 1.1328125,
"learning_rate": 4.919034655987493e-06,
"loss": 0.64795966,
"memory(GiB)": 150.98,
"step": 240,
"train_speed(iter/s)": 0.046852
},
{
"acc": 0.83698711,
"epoch": 1.1322934719815136,
"grad_norm": 1.4375,
"learning_rate": 4.716760614298118e-06,
"loss": 0.5604661,
"memory(GiB)": 150.98,
"step": 245,
"train_speed(iter/s)": 0.046935
},
{
"acc": 0.82200708,
"epoch": 1.1554015020219526,
"grad_norm": 1.2265625,
"learning_rate": 4.51495073572676e-06,
"loss": 0.65722399,
"memory(GiB)": 150.98,
"step": 250,
"train_speed(iter/s)": 0.046964
},
{
"acc": 0.83468876,
"epoch": 1.1785095320623917,
"grad_norm": 1.125,
"learning_rate": 4.313935739471425e-06,
"loss": 0.58888826,
"memory(GiB)": 150.98,
"step": 255,
"train_speed(iter/s)": 0.046962
},
{
"acc": 0.82915096,
"epoch": 1.2016175621028307,
"grad_norm": 1.4296875,
"learning_rate": 4.1140450421038865e-06,
"loss": 0.64913301,
"memory(GiB)": 150.98,
"step": 260,
"train_speed(iter/s)": 0.04701
},
{
"acc": 0.8357687,
"epoch": 1.2247255921432698,
"grad_norm": 1.2578125,
"learning_rate": 3.9156062177329624e-06,
"loss": 0.58918366,
"memory(GiB)": 150.98,
"step": 265,
"train_speed(iter/s)": 0.04707
},
{
"acc": 0.83036833,
"epoch": 1.2478336221837087,
"grad_norm": 1.203125,
"learning_rate": 3.7189444611871383e-06,
"loss": 0.60051751,
"memory(GiB)": 150.98,
"step": 270,
"train_speed(iter/s)": 0.047112
},
{
"acc": 0.8339922,
"epoch": 1.270941652224148,
"grad_norm": 1.3671875,
"learning_rate": 3.524382055096309e-06,
"loss": 0.55478253,
"memory(GiB)": 150.98,
"step": 275,
"train_speed(iter/s)": 0.047186
},
{
"acc": 0.83721056,
"epoch": 1.2940496822645868,
"grad_norm": 1.1328125,
"learning_rate": 3.3322378417458985e-06,
"loss": 0.59262004,
"memory(GiB)": 150.98,
"step": 280,
"train_speed(iter/s)": 0.047233
},
{
"acc": 0.82519064,
"epoch": 1.317157712305026,
"grad_norm": 1.171875,
"learning_rate": 3.1428267005689182e-06,
"loss": 0.68231759,
"memory(GiB)": 150.98,
"step": 285,
"train_speed(iter/s)": 0.047214
},
{
"acc": 0.83670778,
"epoch": 1.3402657423454651,
"grad_norm": 1.203125,
"learning_rate": 2.9564590321322206e-06,
"loss": 0.54285073,
"memory(GiB)": 150.98,
"step": 290,
"train_speed(iter/s)": 0.047272
},
{
"acc": 0.8335681,
"epoch": 1.363373772385904,
"grad_norm": 1.1953125,
"learning_rate": 2.7734402494625395e-06,
"loss": 0.60112472,
"memory(GiB)": 150.98,
"step": 295,
"train_speed(iter/s)": 0.04725
},
{
"acc": 0.83400679,
"epoch": 1.3864818024263432,
"grad_norm": 1.1328125,
"learning_rate": 2.594070277545975e-06,
"loss": 0.61252079,
"memory(GiB)": 150.98,
"step": 300,
"train_speed(iter/s)": 0.047279
},
{
"acc": 0.83710117,
"epoch": 1.4095898324667822,
"grad_norm": 1.2109375,
"learning_rate": 2.4186430618210704e-06,
"loss": 0.53748951,
"memory(GiB)": 150.98,
"step": 305,
"train_speed(iter/s)": 0.047266
},
{
"acc": 0.8276473,
"epoch": 1.4326978625072213,
"grad_norm": 1.3125,
"learning_rate": 2.2474460864709825e-06,
"loss": 0.6252665,
"memory(GiB)": 150.98,
"step": 310,
"train_speed(iter/s)": 0.04732
},
{
"acc": 0.83237705,
"epoch": 1.4558058925476602,
"grad_norm": 1.1328125,
"learning_rate": 2.0807599033041238e-06,
"loss": 0.63391018,
"memory(GiB)": 150.98,
"step": 315,
"train_speed(iter/s)": 0.04733
},
{
"acc": 0.83387976,
"epoch": 1.4789139225880994,
"grad_norm": 1.046875,
"learning_rate": 1.9188576719953635e-06,
"loss": 0.63072858,
"memory(GiB)": 150.98,
"step": 320,
"train_speed(iter/s)": 0.047302
},
{
"acc": 0.83439655,
"epoch": 1.5020219526285383,
"grad_norm": 1.2265625,
"learning_rate": 1.7620047124412e-06,
"loss": 0.615345,
"memory(GiB)": 150.98,
"step": 325,
"train_speed(iter/s)": 0.04731
},
{
"acc": 0.83732977,
"epoch": 1.5251299826689775,
"grad_norm": 1.3671875,
"learning_rate": 1.6104580699624839e-06,
"loss": 0.57706161,
"memory(GiB)": 150.98,
"step": 330,
"train_speed(iter/s)": 0.04735
},
{
"acc": 0.82502375,
"epoch": 1.5482380127094166,
"grad_norm": 1.2578125,
"learning_rate": 1.4644660940672628e-06,
"loss": 0.68620262,
"memory(GiB)": 150.98,
"step": 335,
"train_speed(iter/s)": 0.047298
},
{
"acc": 0.83036242,
"epoch": 1.5713460427498556,
"grad_norm": 1.1015625,
"learning_rate": 1.3242680314639995e-06,
"loss": 0.65163517,
"memory(GiB)": 150.98,
"step": 340,
"train_speed(iter/s)": 0.047314
},
{
"acc": 0.82439814,
"epoch": 1.5944540727902945,
"grad_norm": 1.140625,
"learning_rate": 1.1900936339921692e-06,
"loss": 0.64395781,
"memory(GiB)": 150.98,
"step": 345,
"train_speed(iter/s)": 0.04733
},
{
"acc": 0.83437843,
"epoch": 1.6175621028307337,
"grad_norm": 1.28125,
"learning_rate": 1.062162782112729e-06,
"loss": 0.58336234,
"memory(GiB)": 150.98,
"step": 350,
"train_speed(iter/s)": 0.047379
},
{
"acc": 0.83543835,
"epoch": 1.6406701328711728,
"grad_norm": 1.1640625,
"learning_rate": 9.406851245754478e-07,
"loss": 0.6100008,
"memory(GiB)": 150.98,
"step": 355,
"train_speed(iter/s)": 0.047394
},
{
"acc": 0.82420549,
"epoch": 1.6637781629116117,
"grad_norm": 1.3515625,
"learning_rate": 8.258597348536452e-07,
"loss": 0.67773538,
"memory(GiB)": 150.98,
"step": 360,
"train_speed(iter/s)": 0.047419
},
{
"acc": 0.83056965,
"epoch": 1.686886192952051,
"grad_norm": 1.1484375,
"learning_rate": 7.178747849093093e-07,
"loss": 0.63277826,
"memory(GiB)": 150.98,
"step": 365,
"train_speed(iter/s)": 0.047439
},
{
"acc": 0.83617363,
"epoch": 1.7099942229924898,
"grad_norm": 1.28125,
"learning_rate": 6.16907236823262e-07,
"loss": 0.58432493,
"memory(GiB)": 150.98,
"step": 370,
"train_speed(iter/s)": 0.04747
},
{
"acc": 0.82903013,
"epoch": 1.733102253032929,
"grad_norm": 1.1640625,
"learning_rate": 5.231225527956924e-07,
"loss": 0.6580874,
"memory(GiB)": 150.98,
"step": 375,
"train_speed(iter/s)": 0.047437
},
{
"acc": 0.83327017,
"epoch": 1.7562102830733681,
"grad_norm": 1.1328125,
"learning_rate": 4.3667442399229985e-07,
"loss": 0.58462934,
"memory(GiB)": 150.98,
"step": 380,
"train_speed(iter/s)": 0.047468
},
{
"acc": 0.83651209,
"epoch": 1.779318313113807,
"grad_norm": 1.125,
"learning_rate": 3.577045186804118e-07,
"loss": 0.58018489,
"memory(GiB)": 150.98,
"step": 385,
"train_speed(iter/s)": 0.047511
},
{
"acc": 0.82789068,
"epoch": 1.802426343154246,
"grad_norm": 1.0703125,
"learning_rate": 2.8634225006782867e-07,
"loss": 0.67336974,
"memory(GiB)": 150.98,
"step": 390,
"train_speed(iter/s)": 0.047483
},
{
"acc": 0.84120426,
"epoch": 1.8255343731946851,
"grad_norm": 1.2109375,
"learning_rate": 2.2270456422483655e-07,
"loss": 0.58580966,
"memory(GiB)": 150.98,
"step": 395,
"train_speed(iter/s)": 0.047495
},
{
"acc": 0.82003698,
"epoch": 1.8486424032351243,
"grad_norm": 1.125,
"learning_rate": 1.6689574843694433e-07,
"loss": 0.77600121,
"memory(GiB)": 150.98,
"step": 400,
"train_speed(iter/s)": 0.04747
},
{
"epoch": 1.8486424032351243,
"eval_acc": 0.7489352050202878,
"eval_loss": 0.6084072589874268,
"eval_runtime": 11.0063,
"eval_samples_per_second": 12.538,
"eval_steps_per_second": 12.538,
"step": 400
},
{
"acc": 0.83795338,
"epoch": 1.8717504332755632,
"grad_norm": 1.3671875,
"learning_rate": 1.1900726030241005e-07,
"loss": 0.57120104,
"memory(GiB)": 150.98,
"step": 405,
"train_speed(iter/s)": 0.046796
},
{
"acc": 0.82940121,
"epoch": 1.8948584633160022,
"grad_norm": 1.2109375,
"learning_rate": 7.911757785462882e-08,
"loss": 0.63004894,
"memory(GiB)": 150.98,
"step": 410,
"train_speed(iter/s)": 0.046799
},
{
"acc": 0.82601376,
"epoch": 1.9179664933564413,
"grad_norm": 1.3125,
"learning_rate": 4.729207095498345e-08,
"loss": 0.5973454,
"memory(GiB)": 150.98,
"step": 415,
"train_speed(iter/s)": 0.046849
},
{
"acc": 0.83928432,
"epoch": 1.9410745233968805,
"grad_norm": 1.2578125,
"learning_rate": 2.358289416693027e-08,
"loss": 0.57047229,
"memory(GiB)": 150.98,
"step": 420,
"train_speed(iter/s)": 0.046875
},
{
"acc": 0.83347263,
"epoch": 1.9641825534373196,
"grad_norm": 1.1171875,
"learning_rate": 8.02890128686562e-09,
"loss": 0.56035862,
"memory(GiB)": 150.98,
"step": 425,
"train_speed(iter/s)": 0.046877
},
{
"acc": 0.83463163,
"epoch": 1.9872905834777586,
"grad_norm": 1.28125,
"learning_rate": 6.555816718389896e-10,
"loss": 0.57831531,
"memory(GiB)": 150.98,
"step": 430,
"train_speed(iter/s)": 0.046917
},
{
"epoch": 1.9965337954939342,
"eval_acc": 0.7487839915320447,
"eval_loss": 0.596267580986023,
"eval_runtime": 11.0681,
"eval_samples_per_second": 12.468,
"eval_steps_per_second": 12.468,
"step": 432
}
],
"logging_steps": 5,
"max_steps": 432,
"num_input_tokens_seen": 0,
"num_train_epochs": 2,
"save_steps": 200,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": true
},
"attributes": {}
}
},
"total_flos": 8.150084761585428e+17,
"train_batch_size": 1,
"trial_name": null,
"trial_params": null
}

3
training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8dda763da817afe2fac8e60998f9a920b3aecf4281f9307a159c366a1720deb6
size 7288

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long