初始化项目，由ModelHub XC社区提供模型
Model: swift/MS-LongWriter-Qwen2-7B-Instruct Source: Original Platform
This commit is contained in:
ModelHub XC
38
.gitattributes
vendored
Normal file
38
.gitattributes
vendored
Normal file
@@ -0,0 +1,38 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zstandard filter=lfs diff=lfs merge=lfs -text
|
||||
*.tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
*.db* filter=lfs diff=lfs merge=lfs -text
|
||||
*.ark* filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.gguf* filter=lfs diff=lfs merge=lfs -text
|
||||
*.ggml filter=lfs diff=lfs merge=lfs -text
|
||||
*.llamafile* filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt2 filter=lfs diff=lfs merge=lfs -text
|
||||
76
README.md
Normal file
76
README.md
Normal file
@@ -0,0 +1,76 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
- zh
|
||||
library_name: transformers
|
||||
tags:
|
||||
- Long Context
|
||||
- qwen2
|
||||
|
||||
---
|
||||
# MS-LongWriter-Qwen2-7B-Instruct
|
||||
|
||||
<p align="center">
|
||||
🤖 <a href="https://modelscope.cn/datasets/swift/longwriter-6k-filtered" target="_blank">[LongWriter Dataset] </a> • 💻 <a href="https://github.com/THUDM/LongWriter" target="_blank">[Github Repo]</a> • 📃 <a href="https://arxiv.org/abs/2408.07055" target="_blank">[LongWriter Paper]</a>
|
||||
</p>
|
||||
|
||||
MS-LongWriter-Qwen2-7B-Instruct is trained based on [https://modelscope.cn/models/qwen/Qwen2-7B-Instruct](https://modelscope.cn/models/qwen/Qwen2-7B-Instruct), and is capable of generating 10,000+ words at once.
|
||||
|
||||
MS-LongWriter-Qwen2-7B-Instruct begins training directly from the Qwen2-7B-Instruct, while performing significant distillation on the [LongWriter-6k](https://modelscope.cn/datasets/ZhipuAI/LongWriter-6k) to obtain 666 high-quality samples.
|
||||
|
||||
|
||||
## Datasets
|
||||
1. [LongWriter-6k-filtered](https://modelscope.cn/datasets/swift/longwriter-6k-filtered), which is based on the [LongWriter-6k](https://modelscope.cn/datasets/ZhipuAI/LongWriter-6k)
|
||||
2. [Magpie-Qwen2-Pro-200K-Chinese](https://modelscope.cn/datasets/AI-ModelScope/Magpie-Qwen2-Pro-200K-Chinese) , random sampling 6k examples.
|
||||
3. [Magpie-Qwen2-Pro-200K-English](https://modelscope.cn/datasets/AI-ModelScope/Magpie-Qwen2-Pro-200K-English) , random sampling 6k examples.
|
||||
|
||||
|
||||
## Model
|
||||
|
||||
We use [ms-swift](https://github.com/modelscope/swift) to fine-tune the Qwen2-7B-Instruct model.
|
||||
|
||||
1. Installation
|
||||
```python
|
||||
pip install ms-swift[llm]
|
||||
```
|
||||
|
||||
2. Fine-tuning
|
||||
|
||||
Envs:
|
||||
```text
|
||||
Nvidia A100(80G) x 4
|
||||
```
|
||||
|
||||
|
||||
Run:
|
||||
```shell
|
||||
swift sft \
|
||||
--model_type qwen2-7b-instruct \
|
||||
--dataset longwriter-6k-filtered#666 qwen2-pro-zh#6660 qwen2-pro-en#6660 \
|
||||
--max_length 28672 \
|
||||
--num_train_epochs 2 \
|
||||
--eval_steps 200 \
|
||||
--batch_size 1 \
|
||||
--gradient_accumulation_steps 64 \
|
||||
--gradient_checkpointing true \
|
||||
--warmup_ratio 0.1 \
|
||||
--learning_rate 1e-5 \
|
||||
--sft_type full \
|
||||
--loss_name long-ce \
|
||||
--check_dataset_strategy warning \
|
||||
--save_only_model false \
|
||||
--save_total_limit -1 \
|
||||
--lazy_tokenize true \
|
||||
--dataloader_num_workers 1 \
|
||||
--resume_only_model true \
|
||||
--neftune_noise_alpha 5 \
|
||||
--use_flash_attn true
|
||||
```
|
||||
|
||||
|
||||
## Evaluation
|
||||
|
||||
Refer to [LongWriter Evaluation](https://github.com/modelscope/evalscope/tree/main/evalscope/third_party/longbench_write) from the [EvalScope](https://github.com/modelscope/evalscope).
|
||||
|
||||
|
||||
|
||||
5
added_tokens.json
Normal file
5
added_tokens.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<|endoftext|>": 151643,
|
||||
"<|im_end|>": 151645,
|
||||
"<|im_start|>": 151644
|
||||
}
|
||||
28
config.json
Normal file
28
config.json
Normal file
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"_name_or_path": "/mnt/nas2/huangjintao.hjt/.cache/modelscope/hub/qwen/Qwen2-7B-Instruct",
|
||||
"architectures": [
|
||||
"Qwen2ForCausalLM"
|
||||
],
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 151643,
|
||||
"eos_token_id": 151645,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 3584,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 18944,
|
||||
"max_position_embeddings": 32768,
|
||||
"max_window_layers": 28,
|
||||
"model_type": "qwen2",
|
||||
"num_attention_heads": 28,
|
||||
"num_hidden_layers": 28,
|
||||
"num_key_value_heads": 4,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_theta": 1000000.0,
|
||||
"sliding_window": 131072,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.42.4",
|
||||
"use_cache": false,
|
||||
"use_sliding_window": false,
|
||||
"vocab_size": 152064
|
||||
}
|
||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
||||
{"framework":"Pytorch","task":"text-generation"}
|
||||
12
generation_config.json
Normal file
12
generation_config.json
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"bos_token_id": 151643,
|
||||
"do_sample": true,
|
||||
"eos_token_id": 151645,
|
||||
"max_new_tokens": 2048,
|
||||
"pad_token_id": 151643,
|
||||
"repetition_penalty": 1.05,
|
||||
"temperature": 0.7,
|
||||
"top_k": 20,
|
||||
"top_p": 0.8,
|
||||
"transformers_version": "4.42.4"
|
||||
}
|
||||
151388
merges.txt
Normal file
151388
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model-00001-of-00004.safetensors
Normal file
3
model-00001-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ab0de5a00337c91e54b81ba4a681dd8c31d6dd6cdbfaae70ceda57a195207c96
|
||||
size 4877660776
|
||||
3
model-00002-of-00004.safetensors
Normal file
3
model-00002-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:5582aac209a09cfc25e4195c5f4a2175c1b0149c6e2312b30a9b7b25fa4eeaf7
|
||||
size 4932751008
|
||||
3
model-00003-of-00004.safetensors
Normal file
3
model-00003-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:5614bd6b55243f709f698b3cb4bf043840a62697348875b44ed864b7d31bd008
|
||||
size 4330865200
|
||||
3
model-00004-of-00004.safetensors
Normal file
3
model-00004-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2fff7a607bf2a49d174a90c0dc7b1a1e143352eeb4b2dcdddaa69a58d4295689
|
||||
size 1089994880
|
||||
346
model.safetensors.index.json
Normal file
346
model.safetensors.index.json
Normal file
@@ -0,0 +1,346 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_size": 15231233024
|
||||
},
|
||||
"weight_map": {
|
||||
"lm_head.weight": "model-00004-of-00004.safetensors",
|
||||
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.norm.weight": "model-00003-of-00004.safetensors"
|
||||
}
|
||||
}
|
||||
3
optimizer.pt
Normal file
3
optimizer.pt
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:cc0802fabfbb8c7ba1a1bd45edb754a297fb91fe1cb0ec25743b7c909c2cfc30
|
||||
size 30462761583
|
||||
3
rng_state.pth
Normal file
3
rng_state.pth
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:6416e992797ca06b75b0d21587f32db29da0ffa33ff8f87fd50b62dce973affd
|
||||
size 14244
|
||||
3
scheduler.pt
Normal file
3
scheduler.pt
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:61c64d81104619ca3adfa8f5d94c94c806b06b20a9577221cd2113ae44f70927
|
||||
size 1064
|
||||
250
sft_args.json
Normal file
250
sft_args.json
Normal file
@@ -0,0 +1,250 @@
|
||||
{
|
||||
"model_type": "qwen2-7b-instruct",
|
||||
"model_id_or_path": "qwen/Qwen2-7B-Instruct",
|
||||
"model_revision": "master",
|
||||
"full_determinism": false,
|
||||
"sft_type": "full",
|
||||
"freeze_parameters": [],
|
||||
"freeze_vit": false,
|
||||
"freeze_parameters_ratio": 0.0,
|
||||
"additional_trainable_parameters": [],
|
||||
"tuner_backend": "peft",
|
||||
"template_type": "qwen",
|
||||
"output_dir": "/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455",
|
||||
"add_output_dir_suffix": true,
|
||||
"ddp_backend": null,
|
||||
"ddp_find_unused_parameters": null,
|
||||
"ddp_broadcast_buffers": null,
|
||||
"ddp_timeout": 1800,
|
||||
"seed": 42,
|
||||
"resume_from_checkpoint": null,
|
||||
"resume_only_model": true,
|
||||
"ignore_data_skip": false,
|
||||
"dtype": "bf16",
|
||||
"packing": false,
|
||||
"train_backend": "transformers",
|
||||
"tp": 1,
|
||||
"pp": 1,
|
||||
"min_lr": null,
|
||||
"sequence_parallel": false,
|
||||
"model_kwargs": null,
|
||||
"loss_name": "long-ce",
|
||||
"dataset": [
|
||||
"longwriter-6k-filtered#666",
|
||||
"qwen2-pro-zh#6660",
|
||||
"qwen2-pro-en#6660"
|
||||
],
|
||||
"val_dataset": [],
|
||||
"dataset_seed": 42,
|
||||
"dataset_test_ratio": 0.01,
|
||||
"use_loss_scale": false,
|
||||
"loss_scale_config_path": "/mnt/workspace/xingjun.wxj/swift_work/swift-by/swift/llm/agent/default_loss_scale_config.json",
|
||||
"system": "You are a helpful assistant.",
|
||||
"tools_prompt": "react_en",
|
||||
"max_length": 28672,
|
||||
"truncation_strategy": "delete",
|
||||
"check_dataset_strategy": "warning",
|
||||
"streaming": false,
|
||||
"streaming_val_size": 0,
|
||||
"streaming_buffer_size": 16384,
|
||||
"model_name": [
|
||||
null,
|
||||
null
|
||||
],
|
||||
"model_author": [
|
||||
null,
|
||||
null
|
||||
],
|
||||
"quant_method": null,
|
||||
"quantization_bit": 0,
|
||||
"hqq_axis": 0,
|
||||
"hqq_dynamic_config_path": null,
|
||||
"bnb_4bit_comp_dtype": "bf16",
|
||||
"bnb_4bit_quant_type": "nf4",
|
||||
"bnb_4bit_use_double_quant": true,
|
||||
"bnb_4bit_quant_storage": null,
|
||||
"rescale_image": -1,
|
||||
"target_modules": [
|
||||
"q_proj",
|
||||
"k_proj",
|
||||
"v_proj"
|
||||
],
|
||||
"target_regex": null,
|
||||
"modules_to_save": [],
|
||||
"lora_rank": 8,
|
||||
"lora_alpha": 32,
|
||||
"lora_dropout": 0.05,
|
||||
"lora_bias_trainable": "none",
|
||||
"lora_dtype": "AUTO",
|
||||
"lora_lr_ratio": null,
|
||||
"use_rslora": false,
|
||||
"use_dora": false,
|
||||
"init_lora_weights": "true",
|
||||
"fourier_n_frequency": 2000,
|
||||
"fourier_scaling": 300.0,
|
||||
"rope_scaling": null,
|
||||
"boft_block_size": 4,
|
||||
"boft_block_num": 0,
|
||||
"boft_n_butterfly_factor": 1,
|
||||
"boft_dropout": 0.0,
|
||||
"vera_rank": 256,
|
||||
"vera_projection_prng_key": 0,
|
||||
"vera_dropout": 0.0,
|
||||
"vera_d_initial": 0.1,
|
||||
"adapter_act": "gelu",
|
||||
"adapter_length": 128,
|
||||
"use_galore": false,
|
||||
"galore_target_modules": null,
|
||||
"galore_rank": 128,
|
||||
"galore_update_proj_gap": 50,
|
||||
"galore_scale": 1.0,
|
||||
"galore_proj_type": "std",
|
||||
"galore_optim_per_parameter": false,
|
||||
"galore_with_embedding": false,
|
||||
"galore_quantization": false,
|
||||
"galore_proj_quant": false,
|
||||
"galore_proj_bits": 4,
|
||||
"galore_proj_group_size": 256,
|
||||
"galore_cos_threshold": 0.4,
|
||||
"galore_gamma_proj": 2,
|
||||
"galore_queue_size": 5,
|
||||
"adalora_target_r": 8,
|
||||
"adalora_init_r": 12,
|
||||
"adalora_tinit": 0,
|
||||
"adalora_tfinal": 0,
|
||||
"adalora_deltaT": 1,
|
||||
"adalora_beta1": 0.85,
|
||||
"adalora_beta2": 0.85,
|
||||
"adalora_orth_reg_weight": 0.5,
|
||||
"ia3_feedforward_modules": [],
|
||||
"llamapro_num_new_blocks": 4,
|
||||
"llamapro_num_groups": null,
|
||||
"neftune_noise_alpha": 5.0,
|
||||
"neftune_backend": "transformers",
|
||||
"lisa_activated_layers": 0,
|
||||
"lisa_step_interval": 20,
|
||||
"reft_layer_key": null,
|
||||
"reft_layers": null,
|
||||
"reft_rank": 4,
|
||||
"reft_intervention_type": "LoreftIntervention",
|
||||
"reft_args": null,
|
||||
"use_liger": false,
|
||||
"gradient_checkpointing": true,
|
||||
"deepspeed": null,
|
||||
"batch_size": 1,
|
||||
"eval_batch_size": 1,
|
||||
"auto_find_batch_size": false,
|
||||
"num_train_epochs": 2,
|
||||
"max_steps": -1,
|
||||
"optim": "adamw_torch",
|
||||
"adam_beta1": 0.9,
|
||||
"adam_beta2": 0.95,
|
||||
"adam_epsilon": 1e-08,
|
||||
"learning_rate": 1e-05,
|
||||
"weight_decay": 0.1,
|
||||
"gradient_accumulation_steps": 64,
|
||||
"max_grad_norm": 1,
|
||||
"predict_with_generate": false,
|
||||
"lr_scheduler_type": "cosine",
|
||||
"lr_scheduler_kwargs": {},
|
||||
"warmup_ratio": 0.1,
|
||||
"warmup_steps": 0,
|
||||
"eval_steps": 200,
|
||||
"save_steps": 200,
|
||||
"save_only_model": false,
|
||||
"save_total_limit": null,
|
||||
"logging_steps": 5,
|
||||
"acc_steps": 1,
|
||||
"dataloader_num_workers": 1,
|
||||
"dataloader_pin_memory": true,
|
||||
"dataloader_drop_last": false,
|
||||
"push_to_hub": false,
|
||||
"hub_model_id": null,
|
||||
"hub_token": null,
|
||||
"hub_private_repo": false,
|
||||
"hub_strategy": "every_save",
|
||||
"test_oom_error": false,
|
||||
"disable_tqdm": false,
|
||||
"lazy_tokenize": true,
|
||||
"preprocess_num_proc": 1,
|
||||
"use_flash_attn": true,
|
||||
"ignore_args_error": false,
|
||||
"check_model_is_latest": true,
|
||||
"logging_dir": "/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455/runs",
|
||||
"report_to": [
|
||||
"tensorboard"
|
||||
],
|
||||
"acc_strategy": "token",
|
||||
"save_on_each_node": false,
|
||||
"evaluation_strategy": "steps",
|
||||
"save_strategy": "steps",
|
||||
"save_safetensors": true,
|
||||
"gpu_memory_fraction": null,
|
||||
"include_num_input_tokens_seen": false,
|
||||
"local_repo_path": null,
|
||||
"custom_register_path": null,
|
||||
"custom_dataset_info": null,
|
||||
"device_map_config": null,
|
||||
"device_max_memory": [],
|
||||
"max_new_tokens": 2048,
|
||||
"do_sample": null,
|
||||
"temperature": null,
|
||||
"top_k": null,
|
||||
"top_p": null,
|
||||
"repetition_penalty": null,
|
||||
"num_beams": 1,
|
||||
"fsdp": "",
|
||||
"fsdp_config": null,
|
||||
"sequence_parallel_size": 1,
|
||||
"model_layer_cls_name": null,
|
||||
"metric_warmup_step": 0,
|
||||
"fsdp_num": 1,
|
||||
"per_device_train_batch_size": null,
|
||||
"per_device_eval_batch_size": null,
|
||||
"eval_strategy": null,
|
||||
"self_cognition_sample": 0,
|
||||
"train_dataset_mix_ratio": 0.0,
|
||||
"train_dataset_mix_ds": [
|
||||
"ms-bench"
|
||||
],
|
||||
"train_dataset_sample": -1,
|
||||
"val_dataset_sample": null,
|
||||
"safe_serialization": null,
|
||||
"only_save_model": null,
|
||||
"neftune_alpha": null,
|
||||
"deepspeed_config_path": null,
|
||||
"model_cache_dir": null,
|
||||
"lora_dropout_p": null,
|
||||
"lora_target_modules": [],
|
||||
"lora_target_regex": null,
|
||||
"lora_modules_to_save": [],
|
||||
"boft_target_modules": [],
|
||||
"boft_modules_to_save": [],
|
||||
"vera_target_modules": [],
|
||||
"vera_modules_to_save": [],
|
||||
"ia3_target_modules": [],
|
||||
"ia3_modules_to_save": [],
|
||||
"custom_train_dataset_path": [],
|
||||
"custom_val_dataset_path": [],
|
||||
"device_map_config_path": null,
|
||||
"push_hub_strategy": null,
|
||||
"use_self_cognition": false,
|
||||
"is_multimodal": false,
|
||||
"is_vision": false,
|
||||
"lora_use_embedding": false,
|
||||
"lora_use_all": false,
|
||||
"lora_m2s_use_embedding": false,
|
||||
"lora_m2s_use_ln": false,
|
||||
"torch_dtype": "torch.bfloat16",
|
||||
"fp16": false,
|
||||
"bf16": true,
|
||||
"rank": -1,
|
||||
"local_rank": -1,
|
||||
"world_size": 1,
|
||||
"local_world_size": 1,
|
||||
"bnb_4bit_compute_dtype": "torch.bfloat16",
|
||||
"load_in_4bit": false,
|
||||
"load_in_8bit": false,
|
||||
"train_sampler_random": true,
|
||||
"training_args": "Seq2SeqTrainingArguments(output_dir='/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455', overwrite_output_dir=False, do_train=False, do_eval=True, do_predict=False, eval_strategy=<IntervalStrategy.STEPS: 'steps'>, prediction_loss_only=False, per_device_train_batch_size=1, per_device_eval_batch_size=1, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=64, eval_accumulation_steps=None, eval_delay=0, learning_rate=1e-05, weight_decay=0.1, adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1e-08, max_grad_norm=1, num_train_epochs=2, max_steps=-1, lr_scheduler_type=<SchedulerType.COSINE: 'cosine'>, lr_scheduler_kwargs={}, warmup_ratio=0.1, warmup_steps=0, log_level='passive', log_level_replica='warning', log_on_each_node=True, logging_dir='/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455/runs', logging_strategy=<IntervalStrategy.STEPS: 'steps'>, logging_first_step=True, logging_steps=5, logging_nan_inf_filter=True, save_strategy=<IntervalStrategy.STEPS: 'steps'>, save_steps=200, save_total_limit=None, save_safetensors=True, save_on_each_node=False, save_only_model=False, restore_callback_states_from_checkpoint=False, no_cuda=False, use_cpu=False, use_mps_device=False, seed=42, data_seed=None, jit_mode_eval=False, use_ipex=False, bf16=True, fp16=False, fp16_opt_level='O1', half_precision_backend='auto', bf16_full_eval=False, fp16_full_eval=False, tf32=None, local_rank=0, ddp_backend=None, tpu_num_cores=None, tpu_metrics_debug=False, debug=[], dataloader_drop_last=False, eval_steps=200, dataloader_num_workers=1, dataloader_prefetch_factor=None, past_index=-1, run_name='/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455', disable_tqdm=False, remove_unused_columns=False, label_names=None, load_best_model_at_end=False, metric_for_best_model='loss', greater_is_better=False, ignore_data_skip=False, fsdp=[], fsdp_min_num_params=0, fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, fsdp_transformer_layer_cls_to_wrap=None, accelerator_config=AcceleratorConfig(split_batches=False, dispatch_batches=False, even_batches=True, use_seedable_sampler=True, non_blocking=False, gradient_accumulation_kwargs=None, use_configured_state=False), deepspeed=None, label_smoothing_factor=0.0, optim=<OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, optim_args=None, adafactor=False, group_by_length=False, length_column_name='length', report_to=['tensorboard'], ddp_find_unused_parameters=None, ddp_bucket_cap_mb=None, ddp_broadcast_buffers=None, dataloader_pin_memory=True, dataloader_persistent_workers=False, skip_memory_metrics=True, use_legacy_prediction_loop=False, push_to_hub=False, resume_from_checkpoint=None, hub_model_id=None, hub_strategy=<HubStrategy.EVERY_SAVE: 'every_save'>, hub_token=None, hub_private_repo=False, hub_always_push=False, gradient_checkpointing=True, gradient_checkpointing_kwargs=None, include_inputs_for_metrics=False, eval_do_concat_batches=True, fp16_backend='auto', evaluation_strategy=None, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=None, mp_parameters='', auto_find_batch_size=False, full_determinism=False, torchdynamo=None, ray_scope='last', ddp_timeout=1800, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, dispatch_batches=None, split_batches=None, include_tokens_per_second=False, include_num_input_tokens_seen=False, neftune_noise_alpha=5.0, optim_target_modules=None, batch_eval_metrics=False, eval_on_start=False, sortish_sampler=True, predict_with_generate=False, generation_max_length=None, generation_num_beams=None, generation_config=GenerationConfig {\n \"bos_token_id\": 151643,\n \"do_sample\": true,\n \"eos_token_id\": 151645,\n \"max_new_tokens\": 2048,\n \"pad_token_id\": 151643,\n \"repetition_penalty\": 1.05,\n \"temperature\": 0.7,\n \"top_k\": 20,\n \"top_p\": 0.8\n}\n, train_sampler_random=True, acc_strategy='token', loss_name='long-ce', additional_saved_files=[], metric_warmup_step=0, train_dataset_sample=-1)"
|
||||
}
|
||||
14
special_tokens_map.json
Normal file
14
special_tokens_map.json
Normal file
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>"
|
||||
],
|
||||
"eos_token": "<|im_end|>",
|
||||
"pad_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
303112
tokenizer.json
Normal file
303112
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
43
tokenizer_config.json
Normal file
43
tokenizer_config.json
Normal file
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"151643": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151644": {
|
||||
"content": "<|im_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151645": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>"
|
||||
],
|
||||
"bos_token": null,
|
||||
"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|im_end|>",
|
||||
"errors": "replace",
|
||||
"model_max_length": 131072,
|
||||
"pad_token": "<|endoftext|>",
|
||||
"split_special_tokens": false,
|
||||
"tokenizer_class": "Qwen2Tokenizer",
|
||||
"unk_token": null
|
||||
}
|
||||
930
trainer_state.json
Normal file
930
trainer_state.json
Normal file
@@ -0,0 +1,930 @@
|
||||
{
|
||||
"best_metric": 0.59626758,
|
||||
"best_model_checkpoint": "/mnt/workspace/xingjun.wxj/swift_work/swift-by/output/qwen2-7b-instruct/v7-20240904-182455/checkpoint-432",
|
||||
"epoch": 1.9965337954939342,
|
||||
"eval_steps": 200,
|
||||
"global_step": 432,
|
||||
"is_hyper_param_search": false,
|
||||
"is_local_process_zero": true,
|
||||
"is_world_process_zero": true,
|
||||
"log_history": [
|
||||
{
|
||||
"acc": 0.81453747,
|
||||
"epoch": 0.00462160600808781,
|
||||
"grad_norm": 2.1875,
|
||||
"learning_rate": 2.2727272727272729e-07,
|
||||
"loss": 0.76972306,
|
||||
"memory(GiB)": 94.61,
|
||||
"step": 1,
|
||||
"train_speed(iter/s)": 0.038737
|
||||
},
|
||||
{
|
||||
"acc": 0.8124181,
|
||||
"epoch": 0.023108030040439053,
|
||||
"grad_norm": 2.6875,
|
||||
"learning_rate": 1.1363636363636364e-06,
|
||||
"loss": 0.64217675,
|
||||
"memory(GiB)": 100.77,
|
||||
"step": 5,
|
||||
"train_speed(iter/s)": 0.049551
|
||||
},
|
||||
{
|
||||
"acc": 0.80393114,
|
||||
"epoch": 0.046216060080878106,
|
||||
"grad_norm": 2.40625,
|
||||
"learning_rate": 2.2727272727272728e-06,
|
||||
"loss": 0.72643595,
|
||||
"memory(GiB)": 141.03,
|
||||
"step": 10,
|
||||
"train_speed(iter/s)": 0.049763
|
||||
},
|
||||
{
|
||||
"acc": 0.79872313,
|
||||
"epoch": 0.06932409012131716,
|
||||
"grad_norm": 2.09375,
|
||||
"learning_rate": 3.409090909090909e-06,
|
||||
"loss": 0.73066745,
|
||||
"memory(GiB)": 141.03,
|
||||
"step": 15,
|
||||
"train_speed(iter/s)": 0.049428
|
||||
},
|
||||
{
|
||||
"acc": 0.8086524,
|
||||
"epoch": 0.09243212016175621,
|
||||
"grad_norm": 2.078125,
|
||||
"learning_rate": 4.5454545454545455e-06,
|
||||
"loss": 0.76757164,
|
||||
"memory(GiB)": 184.34,
|
||||
"step": 20,
|
||||
"train_speed(iter/s)": 0.047908
|
||||
},
|
||||
{
|
||||
"acc": 0.80337076,
|
||||
"epoch": 0.11554015020219527,
|
||||
"grad_norm": 1.625,
|
||||
"learning_rate": 5.681818181818183e-06,
|
||||
"loss": 0.81441174,
|
||||
"memory(GiB)": 150.97,
|
||||
"step": 25,
|
||||
"train_speed(iter/s)": 0.046696
|
||||
},
|
||||
{
|
||||
"acc": 0.8120677,
|
||||
"epoch": 0.1386481802426343,
|
||||
"grad_norm": 1.453125,
|
||||
"learning_rate": 6.818181818181818e-06,
|
||||
"loss": 0.7133667,
|
||||
"memory(GiB)": 150.97,
|
||||
"step": 30,
|
||||
"train_speed(iter/s)": 0.046543
|
||||
},
|
||||
{
|
||||
"acc": 0.82139053,
|
||||
"epoch": 0.16175621028307338,
|
||||
"grad_norm": 1.4765625,
|
||||
"learning_rate": 7.954545454545455e-06,
|
||||
"loss": 0.59637189,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 35,
|
||||
"train_speed(iter/s)": 0.047037
|
||||
},
|
||||
{
|
||||
"acc": 0.81568699,
|
||||
"epoch": 0.18486424032351242,
|
||||
"grad_norm": 1.1796875,
|
||||
"learning_rate": 9.090909090909091e-06,
|
||||
"loss": 0.70513887,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 40,
|
||||
"train_speed(iter/s)": 0.046892
|
||||
},
|
||||
{
|
||||
"acc": 0.820224,
|
||||
"epoch": 0.20797227036395147,
|
||||
"grad_norm": 1.171875,
|
||||
"learning_rate": 9.999836101895782e-06,
|
||||
"loss": 0.69312973,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 45,
|
||||
"train_speed(iter/s)": 0.046848
|
||||
},
|
||||
{
|
||||
"acc": 0.81471777,
|
||||
"epoch": 0.23108030040439054,
|
||||
"grad_norm": 1.234375,
|
||||
"learning_rate": 9.994100796397954e-06,
|
||||
"loss": 0.71612887,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 50,
|
||||
"train_speed(iter/s)": 0.04669
|
||||
},
|
||||
{
|
||||
"acc": 0.81478605,
|
||||
"epoch": 0.2541883304448296,
|
||||
"grad_norm": 1.2578125,
|
||||
"learning_rate": 9.98018132755861e-06,
|
||||
"loss": 0.64565916,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 55,
|
||||
"train_speed(iter/s)": 0.046851
|
||||
},
|
||||
{
|
||||
"acc": 0.81756649,
|
||||
"epoch": 0.2772963604852686,
|
||||
"grad_norm": 1.28125,
|
||||
"learning_rate": 9.958100506132127e-06,
|
||||
"loss": 0.66640944,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 60,
|
||||
"train_speed(iter/s)": 0.047121
|
||||
},
|
||||
{
|
||||
"acc": 0.81747818,
|
||||
"epoch": 0.30040439052570767,
|
||||
"grad_norm": 1.4375,
|
||||
"learning_rate": 9.927894517421252e-06,
|
||||
"loss": 0.63872623,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 65,
|
||||
"train_speed(iter/s)": 0.047364
|
||||
},
|
||||
{
|
||||
"acc": 0.81161108,
|
||||
"epoch": 0.32351242056614676,
|
||||
"grad_norm": 1.28125,
|
||||
"learning_rate": 9.889612861977855e-06,
|
||||
"loss": 0.7101274,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 70,
|
||||
"train_speed(iter/s)": 0.047269
|
||||
},
|
||||
{
|
||||
"acc": 0.81774912,
|
||||
"epoch": 0.3466204506065858,
|
||||
"grad_norm": 1.3828125,
|
||||
"learning_rate": 9.843318274483089e-06,
|
||||
"loss": 0.64770298,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 75,
|
||||
"train_speed(iter/s)": 0.047298
|
||||
},
|
||||
{
|
||||
"acc": 0.81703777,
|
||||
"epoch": 0.36972848064702485,
|
||||
"grad_norm": 1.1328125,
|
||||
"learning_rate": 9.789086620939936e-06,
|
||||
"loss": 0.64908137,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 80,
|
||||
"train_speed(iter/s)": 0.047241
|
||||
},
|
||||
{
|
||||
"acc": 0.8161418,
|
||||
"epoch": 0.3928365106874639,
|
||||
"grad_norm": 1.3671875,
|
||||
"learning_rate": 9.727006774346551e-06,
|
||||
"loss": 0.66009436,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 85,
|
||||
"train_speed(iter/s)": 0.047297
|
||||
},
|
||||
{
|
||||
"acc": 0.82398653,
|
||||
"epoch": 0.41594454072790293,
|
||||
"grad_norm": 1.3203125,
|
||||
"learning_rate": 9.657180469054213e-06,
|
||||
"loss": 0.64000554,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 90,
|
||||
"train_speed(iter/s)": 0.047467
|
||||
},
|
||||
{
|
||||
"acc": 0.82110977,
|
||||
"epoch": 0.439052570768342,
|
||||
"grad_norm": 1.28125,
|
||||
"learning_rate": 9.579722134048506e-06,
|
||||
"loss": 0.67605906,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 95,
|
||||
"train_speed(iter/s)": 0.047599
|
||||
},
|
||||
{
|
||||
"acc": 0.82549,
|
||||
"epoch": 0.4621606008087811,
|
||||
"grad_norm": 1.4140625,
|
||||
"learning_rate": 9.494758705426978e-06,
|
||||
"loss": 0.58404155,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 100,
|
||||
"train_speed(iter/s)": 0.047822
|
||||
},
|
||||
{
|
||||
"acc": 0.82027588,
|
||||
"epoch": 0.4852686308492201,
|
||||
"grad_norm": 1.28125,
|
||||
"learning_rate": 9.402429418380555e-06,
|
||||
"loss": 0.69437795,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 105,
|
||||
"train_speed(iter/s)": 0.047539
|
||||
},
|
||||
{
|
||||
"acc": 0.82060165,
|
||||
"epoch": 0.5083766608896592,
|
||||
"grad_norm": 1.4296875,
|
||||
"learning_rate": 9.302885579019626e-06,
|
||||
"loss": 0.60217834,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 110,
|
||||
"train_speed(iter/s)": 0.047767
|
||||
},
|
||||
{
|
||||
"acc": 0.81884346,
|
||||
"epoch": 0.5314846909300982,
|
||||
"grad_norm": 1.40625,
|
||||
"learning_rate": 9.196290316418712e-06,
|
||||
"loss": 0.62260752,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 115,
|
||||
"train_speed(iter/s)": 0.047925
|
||||
},
|
||||
{
|
||||
"acc": 0.81530762,
|
||||
"epoch": 0.5545927209705372,
|
||||
"grad_norm": 1.1796875,
|
||||
"learning_rate": 9.082818315286054e-06,
|
||||
"loss": 0.67854958,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 120,
|
||||
"train_speed(iter/s)": 0.048032
|
||||
},
|
||||
{
|
||||
"acc": 0.8216691,
|
||||
"epoch": 0.5777007510109763,
|
||||
"grad_norm": 1.28125,
|
||||
"learning_rate": 8.962655529696236e-06,
|
||||
"loss": 0.61422577,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 125,
|
||||
"train_speed(iter/s)": 0.048043
|
||||
},
|
||||
{
|
||||
"acc": 0.82413073,
|
||||
"epoch": 0.6008087810514153,
|
||||
"grad_norm": 1.1796875,
|
||||
"learning_rate": 8.83599887835493e-06,
|
||||
"loss": 0.59601698,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 130,
|
||||
"train_speed(iter/s)": 0.048165
|
||||
},
|
||||
{
|
||||
"acc": 0.82421112,
|
||||
"epoch": 0.6239168110918544,
|
||||
"grad_norm": 1.3515625,
|
||||
"learning_rate": 8.7030559218952e-06,
|
||||
"loss": 0.62850995,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 135,
|
||||
"train_speed(iter/s)": 0.048186
|
||||
},
|
||||
{
|
||||
"acc": 0.81220446,
|
||||
"epoch": 0.6470248411322935,
|
||||
"grad_norm": 1.40625,
|
||||
"learning_rate": 8.564044522734147e-06,
|
||||
"loss": 0.65466747,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 140,
|
||||
"train_speed(iter/s)": 0.048296
|
||||
},
|
||||
{
|
||||
"acc": 0.8225975,
|
||||
"epoch": 0.6701328711727326,
|
||||
"grad_norm": 1.3515625,
|
||||
"learning_rate": 8.419192488047369e-06,
|
||||
"loss": 0.68851438,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 145,
|
||||
"train_speed(iter/s)": 0.048275
|
||||
},
|
||||
{
|
||||
"acc": 0.82113705,
|
||||
"epoch": 0.6932409012131716,
|
||||
"grad_norm": 1.3203125,
|
||||
"learning_rate": 8.268737196446264e-06,
|
||||
"loss": 0.66175132,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 150,
|
||||
"train_speed(iter/s)": 0.048236
|
||||
},
|
||||
{
|
||||
"acc": 0.82547045,
|
||||
"epoch": 0.7163489312536107,
|
||||
"grad_norm": 1.1875,
|
||||
"learning_rate": 8.112925208969996e-06,
|
||||
"loss": 0.58082228,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 155,
|
||||
"train_speed(iter/s)": 0.048304
|
||||
},
|
||||
{
|
||||
"acc": 0.81520081,
|
||||
"epoch": 0.7394569612940497,
|
||||
"grad_norm": 1.15625,
|
||||
"learning_rate": 7.952011865029614e-06,
|
||||
"loss": 0.7566462,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 160,
|
||||
"train_speed(iter/s)": 0.04806
|
||||
},
|
||||
{
|
||||
"acc": 0.82244606,
|
||||
"epoch": 0.7625649913344887,
|
||||
"grad_norm": 1.265625,
|
||||
"learning_rate": 7.786260863966468e-06,
|
||||
"loss": 0.64169059,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 165,
|
||||
"train_speed(iter/s)": 0.048128
|
||||
},
|
||||
{
|
||||
"acc": 0.82460566,
|
||||
"epoch": 0.7856730213749278,
|
||||
"grad_norm": 1.203125,
|
||||
"learning_rate": 7.61594383291065e-06,
|
||||
"loss": 0.64750757,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 170,
|
||||
"train_speed(iter/s)": 0.048108
|
||||
},
|
||||
{
|
||||
"acc": 0.82297211,
|
||||
"epoch": 0.8087810514153668,
|
||||
"grad_norm": 1.1484375,
|
||||
"learning_rate": 7.441339881647654e-06,
|
||||
"loss": 0.62678857,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 175,
|
||||
"train_speed(iter/s)": 0.048123
|
||||
},
|
||||
{
|
||||
"acc": 0.82202625,
|
||||
"epoch": 0.8318890814558059,
|
||||
"grad_norm": 1.2109375,
|
||||
"learning_rate": 7.262735145222696e-06,
|
||||
"loss": 0.63359532,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 180,
|
||||
"train_speed(iter/s)": 0.04815
|
||||
},
|
||||
{
|
||||
"acc": 0.81521053,
|
||||
"epoch": 0.8549971114962449,
|
||||
"grad_norm": 1.3984375,
|
||||
"learning_rate": 7.080422315032298e-06,
|
||||
"loss": 0.66546955,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 185,
|
||||
"train_speed(iter/s)": 0.048203
|
||||
},
|
||||
{
|
||||
"acc": 0.83204832,
|
||||
"epoch": 0.878105141536684,
|
||||
"grad_norm": 1.3203125,
|
||||
"learning_rate": 6.894700159171535e-06,
|
||||
"loss": 0.60190206,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 190,
|
||||
"train_speed(iter/s)": 0.04822
|
||||
},
|
||||
{
|
||||
"acc": 0.81509237,
|
||||
"epoch": 0.901213171577123,
|
||||
"grad_norm": 1.15625,
|
||||
"learning_rate": 6.705873032822981e-06,
|
||||
"loss": 0.71670413,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 195,
|
||||
"train_speed(iter/s)": 0.048127
|
||||
},
|
||||
{
|
||||
"acc": 0.82601328,
|
||||
"epoch": 0.9243212016175621,
|
||||
"grad_norm": 1.3984375,
|
||||
"learning_rate": 6.514250379489754e-06,
|
||||
"loss": 0.55743012,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 200,
|
||||
"train_speed(iter/s)": 0.048221
|
||||
},
|
||||
{
|
||||
"epoch": 0.9243212016175621,
|
||||
"eval_acc": 0.7485949746717407,
|
||||
"eval_loss": 0.6565380096435547,
|
||||
"eval_runtime": 11.0891,
|
||||
"eval_samples_per_second": 12.445,
|
||||
"eval_steps_per_second": 12.445,
|
||||
"step": 200
|
||||
},
|
||||
{
|
||||
"acc": 0.82308855,
|
||||
"epoch": 0.9474292316580012,
|
||||
"grad_norm": 1.4375,
|
||||
"learning_rate": 6.320146223889966e-06,
|
||||
"loss": 0.63252063,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 205,
|
||||
"train_speed(iter/s)": 0.046847
|
||||
},
|
||||
{
|
||||
"acc": 0.82361374,
|
||||
"epoch": 0.9705372616984402,
|
||||
"grad_norm": 1.421875,
|
||||
"learning_rate": 6.123878657343648e-06,
|
||||
"loss": 0.59042826,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 210,
|
||||
"train_speed(iter/s)": 0.046942
|
||||
},
|
||||
{
|
||||
"acc": 0.82532425,
|
||||
"epoch": 0.9936452917388793,
|
||||
"grad_norm": 0.98828125,
|
||||
"learning_rate": 5.925769316495461e-06,
|
||||
"loss": 0.60934105,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 215,
|
||||
"train_speed(iter/s)": 0.046992
|
||||
},
|
||||
{
|
||||
"acc": 0.83245983,
|
||||
"epoch": 1.0167533217793183,
|
||||
"grad_norm": 1.1953125,
|
||||
"learning_rate": 5.726142856227453e-06,
|
||||
"loss": 0.65204225,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 220,
|
||||
"train_speed(iter/s)": 0.046872
|
||||
},
|
||||
{
|
||||
"acc": 0.83649168,
|
||||
"epoch": 1.0398613518197575,
|
||||
"grad_norm": 1.2109375,
|
||||
"learning_rate": 5.52532641762562e-06,
|
||||
"loss": 0.66422782,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 225,
|
||||
"train_speed(iter/s)": 0.046806
|
||||
},
|
||||
{
|
||||
"acc": 0.83163223,
|
||||
"epoch": 1.0629693818601964,
|
||||
"grad_norm": 1.140625,
|
||||
"learning_rate": 5.323649091872179e-06,
|
||||
"loss": 0.60072503,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 230,
|
||||
"train_speed(iter/s)": 0.046824
|
||||
},
|
||||
{
|
||||
"acc": 0.83812866,
|
||||
"epoch": 1.0860774119006356,
|
||||
"grad_norm": 1.25,
|
||||
"learning_rate": 5.121441380942066e-06,
|
||||
"loss": 0.58486128,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 235,
|
||||
"train_speed(iter/s)": 0.046865
|
||||
},
|
||||
{
|
||||
"acc": 0.82630444,
|
||||
"epoch": 1.1091854419410745,
|
||||
"grad_norm": 1.1328125,
|
||||
"learning_rate": 4.919034655987493e-06,
|
||||
"loss": 0.64795966,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 240,
|
||||
"train_speed(iter/s)": 0.046852
|
||||
},
|
||||
{
|
||||
"acc": 0.83698711,
|
||||
"epoch": 1.1322934719815136,
|
||||
"grad_norm": 1.4375,
|
||||
"learning_rate": 4.716760614298118e-06,
|
||||
"loss": 0.5604661,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 245,
|
||||
"train_speed(iter/s)": 0.046935
|
||||
},
|
||||
{
|
||||
"acc": 0.82200708,
|
||||
"epoch": 1.1554015020219526,
|
||||
"grad_norm": 1.2265625,
|
||||
"learning_rate": 4.51495073572676e-06,
|
||||
"loss": 0.65722399,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 250,
|
||||
"train_speed(iter/s)": 0.046964
|
||||
},
|
||||
{
|
||||
"acc": 0.83468876,
|
||||
"epoch": 1.1785095320623917,
|
||||
"grad_norm": 1.125,
|
||||
"learning_rate": 4.313935739471425e-06,
|
||||
"loss": 0.58888826,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 255,
|
||||
"train_speed(iter/s)": 0.046962
|
||||
},
|
||||
{
|
||||
"acc": 0.82915096,
|
||||
"epoch": 1.2016175621028307,
|
||||
"grad_norm": 1.4296875,
|
||||
"learning_rate": 4.1140450421038865e-06,
|
||||
"loss": 0.64913301,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 260,
|
||||
"train_speed(iter/s)": 0.04701
|
||||
},
|
||||
{
|
||||
"acc": 0.8357687,
|
||||
"epoch": 1.2247255921432698,
|
||||
"grad_norm": 1.2578125,
|
||||
"learning_rate": 3.9156062177329624e-06,
|
||||
"loss": 0.58918366,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 265,
|
||||
"train_speed(iter/s)": 0.04707
|
||||
},
|
||||
{
|
||||
"acc": 0.83036833,
|
||||
"epoch": 1.2478336221837087,
|
||||
"grad_norm": 1.203125,
|
||||
"learning_rate": 3.7189444611871383e-06,
|
||||
"loss": 0.60051751,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 270,
|
||||
"train_speed(iter/s)": 0.047112
|
||||
},
|
||||
{
|
||||
"acc": 0.8339922,
|
||||
"epoch": 1.270941652224148,
|
||||
"grad_norm": 1.3671875,
|
||||
"learning_rate": 3.524382055096309e-06,
|
||||
"loss": 0.55478253,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 275,
|
||||
"train_speed(iter/s)": 0.047186
|
||||
},
|
||||
{
|
||||
"acc": 0.83721056,
|
||||
"epoch": 1.2940496822645868,
|
||||
"grad_norm": 1.1328125,
|
||||
"learning_rate": 3.3322378417458985e-06,
|
||||
"loss": 0.59262004,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 280,
|
||||
"train_speed(iter/s)": 0.047233
|
||||
},
|
||||
{
|
||||
"acc": 0.82519064,
|
||||
"epoch": 1.317157712305026,
|
||||
"grad_norm": 1.171875,
|
||||
"learning_rate": 3.1428267005689182e-06,
|
||||
"loss": 0.68231759,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 285,
|
||||
"train_speed(iter/s)": 0.047214
|
||||
},
|
||||
{
|
||||
"acc": 0.83670778,
|
||||
"epoch": 1.3402657423454651,
|
||||
"grad_norm": 1.203125,
|
||||
"learning_rate": 2.9564590321322206e-06,
|
||||
"loss": 0.54285073,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 290,
|
||||
"train_speed(iter/s)": 0.047272
|
||||
},
|
||||
{
|
||||
"acc": 0.8335681,
|
||||
"epoch": 1.363373772385904,
|
||||
"grad_norm": 1.1953125,
|
||||
"learning_rate": 2.7734402494625395e-06,
|
||||
"loss": 0.60112472,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 295,
|
||||
"train_speed(iter/s)": 0.04725
|
||||
},
|
||||
{
|
||||
"acc": 0.83400679,
|
||||
"epoch": 1.3864818024263432,
|
||||
"grad_norm": 1.1328125,
|
||||
"learning_rate": 2.594070277545975e-06,
|
||||
"loss": 0.61252079,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 300,
|
||||
"train_speed(iter/s)": 0.047279
|
||||
},
|
||||
{
|
||||
"acc": 0.83710117,
|
||||
"epoch": 1.4095898324667822,
|
||||
"grad_norm": 1.2109375,
|
||||
"learning_rate": 2.4186430618210704e-06,
|
||||
"loss": 0.53748951,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 305,
|
||||
"train_speed(iter/s)": 0.047266
|
||||
},
|
||||
{
|
||||
"acc": 0.8276473,
|
||||
"epoch": 1.4326978625072213,
|
||||
"grad_norm": 1.3125,
|
||||
"learning_rate": 2.2474460864709825e-06,
|
||||
"loss": 0.6252665,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 310,
|
||||
"train_speed(iter/s)": 0.04732
|
||||
},
|
||||
{
|
||||
"acc": 0.83237705,
|
||||
"epoch": 1.4558058925476602,
|
||||
"grad_norm": 1.1328125,
|
||||
"learning_rate": 2.0807599033041238e-06,
|
||||
"loss": 0.63391018,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 315,
|
||||
"train_speed(iter/s)": 0.04733
|
||||
},
|
||||
{
|
||||
"acc": 0.83387976,
|
||||
"epoch": 1.4789139225880994,
|
||||
"grad_norm": 1.046875,
|
||||
"learning_rate": 1.9188576719953635e-06,
|
||||
"loss": 0.63072858,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 320,
|
||||
"train_speed(iter/s)": 0.047302
|
||||
},
|
||||
{
|
||||
"acc": 0.83439655,
|
||||
"epoch": 1.5020219526285383,
|
||||
"grad_norm": 1.2265625,
|
||||
"learning_rate": 1.7620047124412e-06,
|
||||
"loss": 0.615345,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 325,
|
||||
"train_speed(iter/s)": 0.04731
|
||||
},
|
||||
{
|
||||
"acc": 0.83732977,
|
||||
"epoch": 1.5251299826689775,
|
||||
"grad_norm": 1.3671875,
|
||||
"learning_rate": 1.6104580699624839e-06,
|
||||
"loss": 0.57706161,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 330,
|
||||
"train_speed(iter/s)": 0.04735
|
||||
},
|
||||
{
|
||||
"acc": 0.82502375,
|
||||
"epoch": 1.5482380127094166,
|
||||
"grad_norm": 1.2578125,
|
||||
"learning_rate": 1.4644660940672628e-06,
|
||||
"loss": 0.68620262,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 335,
|
||||
"train_speed(iter/s)": 0.047298
|
||||
},
|
||||
{
|
||||
"acc": 0.83036242,
|
||||
"epoch": 1.5713460427498556,
|
||||
"grad_norm": 1.1015625,
|
||||
"learning_rate": 1.3242680314639995e-06,
|
||||
"loss": 0.65163517,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 340,
|
||||
"train_speed(iter/s)": 0.047314
|
||||
},
|
||||
{
|
||||
"acc": 0.82439814,
|
||||
"epoch": 1.5944540727902945,
|
||||
"grad_norm": 1.140625,
|
||||
"learning_rate": 1.1900936339921692e-06,
|
||||
"loss": 0.64395781,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 345,
|
||||
"train_speed(iter/s)": 0.04733
|
||||
},
|
||||
{
|
||||
"acc": 0.83437843,
|
||||
"epoch": 1.6175621028307337,
|
||||
"grad_norm": 1.28125,
|
||||
"learning_rate": 1.062162782112729e-06,
|
||||
"loss": 0.58336234,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 350,
|
||||
"train_speed(iter/s)": 0.047379
|
||||
},
|
||||
{
|
||||
"acc": 0.83543835,
|
||||
"epoch": 1.6406701328711728,
|
||||
"grad_norm": 1.1640625,
|
||||
"learning_rate": 9.406851245754478e-07,
|
||||
"loss": 0.6100008,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 355,
|
||||
"train_speed(iter/s)": 0.047394
|
||||
},
|
||||
{
|
||||
"acc": 0.82420549,
|
||||
"epoch": 1.6637781629116117,
|
||||
"grad_norm": 1.3515625,
|
||||
"learning_rate": 8.258597348536452e-07,
|
||||
"loss": 0.67773538,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 360,
|
||||
"train_speed(iter/s)": 0.047419
|
||||
},
|
||||
{
|
||||
"acc": 0.83056965,
|
||||
"epoch": 1.686886192952051,
|
||||
"grad_norm": 1.1484375,
|
||||
"learning_rate": 7.178747849093093e-07,
|
||||
"loss": 0.63277826,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 365,
|
||||
"train_speed(iter/s)": 0.047439
|
||||
},
|
||||
{
|
||||
"acc": 0.83617363,
|
||||
"epoch": 1.7099942229924898,
|
||||
"grad_norm": 1.28125,
|
||||
"learning_rate": 6.16907236823262e-07,
|
||||
"loss": 0.58432493,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 370,
|
||||
"train_speed(iter/s)": 0.04747
|
||||
},
|
||||
{
|
||||
"acc": 0.82903013,
|
||||
"epoch": 1.733102253032929,
|
||||
"grad_norm": 1.1640625,
|
||||
"learning_rate": 5.231225527956924e-07,
|
||||
"loss": 0.6580874,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 375,
|
||||
"train_speed(iter/s)": 0.047437
|
||||
},
|
||||
{
|
||||
"acc": 0.83327017,
|
||||
"epoch": 1.7562102830733681,
|
||||
"grad_norm": 1.1328125,
|
||||
"learning_rate": 4.3667442399229985e-07,
|
||||
"loss": 0.58462934,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 380,
|
||||
"train_speed(iter/s)": 0.047468
|
||||
},
|
||||
{
|
||||
"acc": 0.83651209,
|
||||
"epoch": 1.779318313113807,
|
||||
"grad_norm": 1.125,
|
||||
"learning_rate": 3.577045186804118e-07,
|
||||
"loss": 0.58018489,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 385,
|
||||
"train_speed(iter/s)": 0.047511
|
||||
},
|
||||
{
|
||||
"acc": 0.82789068,
|
||||
"epoch": 1.802426343154246,
|
||||
"grad_norm": 1.0703125,
|
||||
"learning_rate": 2.8634225006782867e-07,
|
||||
"loss": 0.67336974,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 390,
|
||||
"train_speed(iter/s)": 0.047483
|
||||
},
|
||||
{
|
||||
"acc": 0.84120426,
|
||||
"epoch": 1.8255343731946851,
|
||||
"grad_norm": 1.2109375,
|
||||
"learning_rate": 2.2270456422483655e-07,
|
||||
"loss": 0.58580966,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 395,
|
||||
"train_speed(iter/s)": 0.047495
|
||||
},
|
||||
{
|
||||
"acc": 0.82003698,
|
||||
"epoch": 1.8486424032351243,
|
||||
"grad_norm": 1.125,
|
||||
"learning_rate": 1.6689574843694433e-07,
|
||||
"loss": 0.77600121,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 400,
|
||||
"train_speed(iter/s)": 0.04747
|
||||
},
|
||||
{
|
||||
"epoch": 1.8486424032351243,
|
||||
"eval_acc": 0.7489352050202878,
|
||||
"eval_loss": 0.6084072589874268,
|
||||
"eval_runtime": 11.0063,
|
||||
"eval_samples_per_second": 12.538,
|
||||
"eval_steps_per_second": 12.538,
|
||||
"step": 400
|
||||
},
|
||||
{
|
||||
"acc": 0.83795338,
|
||||
"epoch": 1.8717504332755632,
|
||||
"grad_norm": 1.3671875,
|
||||
"learning_rate": 1.1900726030241005e-07,
|
||||
"loss": 0.57120104,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 405,
|
||||
"train_speed(iter/s)": 0.046796
|
||||
},
|
||||
{
|
||||
"acc": 0.82940121,
|
||||
"epoch": 1.8948584633160022,
|
||||
"grad_norm": 1.2109375,
|
||||
"learning_rate": 7.911757785462882e-08,
|
||||
"loss": 0.63004894,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 410,
|
||||
"train_speed(iter/s)": 0.046799
|
||||
},
|
||||
{
|
||||
"acc": 0.82601376,
|
||||
"epoch": 1.9179664933564413,
|
||||
"grad_norm": 1.3125,
|
||||
"learning_rate": 4.729207095498345e-08,
|
||||
"loss": 0.5973454,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 415,
|
||||
"train_speed(iter/s)": 0.046849
|
||||
},
|
||||
{
|
||||
"acc": 0.83928432,
|
||||
"epoch": 1.9410745233968805,
|
||||
"grad_norm": 1.2578125,
|
||||
"learning_rate": 2.358289416693027e-08,
|
||||
"loss": 0.57047229,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 420,
|
||||
"train_speed(iter/s)": 0.046875
|
||||
},
|
||||
{
|
||||
"acc": 0.83347263,
|
||||
"epoch": 1.9641825534373196,
|
||||
"grad_norm": 1.1171875,
|
||||
"learning_rate": 8.02890128686562e-09,
|
||||
"loss": 0.56035862,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 425,
|
||||
"train_speed(iter/s)": 0.046877
|
||||
},
|
||||
{
|
||||
"acc": 0.83463163,
|
||||
"epoch": 1.9872905834777586,
|
||||
"grad_norm": 1.28125,
|
||||
"learning_rate": 6.555816718389896e-10,
|
||||
"loss": 0.57831531,
|
||||
"memory(GiB)": 150.98,
|
||||
"step": 430,
|
||||
"train_speed(iter/s)": 0.046917
|
||||
},
|
||||
{
|
||||
"epoch": 1.9965337954939342,
|
||||
"eval_acc": 0.7487839915320447,
|
||||
"eval_loss": 0.596267580986023,
|
||||
"eval_runtime": 11.0681,
|
||||
"eval_samples_per_second": 12.468,
|
||||
"eval_steps_per_second": 12.468,
|
||||
"step": 432
|
||||
}
|
||||
],
|
||||
"logging_steps": 5,
|
||||
"max_steps": 432,
|
||||
"num_input_tokens_seen": 0,
|
||||
"num_train_epochs": 2,
|
||||
"save_steps": 200,
|
||||
"stateful_callbacks": {
|
||||
"TrainerControl": {
|
||||
"args": {
|
||||
"should_epoch_stop": false,
|
||||
"should_evaluate": false,
|
||||
"should_log": false,
|
||||
"should_save": true,
|
||||
"should_training_stop": true
|
||||
},
|
||||
"attributes": {}
|
||||
}
|
||||
},
|
||||
"total_flos": 8.150084761585428e+17,
|
||||
"train_batch_size": 1,
|
||||
"trial_name": null,
|
||||
"trial_params": null
|
||||
}
|
||||
3
training_args.bin
Normal file
3
training_args.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8dda763da817afe2fac8e60998f9a920b3aecf4281f9307a159c366a1720deb6
|
||||
size 7288
|
||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user