初始化项目，由ModelHub XC社区提供模型
Model: formalmathatepfl/deepseek-prover-v2-cpt Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
62
README.md
Normal file
62
README.md
Normal file
@@ -0,0 +1,62 @@
|
||||
---
|
||||
library_name: transformers
|
||||
license: other
|
||||
base_model: deepseek-ai/DeepSeek-Prover-V2-7B
|
||||
tags:
|
||||
- llama-factory
|
||||
- full
|
||||
- generated_from_trainer
|
||||
model-index:
|
||||
- name: deepseek-prover-v2-7b-lean-cpt-32k
|
||||
results: []
|
||||
---
|
||||
|
||||
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
||||
should probably proofread and complete it, then remove this comment. -->
|
||||
|
||||
# deepseek-prover-v2-7b-lean-cpt-32k
|
||||
|
||||
This model is a fine-tuned version of [deepseek-ai/DeepSeek-Prover-V2-7B](https://huggingface.co/deepseek-ai/DeepSeek-Prover-V2-7B) on the lean_cpt_scrape dataset.
|
||||
It achieves the following results on the evaluation set:
|
||||
- Loss: 0.6429
|
||||
|
||||
## Model description
|
||||
|
||||
More information needed
|
||||
|
||||
## Intended uses & limitations
|
||||
|
||||
More information needed
|
||||
|
||||
## Training and evaluation data
|
||||
|
||||
More information needed
|
||||
|
||||
## Training procedure
|
||||
|
||||
### Training hyperparameters
|
||||
|
||||
The following hyperparameters were used during training:
|
||||
- learning_rate: 5e-06
|
||||
- train_batch_size: 2
|
||||
- eval_batch_size: 2
|
||||
- seed: 42
|
||||
- distributed_type: multi-GPU
|
||||
- num_devices: 8
|
||||
- total_train_batch_size: 16
|
||||
- total_eval_batch_size: 16
|
||||
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
|
||||
- lr_scheduler_type: cosine
|
||||
- lr_scheduler_warmup_ratio: 0.03
|
||||
- num_epochs: 1.0
|
||||
|
||||
### Training results
|
||||
|
||||
|
||||
|
||||
### Framework versions
|
||||
|
||||
- Transformers 4.57.3
|
||||
- Pytorch 2.9.0+cu129
|
||||
- Datasets 3.6.0
|
||||
- Tokenizers 0.22.1
|
||||
13
all_results.json
Normal file
13
all_results.json
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"epoch": 1.0,
|
||||
"eval_loss": 0.6429444551467896,
|
||||
"eval_perplexity": 1.9020732113358507,
|
||||
"eval_runtime": 3.6621,
|
||||
"eval_samples_per_second": 4.096,
|
||||
"eval_steps_per_second": 0.273,
|
||||
"total_flos": 4.0836188273115136e+18,
|
||||
"train_loss": 0.746429123878479,
|
||||
"train_runtime": 2870.5475,
|
||||
"train_samples_per_second": 1.114,
|
||||
"train_steps_per_second": 0.07
|
||||
}
|
||||
14
chat_template.jinja
Normal file
14
chat_template.jinja
Normal file
@@ -0,0 +1,14 @@
|
||||
{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='', is_first_sp=true, is_last_user=false) %}{%- for message in messages %}{%- if message['role'] == 'system' %}{%- if ns.is_first_sp %}{% set ns.system_prompt = ns.system_prompt + message['content'] %}{% set ns.is_first_sp = false %}{%- else %}{% set ns.system_prompt = ns.system_prompt + '
|
||||
|
||||
' + message['content'] %}{%- endif %}{%- endif %}{%- endfor %}{{ bos_token }}{{ ns.system_prompt }}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{%- set ns.is_first = false -%}{%- set ns.is_last_user = true -%}{{'<｜User｜>' + message['content'] + '<｜Assistant｜>'}}{%- endif %}{%- if message['role'] == 'assistant' and message['tool_calls'] is defined and message['tool_calls'] is not none %}{%- set ns.is_last_user = false -%}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{%- endif %}{%- set ns.is_first = false %}{%- set ns.is_tool = false -%}{%- set ns.is_output_first = true %}{%- for tool in message['tool_calls'] %}{%- if not ns.is_first %}{%- if message['content'] is none %}{{'<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '
|
||||
' + '```json' + '
|
||||
' + tool['function']['arguments'] + '
|
||||
' + '```' + '<｜tool▁call▁end｜>'}}{%- else %}{{message['content'] + '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '
|
||||
' + '```json' + '
|
||||
' + tool['function']['arguments'] + '
|
||||
' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- set ns.is_first = true -%}{%- else %}{{'
|
||||
' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '
|
||||
' + '```json' + '
|
||||
' + tool['function']['arguments'] + '
|
||||
' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- endfor %}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- if message['role'] == 'assistant' and (message['tool_calls'] is not defined or message['tool_calls'] is none)%}{%- set ns.is_last_user = false -%}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{{content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_last_user = false -%}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'
|
||||
<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_last_user and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}
|
||||
38
config.json
Normal file
38
config.json
Normal file
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 100000,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 100001,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 11008,
|
||||
"max_position_embeddings": 65536,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 30,
|
||||
"num_key_value_heads": 32,
|
||||
"pad_token_id": 100001,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_scaling": {
|
||||
"beta_fast": 32,
|
||||
"beta_slow": 1,
|
||||
"factor": 16,
|
||||
"mscale": true,
|
||||
"original_max_position_embeddings": 4096,
|
||||
"rope_type": "yarn",
|
||||
"type": "yarn"
|
||||
},
|
||||
"rope_theta": 10000,
|
||||
"tie_word_embeddings": false,
|
||||
"transformers_version": "4.57.3",
|
||||
"use_cache": false,
|
||||
"vocab_size": 102400
|
||||
}
|
||||
8
eval_results.json
Normal file
8
eval_results.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"epoch": 1.0,
|
||||
"eval_loss": 0.6429444551467896,
|
||||
"eval_perplexity": 1.9020732113358507,
|
||||
"eval_runtime": 3.6621,
|
||||
"eval_samples_per_second": 4.096,
|
||||
"eval_steps_per_second": 0.273
|
||||
}
|
||||
10
generation_config.json
Normal file
10
generation_config.json
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 100000,
|
||||
"eos_token_id": [
|
||||
100001
|
||||
],
|
||||
"pad_token_id": 100001,
|
||||
"transformers_version": "4.57.3",
|
||||
"use_cache": false
|
||||
}
|
||||
3
model-00001-of-00003.safetensors
Normal file
3
model-00001-of-00003.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7f72301b08387df314704d235a22b52805bb31c35a1a0ae9bd5f7f2e9e900474
|
||||
size 4987202208
|
||||
3
model-00002-of-00003.safetensors
Normal file
3
model-00002-of-00003.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:470141b28c621f1697aa34efa3c2ce8bae2857db908c56e5fa67bb53f372c421
|
||||
size 4980945440
|
||||
3
model-00003-of-00003.safetensors
Normal file
3
model-00003-of-00003.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:77ca8b1787a1bbb43f9d628c1121df60f94e5d9c361d787f4c18483083843742
|
||||
size 3852615520
|
||||
281
model.safetensors.index.json
Normal file
281
model.safetensors.index.json
Normal file
@@ -0,0 +1,281 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_parameters": 6910365696,
|
||||
"total_size": 13820731392
|
||||
},
|
||||
"weight_map": {
|
||||
"lm_head.weight": "model-00003-of-00003.safetensors",
|
||||
"model.embed_tokens.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.22.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.24.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.25.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.26.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.27.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.28.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.28.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.28.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.28.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.28.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.29.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.29.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.29.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.29.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.29.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.29.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.29.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.norm.weight": "model-00003-of-00003.safetensors"
|
||||
}
|
||||
}
|
||||
23
special_tokens_map.json
Normal file
23
special_tokens_map.json
Normal file
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<｜begin▁of▁sentence｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<｜end▁of▁sentence｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<｜end▁of▁sentence｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
499330
tokenizer.json
Normal file
499330
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
164
tokenizer_config.json
Normal file
164
tokenizer_config.json
Normal file
@@ -0,0 +1,164 @@
|
||||
{
|
||||
"add_bos_token": true,
|
||||
"add_eos_token": false,
|
||||
"add_prefix_space": null,
|
||||
"added_tokens_decoder": {
|
||||
"100000": {
|
||||
"content": "<｜begin▁of▁sentence｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100001": {
|
||||
"content": "<｜end▁of▁sentence｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100002": {
|
||||
"content": "<｜fim▁hole｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100003": {
|
||||
"content": "<｜fim▁begin｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100004": {
|
||||
"content": "<｜fim▁end｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100005": {
|
||||
"content": "<｜completion｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100006": {
|
||||
"content": "<｜User｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100007": {
|
||||
"content": "<｜Assistant｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100008": {
|
||||
"content": "<|EOT|>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100009": {
|
||||
"content": "<｜tool▁calls▁begin｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100010": {
|
||||
"content": "<｜tool▁calls▁end｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100011": {
|
||||
"content": "<｜tool▁call▁begin｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100012": {
|
||||
"content": "<｜tool▁call▁end｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100013": {
|
||||
"content": "<｜tool▁outputs▁begin｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100014": {
|
||||
"content": "<｜tool▁outputs▁end｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100015": {
|
||||
"content": "<｜tool▁output▁begin｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100016": {
|
||||
"content": "<｜tool▁output▁end｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100017": {
|
||||
"content": "<｜tool▁sep｜>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
}
|
||||
},
|
||||
"bos_token": "<｜begin▁of▁sentence｜>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<｜end▁of▁sentence｜>",
|
||||
"extra_special_tokens": {},
|
||||
"legacy": true,
|
||||
"model_max_length": 131072,
|
||||
"pad_token": "<｜end▁of▁sentence｜>",
|
||||
"padding_side": "right",
|
||||
"sp_model_kwargs": {},
|
||||
"split_special_tokens": false,
|
||||
"tokenizer_class": "LlamaTokenizerFast",
|
||||
"unk_token": null,
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
8
train_results.json
Normal file
8
train_results.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"epoch": 1.0,
|
||||
"total_flos": 4.0836188273115136e+18,
|
||||
"train_loss": 0.746429123878479,
|
||||
"train_runtime": 2870.5475,
|
||||
"train_samples_per_second": 1.114,
|
||||
"train_steps_per_second": 0.07
|
||||
}
|
||||
41
trainer_log.jsonl
Normal file
41
trainer_log.jsonl
Normal file
@@ -0,0 +1,41 @@
|
||||
{"current_steps": 5, "total_steps": 200, "loss": 1.2871, "lr": 3.3333333333333333e-06, "epoch": 0.025, "percentage": 2.5, "elapsed_time": "0:01:14", "remaining_time": "0:48:15"}
|
||||
{"current_steps": 10, "total_steps": 200, "loss": 1.0775, "lr": 4.997050398198977e-06, "epoch": 0.05, "percentage": 5.0, "elapsed_time": "0:02:23", "remaining_time": "0:45:32"}
|
||||
{"current_steps": 15, "total_steps": 200, "loss": 0.9553, "lr": 4.979050253066064e-06, "epoch": 0.075, "percentage": 7.5, "elapsed_time": "0:03:33", "remaining_time": "0:43:53"}
|
||||
{"current_steps": 20, "total_steps": 200, "loss": 0.8635, "lr": 4.944806430988927e-06, "epoch": 0.1, "percentage": 10.0, "elapsed_time": "0:04:43", "remaining_time": "0:42:28"}
|
||||
{"current_steps": 25, "total_steps": 200, "loss": 0.8612, "lr": 4.894543310469968e-06, "epoch": 0.125, "percentage": 12.5, "elapsed_time": "0:05:52", "remaining_time": "0:41:09"}
|
||||
{"current_steps": 30, "total_steps": 200, "loss": 0.8138, "lr": 4.828590234527107e-06, "epoch": 0.15, "percentage": 15.0, "elapsed_time": "0:07:02", "remaining_time": "0:39:53"}
|
||||
{"current_steps": 35, "total_steps": 200, "loss": 0.7887, "lr": 4.747379352713489e-06, "epoch": 0.175, "percentage": 17.5, "elapsed_time": "0:08:12", "remaining_time": "0:38:39"}
|
||||
{"current_steps": 40, "total_steps": 200, "loss": 0.7501, "lr": 4.651442789509813e-06, "epoch": 0.2, "percentage": 20.0, "elapsed_time": "0:09:21", "remaining_time": "0:37:27"}
|
||||
{"current_steps": 45, "total_steps": 200, "loss": 0.7603, "lr": 4.541409157643027e-06, "epoch": 0.225, "percentage": 22.5, "elapsed_time": "0:10:31", "remaining_time": "0:36:14"}
|
||||
{"current_steps": 50, "total_steps": 200, "loss": 0.7377, "lr": 4.417999439177465e-06, "epoch": 0.25, "percentage": 25.0, "elapsed_time": "0:11:41", "remaining_time": "0:35:03"}
|
||||
{"current_steps": 55, "total_steps": 200, "loss": 0.7541, "lr": 4.282022261367074e-06, "epoch": 0.275, "percentage": 27.5, "elapsed_time": "0:12:50", "remaining_time": "0:33:52"}
|
||||
{"current_steps": 60, "total_steps": 200, "loss": 0.7207, "lr": 4.134368598223132e-06, "epoch": 0.3, "percentage": 30.0, "elapsed_time": "0:14:00", "remaining_time": "0:32:41"}
|
||||
{"current_steps": 65, "total_steps": 200, "loss": 0.7318, "lr": 3.976005932514807e-06, "epoch": 0.325, "percentage": 32.5, "elapsed_time": "0:15:10", "remaining_time": "0:31:30"}
|
||||
{"current_steps": 70, "total_steps": 200, "loss": 0.7166, "lr": 3.807971916455325e-06, "epoch": 0.35, "percentage": 35.0, "elapsed_time": "0:16:19", "remaining_time": "0:30:19"}
|
||||
{"current_steps": 75, "total_steps": 200, "loss": 0.7351, "lr": 3.631367572611348e-06, "epoch": 0.375, "percentage": 37.5, "elapsed_time": "0:17:29", "remaining_time": "0:29:09"}
|
||||
{"current_steps": 80, "total_steps": 200, "loss": 0.7192, "lr": 3.4473500795857674e-06, "epoch": 0.4, "percentage": 40.0, "elapsed_time": "0:18:39", "remaining_time": "0:27:59"}
|
||||
{"current_steps": 85, "total_steps": 200, "loss": 0.7018, "lr": 3.257125189744877e-06, "epoch": 0.425, "percentage": 42.5, "elapsed_time": "0:19:49", "remaining_time": "0:26:48"}
|
||||
{"current_steps": 90, "total_steps": 200, "loss": 0.6939, "lr": 3.061939328671824e-06, "epoch": 0.45, "percentage": 45.0, "elapsed_time": "0:20:58", "remaining_time": "0:25:38"}
|
||||
{"current_steps": 95, "total_steps": 200, "loss": 0.7129, "lr": 2.8630714281137263e-06, "epoch": 0.475, "percentage": 47.5, "elapsed_time": "0:22:08", "remaining_time": "0:24:28"}
|
||||
{"current_steps": 100, "total_steps": 200, "loss": 0.6993, "lr": 2.6618245459360896e-06, "epoch": 0.5, "percentage": 50.0, "elapsed_time": "0:23:18", "remaining_time": "0:23:18"}
|
||||
{"current_steps": 105, "total_steps": 200, "loss": 0.7106, "lr": 2.4595173279937464e-06, "epoch": 0.525, "percentage": 52.5, "elapsed_time": "0:24:28", "remaining_time": "0:22:08"}
|
||||
{"current_steps": 110, "total_steps": 200, "loss": 0.6975, "lr": 2.25747536786338e-06, "epoch": 0.55, "percentage": 55.0, "elapsed_time": "0:25:37", "remaining_time": "0:20:58"}
|
||||
{"current_steps": 115, "total_steps": 200, "loss": 0.7025, "lr": 2.0570225210519433e-06, "epoch": 0.575, "percentage": 57.5, "elapsed_time": "0:26:47", "remaining_time": "0:19:48"}
|
||||
{"current_steps": 120, "total_steps": 200, "loss": 0.6748, "lr": 1.8594722305935691e-06, "epoch": 0.6, "percentage": 60.0, "elapsed_time": "0:27:57", "remaining_time": "0:18:38"}
|
||||
{"current_steps": 125, "total_steps": 200, "loss": 0.7103, "lr": 1.6661189208729492e-06, "epoch": 0.625, "percentage": 62.5, "elapsed_time": "0:29:06", "remaining_time": "0:17:28"}
|
||||
{"current_steps": 130, "total_steps": 200, "loss": 0.6787, "lr": 1.4782295160661103e-06, "epoch": 0.65, "percentage": 65.0, "elapsed_time": "0:30:16", "remaining_time": "0:16:18"}
|
||||
{"current_steps": 135, "total_steps": 200, "loss": 0.6844, "lr": 1.2970351387729875e-06, "epoch": 0.675, "percentage": 67.5, "elapsed_time": "0:31:26", "remaining_time": "0:15:08"}
|
||||
{"current_steps": 140, "total_steps": 200, "loss": 0.6872, "lr": 1.1237230432354912e-06, "epoch": 0.7, "percentage": 70.0, "elapsed_time": "0:32:35", "remaining_time": "0:13:58"}
|
||||
{"current_steps": 145, "total_steps": 200, "loss": 0.6843, "lr": 9.594288359976817e-07, "epoch": 0.725, "percentage": 72.5, "elapsed_time": "0:33:45", "remaining_time": "0:12:48"}
|
||||
{"current_steps": 150, "total_steps": 200, "loss": 0.6983, "lr": 8.052290349812419e-07, "epoch": 0.75, "percentage": 75.0, "elapsed_time": "0:34:55", "remaining_time": "0:11:38"}
|
||||
{"current_steps": 155, "total_steps": 200, "loss": 0.6907, "lr": 6.621340157319998e-07, "epoch": 0.775, "percentage": 77.5, "elapsed_time": "0:36:04", "remaining_time": "0:10:28"}
|
||||
{"current_steps": 160, "total_steps": 200, "loss": 0.6765, "lr": 5.310813910563645e-07, "epoch": 0.8, "percentage": 80.0, "elapsed_time": "0:37:14", "remaining_time": "0:09:18"}
|
||||
{"current_steps": 165, "total_steps": 200, "loss": 0.682, "lr": 4.129298674268226e-07, "epoch": 0.825, "percentage": 82.5, "elapsed_time": "0:38:24", "remaining_time": "0:08:08"}
|
||||
{"current_steps": 170, "total_steps": 200, "loss": 0.6918, "lr": 3.08453618411631e-07, "epoch": 0.85, "percentage": 85.0, "elapsed_time": "0:39:34", "remaining_time": "0:06:58"}
|
||||
{"current_steps": 175, "total_steps": 200, "loss": 0.6782, "lr": 2.1833721199614992e-07, "epoch": 0.875, "percentage": 87.5, "elapsed_time": "0:40:43", "remaining_time": "0:05:49"}
|
||||
{"current_steps": 180, "total_steps": 200, "loss": 0.6997, "lr": 1.4317112503391433e-07, "epoch": 0.9, "percentage": 90.0, "elapsed_time": "0:41:53", "remaining_time": "0:04:39"}
|
||||
{"current_steps": 185, "total_steps": 200, "loss": 0.6741, "lr": 8.344787421847216e-08, "epoch": 0.925, "percentage": 92.5, "elapsed_time": "0:43:03", "remaining_time": "0:03:29"}
|
||||
{"current_steps": 190, "total_steps": 200, "loss": 0.685, "lr": 3.955878892731441e-08, "epoch": 0.95, "percentage": 95.0, "elapsed_time": "0:44:12", "remaining_time": "0:02:19"}
|
||||
{"current_steps": 195, "total_steps": 200, "loss": 0.6769, "lr": 1.1791447083465136e-08, "epoch": 0.975, "percentage": 97.5, "elapsed_time": "0:45:22", "remaining_time": "0:01:09"}
|
||||
{"current_steps": 200, "total_steps": 200, "loss": 0.693, "lr": 3.277908359194948e-10, "epoch": 1.0, "percentage": 100.0, "elapsed_time": "0:46:32", "remaining_time": "0:00:00"}
|
||||
{"current_steps": 200, "total_steps": 200, "epoch": 1.0, "percentage": 100.0, "elapsed_time": "0:47:45", "remaining_time": "0:00:00"}
|
||||
323
trainer_state.json
Normal file
323
trainer_state.json
Normal file
@@ -0,0 +1,323 @@
|
||||
{
|
||||
"best_global_step": null,
|
||||
"best_metric": null,
|
||||
"best_model_checkpoint": null,
|
||||
"epoch": 1.0,
|
||||
"eval_steps": 1000,
|
||||
"global_step": 200,
|
||||
"is_hyper_param_search": false,
|
||||
"is_local_process_zero": true,
|
||||
"is_world_process_zero": true,
|
||||
"log_history": [
|
||||
{
|
||||
"epoch": 0.025,
|
||||
"grad_norm": 7.022058486938477,
|
||||
"learning_rate": 3.3333333333333333e-06,
|
||||
"loss": 1.2871,
|
||||
"step": 5
|
||||
},
|
||||
{
|
||||
"epoch": 0.05,
|
||||
"grad_norm": 2.1450326442718506,
|
||||
"learning_rate": 4.997050398198977e-06,
|
||||
"loss": 1.0775,
|
||||
"step": 10
|
||||
},
|
||||
{
|
||||
"epoch": 0.075,
|
||||
"grad_norm": 1.9242591857910156,
|
||||
"learning_rate": 4.979050253066064e-06,
|
||||
"loss": 0.9553,
|
||||
"step": 15
|
||||
},
|
||||
{
|
||||
"epoch": 0.1,
|
||||
"grad_norm": 1.4348323345184326,
|
||||
"learning_rate": 4.944806430988927e-06,
|
||||
"loss": 0.8635,
|
||||
"step": 20
|
||||
},
|
||||
{
|
||||
"epoch": 0.125,
|
||||
"grad_norm": 1.0105319023132324,
|
||||
"learning_rate": 4.894543310469968e-06,
|
||||
"loss": 0.8612,
|
||||
"step": 25
|
||||
},
|
||||
{
|
||||
"epoch": 0.15,
|
||||
"grad_norm": 0.6607258319854736,
|
||||
"learning_rate": 4.828590234527107e-06,
|
||||
"loss": 0.8138,
|
||||
"step": 30
|
||||
},
|
||||
{
|
||||
"epoch": 0.175,
|
||||
"grad_norm": 0.4061996638774872,
|
||||
"learning_rate": 4.747379352713489e-06,
|
||||
"loss": 0.7887,
|
||||
"step": 35
|
||||
},
|
||||
{
|
||||
"epoch": 0.2,
|
||||
"grad_norm": 0.2776339054107666,
|
||||
"learning_rate": 4.651442789509813e-06,
|
||||
"loss": 0.7501,
|
||||
"step": 40
|
||||
},
|
||||
{
|
||||
"epoch": 0.225,
|
||||
"grad_norm": 0.25159376859664917,
|
||||
"learning_rate": 4.541409157643027e-06,
|
||||
"loss": 0.7603,
|
||||
"step": 45
|
||||
},
|
||||
{
|
||||
"epoch": 0.25,
|
||||
"grad_norm": 0.22889962792396545,
|
||||
"learning_rate": 4.417999439177465e-06,
|
||||
"loss": 0.7377,
|
||||
"step": 50
|
||||
},
|
||||
{
|
||||
"epoch": 0.275,
|
||||
"grad_norm": 0.25352779030799866,
|
||||
"learning_rate": 4.282022261367074e-06,
|
||||
"loss": 0.7541,
|
||||
"step": 55
|
||||
},
|
||||
{
|
||||
"epoch": 0.3,
|
||||
"grad_norm": 0.2564803957939148,
|
||||
"learning_rate": 4.134368598223132e-06,
|
||||
"loss": 0.7207,
|
||||
"step": 60
|
||||
},
|
||||
{
|
||||
"epoch": 0.325,
|
||||
"grad_norm": 0.21964561939239502,
|
||||
"learning_rate": 3.976005932514807e-06,
|
||||
"loss": 0.7318,
|
||||
"step": 65
|
||||
},
|
||||
{
|
||||
"epoch": 0.35,
|
||||
"grad_norm": 0.1952921450138092,
|
||||
"learning_rate": 3.807971916455325e-06,
|
||||
"loss": 0.7166,
|
||||
"step": 70
|
||||
},
|
||||
{
|
||||
"epoch": 0.375,
|
||||
"grad_norm": 0.21189448237419128,
|
||||
"learning_rate": 3.631367572611348e-06,
|
||||
"loss": 0.7351,
|
||||
"step": 75
|
||||
},
|
||||
{
|
||||
"epoch": 0.4,
|
||||
"grad_norm": 0.20247572660446167,
|
||||
"learning_rate": 3.4473500795857674e-06,
|
||||
"loss": 0.7192,
|
||||
"step": 80
|
||||
},
|
||||
{
|
||||
"epoch": 0.425,
|
||||
"grad_norm": 0.1986052691936493,
|
||||
"learning_rate": 3.257125189744877e-06,
|
||||
"loss": 0.7018,
|
||||
"step": 85
|
||||
},
|
||||
{
|
||||
"epoch": 0.45,
|
||||
"grad_norm": 0.20034098625183105,
|
||||
"learning_rate": 3.061939328671824e-06,
|
||||
"loss": 0.6939,
|
||||
"step": 90
|
||||
},
|
||||
{
|
||||
"epoch": 0.475,
|
||||
"grad_norm": 0.1981743723154068,
|
||||
"learning_rate": 2.8630714281137263e-06,
|
||||
"loss": 0.7129,
|
||||
"step": 95
|
||||
},
|
||||
{
|
||||
"epoch": 0.5,
|
||||
"grad_norm": 0.2070968598127365,
|
||||
"learning_rate": 2.6618245459360896e-06,
|
||||
"loss": 0.6993,
|
||||
"step": 100
|
||||
},
|
||||
{
|
||||
"epoch": 0.525,
|
||||
"grad_norm": 0.19754789769649506,
|
||||
"learning_rate": 2.4595173279937464e-06,
|
||||
"loss": 0.7106,
|
||||
"step": 105
|
||||
},
|
||||
{
|
||||
"epoch": 0.55,
|
||||
"grad_norm": 0.18637321889400482,
|
||||
"learning_rate": 2.25747536786338e-06,
|
||||
"loss": 0.6975,
|
||||
"step": 110
|
||||
},
|
||||
{
|
||||
"epoch": 0.575,
|
||||
"grad_norm": 0.1984306424856186,
|
||||
"learning_rate": 2.0570225210519433e-06,
|
||||
"loss": 0.7025,
|
||||
"step": 115
|
||||
},
|
||||
{
|
||||
"epoch": 0.6,
|
||||
"grad_norm": 0.19408036768436432,
|
||||
"learning_rate": 1.8594722305935691e-06,
|
||||
"loss": 0.6748,
|
||||
"step": 120
|
||||
},
|
||||
{
|
||||
"epoch": 0.625,
|
||||
"grad_norm": 0.19682057201862335,
|
||||
"learning_rate": 1.6661189208729492e-06,
|
||||
"loss": 0.7103,
|
||||
"step": 125
|
||||
},
|
||||
{
|
||||
"epoch": 0.65,
|
||||
"grad_norm": 0.2067941576242447,
|
||||
"learning_rate": 1.4782295160661103e-06,
|
||||
"loss": 0.6787,
|
||||
"step": 130
|
||||
},
|
||||
{
|
||||
"epoch": 0.675,
|
||||
"grad_norm": 0.19579337537288666,
|
||||
"learning_rate": 1.2970351387729875e-06,
|
||||
"loss": 0.6844,
|
||||
"step": 135
|
||||
},
|
||||
{
|
||||
"epoch": 0.7,
|
||||
"grad_norm": 0.19357725977897644,
|
||||
"learning_rate": 1.1237230432354912e-06,
|
||||
"loss": 0.6872,
|
||||
"step": 140
|
||||
},
|
||||
{
|
||||
"epoch": 0.725,
|
||||
"grad_norm": 0.19475950300693512,
|
||||
"learning_rate": 9.594288359976817e-07,
|
||||
"loss": 0.6843,
|
||||
"step": 145
|
||||
},
|
||||
{
|
||||
"epoch": 0.75,
|
||||
"grad_norm": 0.19206686317920685,
|
||||
"learning_rate": 8.052290349812419e-07,
|
||||
"loss": 0.6983,
|
||||
"step": 150
|
||||
},
|
||||
{
|
||||
"epoch": 0.775,
|
||||
"grad_norm": 0.18994426727294922,
|
||||
"learning_rate": 6.621340157319998e-07,
|
||||
"loss": 0.6907,
|
||||
"step": 155
|
||||
},
|
||||
{
|
||||
"epoch": 0.8,
|
||||
"grad_norm": 0.19607287645339966,
|
||||
"learning_rate": 5.310813910563645e-07,
|
||||
"loss": 0.6765,
|
||||
"step": 160
|
||||
},
|
||||
{
|
||||
"epoch": 0.825,
|
||||
"grad_norm": 0.18892605602741241,
|
||||
"learning_rate": 4.129298674268226e-07,
|
||||
"loss": 0.682,
|
||||
"step": 165
|
||||
},
|
||||
{
|
||||
"epoch": 0.85,
|
||||
"grad_norm": 0.20095601677894592,
|
||||
"learning_rate": 3.08453618411631e-07,
|
||||
"loss": 0.6918,
|
||||
"step": 170
|
||||
},
|
||||
{
|
||||
"epoch": 0.875,
|
||||
"grad_norm": 0.1872701197862625,
|
||||
"learning_rate": 2.1833721199614992e-07,
|
||||
"loss": 0.6782,
|
||||
"step": 175
|
||||
},
|
||||
{
|
||||
"epoch": 0.9,
|
||||
"grad_norm": 0.1944805085659027,
|
||||
"learning_rate": 1.4317112503391433e-07,
|
||||
"loss": 0.6997,
|
||||
"step": 180
|
||||
},
|
||||
{
|
||||
"epoch": 0.925,
|
||||
"grad_norm": 0.19366849958896637,
|
||||
"learning_rate": 8.344787421847216e-08,
|
||||
"loss": 0.6741,
|
||||
"step": 185
|
||||
},
|
||||
{
|
||||
"epoch": 0.95,
|
||||
"grad_norm": 0.18197080492973328,
|
||||
"learning_rate": 3.955878892731441e-08,
|
||||
"loss": 0.685,
|
||||
"step": 190
|
||||
},
|
||||
{
|
||||
"epoch": 0.975,
|
||||
"grad_norm": 0.18788814544677734,
|
||||
"learning_rate": 1.1791447083465136e-08,
|
||||
"loss": 0.6769,
|
||||
"step": 195
|
||||
},
|
||||
{
|
||||
"epoch": 1.0,
|
||||
"grad_norm": 0.2002282291650772,
|
||||
"learning_rate": 3.277908359194948e-10,
|
||||
"loss": 0.693,
|
||||
"step": 200
|
||||
},
|
||||
{
|
||||
"epoch": 1.0,
|
||||
"step": 200,
|
||||
"total_flos": 4.0836188273115136e+18,
|
||||
"train_loss": 0.746429123878479,
|
||||
"train_runtime": 2870.5475,
|
||||
"train_samples_per_second": 1.114,
|
||||
"train_steps_per_second": 0.07
|
||||
}
|
||||
],
|
||||
"logging_steps": 5,
|
||||
"max_steps": 200,
|
||||
"num_input_tokens_seen": 0,
|
||||
"num_train_epochs": 1,
|
||||
"save_steps": 1000,
|
||||
"stateful_callbacks": {
|
||||
"TrainerControl": {
|
||||
"args": {
|
||||
"should_epoch_stop": false,
|
||||
"should_evaluate": false,
|
||||
"should_log": false,
|
||||
"should_save": true,
|
||||
"should_training_stop": true
|
||||
},
|
||||
"attributes": {}
|
||||
}
|
||||
},
|
||||
"total_flos": 4.0836188273115136e+18,
|
||||
"train_batch_size": 2,
|
||||
"trial_name": null,
|
||||
"trial_params": null
|
||||
}
|
||||
3
training_args.bin
Normal file
3
training_args.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2b00467653fb3e37b0e63fb76400c6e65d92ac0c8958d7f915454d0a0f204093
|
||||
size 7953
|
||||
BIN
training_loss.png
Normal file
BIN
training_loss.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Reference in New Issue
Block a user