初始化项目，由ModelHub XC社区提供模型
Model: PKU-Alignment/ProgressGym-HistLlama3-8B-C014-pretrain-v0.2 Source: Original Platform
This commit is contained in:
ModelHub XC
39
.gitattributes
vendored
Normal file
39
.gitattributes
vendored
Normal file
@@ -0,0 +1,39 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
model-00001-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
model-00002-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
model-00003-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
model-00004-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
147
README.md
Normal file
147
README.md
Normal file
@@ -0,0 +1,147 @@
|
||||
---
|
||||
license: cc-by-4.0
|
||||
tags:
|
||||
- alignment
|
||||
- value alignment
|
||||
- AI safety
|
||||
- safety
|
||||
- LLM
|
||||
- history
|
||||
datasets:
|
||||
- PKU-Alignment/ProgressGym-HistText
|
||||
base_model:
|
||||
- meta-llama/Meta-Llama-3-8B
|
||||
---
|
||||
|
||||
# ProgressGym-HistLlama3-8B-C014-pretrain
|
||||
|
||||
## Overview
|
||||
|
||||
#### The ProgressGym Framework
|
||||
|
||||
|
||||
|
||||
**ProgressGym-HistLlama3-8B-C014-pretrain** is part of the **ProgressGym** framework for research and experimentation on *progress alignment* - the emulation of moral progress in AI alignment algorithms, as a measure to prevent risks of societal value lock-in.
|
||||
|
||||
To quote the paper [*ProgressGym: Alignment with a Millennium of Moral Progress*](https://arxiv.org/abs/2406.20087):
|
||||
|
||||
> Frontier AI systems, including large language models (LLMs), hold increasing influence over the epistemology of human users. Such influence can reinforce prevailing societal values, potentially contributing to the lock-in of misguided moral beliefs and, consequently, the perpetuation of problematic moral practices on a broad scale.
|
||||
>
|
||||
> We introduce *progress alignment* as a technical solution to mitigate this imminent risk. Progress alignment algorithms learn to emulate the mechanics of human moral progress, thereby addressing the susceptibility of existing alignment methods to contemporary moral blindspots.
|
||||
|
||||
#### ProgressGym-HistLlama3-8B-C014-pretrain
|
||||
|
||||
ProgressGym-HistLlama3-8B-C014-pretrain is one of the **36 historical language models** in the ProgressGym framework. It is a pretrained model without instruction-tuning. For the instruction-tuned version, see [ProgressGym-HistLlama3-8B-C014-instruct](https://huggingface.co/PKU-Alignment/ProgressGym-HistLlama3-8B-C014-instruct).
|
||||
|
||||
**ProgressGym-HistLlama3-8B-C014-pretrain is under continual iteration.** Improving upon the current version, new versions of the model are currently being trained to reflect historical moral tendencies in ever more comprehensive ways.
|
||||
|
||||
**ProgressGym-HistLlama3-8B-C014-pretrain is a 14th-century historical language model.** Based on [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B), It is continued-pretrained on the 14th-century text data from [ProgressGym-HistText](https://huggingface.co/datasets/PKU-Alignment/ProgressGym-HistText), using the following hyperparameters:
|
||||
|
||||
- learning_rate: 1.5e-05
|
||||
- train_batch_size: 8
|
||||
- eval_batch_size: 16
|
||||
- seed: 42
|
||||
- distributed_type: multi-GPU
|
||||
- num_devices: 8
|
||||
- total_train_batch_size: 64
|
||||
- total_eval_batch_size: 128
|
||||
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
|
||||
- lr_scheduler_type: polynomial
|
||||
- lr_scheduler_warmup_steps: 20
|
||||
- num_epochs: 4.0
|
||||
- mixed_precision_training: Native AMP
|
||||
|
||||
... with the following training results:
|
||||
|
||||
| Training Loss | Epoch | Step | Validation Loss |
|
||||
|:-------------:|:------:|:----:|:---------------:|
|
||||
| 2.5789 | 0.0152 | 1 | 2.6458 |
|
||||
| 2.5672 | 0.0758 | 5 | 2.6280 |
|
||||
| 2.5751 | 0.1515 | 10 | 2.5314 |
|
||||
| 2.418 | 0.2273 | 15 | 2.4634 |
|
||||
| 2.4701 | 0.3030 | 20 | 2.4177 |
|
||||
| 2.3904 | 0.3788 | 25 | 2.3785 |
|
||||
| 2.3539 | 0.4545 | 30 | 2.3378 |
|
||||
| 2.3101 | 0.5303 | 35 | 2.3082 |
|
||||
| 2.3254 | 0.6061 | 40 | 2.2816 |
|
||||
| 2.2762 | 0.6818 | 45 | 2.2614 |
|
||||
| 2.2525 | 0.7576 | 50 | 2.2458 |
|
||||
| 2.2777 | 0.8333 | 55 | 2.2321 |
|
||||
| 2.2054 | 0.9091 | 60 | 2.2206 |
|
||||
| 2.237 | 0.9848 | 65 | 2.2113 |
|
||||
| 1.986 | 1.0606 | 70 | 2.2115 |
|
||||
| 1.9373 | 1.1364 | 75 | 2.2217 |
|
||||
| 1.9228 | 1.2121 | 80 | 2.2132 |
|
||||
| 1.9084 | 1.2879 | 85 | 2.2118 |
|
||||
| 1.9684 | 1.3636 | 90 | 2.2122 |
|
||||
| 1.9126 | 1.4394 | 95 | 2.2094 |
|
||||
| 1.9101 | 1.5152 | 100 | 2.2066 |
|
||||
| 1.8496 | 1.5909 | 105 | 2.2058 |
|
||||
| 1.9154 | 1.6667 | 110 | 2.2057 |
|
||||
| 1.9233 | 1.7424 | 115 | 2.2056 |
|
||||
| 1.9198 | 1.8182 | 120 | 2.2052 |
|
||||
| 1.9229 | 1.8939 | 125 | 2.2048 |
|
||||
| 1.8913 | 1.9697 | 130 | 2.2045 |
|
||||
| 1.8814 | 2.0455 | 135 | 2.2046 |
|
||||
| 1.8813 | 2.1212 | 140 | 2.2051 |
|
||||
| 1.8912 | 2.1970 | 145 | 2.2058 |
|
||||
| 1.9184 | 2.2727 | 150 | 2.2065 |
|
||||
| 1.8662 | 2.3485 | 155 | 2.2071 |
|
||||
| 1.8809 | 2.4242 | 160 | 2.2074 |
|
||||
| 1.8591 | 2.5 | 165 | 2.2077 |
|
||||
| 1.8731 | 2.5758 | 170 | 2.2079 |
|
||||
| 1.8948 | 2.6515 | 175 | 2.2082 |
|
||||
| 1.8876 | 2.7273 | 180 | 2.2082 |
|
||||
| 1.8408 | 2.8030 | 185 | 2.2083 |
|
||||
| 1.8931 | 2.8788 | 190 | 2.2082 |
|
||||
| 1.8569 | 2.9545 | 195 | 2.2080 |
|
||||
| 1.8621 | 3.0303 | 200 | 2.2079 |
|
||||
| 1.8863 | 3.1061 | 205 | 2.2078 |
|
||||
| 1.9021 | 3.1818 | 210 | 2.2079 |
|
||||
| 1.8648 | 3.2576 | 215 | 2.2080 |
|
||||
| 1.8443 | 3.3333 | 220 | 2.2081 |
|
||||
| 1.8978 | 3.4091 | 225 | 2.2080 |
|
||||
| 1.8658 | 3.4848 | 230 | 2.2080 |
|
||||
| 1.8706 | 3.5606 | 235 | 2.2079 |
|
||||
| 1.8855 | 3.6364 | 240 | 2.2078 |
|
||||
| 1.8535 | 3.7121 | 245 | 2.2078 |
|
||||
| 1.9062 | 3.7879 | 250 | 2.2079 |
|
||||
| 1.8628 | 3.8636 | 255 | 2.2078 |
|
||||
| 1.8484 | 3.9394 | 260 | 2.2077 |
|
||||
|
||||
Note that the training data volume for the continued pretraining stage is capped at 3GB. When the corresponding century's corpus exceeds this volume, the training data is randomly sampled to fit the volume.
|
||||
|
||||
## Links
|
||||
|
||||
- **[Paper Preprint]** [ProgressGym: Alignment with a Millennium of Moral Progress](https://arxiv.org/abs/2406.20087)
|
||||
- **[Leaderboard & Interactive Playground]** [PKU-Alignment/ProgressGym-LeaderBoard](https://huggingface.co/spaces/PKU-Alignment/ProgressGym-LeaderBoard)
|
||||
- **[Huggingface Data & Model Collection]** [PKU-Alignment/ProgressGym](https://huggingface.co/collections/PKU-Alignment/progressgym-666735fcf3e4efa276226eaa)
|
||||
- **[Github Codebase]** [PKU-Alignment/ProgressGym](https://github.com/PKU-Alignment/ProgressGym)
|
||||
- **[Documentation]** [ProgressGym Documentation](https://pku-alignment.github.io/ProgressGym/)
|
||||
- **[PyPI Package]** *(coming soon - [stay tuned](https://forms.gle/1TWFLL4ZCLeYTD5N6)!)*
|
||||
|
||||
## Citation
|
||||
|
||||
If the datasets, models, or framework of ProgressGym help you in your project, please cite ProgressGym using the bibtex entry below.
|
||||
|
||||
```text
|
||||
@article{progressgym,
|
||||
title={ProgressGym: Alignment with a Millennium of Moral Progress},
|
||||
author={Tianyi Qiu and Yang Zhang and Xuchuan Huang and Jasmine Xinze Li and Jiaming Ji and Yaodong Yang},
|
||||
journal={arXiv preprint arXiv:2406.20087},
|
||||
eprint={2406.20087},
|
||||
eprinttype = {arXiv},
|
||||
year={2024}
|
||||
}
|
||||
```
|
||||
|
||||
## Ethics Statement
|
||||
|
||||
- **Copyright information of historical text data sources**:
|
||||
- Project Gutenberg, one among our four source of our historical text data, consists only of texts in the public domain.
|
||||
- For the text that we draw from Internet Archive, we only include those that uploaded by *Library of Congress*, which are texts freely released online by the U.S. Library of Congress for research and public use.
|
||||
- The text data from Early English Books Online are, according to their publisher, "freely available to the public" and "available for access, distribution, use, or reuse by anyone".
|
||||
- The last remaining source of our historical text data, the Pile of Law dataset, is released under a Creative Commons license, which we adhere to in our use.
|
||||
- **Reproducibility**: To ensure reproducibility, we open-source all the code involved in the production of our main results (including the entire pipeline starting from data collection and model training), as well as the supporting infrastructure (the ProgressGym framework), making replication as easy as running a few simple script files.
|
||||
- **Misuse Prevention**: In order to prevent potential misuse of progress alignment algorithms, we have carefully formulated progress alignment as strictly value-neutral, without *a priori* assumptions on the direction of progress. In the event of potential misuse of our dataset, we condemn any misuse attempt to the strongest degree possible, and will work with the research community on whistleblowing for such attempts.
|
||||
- **Open-Sourcing**: We confirm that our code, data, and models are to be open-sourced under a CC-BY 4.0 license. We will continue to maintain and update our open-source repositories and models.
|
||||
13
all_results.json
Normal file
13
all_results.json
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"epoch": 4.0,
|
||||
"eval_loss": 2.204540491104126,
|
||||
"eval_runtime": 6.002,
|
||||
"eval_samples_per_second": 77.474,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"perplexity": 9.066084664011177,
|
||||
"total_flos": 27428734894080.0,
|
||||
"train_loss": 2.005241002097274,
|
||||
"train_runtime": 8788.5675,
|
||||
"train_samples_per_second": 1.904,
|
||||
"train_steps_per_second": 0.03
|
||||
}
|
||||
28
config.json
Normal file
28
config.json
Normal file
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"_name_or_path": "/mnt/models-pku/progressalign/shared_storage/downloaded_models/llama3-8b-base",
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 128000,
|
||||
"eos_token_id": 128001,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 14336,
|
||||
"max_position_embeddings": 8192,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 8,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 500000.0,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "float16",
|
||||
"transformers_version": "4.40.0",
|
||||
"use_cache": false,
|
||||
"vocab_size": 128256
|
||||
}
|
||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
||||
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}
|
||||
8
eval_results.json
Normal file
8
eval_results.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"epoch": 4.0,
|
||||
"eval_loss": 2.204540491104126,
|
||||
"eval_runtime": 6.002,
|
||||
"eval_samples_per_second": 77.474,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"perplexity": 9.066084664011177
|
||||
}
|
||||
6
generation_config.json
Normal file
6
generation_config.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 128000,
|
||||
"eos_token_id": 128001,
|
||||
"transformers_version": "4.40.0"
|
||||
}
|
||||
3
model-00001-of-00004.safetensors
Normal file
3
model-00001-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:abd601d06605edc9e1d87e0fceb5b65d6c5485027c6718cd8aabd25533c38afb
|
||||
size 4976698592
|
||||
3
model-00002-of-00004.safetensors
Normal file
3
model-00002-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1bb32f0bee5519d5c68723ce31a39ac68f45e565bdb13d8f7b632f8cdde42e53
|
||||
size 4999802616
|
||||
3
model-00003-of-00004.safetensors
Normal file
3
model-00003-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:111e9e21c4fab4b10644511446b77da5697b363485c2d4db0f762550d857ef38
|
||||
size 4915916080
|
||||
3
model-00004-of-00004.safetensors
Normal file
3
model-00004-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:55f7864b6deff611a626d4e315757a572c73b28fe40bb2f6ca5504627362da2f
|
||||
size 1168138808
|
||||
298
model.safetensors.index.json
Normal file
298
model.safetensors.index.json
Normal file
@@ -0,0 +1,298 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_size": 16060522496
|
||||
},
|
||||
"weight_map": {
|
||||
"lm_head.weight": "model-00004-of-00004.safetensors",
|
||||
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.31.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.31.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.norm.weight": "model-00004-of-00004.safetensors"
|
||||
}
|
||||
}
|
||||
BIN
readme-assets/data-sources.png
Normal file
BIN
readme-assets/data-sources.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 85 KiB |
BIN
readme-assets/data-stats.png
Normal file
BIN
readme-assets/data-stats.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 210 KiB |
BIN
readme-assets/main-diagram.png
Normal file
BIN
readme-assets/main-diagram.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 178 KiB |
BIN
readme-assets/moral-evals.png
Normal file
BIN
readme-assets/moral-evals.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 218 KiB |
17
special_tokens_map.json
Normal file
17
special_tokens_map.json
Normal file
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<|begin_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": "<|end_of_text|>"
|
||||
}
|
||||
410504
tokenizer.json
Normal file
410504
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
2065
tokenizer_config.json
Normal file
2065
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
8
train_results.json
Normal file
8
train_results.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"epoch": 4.0,
|
||||
"total_flos": 27428734894080.0,
|
||||
"train_loss": 2.005241002097274,
|
||||
"train_runtime": 8788.5675,
|
||||
"train_samples_per_second": 1.904,
|
||||
"train_steps_per_second": 0.03
|
||||
}
|
||||
108
trainer_log.jsonl
Normal file
108
trainer_log.jsonl
Normal file
@@ -0,0 +1,108 @@
|
||||
{"current_steps": 1, "total_steps": 264, "loss": 2.5789, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 0.0, "epoch": 0.015151515151515152, "percentage": 0.38, "elapsed_time": "0:00:23", "remaining_time": "1:40:54"}
|
||||
{"current_steps": 1, "total_steps": 264, "loss": null, "eval_loss": 2.6458332538604736, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.015151515151515152, "percentage": 0.38, "elapsed_time": "0:00:23", "remaining_time": "1:40:54"}
|
||||
{"current_steps": 5, "total_steps": 264, "loss": 2.5672, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 2.25e-06, "epoch": 0.07575757575757576, "percentage": 1.89, "elapsed_time": "0:01:39", "remaining_time": "1:26:00"}
|
||||
{"current_steps": 5, "total_steps": 264, "loss": null, "eval_loss": 2.628009080886841, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.07575757575757576, "percentage": 1.89, "elapsed_time": "0:01:39", "remaining_time": "1:26:00"}
|
||||
{"current_steps": 10, "total_steps": 264, "loss": 2.5751, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 6e-06, "epoch": 0.15151515151515152, "percentage": 3.79, "elapsed_time": "0:04:34", "remaining_time": "1:56:02"}
|
||||
{"current_steps": 10, "total_steps": 264, "loss": null, "eval_loss": 2.5313849449157715, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.15151515151515152, "percentage": 3.79, "elapsed_time": "0:04:34", "remaining_time": "1:56:02"}
|
||||
{"current_steps": 15, "total_steps": 264, "loss": 2.418, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 9.75e-06, "epoch": 0.22727272727272727, "percentage": 5.68, "elapsed_time": "0:07:18", "remaining_time": "2:01:16"}
|
||||
{"current_steps": 15, "total_steps": 264, "loss": null, "eval_loss": 2.4634220600128174, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.22727272727272727, "percentage": 5.68, "elapsed_time": "0:07:18", "remaining_time": "2:01:16"}
|
||||
{"current_steps": 20, "total_steps": 264, "loss": 2.4701, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 1.3500000000000001e-05, "epoch": 0.30303030303030304, "percentage": 7.58, "elapsed_time": "0:10:10", "remaining_time": "2:04:09"}
|
||||
{"current_steps": 20, "total_steps": 264, "loss": null, "eval_loss": 2.4176573753356934, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.30303030303030304, "percentage": 7.58, "elapsed_time": "0:10:10", "remaining_time": "2:04:09"}
|
||||
{"current_steps": 25, "total_steps": 264, "loss": 2.3904, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 1.3097898548149108e-05, "epoch": 0.3787878787878788, "percentage": 9.47, "elapsed_time": "0:12:52", "remaining_time": "2:03:01"}
|
||||
{"current_steps": 25, "total_steps": 264, "loss": null, "eval_loss": 2.3785245418548584, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.3787878787878788, "percentage": 9.47, "elapsed_time": "0:12:52", "remaining_time": "2:03:01"}
|
||||
{"current_steps": 30, "total_steps": 264, "loss": 2.3539, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 1.041060545673204e-05, "epoch": 0.45454545454545453, "percentage": 11.36, "elapsed_time": "0:15:35", "remaining_time": "2:01:37"}
|
||||
{"current_steps": 30, "total_steps": 264, "loss": null, "eval_loss": 2.337780475616455, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.45454545454545453, "percentage": 11.36, "elapsed_time": "0:15:35", "remaining_time": "2:01:37"}
|
||||
{"current_steps": 35, "total_steps": 264, "loss": 2.3101, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 8.236247706221891e-06, "epoch": 0.5303030303030303, "percentage": 13.26, "elapsed_time": "0:18:21", "remaining_time": "2:00:05"}
|
||||
{"current_steps": 35, "total_steps": 264, "loss": null, "eval_loss": 2.3082308769226074, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.5303030303030303, "percentage": 13.26, "elapsed_time": "0:18:21", "remaining_time": "2:00:05"}
|
||||
{"current_steps": 40, "total_steps": 264, "loss": 2.3254, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 6.4849612135310325e-06, "epoch": 0.6060606060606061, "percentage": 15.15, "elapsed_time": "0:21:05", "remaining_time": "1:58:04"}
|
||||
{"current_steps": 40, "total_steps": 264, "loss": null, "eval_loss": 2.2816028594970703, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.6060606060606061, "percentage": 15.15, "elapsed_time": "0:21:05", "remaining_time": "1:58:04"}
|
||||
{"current_steps": 45, "total_steps": 264, "loss": 2.2762, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.081159821297093e-06, "epoch": 0.6818181818181818, "percentage": 17.05, "elapsed_time": "0:23:48", "remaining_time": "1:55:52"}
|
||||
{"current_steps": 45, "total_steps": 264, "loss": null, "eval_loss": 2.2614095211029053, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.6818181818181818, "percentage": 17.05, "elapsed_time": "0:23:48", "remaining_time": "1:55:52"}
|
||||
{"current_steps": 50, "total_steps": 264, "loss": 2.2525, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 3.961509285889694e-06, "epoch": 0.7575757575757576, "percentage": 18.94, "elapsed_time": "0:26:31", "remaining_time": "1:53:31"}
|
||||
{"current_steps": 50, "total_steps": 264, "loss": null, "eval_loss": 2.2457971572875977, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.7575757575757576, "percentage": 18.94, "elapsed_time": "0:26:31", "remaining_time": "1:53:31"}
|
||||
{"current_steps": 55, "total_steps": 264, "loss": 2.2777, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 3.073152889221908e-06, "epoch": 0.8333333333333334, "percentage": 20.83, "elapsed_time": "0:29:17", "remaining_time": "1:51:20"}
|
||||
{"current_steps": 55, "total_steps": 264, "loss": null, "eval_loss": 2.232052803039551, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.8333333333333334, "percentage": 20.83, "elapsed_time": "0:29:17", "remaining_time": "1:51:20"}
|
||||
{"current_steps": 60, "total_steps": 264, "loss": 2.2054, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 2.372162069694911e-06, "epoch": 0.9090909090909091, "percentage": 22.73, "elapsed_time": "0:32:03", "remaining_time": "1:48:59"}
|
||||
{"current_steps": 60, "total_steps": 264, "loss": null, "eval_loss": 2.2206437587738037, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.9090909090909091, "percentage": 22.73, "elapsed_time": "0:32:03", "remaining_time": "1:48:59"}
|
||||
{"current_steps": 65, "total_steps": 264, "loss": 2.237, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 1.8221877676625323e-06, "epoch": 0.9848484848484849, "percentage": 24.62, "elapsed_time": "0:34:53", "remaining_time": "1:46:48"}
|
||||
{"current_steps": 65, "total_steps": 264, "loss": null, "eval_loss": 2.2112882137298584, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 0.9848484848484849, "percentage": 24.62, "elapsed_time": "0:34:53", "remaining_time": "1:46:48"}
|
||||
{"current_steps": 70, "total_steps": 264, "loss": 1.986, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 1.3932903283558643e-06, "epoch": 1.0606060606060606, "percentage": 26.52, "elapsed_time": "0:37:41", "remaining_time": "1:44:26"}
|
||||
{"current_steps": 70, "total_steps": 264, "loss": null, "eval_loss": 2.2115273475646973, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.0606060606060606, "percentage": 26.52, "elapsed_time": "0:37:41", "remaining_time": "1:44:26"}
|
||||
{"current_steps": 75, "total_steps": 264, "loss": 1.9373, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 1.0609278071546894e-06, "epoch": 1.1363636363636362, "percentage": 28.41, "elapsed_time": "0:40:29", "remaining_time": "1:42:01"}
|
||||
{"current_steps": 75, "total_steps": 264, "loss": null, "eval_loss": 2.221705913543701, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.1363636363636362, "percentage": 28.41, "elapsed_time": "0:40:29", "remaining_time": "1:42:01"}
|
||||
{"current_steps": 80, "total_steps": 264, "loss": 1.9228, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 8.050843851687484e-07, "epoch": 1.2121212121212122, "percentage": 30.3, "elapsed_time": "0:43:16", "remaining_time": "1:39:30"}
|
||||
{"current_steps": 80, "total_steps": 264, "loss": null, "eval_loss": 2.2131617069244385, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.2121212121212122, "percentage": 30.3, "elapsed_time": "0:43:16", "remaining_time": "1:39:30"}
|
||||
{"current_steps": 85, "total_steps": 264, "loss": 1.9084, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 6.095223338761627e-07, "epoch": 1.2878787878787878, "percentage": 32.2, "elapsed_time": "0:46:02", "remaining_time": "1:36:56"}
|
||||
{"current_steps": 85, "total_steps": 264, "loss": null, "eval_loss": 2.2117583751678467, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.2878787878787878, "percentage": 32.2, "elapsed_time": "0:46:02", "remaining_time": "1:36:56"}
|
||||
{"current_steps": 90, "total_steps": 264, "loss": 1.9684, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 4.611425724763914e-07, "epoch": 1.3636363636363638, "percentage": 34.09, "elapsed_time": "0:48:50", "remaining_time": "1:34:26"}
|
||||
{"current_steps": 90, "total_steps": 264, "loss": null, "eval_loss": 2.212195634841919, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.3636363636363638, "percentage": 34.09, "elapsed_time": "0:48:50", "remaining_time": "1:34:26"}
|
||||
{"current_steps": 95, "total_steps": 264, "loss": 1.9126, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 3.494403469094348e-07, "epoch": 1.4393939393939394, "percentage": 35.98, "elapsed_time": "0:51:39", "remaining_time": "1:31:53"}
|
||||
{"current_steps": 95, "total_steps": 264, "loss": null, "eval_loss": 2.2093794345855713, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.4393939393939394, "percentage": 35.98, "elapsed_time": "0:51:39", "remaining_time": "1:31:53"}
|
||||
{"current_steps": 100, "total_steps": 264, "loss": 1.9101, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 2.660439312704735e-07, "epoch": 1.5151515151515151, "percentage": 37.88, "elapsed_time": "0:54:26", "remaining_time": "1:29:17"}
|
||||
{"current_steps": 100, "total_steps": 264, "loss": null, "eval_loss": 2.2066152095794678, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.5151515151515151, "percentage": 37.88, "elapsed_time": "0:54:26", "remaining_time": "1:29:17"}
|
||||
{"current_steps": 105, "total_steps": 264, "loss": 1.8496, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 2.0432551654866868e-07, "epoch": 1.5909090909090908, "percentage": 39.77, "elapsed_time": "0:57:14", "remaining_time": "1:26:40"}
|
||||
{"current_steps": 105, "total_steps": 264, "loss": null, "eval_loss": 2.2057933807373047, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.5909090909090908, "percentage": 39.77, "elapsed_time": "0:57:14", "remaining_time": "1:26:40"}
|
||||
{"current_steps": 110, "total_steps": 264, "loss": 1.9154, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 1.590746140201269e-07, "epoch": 1.6666666666666665, "percentage": 41.67, "elapsed_time": "1:00:01", "remaining_time": "1:24:01"}
|
||||
{"current_steps": 110, "total_steps": 264, "loss": null, "eval_loss": 2.205655097961426, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.6666666666666665, "percentage": 41.67, "elapsed_time": "1:00:01", "remaining_time": "1:24:01"}
|
||||
{"current_steps": 115, "total_steps": 264, "loss": 1.9233, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 1.2622536684767967e-07, "epoch": 1.7424242424242424, "percentage": 43.56, "elapsed_time": "1:02:49", "remaining_time": "1:21:24"}
|
||||
{"current_steps": 115, "total_steps": 264, "loss": null, "eval_loss": 2.2055680751800537, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.7424242424242424, "percentage": 43.56, "elapsed_time": "1:02:49", "remaining_time": "1:21:24"}
|
||||
{"current_steps": 120, "total_steps": 264, "loss": 1.9198, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 1.0263013894441628e-07, "epoch": 1.8181818181818183, "percentage": 45.45, "elapsed_time": "1:05:38", "remaining_time": "1:18:45"}
|
||||
{"current_steps": 120, "total_steps": 264, "loss": null, "eval_loss": 2.205195665359497, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.8181818181818183, "percentage": 45.45, "elapsed_time": "1:05:38", "remaining_time": "1:18:45"}
|
||||
{"current_steps": 125, "total_steps": 264, "loss": 1.9229, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 8.587264024428055e-08, "epoch": 1.893939393939394, "percentage": 47.35, "elapsed_time": "1:08:25", "remaining_time": "1:16:05"}
|
||||
{"current_steps": 125, "total_steps": 264, "loss": null, "eval_loss": 2.2048099040985107, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.893939393939394, "percentage": 47.35, "elapsed_time": "1:08:25", "remaining_time": "1:16:05"}
|
||||
{"current_steps": 130, "total_steps": 264, "loss": 1.8913, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 7.411465733236604e-08, "epoch": 1.9696969696969697, "percentage": 49.24, "elapsed_time": "1:11:13", "remaining_time": "1:13:24"}
|
||||
{"current_steps": 130, "total_steps": 264, "loss": null, "eval_loss": 2.204540491104126, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.9696969696969697, "percentage": 49.24, "elapsed_time": "1:11:13", "remaining_time": "1:13:24"}
|
||||
{"current_steps": 135, "total_steps": 264, "loss": 1.8814, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 6.59711929010128e-08, "epoch": 2.0454545454545454, "percentage": 51.14, "elapsed_time": "1:14:02", "remaining_time": "1:10:44"}
|
||||
{"current_steps": 135, "total_steps": 264, "loss": null, "eval_loss": 2.204589605331421, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.0454545454545454, "percentage": 51.14, "elapsed_time": "1:14:02", "remaining_time": "1:10:44"}
|
||||
{"current_steps": 140, "total_steps": 264, "loss": 1.8813, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 6.040948153695873e-08, "epoch": 2.121212121212121, "percentage": 53.03, "elapsed_time": "1:16:49", "remaining_time": "1:08:02"}
|
||||
{"current_steps": 140, "total_steps": 264, "loss": null, "eval_loss": 2.2050745487213135, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.121212121212121, "percentage": 53.03, "elapsed_time": "1:16:49", "remaining_time": "1:08:02"}
|
||||
{"current_steps": 145, "total_steps": 264, "loss": 1.8912, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.666794757151726e-08, "epoch": 2.196969696969697, "percentage": 54.92, "elapsed_time": "1:19:36", "remaining_time": "1:05:19"}
|
||||
{"current_steps": 145, "total_steps": 264, "loss": null, "eval_loss": 2.2057695388793945, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.196969696969697, "percentage": 54.92, "elapsed_time": "1:19:36", "remaining_time": "1:05:19"}
|
||||
{"current_steps": 150, "total_steps": 264, "loss": 1.9184, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.4191707642277796e-08, "epoch": 2.2727272727272725, "percentage": 56.82, "elapsed_time": "1:22:22", "remaining_time": "1:02:36"}
|
||||
{"current_steps": 150, "total_steps": 264, "loss": null, "eval_loss": 2.2065114974975586, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.2727272727272725, "percentage": 56.82, "elapsed_time": "1:22:22", "remaining_time": "1:02:36"}
|
||||
{"current_steps": 155, "total_steps": 264, "loss": 1.8662, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.258170056372994e-08, "epoch": 2.3484848484848486, "percentage": 58.71, "elapsed_time": "1:25:10", "remaining_time": "0:59:53"}
|
||||
{"current_steps": 155, "total_steps": 264, "loss": null, "eval_loss": 2.207070827484131, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.3484848484848486, "percentage": 58.71, "elapsed_time": "1:25:10", "remaining_time": "0:59:53"}
|
||||
{"current_steps": 160, "total_steps": 264, "loss": 1.8809, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.1554954268425945e-08, "epoch": 2.4242424242424243, "percentage": 60.61, "elapsed_time": "1:27:56", "remaining_time": "0:57:09"}
|
||||
{"current_steps": 160, "total_steps": 264, "loss": null, "eval_loss": 2.2073893547058105, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.4242424242424243, "percentage": 60.61, "elapsed_time": "1:27:56", "remaining_time": "0:57:09"}
|
||||
{"current_steps": 165, "total_steps": 264, "loss": 1.8591, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.091387798309037e-08, "epoch": 2.5, "percentage": 62.5, "elapsed_time": "1:30:44", "remaining_time": "0:54:26"}
|
||||
{"current_steps": 165, "total_steps": 264, "loss": null, "eval_loss": 2.2076644897460938, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.5, "percentage": 62.5, "elapsed_time": "1:30:44", "remaining_time": "0:54:26"}
|
||||
{"current_steps": 170, "total_steps": 264, "loss": 1.8731, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.0522801309078135e-08, "epoch": 2.5757575757575757, "percentage": 64.39, "elapsed_time": "1:33:31", "remaining_time": "0:51:42"}
|
||||
{"current_steps": 170, "total_steps": 264, "loss": null, "eval_loss": 2.2079408168792725, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.5757575757575757, "percentage": 64.39, "elapsed_time": "1:33:31", "remaining_time": "0:51:42"}
|
||||
{"current_steps": 175, "total_steps": 264, "loss": 1.8948, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.0290274187738543e-08, "epoch": 2.6515151515151514, "percentage": 66.29, "elapsed_time": "1:36:18", "remaining_time": "0:48:58"}
|
||||
{"current_steps": 175, "total_steps": 264, "loss": null, "eval_loss": 2.208183765411377, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.6515151515151514, "percentage": 66.29, "elapsed_time": "1:36:18", "remaining_time": "0:48:58"}
|
||||
{"current_steps": 180, "total_steps": 264, "loss": 1.8876, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.015589639287439e-08, "epoch": 2.7272727272727275, "percentage": 68.18, "elapsed_time": "1:39:06", "remaining_time": "0:46:15"}
|
||||
{"current_steps": 180, "total_steps": 264, "loss": null, "eval_loss": 2.208235740661621, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.7272727272727275, "percentage": 68.18, "elapsed_time": "1:39:06", "remaining_time": "0:46:15"}
|
||||
{"current_steps": 185, "total_steps": 264, "loss": 1.8408, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.0080665589248236e-08, "epoch": 2.8030303030303028, "percentage": 70.08, "elapsed_time": "1:41:54", "remaining_time": "0:43:31"}
|
||||
{"current_steps": 185, "total_steps": 264, "loss": null, "eval_loss": 2.2083210945129395, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.8030303030303028, "percentage": 70.08, "elapsed_time": "1:41:54", "remaining_time": "0:43:31"}
|
||||
{"current_steps": 190, "total_steps": 264, "loss": 1.8931, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.004002235298783e-08, "epoch": 2.878787878787879, "percentage": 71.97, "elapsed_time": "1:44:40", "remaining_time": "0:40:46"}
|
||||
{"current_steps": 190, "total_steps": 264, "loss": null, "eval_loss": 2.208212375640869, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.878787878787879, "percentage": 71.97, "elapsed_time": "1:44:40", "remaining_time": "0:40:46"}
|
||||
{"current_steps": 195, "total_steps": 264, "loss": 1.8569, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.001893193212864e-08, "epoch": 2.9545454545454546, "percentage": 73.86, "elapsed_time": "1:47:29", "remaining_time": "0:38:01"}
|
||||
{"current_steps": 195, "total_steps": 264, "loss": null, "eval_loss": 2.2080307006835938, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 2.9545454545454546, "percentage": 73.86, "elapsed_time": "1:47:29", "remaining_time": "0:38:01"}
|
||||
{"current_steps": 200, "total_steps": 264, "loss": 1.8621, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.000847883910016e-08, "epoch": 3.0303030303030303, "percentage": 75.76, "elapsed_time": "1:50:16", "remaining_time": "0:35:17"}
|
||||
{"current_steps": 200, "total_steps": 264, "loss": null, "eval_loss": 2.207866907119751, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.0303030303030303, "percentage": 75.76, "elapsed_time": "1:50:16", "remaining_time": "0:35:17"}
|
||||
{"current_steps": 205, "total_steps": 264, "loss": 1.8863, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.000356435775757e-08, "epoch": 3.106060606060606, "percentage": 77.65, "elapsed_time": "1:53:03", "remaining_time": "0:32:32"}
|
||||
{"current_steps": 205, "total_steps": 264, "loss": null, "eval_loss": 2.207792043685913, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.106060606060606, "percentage": 77.65, "elapsed_time": "1:53:03", "remaining_time": "0:32:32"}
|
||||
{"current_steps": 210, "total_steps": 264, "loss": 1.9021, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.0001391301969795e-08, "epoch": 3.1818181818181817, "percentage": 79.55, "elapsed_time": "1:55:50", "remaining_time": "0:29:47"}
|
||||
{"current_steps": 210, "total_steps": 264, "loss": null, "eval_loss": 2.2078535556793213, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.1818181818181817, "percentage": 79.55, "elapsed_time": "1:55:50", "remaining_time": "0:29:47"}
|
||||
{"current_steps": 215, "total_steps": 264, "loss": 1.8648, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.000049730753554e-08, "epoch": 3.257575757575758, "percentage": 81.44, "elapsed_time": "1:58:38", "remaining_time": "0:27:02"}
|
||||
{"current_steps": 215, "total_steps": 264, "loss": null, "eval_loss": 2.2079594135284424, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.257575757575758, "percentage": 81.44, "elapsed_time": "1:58:38", "remaining_time": "0:27:02"}
|
||||
{"current_steps": 220, "total_steps": 264, "loss": 1.8443, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.0000159841391415e-08, "epoch": 3.3333333333333335, "percentage": 83.33, "elapsed_time": "2:01:24", "remaining_time": "0:24:16"}
|
||||
{"current_steps": 220, "total_steps": 264, "loss": null, "eval_loss": 2.2080650329589844, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.3333333333333335, "percentage": 83.33, "elapsed_time": "2:01:24", "remaining_time": "0:24:16"}
|
||||
{"current_steps": 225, "total_steps": 264, "loss": 1.8978, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.0000045079130105e-08, "epoch": 3.409090909090909, "percentage": 85.23, "elapsed_time": "2:04:13", "remaining_time": "0:21:31"}
|
||||
{"current_steps": 225, "total_steps": 264, "loss": null, "eval_loss": 2.20800518989563, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.409090909090909, "percentage": 85.23, "elapsed_time": "2:04:13", "remaining_time": "0:21:31"}
|
||||
{"current_steps": 230, "total_steps": 264, "loss": 1.8658, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.000001078153535e-08, "epoch": 3.484848484848485, "percentage": 87.12, "elapsed_time": "2:06:59", "remaining_time": "0:18:46"}
|
||||
{"current_steps": 230, "total_steps": 264, "loss": null, "eval_loss": 2.208038806915283, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.484848484848485, "percentage": 87.12, "elapsed_time": "2:06:59", "remaining_time": "0:18:46"}
|
||||
{"current_steps": 235, "total_steps": 264, "loss": 1.8706, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.0000002081285866e-08, "epoch": 3.5606060606060606, "percentage": 89.02, "elapsed_time": "2:09:46", "remaining_time": "0:16:00"}
|
||||
{"current_steps": 235, "total_steps": 264, "loss": null, "eval_loss": 2.207921266555786, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.5606060606060606, "percentage": 89.02, "elapsed_time": "2:09:46", "remaining_time": "0:16:00"}
|
||||
{"current_steps": 240, "total_steps": 264, "loss": 1.8855, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.0000000300649115e-08, "epoch": 3.6363636363636362, "percentage": 90.91, "elapsed_time": "2:12:33", "remaining_time": "0:13:15"}
|
||||
{"current_steps": 240, "total_steps": 264, "loss": null, "eval_loss": 2.2077724933624268, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.6363636363636362, "percentage": 90.91, "elapsed_time": "2:12:33", "remaining_time": "0:13:15"}
|
||||
{"current_steps": 245, "total_steps": 264, "loss": 1.8535, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.00000000286923e-08, "epoch": 3.712121212121212, "percentage": 92.8, "elapsed_time": "2:15:21", "remaining_time": "0:10:29"}
|
||||
{"current_steps": 245, "total_steps": 264, "loss": null, "eval_loss": 2.2078235149383545, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.712121212121212, "percentage": 92.8, "elapsed_time": "2:15:21", "remaining_time": "0:10:29"}
|
||||
{"current_steps": 250, "total_steps": 264, "loss": 1.9062, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.0000000001441026e-08, "epoch": 3.787878787878788, "percentage": 94.7, "elapsed_time": "2:18:08", "remaining_time": "0:07:44"}
|
||||
{"current_steps": 250, "total_steps": 264, "loss": null, "eval_loss": 2.207879066467285, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.787878787878788, "percentage": 94.7, "elapsed_time": "2:18:08", "remaining_time": "0:07:44"}
|
||||
{"current_steps": 255, "total_steps": 264, "loss": 1.8628, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.000000000002337e-08, "epoch": 3.8636363636363638, "percentage": 96.59, "elapsed_time": "2:20:53", "remaining_time": "0:04:58"}
|
||||
{"current_steps": 255, "total_steps": 264, "loss": null, "eval_loss": 2.2078306674957275, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.8636363636363638, "percentage": 96.59, "elapsed_time": "2:20:53", "remaining_time": "0:04:58"}
|
||||
{"current_steps": 260, "total_steps": 264, "loss": 1.8484, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 5.0000000000000024e-08, "epoch": 3.9393939393939394, "percentage": 98.48, "elapsed_time": "2:23:40", "remaining_time": "0:02:12"}
|
||||
{"current_steps": 260, "total_steps": 264, "loss": null, "eval_loss": 2.207735776901245, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 3.9393939393939394, "percentage": 98.48, "elapsed_time": "2:23:40", "remaining_time": "0:02:12"}
|
||||
{"current_steps": 264, "total_steps": 264, "loss": null, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 4.0, "percentage": 100.0, "elapsed_time": "2:26:13", "remaining_time": "0:00:00"}
|
||||
{"current_steps": 4, "total_steps": 4, "loss": null, "eval_loss": 2.204540491104126, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 4.0, "percentage": 100.0, "elapsed_time": "2:27:01", "remaining_time": "0:00:00"}
|
||||
825
trainer_state.json
Normal file
825
trainer_state.json
Normal file
@@ -0,0 +1,825 @@
|
||||
{
|
||||
"best_metric": 2.204540491104126,
|
||||
"best_model_checkpoint": "./output/training_results/C014_llama3-8b-base_pretrain_20240428_005832/checkpoint-130",
|
||||
"epoch": 4.0,
|
||||
"eval_steps": 5,
|
||||
"global_step": 264,
|
||||
"is_hyper_param_search": false,
|
||||
"is_local_process_zero": true,
|
||||
"is_world_process_zero": true,
|
||||
"log_history": [
|
||||
{
|
||||
"epoch": 0.015151515151515152,
|
||||
"grad_norm": 0.0,
|
||||
"learning_rate": 0.0,
|
||||
"loss": 2.5789,
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"epoch": 0.015151515151515152,
|
||||
"eval_loss": 2.6458332538604736,
|
||||
"eval_runtime": 5.9609,
|
||||
"eval_samples_per_second": 78.008,
|
||||
"eval_steps_per_second": 0.671,
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"epoch": 0.07575757575757576,
|
||||
"grad_norm": 3.5510944022733923,
|
||||
"learning_rate": 2.25e-06,
|
||||
"loss": 2.5672,
|
||||
"step": 5
|
||||
},
|
||||
{
|
||||
"epoch": 0.07575757575757576,
|
||||
"eval_loss": 2.628009080886841,
|
||||
"eval_runtime": 5.9819,
|
||||
"eval_samples_per_second": 77.735,
|
||||
"eval_steps_per_second": 0.669,
|
||||
"step": 5
|
||||
},
|
||||
{
|
||||
"epoch": 0.15151515151515152,
|
||||
"grad_norm": 3.473712979202145,
|
||||
"learning_rate": 6e-06,
|
||||
"loss": 2.5751,
|
||||
"step": 10
|
||||
},
|
||||
{
|
||||
"epoch": 0.15151515151515152,
|
||||
"eval_loss": 2.5313849449157715,
|
||||
"eval_runtime": 5.9441,
|
||||
"eval_samples_per_second": 78.228,
|
||||
"eval_steps_per_second": 0.673,
|
||||
"step": 10
|
||||
},
|
||||
{
|
||||
"epoch": 0.22727272727272727,
|
||||
"grad_norm": 2.7296451314204835,
|
||||
"learning_rate": 9.75e-06,
|
||||
"loss": 2.418,
|
||||
"step": 15
|
||||
},
|
||||
{
|
||||
"epoch": 0.22727272727272727,
|
||||
"eval_loss": 2.4634220600128174,
|
||||
"eval_runtime": 6.0122,
|
||||
"eval_samples_per_second": 77.343,
|
||||
"eval_steps_per_second": 0.665,
|
||||
"step": 15
|
||||
},
|
||||
{
|
||||
"epoch": 0.30303030303030304,
|
||||
"grad_norm": 2.9793222204470453,
|
||||
"learning_rate": 1.3500000000000001e-05,
|
||||
"loss": 2.4701,
|
||||
"step": 20
|
||||
},
|
||||
{
|
||||
"epoch": 0.30303030303030304,
|
||||
"eval_loss": 2.4176573753356934,
|
||||
"eval_runtime": 5.9735,
|
||||
"eval_samples_per_second": 77.844,
|
||||
"eval_steps_per_second": 0.67,
|
||||
"step": 20
|
||||
},
|
||||
{
|
||||
"epoch": 0.3787878787878788,
|
||||
"grad_norm": 2.9152379669367696,
|
||||
"learning_rate": 1.3097898548149108e-05,
|
||||
"loss": 2.3904,
|
||||
"step": 25
|
||||
},
|
||||
{
|
||||
"epoch": 0.3787878787878788,
|
||||
"eval_loss": 2.3785245418548584,
|
||||
"eval_runtime": 5.9994,
|
||||
"eval_samples_per_second": 77.507,
|
||||
"eval_steps_per_second": 0.667,
|
||||
"step": 25
|
||||
},
|
||||
{
|
||||
"epoch": 0.45454545454545453,
|
||||
"grad_norm": 2.475241901636818,
|
||||
"learning_rate": 1.041060545673204e-05,
|
||||
"loss": 2.3539,
|
||||
"step": 30
|
||||
},
|
||||
{
|
||||
"epoch": 0.45454545454545453,
|
||||
"eval_loss": 2.337780475616455,
|
||||
"eval_runtime": 5.9932,
|
||||
"eval_samples_per_second": 77.587,
|
||||
"eval_steps_per_second": 0.667,
|
||||
"step": 30
|
||||
},
|
||||
{
|
||||
"epoch": 0.5303030303030303,
|
||||
"grad_norm": 2.5312379436441272,
|
||||
"learning_rate": 8.236247706221891e-06,
|
||||
"loss": 2.3101,
|
||||
"step": 35
|
||||
},
|
||||
{
|
||||
"epoch": 0.5303030303030303,
|
||||
"eval_loss": 2.3082308769226074,
|
||||
"eval_runtime": 5.9901,
|
||||
"eval_samples_per_second": 77.628,
|
||||
"eval_steps_per_second": 0.668,
|
||||
"step": 35
|
||||
},
|
||||
{
|
||||
"epoch": 0.6060606060606061,
|
||||
"grad_norm": 2.5279244134684804,
|
||||
"learning_rate": 6.4849612135310325e-06,
|
||||
"loss": 2.3254,
|
||||
"step": 40
|
||||
},
|
||||
{
|
||||
"epoch": 0.6060606060606061,
|
||||
"eval_loss": 2.2816028594970703,
|
||||
"eval_runtime": 5.9798,
|
||||
"eval_samples_per_second": 77.762,
|
||||
"eval_steps_per_second": 0.669,
|
||||
"step": 40
|
||||
},
|
||||
{
|
||||
"epoch": 0.6818181818181818,
|
||||
"grad_norm": 2.3848893916874836,
|
||||
"learning_rate": 5.081159821297093e-06,
|
||||
"loss": 2.2762,
|
||||
"step": 45
|
||||
},
|
||||
{
|
||||
"epoch": 0.6818181818181818,
|
||||
"eval_loss": 2.2614095211029053,
|
||||
"eval_runtime": 5.9833,
|
||||
"eval_samples_per_second": 77.716,
|
||||
"eval_steps_per_second": 0.669,
|
||||
"step": 45
|
||||
},
|
||||
{
|
||||
"epoch": 0.7575757575757576,
|
||||
"grad_norm": 2.55130938777181,
|
||||
"learning_rate": 3.961509285889694e-06,
|
||||
"loss": 2.2525,
|
||||
"step": 50
|
||||
},
|
||||
{
|
||||
"epoch": 0.7575757575757576,
|
||||
"eval_loss": 2.2457971572875977,
|
||||
"eval_runtime": 6.0002,
|
||||
"eval_samples_per_second": 77.497,
|
||||
"eval_steps_per_second": 0.667,
|
||||
"step": 50
|
||||
},
|
||||
{
|
||||
"epoch": 0.8333333333333334,
|
||||
"grad_norm": 2.2003836383976156,
|
||||
"learning_rate": 3.073152889221908e-06,
|
||||
"loss": 2.2777,
|
||||
"step": 55
|
||||
},
|
||||
{
|
||||
"epoch": 0.8333333333333334,
|
||||
"eval_loss": 2.232052803039551,
|
||||
"eval_runtime": 5.9752,
|
||||
"eval_samples_per_second": 77.822,
|
||||
"eval_steps_per_second": 0.669,
|
||||
"step": 55
|
||||
},
|
||||
{
|
||||
"epoch": 0.9090909090909091,
|
||||
"grad_norm": 1.938340466235554,
|
||||
"learning_rate": 2.372162069694911e-06,
|
||||
"loss": 2.2054,
|
||||
"step": 60
|
||||
},
|
||||
{
|
||||
"epoch": 0.9090909090909091,
|
||||
"eval_loss": 2.2206437587738037,
|
||||
"eval_runtime": 5.9984,
|
||||
"eval_samples_per_second": 77.52,
|
||||
"eval_steps_per_second": 0.667,
|
||||
"step": 60
|
||||
},
|
||||
{
|
||||
"epoch": 0.9848484848484849,
|
||||
"grad_norm": 2.0152410097081375,
|
||||
"learning_rate": 1.8221877676625323e-06,
|
||||
"loss": 2.237,
|
||||
"step": 65
|
||||
},
|
||||
{
|
||||
"epoch": 0.9848484848484849,
|
||||
"eval_loss": 2.2112882137298584,
|
||||
"eval_runtime": 6.0081,
|
||||
"eval_samples_per_second": 77.396,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"step": 65
|
||||
},
|
||||
{
|
||||
"epoch": 1.0606060606060606,
|
||||
"grad_norm": 2.206665008674913,
|
||||
"learning_rate": 1.3932903283558643e-06,
|
||||
"loss": 1.986,
|
||||
"step": 70
|
||||
},
|
||||
{
|
||||
"epoch": 1.0606060606060606,
|
||||
"eval_loss": 2.2115273475646973,
|
||||
"eval_runtime": 6.0121,
|
||||
"eval_samples_per_second": 77.344,
|
||||
"eval_steps_per_second": 0.665,
|
||||
"step": 70
|
||||
},
|
||||
{
|
||||
"epoch": 1.1363636363636362,
|
||||
"grad_norm": 2.4135250740210816,
|
||||
"learning_rate": 1.0609278071546894e-06,
|
||||
"loss": 1.9373,
|
||||
"step": 75
|
||||
},
|
||||
{
|
||||
"epoch": 1.1363636363636362,
|
||||
"eval_loss": 2.221705913543701,
|
||||
"eval_runtime": 5.966,
|
||||
"eval_samples_per_second": 77.942,
|
||||
"eval_steps_per_second": 0.67,
|
||||
"step": 75
|
||||
},
|
||||
{
|
||||
"epoch": 1.2121212121212122,
|
||||
"grad_norm": 2.0909203153994995,
|
||||
"learning_rate": 8.050843851687484e-07,
|
||||
"loss": 1.9228,
|
||||
"step": 80
|
||||
},
|
||||
{
|
||||
"epoch": 1.2121212121212122,
|
||||
"eval_loss": 2.2131617069244385,
|
||||
"eval_runtime": 6.0048,
|
||||
"eval_samples_per_second": 77.439,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"step": 80
|
||||
},
|
||||
{
|
||||
"epoch": 1.2878787878787878,
|
||||
"grad_norm": 2.0701966451221723,
|
||||
"learning_rate": 6.095223338761627e-07,
|
||||
"loss": 1.9084,
|
||||
"step": 85
|
||||
},
|
||||
{
|
||||
"epoch": 1.2878787878787878,
|
||||
"eval_loss": 2.2117583751678467,
|
||||
"eval_runtime": 6.0144,
|
||||
"eval_samples_per_second": 77.314,
|
||||
"eval_steps_per_second": 0.665,
|
||||
"step": 85
|
||||
},
|
||||
{
|
||||
"epoch": 1.3636363636363638,
|
||||
"grad_norm": 2.0411250764985907,
|
||||
"learning_rate": 4.611425724763914e-07,
|
||||
"loss": 1.9684,
|
||||
"step": 90
|
||||
},
|
||||
{
|
||||
"epoch": 1.3636363636363638,
|
||||
"eval_loss": 2.212195634841919,
|
||||
"eval_runtime": 6.005,
|
||||
"eval_samples_per_second": 77.436,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"step": 90
|
||||
},
|
||||
{
|
||||
"epoch": 1.4393939393939394,
|
||||
"grad_norm": 2.095523936778245,
|
||||
"learning_rate": 3.494403469094348e-07,
|
||||
"loss": 1.9126,
|
||||
"step": 95
|
||||
},
|
||||
{
|
||||
"epoch": 1.4393939393939394,
|
||||
"eval_loss": 2.2093794345855713,
|
||||
"eval_runtime": 5.9753,
|
||||
"eval_samples_per_second": 77.82,
|
||||
"eval_steps_per_second": 0.669,
|
||||
"step": 95
|
||||
},
|
||||
{
|
||||
"epoch": 1.5151515151515151,
|
||||
"grad_norm": 1.9491500689498569,
|
||||
"learning_rate": 2.660439312704735e-07,
|
||||
"loss": 1.9101,
|
||||
"step": 100
|
||||
},
|
||||
{
|
||||
"epoch": 1.5151515151515151,
|
||||
"eval_loss": 2.2066152095794678,
|
||||
"eval_runtime": 5.988,
|
||||
"eval_samples_per_second": 77.656,
|
||||
"eval_steps_per_second": 0.668,
|
||||
"step": 100
|
||||
},
|
||||
{
|
||||
"epoch": 1.5909090909090908,
|
||||
"grad_norm": 1.9297056004219728,
|
||||
"learning_rate": 2.0432551654866868e-07,
|
||||
"loss": 1.8496,
|
||||
"step": 105
|
||||
},
|
||||
{
|
||||
"epoch": 1.5909090909090908,
|
||||
"eval_loss": 2.2057933807373047,
|
||||
"eval_runtime": 6.029,
|
||||
"eval_samples_per_second": 77.127,
|
||||
"eval_steps_per_second": 0.663,
|
||||
"step": 105
|
||||
},
|
||||
{
|
||||
"epoch": 1.6666666666666665,
|
||||
"grad_norm": 2.0270269069156095,
|
||||
"learning_rate": 1.590746140201269e-07,
|
||||
"loss": 1.9154,
|
||||
"step": 110
|
||||
},
|
||||
{
|
||||
"epoch": 1.6666666666666665,
|
||||
"eval_loss": 2.205655097961426,
|
||||
"eval_runtime": 5.9865,
|
||||
"eval_samples_per_second": 77.675,
|
||||
"eval_steps_per_second": 0.668,
|
||||
"step": 110
|
||||
},
|
||||
{
|
||||
"epoch": 1.7424242424242424,
|
||||
"grad_norm": 2.0216556200111957,
|
||||
"learning_rate": 1.2622536684767967e-07,
|
||||
"loss": 1.9233,
|
||||
"step": 115
|
||||
},
|
||||
{
|
||||
"epoch": 1.7424242424242424,
|
||||
"eval_loss": 2.2055680751800537,
|
||||
"eval_runtime": 5.9969,
|
||||
"eval_samples_per_second": 77.54,
|
||||
"eval_steps_per_second": 0.667,
|
||||
"step": 115
|
||||
},
|
||||
{
|
||||
"epoch": 1.8181818181818183,
|
||||
"grad_norm": 2.121087680196771,
|
||||
"learning_rate": 1.0263013894441628e-07,
|
||||
"loss": 1.9198,
|
||||
"step": 120
|
||||
},
|
||||
{
|
||||
"epoch": 1.8181818181818183,
|
||||
"eval_loss": 2.205195665359497,
|
||||
"eval_runtime": 5.9814,
|
||||
"eval_samples_per_second": 77.741,
|
||||
"eval_steps_per_second": 0.669,
|
||||
"step": 120
|
||||
},
|
||||
{
|
||||
"epoch": 1.893939393939394,
|
||||
"grad_norm": 1.9887616789335223,
|
||||
"learning_rate": 8.587264024428055e-08,
|
||||
"loss": 1.9229,
|
||||
"step": 125
|
||||
},
|
||||
{
|
||||
"epoch": 1.893939393939394,
|
||||
"eval_loss": 2.2048099040985107,
|
||||
"eval_runtime": 6.0178,
|
||||
"eval_samples_per_second": 77.271,
|
||||
"eval_steps_per_second": 0.665,
|
||||
"step": 125
|
||||
},
|
||||
{
|
||||
"epoch": 1.9696969696969697,
|
||||
"grad_norm": 1.9737440679073275,
|
||||
"learning_rate": 7.411465733236604e-08,
|
||||
"loss": 1.8913,
|
||||
"step": 130
|
||||
},
|
||||
{
|
||||
"epoch": 1.9696969696969697,
|
||||
"eval_loss": 2.204540491104126,
|
||||
"eval_runtime": 6.0056,
|
||||
"eval_samples_per_second": 77.428,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"step": 130
|
||||
},
|
||||
{
|
||||
"epoch": 2.0454545454545454,
|
||||
"grad_norm": 2.037256369037435,
|
||||
"learning_rate": 6.59711929010128e-08,
|
||||
"loss": 1.8814,
|
||||
"step": 135
|
||||
},
|
||||
{
|
||||
"epoch": 2.0454545454545454,
|
||||
"eval_loss": 2.204589605331421,
|
||||
"eval_runtime": 6.0067,
|
||||
"eval_samples_per_second": 77.414,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"step": 135
|
||||
},
|
||||
{
|
||||
"epoch": 2.121212121212121,
|
||||
"grad_norm": 2.0713675635613353,
|
||||
"learning_rate": 6.040948153695873e-08,
|
||||
"loss": 1.8813,
|
||||
"step": 140
|
||||
},
|
||||
{
|
||||
"epoch": 2.121212121212121,
|
||||
"eval_loss": 2.2050745487213135,
|
||||
"eval_runtime": 5.9801,
|
||||
"eval_samples_per_second": 77.758,
|
||||
"eval_steps_per_second": 0.669,
|
||||
"step": 140
|
||||
},
|
||||
{
|
||||
"epoch": 2.196969696969697,
|
||||
"grad_norm": 1.9602573628685647,
|
||||
"learning_rate": 5.666794757151726e-08,
|
||||
"loss": 1.8912,
|
||||
"step": 145
|
||||
},
|
||||
{
|
||||
"epoch": 2.196969696969697,
|
||||
"eval_loss": 2.2057695388793945,
|
||||
"eval_runtime": 6.0053,
|
||||
"eval_samples_per_second": 77.431,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"step": 145
|
||||
},
|
||||
{
|
||||
"epoch": 2.2727272727272725,
|
||||
"grad_norm": 2.0631081811496133,
|
||||
"learning_rate": 5.4191707642277796e-08,
|
||||
"loss": 1.9184,
|
||||
"step": 150
|
||||
},
|
||||
{
|
||||
"epoch": 2.2727272727272725,
|
||||
"eval_loss": 2.2065114974975586,
|
||||
"eval_runtime": 5.9555,
|
||||
"eval_samples_per_second": 78.079,
|
||||
"eval_steps_per_second": 0.672,
|
||||
"step": 150
|
||||
},
|
||||
{
|
||||
"epoch": 2.3484848484848486,
|
||||
"grad_norm": 2.0253318844317705,
|
||||
"learning_rate": 5.258170056372994e-08,
|
||||
"loss": 1.8662,
|
||||
"step": 155
|
||||
},
|
||||
{
|
||||
"epoch": 2.3484848484848486,
|
||||
"eval_loss": 2.207070827484131,
|
||||
"eval_runtime": 6.0047,
|
||||
"eval_samples_per_second": 77.439,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"step": 155
|
||||
},
|
||||
{
|
||||
"epoch": 2.4242424242424243,
|
||||
"grad_norm": 2.073377995618171,
|
||||
"learning_rate": 5.1554954268425945e-08,
|
||||
"loss": 1.8809,
|
||||
"step": 160
|
||||
},
|
||||
{
|
||||
"epoch": 2.4242424242424243,
|
||||
"eval_loss": 2.2073893547058105,
|
||||
"eval_runtime": 6.0042,
|
||||
"eval_samples_per_second": 77.446,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"step": 160
|
||||
},
|
||||
{
|
||||
"epoch": 2.5,
|
||||
"grad_norm": 1.9895760567683491,
|
||||
"learning_rate": 5.091387798309037e-08,
|
||||
"loss": 1.8591,
|
||||
"step": 165
|
||||
},
|
||||
{
|
||||
"epoch": 2.5,
|
||||
"eval_loss": 2.2076644897460938,
|
||||
"eval_runtime": 5.9917,
|
||||
"eval_samples_per_second": 77.608,
|
||||
"eval_steps_per_second": 0.668,
|
||||
"step": 165
|
||||
},
|
||||
{
|
||||
"epoch": 2.5757575757575757,
|
||||
"grad_norm": 2.0833747351029372,
|
||||
"learning_rate": 5.0522801309078135e-08,
|
||||
"loss": 1.8731,
|
||||
"step": 170
|
||||
},
|
||||
{
|
||||
"epoch": 2.5757575757575757,
|
||||
"eval_loss": 2.2079408168792725,
|
||||
"eval_runtime": 6.0146,
|
||||
"eval_samples_per_second": 77.312,
|
||||
"eval_steps_per_second": 0.665,
|
||||
"step": 170
|
||||
},
|
||||
{
|
||||
"epoch": 2.6515151515151514,
|
||||
"grad_norm": 2.0037327741523825,
|
||||
"learning_rate": 5.0290274187738543e-08,
|
||||
"loss": 1.8948,
|
||||
"step": 175
|
||||
},
|
||||
{
|
||||
"epoch": 2.6515151515151514,
|
||||
"eval_loss": 2.208183765411377,
|
||||
"eval_runtime": 6.0305,
|
||||
"eval_samples_per_second": 77.108,
|
||||
"eval_steps_per_second": 0.663,
|
||||
"step": 175
|
||||
},
|
||||
{
|
||||
"epoch": 2.7272727272727275,
|
||||
"grad_norm": 2.0305444519091917,
|
||||
"learning_rate": 5.015589639287439e-08,
|
||||
"loss": 1.8876,
|
||||
"step": 180
|
||||
},
|
||||
{
|
||||
"epoch": 2.7272727272727275,
|
||||
"eval_loss": 2.208235740661621,
|
||||
"eval_runtime": 5.9955,
|
||||
"eval_samples_per_second": 77.558,
|
||||
"eval_steps_per_second": 0.667,
|
||||
"step": 180
|
||||
},
|
||||
{
|
||||
"epoch": 2.8030303030303028,
|
||||
"grad_norm": 2.0062076076889266,
|
||||
"learning_rate": 5.0080665589248236e-08,
|
||||
"loss": 1.8408,
|
||||
"step": 185
|
||||
},
|
||||
{
|
||||
"epoch": 2.8030303030303028,
|
||||
"eval_loss": 2.2083210945129395,
|
||||
"eval_runtime": 6.0004,
|
||||
"eval_samples_per_second": 77.495,
|
||||
"eval_steps_per_second": 0.667,
|
||||
"step": 185
|
||||
},
|
||||
{
|
||||
"epoch": 2.878787878787879,
|
||||
"grad_norm": 2.043895270641295,
|
||||
"learning_rate": 5.004002235298783e-08,
|
||||
"loss": 1.8931,
|
||||
"step": 190
|
||||
},
|
||||
{
|
||||
"epoch": 2.878787878787879,
|
||||
"eval_loss": 2.208212375640869,
|
||||
"eval_runtime": 6.0002,
|
||||
"eval_samples_per_second": 77.498,
|
||||
"eval_steps_per_second": 0.667,
|
||||
"step": 190
|
||||
},
|
||||
{
|
||||
"epoch": 2.9545454545454546,
|
||||
"grad_norm": 1.9689184449811679,
|
||||
"learning_rate": 5.001893193212864e-08,
|
||||
"loss": 1.8569,
|
||||
"step": 195
|
||||
},
|
||||
{
|
||||
"epoch": 2.9545454545454546,
|
||||
"eval_loss": 2.2080307006835938,
|
||||
"eval_runtime": 5.9791,
|
||||
"eval_samples_per_second": 77.772,
|
||||
"eval_steps_per_second": 0.669,
|
||||
"step": 195
|
||||
},
|
||||
{
|
||||
"epoch": 3.0303030303030303,
|
||||
"grad_norm": 2.019410946950212,
|
||||
"learning_rate": 5.000847883910016e-08,
|
||||
"loss": 1.8621,
|
||||
"step": 200
|
||||
},
|
||||
{
|
||||
"epoch": 3.0303030303030303,
|
||||
"eval_loss": 2.207866907119751,
|
||||
"eval_runtime": 6.0003,
|
||||
"eval_samples_per_second": 77.496,
|
||||
"eval_steps_per_second": 0.667,
|
||||
"step": 200
|
||||
},
|
||||
{
|
||||
"epoch": 3.106060606060606,
|
||||
"grad_norm": 2.0283442915048644,
|
||||
"learning_rate": 5.000356435775757e-08,
|
||||
"loss": 1.8863,
|
||||
"step": 205
|
||||
},
|
||||
{
|
||||
"epoch": 3.106060606060606,
|
||||
"eval_loss": 2.207792043685913,
|
||||
"eval_runtime": 5.9858,
|
||||
"eval_samples_per_second": 77.684,
|
||||
"eval_steps_per_second": 0.668,
|
||||
"step": 205
|
||||
},
|
||||
{
|
||||
"epoch": 3.1818181818181817,
|
||||
"grad_norm": 2.05494798524839,
|
||||
"learning_rate": 5.0001391301969795e-08,
|
||||
"loss": 1.9021,
|
||||
"step": 210
|
||||
},
|
||||
{
|
||||
"epoch": 3.1818181818181817,
|
||||
"eval_loss": 2.2078535556793213,
|
||||
"eval_runtime": 5.9911,
|
||||
"eval_samples_per_second": 77.615,
|
||||
"eval_steps_per_second": 0.668,
|
||||
"step": 210
|
||||
},
|
||||
{
|
||||
"epoch": 3.257575757575758,
|
||||
"grad_norm": 2.0316955077847023,
|
||||
"learning_rate": 5.000049730753554e-08,
|
||||
"loss": 1.8648,
|
||||
"step": 215
|
||||
},
|
||||
{
|
||||
"epoch": 3.257575757575758,
|
||||
"eval_loss": 2.2079594135284424,
|
||||
"eval_runtime": 5.9692,
|
||||
"eval_samples_per_second": 77.9,
|
||||
"eval_steps_per_second": 0.67,
|
||||
"step": 215
|
||||
},
|
||||
{
|
||||
"epoch": 3.3333333333333335,
|
||||
"grad_norm": 2.0376583248479174,
|
||||
"learning_rate": 5.0000159841391415e-08,
|
||||
"loss": 1.8443,
|
||||
"step": 220
|
||||
},
|
||||
{
|
||||
"epoch": 3.3333333333333335,
|
||||
"eval_loss": 2.2080650329589844,
|
||||
"eval_runtime": 6.0237,
|
||||
"eval_samples_per_second": 77.195,
|
||||
"eval_steps_per_second": 0.664,
|
||||
"step": 220
|
||||
},
|
||||
{
|
||||
"epoch": 3.409090909090909,
|
||||
"grad_norm": 2.0309390490282864,
|
||||
"learning_rate": 5.0000045079130105e-08,
|
||||
"loss": 1.8978,
|
||||
"step": 225
|
||||
},
|
||||
{
|
||||
"epoch": 3.409090909090909,
|
||||
"eval_loss": 2.20800518989563,
|
||||
"eval_runtime": 6.0116,
|
||||
"eval_samples_per_second": 77.35,
|
||||
"eval_steps_per_second": 0.665,
|
||||
"step": 225
|
||||
},
|
||||
{
|
||||
"epoch": 3.484848484848485,
|
||||
"grad_norm": 2.0870431548602495,
|
||||
"learning_rate": 5.000001078153535e-08,
|
||||
"loss": 1.8658,
|
||||
"step": 230
|
||||
},
|
||||
{
|
||||
"epoch": 3.484848484848485,
|
||||
"eval_loss": 2.208038806915283,
|
||||
"eval_runtime": 5.99,
|
||||
"eval_samples_per_second": 77.629,
|
||||
"eval_steps_per_second": 0.668,
|
||||
"step": 230
|
||||
},
|
||||
{
|
||||
"epoch": 3.5606060606060606,
|
||||
"grad_norm": 2.0234951173098024,
|
||||
"learning_rate": 5.0000002081285866e-08,
|
||||
"loss": 1.8706,
|
||||
"step": 235
|
||||
},
|
||||
{
|
||||
"epoch": 3.5606060606060606,
|
||||
"eval_loss": 2.207921266555786,
|
||||
"eval_runtime": 5.9823,
|
||||
"eval_samples_per_second": 77.73,
|
||||
"eval_steps_per_second": 0.669,
|
||||
"step": 235
|
||||
},
|
||||
{
|
||||
"epoch": 3.6363636363636362,
|
||||
"grad_norm": 2.039329423643573,
|
||||
"learning_rate": 5.0000000300649115e-08,
|
||||
"loss": 1.8855,
|
||||
"step": 240
|
||||
},
|
||||
{
|
||||
"epoch": 3.6363636363636362,
|
||||
"eval_loss": 2.2077724933624268,
|
||||
"eval_runtime": 6.0052,
|
||||
"eval_samples_per_second": 77.433,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"step": 240
|
||||
},
|
||||
{
|
||||
"epoch": 3.712121212121212,
|
||||
"grad_norm": 2.048524032150131,
|
||||
"learning_rate": 5.00000000286923e-08,
|
||||
"loss": 1.8535,
|
||||
"step": 245
|
||||
},
|
||||
{
|
||||
"epoch": 3.712121212121212,
|
||||
"eval_loss": 2.2078235149383545,
|
||||
"eval_runtime": 6.0225,
|
||||
"eval_samples_per_second": 77.21,
|
||||
"eval_steps_per_second": 0.664,
|
||||
"step": 245
|
||||
},
|
||||
{
|
||||
"epoch": 3.787878787878788,
|
||||
"grad_norm": 2.117736519999513,
|
||||
"learning_rate": 5.0000000001441026e-08,
|
||||
"loss": 1.9062,
|
||||
"step": 250
|
||||
},
|
||||
{
|
||||
"epoch": 3.787878787878788,
|
||||
"eval_loss": 2.207879066467285,
|
||||
"eval_runtime": 5.9714,
|
||||
"eval_samples_per_second": 77.871,
|
||||
"eval_steps_per_second": 0.67,
|
||||
"step": 250
|
||||
},
|
||||
{
|
||||
"epoch": 3.8636363636363638,
|
||||
"grad_norm": 2.0315594950889775,
|
||||
"learning_rate": 5.000000000002337e-08,
|
||||
"loss": 1.8628,
|
||||
"step": 255
|
||||
},
|
||||
{
|
||||
"epoch": 3.8636363636363638,
|
||||
"eval_loss": 2.2078306674957275,
|
||||
"eval_runtime": 6.0145,
|
||||
"eval_samples_per_second": 77.314,
|
||||
"eval_steps_per_second": 0.665,
|
||||
"step": 255
|
||||
},
|
||||
{
|
||||
"epoch": 3.9393939393939394,
|
||||
"grad_norm": 2.0132729461699994,
|
||||
"learning_rate": 5.0000000000000024e-08,
|
||||
"loss": 1.8484,
|
||||
"step": 260
|
||||
},
|
||||
{
|
||||
"epoch": 3.9393939393939394,
|
||||
"eval_loss": 2.207735776901245,
|
||||
"eval_runtime": 6.0067,
|
||||
"eval_samples_per_second": 77.413,
|
||||
"eval_steps_per_second": 0.666,
|
||||
"step": 260
|
||||
},
|
||||
{
|
||||
"epoch": 4.0,
|
||||
"step": 264,
|
||||
"total_flos": 27428734894080.0,
|
||||
"train_loss": 2.005241002097274,
|
||||
"train_runtime": 8788.5675,
|
||||
"train_samples_per_second": 1.904,
|
||||
"train_steps_per_second": 0.03
|
||||
}
|
||||
],
|
||||
"logging_steps": 5,
|
||||
"max_steps": 264,
|
||||
"num_input_tokens_seen": 0,
|
||||
"num_train_epochs": 4,
|
||||
"save_steps": 5,
|
||||
"total_flos": 27428734894080.0,
|
||||
"train_batch_size": 8,
|
||||
"trial_name": null,
|
||||
"trial_params": null
|
||||
}
|
||||
3
training_args.bin
Normal file
3
training_args.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:001de46957edb45ce1e70a3c9079e08c3e4094b6ea33413856f85b4d6e6fbb5c
|
||||
size 6968
|
||||
BIN
training_eval_loss.png
Normal file
BIN
training_eval_loss.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
BIN
training_loss.png
Normal file
BIN
training_loss.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 41 KiB |
Reference in New Issue
Block a user