Josephgflowers/TinyLlama-Cinder-Agent-v1
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: Josephgflowers/TinyLlama-Cinder-Agent-v1
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-24 20:22:13 +08:00
commit b60fb39154
13 changed files with 1189 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
.gitattributes vendored Normal file
View File

@@ -0,0 +1,49 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.gguf* filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

67
README.md Normal file
View File

@@ -0,0 +1,67 @@
---
license: mit
tags:
- generated_from_trainer
base_model: Josephgflowers/TinyLlama-Cinder-Tiny-Agent
model-index:
- name: TinyLlama-Cinder-Agent-v1
results: []
---
The goal of this Model is to build a Tinyllama model that can be used for tool usage, RAG, take system instructions, and as a general assistant.
This model is a fine-tuned version of [Josephgflowers/TinyLlama-Cinder-Tiny-Agent](https://huggingface.co/Josephgflowers/TinyLlama-Cinder-Tiny-Agent).
Special Thanks to https://nationtech.io/ for their generous sponorship in training this model.
![image/png](https://cdn-uploads.huggingface.co/production/uploads/6328952f798f8d122ce62a44/MbN_SXChmMxuHO8GjdUSc.png)
This model is a fine-tuned version of [Josephgflowers/TinyLlama-3T-Cinder-v1.2](https://huggingface.co/Josephgflowers/TinyLlama-3T-Cinder-v1.2) on https://huggingface.co/datasets/Josephgflowers/agent_1.
## Model description
This models is trained for RAG, Summary, Function Calling and Tool usage. Trained off of Cinder. Cinder is a chatbot designed for chat about STEM topics, space adventure RP and storytelling.
This model does well at the IFEval (following instuctions) for its size. It is great at summary and RAG. Due to the formatting of the Glaive function calling dataset the JSON
output is not what I was expecting for doing regular JSON dumps but does follow their standard strictly.
*********************************************
10 X the original tinyllama model on GSM8K!!!
*********************************************
To do this I started with all the normal open math datasets EG Orca Math, All the Meta Math, camel AI math qa, ect and as many reasoning datasets as I could make or find.
But what really made it go the extra mile was adding in TIGER-Lab/WebInstructSub along with all of the RAG and Summary data.
So special thanks to TIGER-Lab. I found that as math perfomance improved so did the model's ability to handle extracting relevant data in RAG.
### Help Here
Like my work? Want to see more? Custom request? Message me on discord: joseph.flowers.ra Donate here: https://buymeacoffee.com/josephgflowers
See https://huggingface.co/Josephgflowers/TinyLlama-Cinder-Agent-Rag/blob/main/tinyllama_agent_cinder_txtai-rag.py
For usage example with wiki rag.
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Josephgflowers__TinyLlama-Cinder-Agent-v1)
| Metric |Value|
|---------------------------------|----:|
|Avg. |39.17|
|AI2 Reasoning Challenge (25-Shot)|34.90|
|HellaSwag (10-Shot) |53.87|
|MMLU (5-Shot) |26.89|
|TruthfulQA (0-shot) |39.08|
|Winogrande (5-shot) |59.12|
|GSM8k (5-shot) |21.15|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Josephgflowers__TinyLlama-Cinder-Agent-v1)
| Metric |Value|
|-------------------|----:|
|Avg. | 5.82|
|IFEval (0-Shot) |26.70|
|BBH (3-Shot) | 3.80|
|MATH Lvl 5 (4-Shot)| 0.38|
|GPQA (0-shot) | 0.00|
|MuSR (0-shot) | 2.23|
|MMLU-PRO (5-shot) | 1.79|

9
all_results.json Normal file
View File

@@ -0,0 +1,9 @@
{
"epoch": 1.0,
"total_flos": 1.955920022807249e+18,
"train_loss": 0.8700304180128008,
"train_runtime": 179145.9612,
"train_samples": 153863,
"train_samples_per_second": 0.859,
"train_steps_per_second": 0.072
}

29
config.json Normal file
View File

@@ -0,0 +1,29 @@
{
"_name_or_path": "Josephgflowers/TinyLlama-Cinder-Tiny-Agent",
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 2048,
"initializer_range": 0.02,
"intermediate_size": 5632,
"max_position_embeddings": 2048,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 22,
"num_key_value_heads": 4,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 10000.0,
"tie_word_embeddings": false,
"torch_dtype": "float32",
"transformers_version": "4.41.0.dev0",
"use_cache": false,
"vocab_size": 32000
}

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}

7
generation_config.json Normal file
View File

@@ -0,0 +1,7 @@
{
"bos_token_id": 1,
"eos_token_id": 2,
"max_length": 2048,
"pad_token_id": 0,
"transformers_version": "4.41.0.dev0"
}

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9c86cd1858d9891e7cd52ccb63a868d208a7a0731b84df044c8506a2f0a7e1fb
size 4400216536

30
special_tokens_map.json Normal file
View File

@@ -0,0 +1,30 @@
{
"bos_token": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bf467c9e0f536bda271283c6ef85eb1a943e3196b621c8a912d64953b205df83
size 1842795

41
tokenizer_config.json Normal file
View File

@@ -0,0 +1,41 @@
{
"add_bos_token": true,
"add_eos_token": false,
"added_tokens_decoder": {
"0": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"bos_token": "<s>",
"chat_template": "{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '<|user|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '<|system|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '<|assistant|>\n' + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '<|assistant|>' }}\n{% endif %}\n{% endfor %}",
"clean_up_tokenization_spaces": false,
"eos_token": "</s>",
"model_max_length": 2048,
"pad_token": "</s>",
"padding_side": "right",
"sp_model_kwargs": {},
"tokenizer_class": "LlamaTokenizer",
"unk_token": "<unk>",
"use_default_system_prompt": false
}

9
train_results.json Normal file
View File

@@ -0,0 +1,9 @@
{
"epoch": 1.0,
"total_flos": 1.955920022807249e+18,
"train_loss": 0.8700304180128008,
"train_runtime": 179145.9612,
"train_samples": 153863,
"train_samples_per_second": 0.859,
"train_steps_per_second": 0.072
}

938
trainer_state.json Normal file
View File

@@ -0,0 +1,938 @@
{
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 1.0,
"eval_steps": 500,
"global_step": 12822,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.007799095304944627,
"grad_norm": 63896.45703125,
"learning_rate": 4.961004523475277e-05,
"loss": 0.929,
"step": 100
},
{
"epoch": 0.015598190609889253,
"grad_norm": 79284.765625,
"learning_rate": 4.9220090469505544e-05,
"loss": 0.9281,
"step": 200
},
{
"epoch": 0.02339728591483388,
"grad_norm": 80169.203125,
"learning_rate": 4.883013570425831e-05,
"loss": 0.9385,
"step": 300
},
{
"epoch": 0.031196381219778507,
"grad_norm": 65649.6796875,
"learning_rate": 4.844018093901108e-05,
"loss": 0.9454,
"step": 400
},
{
"epoch": 0.038995476524723134,
"grad_norm": 82255.6953125,
"learning_rate": 4.805022617376384e-05,
"loss": 0.9368,
"step": 500
},
{
"epoch": 0.04679457182966776,
"grad_norm": 78376.4296875,
"learning_rate": 4.7660271408516614e-05,
"loss": 0.9318,
"step": 600
},
{
"epoch": 0.05459366713461238,
"grad_norm": 92200.078125,
"learning_rate": 4.7270316643269384e-05,
"loss": 0.9272,
"step": 700
},
{
"epoch": 0.06239276243955701,
"grad_norm": 88060.359375,
"learning_rate": 4.688036187802215e-05,
"loss": 0.9097,
"step": 800
},
{
"epoch": 0.07019185774450164,
"grad_norm": 76161.1796875,
"learning_rate": 4.649040711277492e-05,
"loss": 0.9475,
"step": 900
},
{
"epoch": 0.07799095304944627,
"grad_norm": 90524.1796875,
"learning_rate": 4.610045234752769e-05,
"loss": 0.9481,
"step": 1000
},
{
"epoch": 0.08579004835439089,
"grad_norm": 100124.8359375,
"learning_rate": 4.571049758228046e-05,
"loss": 0.9333,
"step": 1100
},
{
"epoch": 0.09358914365933552,
"grad_norm": 65681.7734375,
"learning_rate": 4.5320542817033225e-05,
"loss": 0.9247,
"step": 1200
},
{
"epoch": 0.10138823896428015,
"grad_norm": 76333.234375,
"learning_rate": 4.4930588051785995e-05,
"loss": 0.9098,
"step": 1300
},
{
"epoch": 0.10918733426922476,
"grad_norm": 77002.0703125,
"learning_rate": 4.4540633286538766e-05,
"loss": 0.9034,
"step": 1400
},
{
"epoch": 0.1169864295741694,
"grad_norm": 72669.1875,
"learning_rate": 4.415067852129153e-05,
"loss": 0.9149,
"step": 1500
},
{
"epoch": 0.12478552487911403,
"grad_norm": 71785.34375,
"learning_rate": 4.37607237560443e-05,
"loss": 0.9051,
"step": 1600
},
{
"epoch": 0.13258462018405864,
"grad_norm": 72728.5859375,
"learning_rate": 4.3370768990797065e-05,
"loss": 0.9268,
"step": 1700
},
{
"epoch": 0.14038371548900327,
"grad_norm": 69223.3203125,
"learning_rate": 4.2980814225549836e-05,
"loss": 0.9294,
"step": 1800
},
{
"epoch": 0.1481828107939479,
"grad_norm": 62745.203125,
"learning_rate": 4.2590859460302607e-05,
"loss": 0.9038,
"step": 1900
},
{
"epoch": 0.15598190609889254,
"grad_norm": 79897.8671875,
"learning_rate": 4.220090469505538e-05,
"loss": 0.8994,
"step": 2000
},
{
"epoch": 0.16378100140383717,
"grad_norm": 74131.546875,
"learning_rate": 4.181094992980815e-05,
"loss": 0.9144,
"step": 2100
},
{
"epoch": 0.17158009670878177,
"grad_norm": 67760.7109375,
"learning_rate": 4.142099516456091e-05,
"loss": 0.9285,
"step": 2200
},
{
"epoch": 0.1793791920137264,
"grad_norm": 78868.1484375,
"learning_rate": 4.103104039931368e-05,
"loss": 0.8973,
"step": 2300
},
{
"epoch": 0.18717828731867103,
"grad_norm": 71780.2109375,
"learning_rate": 4.064108563406645e-05,
"loss": 0.8961,
"step": 2400
},
{
"epoch": 0.19497738262361566,
"grad_norm": 77858.109375,
"learning_rate": 4.025113086881922e-05,
"loss": 0.9171,
"step": 2500
},
{
"epoch": 0.2027764779285603,
"grad_norm": 70062.796875,
"learning_rate": 3.986117610357199e-05,
"loss": 0.8903,
"step": 2600
},
{
"epoch": 0.21057557323350493,
"grad_norm": 171091.046875,
"learning_rate": 3.947122133832475e-05,
"loss": 0.9061,
"step": 2700
},
{
"epoch": 0.21837466853844953,
"grad_norm": 84371.34375,
"learning_rate": 3.908126657307752e-05,
"loss": 0.9315,
"step": 2800
},
{
"epoch": 0.22617376384339416,
"grad_norm": 62273.2578125,
"learning_rate": 3.8691311807830294e-05,
"loss": 0.9115,
"step": 2900
},
{
"epoch": 0.2339728591483388,
"grad_norm": 60464.15625,
"learning_rate": 3.8301357042583065e-05,
"loss": 0.8888,
"step": 3000
},
{
"epoch": 0.24177195445328342,
"grad_norm": 67546.828125,
"learning_rate": 3.7911402277335836e-05,
"loss": 0.9031,
"step": 3100
},
{
"epoch": 0.24957104975822805,
"grad_norm": 75904.6328125,
"learning_rate": 3.75214475120886e-05,
"loss": 0.9146,
"step": 3200
},
{
"epoch": 0.2573701450631727,
"grad_norm": 81015.3828125,
"learning_rate": 3.713149274684137e-05,
"loss": 0.8935,
"step": 3300
},
{
"epoch": 0.2651692403681173,
"grad_norm": 69763.3203125,
"learning_rate": 3.6741537981594134e-05,
"loss": 0.9129,
"step": 3400
},
{
"epoch": 0.27296833567306195,
"grad_norm": 68604.5,
"learning_rate": 3.6351583216346905e-05,
"loss": 0.8834,
"step": 3500
},
{
"epoch": 0.28076743097800655,
"grad_norm": 66758.03125,
"learning_rate": 3.596162845109967e-05,
"loss": 0.8925,
"step": 3600
},
{
"epoch": 0.28856652628295115,
"grad_norm": 72424.8984375,
"learning_rate": 3.557167368585244e-05,
"loss": 0.915,
"step": 3700
},
{
"epoch": 0.2963656215878958,
"grad_norm": 65904.9453125,
"learning_rate": 3.518171892060521e-05,
"loss": 0.8833,
"step": 3800
},
{
"epoch": 0.3041647168928404,
"grad_norm": 62548.9453125,
"learning_rate": 3.479176415535798e-05,
"loss": 0.8991,
"step": 3900
},
{
"epoch": 0.3119638121977851,
"grad_norm": 85173.90625,
"learning_rate": 3.440180939011075e-05,
"loss": 0.9111,
"step": 4000
},
{
"epoch": 0.3197629075027297,
"grad_norm": 65947.765625,
"learning_rate": 3.4011854624863516e-05,
"loss": 0.8774,
"step": 4100
},
{
"epoch": 0.32756200280767434,
"grad_norm": 69672.1875,
"learning_rate": 3.362189985961629e-05,
"loss": 0.8816,
"step": 4200
},
{
"epoch": 0.33536109811261894,
"grad_norm": 68125.9609375,
"learning_rate": 3.323194509436906e-05,
"loss": 0.8784,
"step": 4300
},
{
"epoch": 0.34316019341756354,
"grad_norm": 76767.6796875,
"learning_rate": 3.284199032912182e-05,
"loss": 0.899,
"step": 4400
},
{
"epoch": 0.3509592887225082,
"grad_norm": 64417.96484375,
"learning_rate": 3.245203556387459e-05,
"loss": 0.8944,
"step": 4500
},
{
"epoch": 0.3587583840274528,
"grad_norm": 62896.37109375,
"learning_rate": 3.206208079862736e-05,
"loss": 0.8745,
"step": 4600
},
{
"epoch": 0.36655747933239746,
"grad_norm": 58443.88671875,
"learning_rate": 3.167212603338013e-05,
"loss": 0.853,
"step": 4700
},
{
"epoch": 0.37435657463734207,
"grad_norm": 75019.8515625,
"learning_rate": 3.12821712681329e-05,
"loss": 0.8728,
"step": 4800
},
{
"epoch": 0.38215566994228667,
"grad_norm": 82645.0078125,
"learning_rate": 3.089221650288567e-05,
"loss": 0.8617,
"step": 4900
},
{
"epoch": 0.3899547652472313,
"grad_norm": 57014.08984375,
"learning_rate": 3.0502261737638436e-05,
"loss": 0.8566,
"step": 5000
},
{
"epoch": 0.39775386055217593,
"grad_norm": 71623.7578125,
"learning_rate": 3.01123069723912e-05,
"loss": 0.879,
"step": 5100
},
{
"epoch": 0.4055529558571206,
"grad_norm": 75911.140625,
"learning_rate": 2.972235220714397e-05,
"loss": 0.8889,
"step": 5200
},
{
"epoch": 0.4133520511620652,
"grad_norm": 69251.0703125,
"learning_rate": 2.9332397441896742e-05,
"loss": 0.8685,
"step": 5300
},
{
"epoch": 0.42115114646700985,
"grad_norm": 72434.6484375,
"learning_rate": 2.8942442676649513e-05,
"loss": 0.8895,
"step": 5400
},
{
"epoch": 0.42895024177195445,
"grad_norm": 69133.78125,
"learning_rate": 2.855248791140228e-05,
"loss": 0.8845,
"step": 5500
},
{
"epoch": 0.43674933707689906,
"grad_norm": 67216.6875,
"learning_rate": 2.8162533146155044e-05,
"loss": 0.8661,
"step": 5600
},
{
"epoch": 0.4445484323818437,
"grad_norm": 61408.3515625,
"learning_rate": 2.7772578380907815e-05,
"loss": 0.8743,
"step": 5700
},
{
"epoch": 0.4523475276867883,
"grad_norm": 66496.1640625,
"learning_rate": 2.7382623615660586e-05,
"loss": 0.8805,
"step": 5800
},
{
"epoch": 0.460146622991733,
"grad_norm": 62693.109375,
"learning_rate": 2.6992668850413357e-05,
"loss": 0.8634,
"step": 5900
},
{
"epoch": 0.4679457182966776,
"grad_norm": 77037.4140625,
"learning_rate": 2.660271408516612e-05,
"loss": 0.8834,
"step": 6000
},
{
"epoch": 0.4757448136016222,
"grad_norm": 67157.15625,
"learning_rate": 2.6212759319918888e-05,
"loss": 0.8993,
"step": 6100
},
{
"epoch": 0.48354390890656684,
"grad_norm": 77155.8125,
"learning_rate": 2.582280455467166e-05,
"loss": 0.854,
"step": 6200
},
{
"epoch": 0.49134300421151145,
"grad_norm": 65097.22265625,
"learning_rate": 2.543284978942443e-05,
"loss": 0.8716,
"step": 6300
},
{
"epoch": 0.4991420995164561,
"grad_norm": 89373.515625,
"learning_rate": 2.50428950241772e-05,
"loss": 0.873,
"step": 6400
},
{
"epoch": 0.5069411948214008,
"grad_norm": 65231.74609375,
"learning_rate": 2.4652940258929964e-05,
"loss": 0.8698,
"step": 6500
},
{
"epoch": 0.5147402901263454,
"grad_norm": 60228.23828125,
"learning_rate": 2.426298549368273e-05,
"loss": 0.8374,
"step": 6600
},
{
"epoch": 0.52253938543129,
"grad_norm": 73940.3125,
"learning_rate": 2.3873030728435502e-05,
"loss": 0.8669,
"step": 6700
},
{
"epoch": 0.5303384807362346,
"grad_norm": 80768.6953125,
"learning_rate": 2.3483075963188273e-05,
"loss": 0.8708,
"step": 6800
},
{
"epoch": 0.5381375760411792,
"grad_norm": 67096.984375,
"learning_rate": 2.309312119794104e-05,
"loss": 0.8797,
"step": 6900
},
{
"epoch": 0.5459366713461239,
"grad_norm": 56897.71875,
"learning_rate": 2.2703166432693808e-05,
"loss": 0.8728,
"step": 7000
},
{
"epoch": 0.5537357666510685,
"grad_norm": 64982.1015625,
"learning_rate": 2.2313211667446575e-05,
"loss": 0.8735,
"step": 7100
},
{
"epoch": 0.5615348619560131,
"grad_norm": 62320.35546875,
"learning_rate": 2.1923256902199346e-05,
"loss": 0.8519,
"step": 7200
},
{
"epoch": 0.5693339572609577,
"grad_norm": 74915.2578125,
"learning_rate": 2.1533302136952117e-05,
"loss": 0.849,
"step": 7300
},
{
"epoch": 0.5771330525659023,
"grad_norm": 69091.40625,
"learning_rate": 2.1143347371704884e-05,
"loss": 0.8451,
"step": 7400
},
{
"epoch": 0.584932147870847,
"grad_norm": 75015.2109375,
"learning_rate": 2.0753392606457652e-05,
"loss": 0.8757,
"step": 7500
},
{
"epoch": 0.5927312431757916,
"grad_norm": 59275.14453125,
"learning_rate": 2.036343784121042e-05,
"loss": 0.8657,
"step": 7600
},
{
"epoch": 0.6005303384807362,
"grad_norm": 64822.92578125,
"learning_rate": 1.997348307596319e-05,
"loss": 0.8555,
"step": 7700
},
{
"epoch": 0.6083294337856808,
"grad_norm": 63732.34765625,
"learning_rate": 1.9583528310715957e-05,
"loss": 0.8415,
"step": 7800
},
{
"epoch": 0.6161285290906254,
"grad_norm": 66452.859375,
"learning_rate": 1.9193573545468728e-05,
"loss": 0.8475,
"step": 7900
},
{
"epoch": 0.6239276243955701,
"grad_norm": 69227.2890625,
"learning_rate": 1.8803618780221496e-05,
"loss": 0.8333,
"step": 8000
},
{
"epoch": 0.6317267197005147,
"grad_norm": 67040.7578125,
"learning_rate": 1.8413664014974263e-05,
"loss": 0.8317,
"step": 8100
},
{
"epoch": 0.6395258150054594,
"grad_norm": 74801.2734375,
"learning_rate": 1.8023709249727034e-05,
"loss": 0.8726,
"step": 8200
},
{
"epoch": 0.647324910310404,
"grad_norm": 63112.703125,
"learning_rate": 1.76337544844798e-05,
"loss": 0.8442,
"step": 8300
},
{
"epoch": 0.6551240056153487,
"grad_norm": 81685.2265625,
"learning_rate": 1.724379971923257e-05,
"loss": 0.8316,
"step": 8400
},
{
"epoch": 0.6629231009202933,
"grad_norm": 69446.4921875,
"learning_rate": 1.6853844953985336e-05,
"loss": 0.8391,
"step": 8500
},
{
"epoch": 0.6707221962252379,
"grad_norm": 66903.671875,
"learning_rate": 1.6463890188738107e-05,
"loss": 0.8144,
"step": 8600
},
{
"epoch": 0.6785212915301825,
"grad_norm": 76092.6328125,
"learning_rate": 1.6073935423490877e-05,
"loss": 0.8441,
"step": 8700
},
{
"epoch": 0.6863203868351271,
"grad_norm": 71659.8515625,
"learning_rate": 1.5683980658243645e-05,
"loss": 0.8388,
"step": 8800
},
{
"epoch": 0.6941194821400718,
"grad_norm": 61097.1171875,
"learning_rate": 1.5294025892996412e-05,
"loss": 0.8383,
"step": 8900
},
{
"epoch": 0.7019185774450164,
"grad_norm": 60165.2734375,
"learning_rate": 1.4904071127749181e-05,
"loss": 0.8442,
"step": 9000
},
{
"epoch": 0.709717672749961,
"grad_norm": 71296.1171875,
"learning_rate": 1.451411636250195e-05,
"loss": 0.8537,
"step": 9100
},
{
"epoch": 0.7175167680549056,
"grad_norm": 67209.6328125,
"learning_rate": 1.4124161597254721e-05,
"loss": 0.8575,
"step": 9200
},
{
"epoch": 0.7253158633598502,
"grad_norm": 63280.3203125,
"learning_rate": 1.3734206832007487e-05,
"loss": 0.8358,
"step": 9300
},
{
"epoch": 0.7331149586647949,
"grad_norm": 69636.6875,
"learning_rate": 1.3344252066760258e-05,
"loss": 0.8475,
"step": 9400
},
{
"epoch": 0.7409140539697395,
"grad_norm": 68302.3828125,
"learning_rate": 1.2954297301513025e-05,
"loss": 0.8305,
"step": 9500
},
{
"epoch": 0.7487131492746841,
"grad_norm": 68986.78125,
"learning_rate": 1.2564342536265794e-05,
"loss": 0.8667,
"step": 9600
},
{
"epoch": 0.7565122445796287,
"grad_norm": 69115.0546875,
"learning_rate": 1.2174387771018563e-05,
"loss": 0.8386,
"step": 9700
},
{
"epoch": 0.7643113398845733,
"grad_norm": 58786.73046875,
"learning_rate": 1.178443300577133e-05,
"loss": 0.8306,
"step": 9800
},
{
"epoch": 0.772110435189518,
"grad_norm": 75233.265625,
"learning_rate": 1.13944782405241e-05,
"loss": 0.8611,
"step": 9900
},
{
"epoch": 0.7799095304944627,
"grad_norm": 65838.609375,
"learning_rate": 1.1004523475276867e-05,
"loss": 0.8191,
"step": 10000
},
{
"epoch": 0.7877086257994073,
"grad_norm": 55231.33984375,
"learning_rate": 1.0614568710029638e-05,
"loss": 0.8551,
"step": 10100
},
{
"epoch": 0.7955077211043519,
"grad_norm": 63516.97265625,
"learning_rate": 1.0224613944782405e-05,
"loss": 0.8685,
"step": 10200
},
{
"epoch": 0.8033068164092966,
"grad_norm": 63661.3671875,
"learning_rate": 9.834659179535174e-06,
"loss": 0.8009,
"step": 10300
},
{
"epoch": 0.8111059117142412,
"grad_norm": 71024.375,
"learning_rate": 9.444704414287943e-06,
"loss": 0.8331,
"step": 10400
},
{
"epoch": 0.8189050070191858,
"grad_norm": 80821.6640625,
"learning_rate": 9.05474964904071e-06,
"loss": 0.8344,
"step": 10500
},
{
"epoch": 0.8267041023241304,
"grad_norm": 61162.19140625,
"learning_rate": 8.66479488379348e-06,
"loss": 0.8454,
"step": 10600
},
{
"epoch": 0.834503197629075,
"grad_norm": 64971.5390625,
"learning_rate": 8.274840118546249e-06,
"loss": 0.8247,
"step": 10700
},
{
"epoch": 0.8423022929340197,
"grad_norm": 64816.95703125,
"learning_rate": 7.884885353299018e-06,
"loss": 0.8142,
"step": 10800
},
{
"epoch": 0.8501013882389643,
"grad_norm": 60450.6875,
"learning_rate": 7.494930588051786e-06,
"loss": 0.8128,
"step": 10900
},
{
"epoch": 0.8579004835439089,
"grad_norm": 71442.4375,
"learning_rate": 7.104975822804555e-06,
"loss": 0.8166,
"step": 11000
},
{
"epoch": 0.8656995788488535,
"grad_norm": 76537.1875,
"learning_rate": 6.715021057557324e-06,
"loss": 0.852,
"step": 11100
},
{
"epoch": 0.8734986741537981,
"grad_norm": 65081.37890625,
"learning_rate": 6.325066292310092e-06,
"loss": 0.8394,
"step": 11200
},
{
"epoch": 0.8812977694587428,
"grad_norm": 73682.296875,
"learning_rate": 5.935111527062861e-06,
"loss": 0.8165,
"step": 11300
},
{
"epoch": 0.8890968647636874,
"grad_norm": 73644.78125,
"learning_rate": 5.54515676181563e-06,
"loss": 0.8312,
"step": 11400
},
{
"epoch": 0.896895960068632,
"grad_norm": 79621.890625,
"learning_rate": 5.155201996568398e-06,
"loss": 0.8404,
"step": 11500
},
{
"epoch": 0.9046950553735766,
"grad_norm": 56045.7734375,
"learning_rate": 4.7652472313211666e-06,
"loss": 0.8295,
"step": 11600
},
{
"epoch": 0.9124941506785212,
"grad_norm": 66463.71875,
"learning_rate": 4.375292466073936e-06,
"loss": 0.8256,
"step": 11700
},
{
"epoch": 0.920293245983466,
"grad_norm": 58425.23046875,
"learning_rate": 3.985337700826705e-06,
"loss": 0.8253,
"step": 11800
},
{
"epoch": 0.9280923412884106,
"grad_norm": 68089.9609375,
"learning_rate": 3.595382935579473e-06,
"loss": 0.8161,
"step": 11900
},
{
"epoch": 0.9358914365933552,
"grad_norm": 63161.6796875,
"learning_rate": 3.2054281703322412e-06,
"loss": 0.8322,
"step": 12000
},
{
"epoch": 0.9436905318982998,
"grad_norm": 56655.69921875,
"learning_rate": 2.8154734050850103e-06,
"loss": 0.8117,
"step": 12100
},
{
"epoch": 0.9514896272032444,
"grad_norm": 72667.96875,
"learning_rate": 2.425518639837779e-06,
"loss": 0.8181,
"step": 12200
},
{
"epoch": 0.9592887225081891,
"grad_norm": 64209.17578125,
"learning_rate": 2.0355638745905476e-06,
"loss": 0.813,
"step": 12300
},
{
"epoch": 0.9670878178131337,
"grad_norm": 68720.7109375,
"learning_rate": 1.6456091093433163e-06,
"loss": 0.8343,
"step": 12400
},
{
"epoch": 0.9748869131180783,
"grad_norm": 62702.68359375,
"learning_rate": 1.255654344096085e-06,
"loss": 0.8156,
"step": 12500
},
{
"epoch": 0.9826860084230229,
"grad_norm": 62949.0625,
"learning_rate": 8.656995788488536e-07,
"loss": 0.8343,
"step": 12600
},
{
"epoch": 0.9904851037279676,
"grad_norm": 64140.0703125,
"learning_rate": 4.757448136016222e-07,
"loss": 0.8189,
"step": 12700
},
{
"epoch": 0.9982841990329122,
"grad_norm": 72402.578125,
"learning_rate": 8.579004835439089e-08,
"loss": 0.823,
"step": 12800
},
{
"epoch": 1.0,
"step": 12822,
"total_flos": 1.955920022807249e+18,
"train_loss": 0.8700304180128008,
"train_runtime": 179145.9612,
"train_samples_per_second": 0.859,
"train_steps_per_second": 0.072
}
],
"logging_steps": 100,
"max_steps": 12822,
"num_input_tokens_seen": 0,
"num_train_epochs": 1,
"save_steps": 12822,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": true
},
"attributes": {}
}
},
"total_flos": 1.955920022807249e+18,
"train_batch_size": 12,
"trial_name": null,
"trial_params": null
}

3
training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0c44c38eb09a6b2599756766edbc167c8760d5ab86cbaedecff96098292537cf
size 5112