pszemraj/distilgpt2-HC3
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: pszemraj/distilgpt2-HC3
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-10 11:14:18 +08:00
commit 68db0b272d
17 changed files with 151432 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
.gitattributes vendored Normal file
View File

@@ -0,0 +1,34 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

1
.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
checkpoint-*/

127
README.md Normal file
View File

@@ -0,0 +1,127 @@
---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- generated_from_trainer
- chatgpt
- HC3
datasets:
- pszemraj/HC3-textgen-qa
metrics:
- accuracy
widget:
- text: 'Review: Best cast iron skillet you will ever buy. Is this review positive
or negative? <answer>'
example_title: Sentiment analysis
- text: Barack Obama nominated Hilary Clinton as his secretary of state on Monday.
He chose her because <answer>
example_title: Coreference resolution
- text: 'On a shelf, there are five books: a gray book, a red book, a purple book,
a blue book, and a black book. Here''s the puzzle, <answer>'
example_title: Logic puzzles
- text: The two men running to become New York City's next mayor will face off in
their first debate Wednesday night <answer>
example_title: Reading comprehension
- text: Is it true that if I have five 5-hour energy drinks in a single 24-hour period,
I get 25 hours of energy and spontaneously explode? <answer>
example_title: 5 hour energy
- text: what happens if you train a smaller model on a dataset of reinforcement-learning
optimized model responses? <answer>
example_title: deep learning advice
inference:
parameters:
temperature: 0.6
max_length: 96
no_repeat_ngram_size: 4
repetition_penalty: 1.5
eta_cutoff: 0.0008
renormalize_logits: true
pipeline_tag: text-generation
model-index:
- name: distilgpt2-HC3
results: []
---
# distilgpt2-HC3
> what happens if you train a smaller model on a dataset of chatGPT responses?
This happens.
![example](https://i.imgur.com/i5snxQJ.png)
## Model description
This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on the "chatgpt answers" column of the `Hello-SimpleAI/HC3` dataset.
It achieves the following results on the evaluation set:
- Loss: 1.9983
- Accuracy: 0.5441
## Intended uses & limitations
Despite how it sounds, this model only has 80m parameters and will likely not be factually accurate most of the time.
## Training and evaluation data
Modifications made w.r.t. original dataset:
- drop all rows that did not have a chatGPT answer
- if a row (_i.e. ELI5 question, etc_) had more than one response (_from chatGPT_), randomly choose one of the responses as the answer to the question
- the "question" and chatGPT answer were combined into a single string for that row as follows: `QUESTION_TEXT <answer> CHATGPT_ANSWER_TEXT <end_answer>`
- `<answer>` and `<end_answer>` serve as added tokens to help the model learn "turns" in the conversation
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 8
- eval_batch_size: 4
- seed: 3208
- gradient_accumulation_steps: 16
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 6.0
- mixed_precision_training: Native AMP
### Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 2.2485 | 0.98 | 41 | 2.1457 | 0.5158 |
| 2.0757 | 1.98 | 82 | 2.0584 | 0.5304 |
| 1.966 | 2.98 | 123 | 2.0210 | 0.5376 |
| 1.8602 | 3.98 | 164 | 2.0012 | 0.5422 |
| 1.8089 | 4.98 | 205 | 1.9977 | 0.5436 |
| 1.7698 | 5.98 | 246 | 1.9983 | 0.5441 |
### Framework versions
- Transformers 4.27.0.dev0
- Pytorch 1.11.0+cu113
- Datasets 2.6.1
- Tokenizers 0.12.1
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_pszemraj__distilgpt2-HC3)
| Metric |Value|
|---------------------------------|----:|
|Avg. |28.18|
|AI2 Reasoning Challenge (25-Shot)|24.66|
|HellaSwag (10-Shot) |27.99|
|MMLU (5-Shot) |23.95|
|TruthfulQA (0-shot) |42.10|
|Winogrande (5-shot) |50.36|
|GSM8k (5-shot) | 0.00|

29
all_results.json Normal file
View File

@@ -0,0 +1,29 @@
{
"before_init_mem_cpu": 699817984,
"before_init_mem_gpu": 0,
"epoch": 5.98,
"eval_accuracy": 0.544073054441718,
"eval_loss": 1.9982993602752686,
"eval_mem_cpu_alloc_delta": 9977856,
"eval_mem_cpu_peaked_delta": 60305408,
"eval_mem_gpu_alloc_delta": 0,
"eval_mem_gpu_peaked_delta": 2357757952,
"eval_runtime": 481.5484,
"eval_samples": 588,
"eval_samples_per_second": 1.221,
"eval_steps_per_second": 0.305,
"init_mem_cpu_alloc_delta": -283938816,
"init_mem_cpu_peaked_delta": 332918784,
"init_mem_gpu_alloc_delta": 334744576,
"init_mem_gpu_peaked_delta": 0,
"perplexity": 7.376500655769286,
"train_loss": 2.0183091745143984,
"train_mem_cpu_alloc_delta": 1155149824,
"train_mem_cpu_peaked_delta": 308719616,
"train_mem_gpu_alloc_delta": 988995584,
"train_mem_gpu_peaked_delta": 4500811776,
"train_runtime": 35067.5673,
"train_samples": 5325,
"train_samples_per_second": 0.911,
"train_steps_per_second": 0.007
}

46
config.json Normal file
View File

@@ -0,0 +1,46 @@
{
"_name_or_path": "distilgpt2",
"_num_labels": 1,
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 50256,
"embd_pdrop": 0.1,
"eos_token_id": 50256,
"id2label": {
"0": "LABEL_0"
},
"initializer_range": 0.02,
"label2id": {
"LABEL_0": 0
},
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 768,
"n_head": 12,
"n_inner": null,
"n_layer": 6,
"n_positions": 1024,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"torch_dtype": "float32",
"transformers_version": "4.27.0.dev0",
"use_cache": true,
"vocab_size": 50257
}

14
eval_results.json Normal file
View File

@@ -0,0 +1,14 @@
{
"epoch": 5.98,
"eval_accuracy": 0.544073054441718,
"eval_loss": 1.9982993602752686,
"eval_mem_cpu_alloc_delta": 9977856,
"eval_mem_cpu_peaked_delta": 60305408,
"eval_mem_gpu_alloc_delta": 0,
"eval_mem_gpu_peaked_delta": 2357757952,
"eval_runtime": 481.5484,
"eval_samples": 588,
"eval_samples_per_second": 1.221,
"eval_steps_per_second": 0.305,
"perplexity": 7.376500655769286
}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 50256,
"eos_token_id": 50256,
"transformers_version": "4.27.0.dev0"
}

50001
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1ee10d542f75f47068524d5520284b4da082eb3e5c6f5486b19aa6cbeb2ebb4d
size 333950622

3
pytorch_model.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a2c6b228db6fefedb801511d458e1f2092fd083a858935b4c9abb6132985d2a3
size 333969117

5
special_tokens_map.json Normal file
View File

@@ -0,0 +1,5 @@
{
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"unk_token": "<|endoftext|>"
}

100304
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

10
tokenizer_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"add_prefix_space": false,
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"model_max_length": 1024,
"name_or_path": "distilgpt2",
"special_tokens_map_file": null,
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>"
}

18
train_results.json Normal file
View File

@@ -0,0 +1,18 @@
{
"before_init_mem_cpu": 699817984,
"before_init_mem_gpu": 0,
"epoch": 5.98,
"init_mem_cpu_alloc_delta": -283938816,
"init_mem_cpu_peaked_delta": 332918784,
"init_mem_gpu_alloc_delta": 334744576,
"init_mem_gpu_peaked_delta": 0,
"train_loss": 2.0183091745143984,
"train_mem_cpu_alloc_delta": 1155149824,
"train_mem_cpu_peaked_delta": 308719616,
"train_mem_gpu_alloc_delta": 988995584,
"train_mem_gpu_peaked_delta": 4500811776,
"train_runtime": 35067.5673,
"train_samples": 5325,
"train_samples_per_second": 0.911,
"train_steps_per_second": 0.007
}

827
trainer_state.json Normal file
View File

@@ -0,0 +1,827 @@
{
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 5.984984984984985,
"global_step": 246,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.05,
"learning_rate": 0.00015384615384615385,
"loss": 2.9074,
"step": 2
},
{
"epoch": 0.1,
"learning_rate": 0.0003076923076923077,
"loss": 2.7922,
"step": 4
},
{
"epoch": 0.14,
"learning_rate": 0.0004615384615384616,
"loss": 2.6824,
"step": 6
},
{
"epoch": 0.19,
"learning_rate": 0.0006153846153846154,
"loss": 2.6036,
"step": 8
},
{
"epoch": 0.24,
"learning_rate": 0.0007692307692307693,
"loss": 2.5524,
"step": 10
},
{
"epoch": 0.29,
"learning_rate": 0.0009230769230769232,
"loss": 2.5342,
"step": 12
},
{
"epoch": 0.34,
"learning_rate": 0.0009999545513138964,
"loss": 2.5052,
"step": 14
},
{
"epoch": 0.38,
"learning_rate": 0.000999591011397558,
"loss": 2.4689,
"step": 16
},
{
"epoch": 0.43,
"learning_rate": 0.000998864195911451,
"loss": 2.4487,
"step": 18
},
{
"epoch": 0.48,
"learning_rate": 0.000997774633356497,
"loss": 2.4068,
"step": 20
},
{
"epoch": 0.53,
"learning_rate": 0.0009963231160036714,
"loss": 2.3813,
"step": 22
},
{
"epoch": 0.58,
"learning_rate": 0.0009945106993179073,
"loss": 2.3558,
"step": 24
},
{
"epoch": 0.62,
"learning_rate": 0.000992338701190618,
"loss": 2.3425,
"step": 26
},
{
"epoch": 0.67,
"learning_rate": 0.0009898087009813986,
"loss": 2.3254,
"step": 28
},
{
"epoch": 0.72,
"learning_rate": 0.000986922538369599,
"loss": 2.3128,
"step": 30
},
{
"epoch": 0.77,
"learning_rate": 0.0009836823120166116,
"loss": 2.3026,
"step": 32
},
{
"epoch": 0.82,
"learning_rate": 0.0009800903780398356,
"loss": 2.3005,
"step": 34
},
{
"epoch": 0.86,
"learning_rate": 0.0009761493482994372,
"loss": 2.2828,
"step": 36
},
{
"epoch": 0.91,
"learning_rate": 0.0009718620884991454,
"loss": 2.2779,
"step": 38
},
{
"epoch": 0.96,
"learning_rate": 0.0009672317161024677,
"loss": 2.2485,
"step": 40
},
{
"epoch": 0.98,
"eval_accuracy": 0.5157865687819605,
"eval_loss": 2.1457464694976807,
"eval_runtime": 496.7153,
"eval_samples_per_second": 1.184,
"eval_steps_per_second": 0.296,
"step": 41
},
{
"epoch": 1.02,
"learning_rate": 0.000962261598065839,
"loss": 2.9045,
"step": 42
},
{
"epoch": 1.07,
"learning_rate": 0.000956955348390353,
"loss": 2.16,
"step": 44
},
{
"epoch": 1.12,
"learning_rate": 0.000951316825493856,
"loss": 2.1395,
"step": 46
},
{
"epoch": 1.17,
"learning_rate": 0.0009453501294053137,
"loss": 2.1327,
"step": 48
},
{
"epoch": 1.22,
"learning_rate": 0.0009390595987834929,
"loss": 2.128,
"step": 50
},
{
"epoch": 1.26,
"learning_rate": 0.000932449807762122,
"loss": 2.1251,
"step": 52
},
{
"epoch": 1.31,
"learning_rate": 0.0009255255626238293,
"loss": 2.1138,
"step": 54
},
{
"epoch": 1.36,
"learning_rate": 0.0009182918983052741,
"loss": 2.118,
"step": 56
},
{
"epoch": 1.41,
"learning_rate": 0.0009107540747360124,
"loss": 2.1034,
"step": 58
},
{
"epoch": 1.46,
"learning_rate": 0.0009029175730137609,
"loss": 2.1016,
"step": 60
},
{
"epoch": 1.5,
"learning_rate": 0.0008947880914188397,
"loss": 2.1027,
"step": 62
},
{
"epoch": 1.55,
"learning_rate": 0.0008863715412706897,
"loss": 2.0969,
"step": 64
},
{
"epoch": 1.6,
"learning_rate": 0.0008776740426294817,
"loss": 2.0926,
"step": 66
},
{
"epoch": 1.65,
"learning_rate": 0.0008687019198459394,
"loss": 2.0946,
"step": 68
},
{
"epoch": 1.7,
"learning_rate": 0.0008594616969626134,
"loss": 2.0968,
"step": 70
},
{
"epoch": 1.74,
"learning_rate": 0.0008499600929699501,
"loss": 2.0932,
"step": 72
},
{
"epoch": 1.79,
"learning_rate": 0.0008402040169206053,
"loss": 2.0813,
"step": 74
},
{
"epoch": 1.84,
"learning_rate": 0.0008302005629055549,
"loss": 2.0815,
"step": 76
},
{
"epoch": 1.89,
"learning_rate": 0.0008199570048956552,
"loss": 2.0809,
"step": 78
},
{
"epoch": 1.94,
"learning_rate": 0.0008094807914524047,
"loss": 2.0704,
"step": 80
},
{
"epoch": 1.98,
"learning_rate": 0.0007987795403117528,
"loss": 2.0757,
"step": 82
},
{
"epoch": 1.98,
"eval_accuracy": 0.530352903624793,
"eval_loss": 2.058377742767334,
"eval_runtime": 493.7321,
"eval_samples_per_second": 1.191,
"eval_steps_per_second": 0.298,
"step": 82
},
{
"epoch": 2.05,
"learning_rate": 0.0007878610328448947,
"loss": 2.615,
"step": 84
},
{
"epoch": 2.1,
"learning_rate": 0.0007767332084000785,
"loss": 1.9638,
"step": 86
},
{
"epoch": 2.14,
"learning_rate": 0.0007654041585295399,
"loss": 1.979,
"step": 88
},
{
"epoch": 2.19,
"learning_rate": 0.0007538821211057647,
"loss": 1.9628,
"step": 90
},
{
"epoch": 2.24,
"learning_rate": 0.0007421754743313514,
"loss": 1.9717,
"step": 92
},
{
"epoch": 2.29,
"learning_rate": 0.0007302927306468364,
"loss": 1.9625,
"step": 94
},
{
"epoch": 2.34,
"learning_rate": 0.0007182425305409072,
"loss": 1.9683,
"step": 96
},
{
"epoch": 2.38,
"learning_rate": 0.0007060336362675068,
"loss": 1.9556,
"step": 98
},
{
"epoch": 2.43,
"learning_rate": 0.0006936749254743951,
"loss": 1.9565,
"step": 100
},
{
"epoch": 2.48,
"learning_rate": 0.000681175384747805,
"loss": 1.9601,
"step": 102
},
{
"epoch": 2.53,
"learning_rate": 0.0006685441030778817,
"loss": 1.9612,
"step": 104
},
{
"epoch": 2.58,
"learning_rate": 0.0006557902652496611,
"loss": 1.9539,
"step": 106
},
{
"epoch": 2.62,
"learning_rate": 0.0006429231451643907,
"loss": 1.9563,
"step": 108
},
{
"epoch": 2.67,
"learning_rate": 0.0006299520990960496,
"loss": 1.9597,
"step": 110
},
{
"epoch": 2.72,
"learning_rate": 0.000616886558887973,
"loss": 1.9625,
"step": 112
},
{
"epoch": 2.77,
"learning_rate": 0.0006037360250945242,
"loss": 1.965,
"step": 114
},
{
"epoch": 2.82,
"learning_rate": 0.0005905100600728067,
"loss": 1.9588,
"step": 116
},
{
"epoch": 2.86,
"learning_rate": 0.0005772182810294344,
"loss": 1.965,
"step": 118
},
{
"epoch": 2.91,
"learning_rate": 0.0005638703530274187,
"loss": 1.9595,
"step": 120
},
{
"epoch": 2.96,
"learning_rate": 0.0005504759819582581,
"loss": 1.966,
"step": 122
},
{
"epoch": 2.98,
"eval_accuracy": 0.5375845352803879,
"eval_loss": 2.0210342407226562,
"eval_runtime": 475.4248,
"eval_samples_per_second": 1.237,
"eval_steps_per_second": 0.309,
"step": 123
},
{
"epoch": 3.02,
"learning_rate": 0.000537044907484338,
"loss": 2.5101,
"step": 124
},
{
"epoch": 3.07,
"learning_rate": 0.0005235868959567755,
"loss": 1.8782,
"step": 126
},
{
"epoch": 3.12,
"learning_rate": 0.0005101117333138558,
"loss": 1.8556,
"step": 128
},
{
"epoch": 3.17,
"learning_rate": 0.0004966292179652285,
"loss": 1.8622,
"step": 130
},
{
"epoch": 3.22,
"learning_rate": 0.00048314915366703335,
"loss": 1.8668,
"step": 132
},
{
"epoch": 3.26,
"learning_rate": 0.0004696813423931381,
"loss": 1.873,
"step": 134
},
{
"epoch": 3.31,
"learning_rate": 0.000456235577207674,
"loss": 1.8694,
"step": 136
},
{
"epoch": 3.36,
"learning_rate": 0.00044282163514404915,
"loss": 1.8742,
"step": 138
},
{
"epoch": 3.41,
"learning_rate": 0.00042944927009561784,
"loss": 1.8694,
"step": 140
},
{
"epoch": 3.46,
"learning_rate": 0.00041612820572317757,
"loss": 1.8569,
"step": 142
},
{
"epoch": 3.5,
"learning_rate": 0.0004028681283844471,
"loss": 1.8619,
"step": 144
},
{
"epoch": 3.55,
"learning_rate": 0.0003896786800906733,
"loss": 1.8734,
"step": 146
},
{
"epoch": 3.6,
"learning_rate": 0.00037656945149547956,
"loss": 1.8677,
"step": 148
},
{
"epoch": 3.65,
"learning_rate": 0.00036354997492106257,
"loss": 1.872,
"step": 150
},
{
"epoch": 3.7,
"learning_rate": 0.00035062971742680246,
"loss": 1.847,
"step": 152
},
{
"epoch": 3.74,
"learning_rate": 0.00033781807392532895,
"loss": 1.8733,
"step": 154
},
{
"epoch": 3.79,
"learning_rate": 0.00032512436035104966,
"loss": 1.8657,
"step": 156
},
{
"epoch": 3.84,
"learning_rate": 0.00031255780688610506,
"loss": 1.8581,
"step": 158
},
{
"epoch": 3.89,
"learning_rate": 0.00030012755124868095,
"loss": 1.8672,
"step": 160
},
{
"epoch": 3.94,
"learning_rate": 0.00028784263204855175,
"loss": 1.8585,
"step": 162
},
{
"epoch": 3.98,
"learning_rate": 0.00027571198221469395,
"loss": 1.8602,
"step": 164
},
{
"epoch": 3.98,
"eval_accuracy": 0.5422011424315571,
"eval_loss": 2.0011844635009766,
"eval_runtime": 493.1263,
"eval_samples_per_second": 1.192,
"eval_steps_per_second": 0.298,
"step": 164
},
{
"epoch": 4.05,
"learning_rate": 0.0002637444224997421,
"loss": 2.4006,
"step": 166
},
{
"epoch": 4.1,
"learning_rate": 0.00025194865506601504,
"loss": 1.8118,
"step": 168
},
{
"epoch": 4.14,
"learning_rate": 0.00024033325715777377,
"loss": 1.7888,
"step": 170
},
{
"epoch": 4.19,
"learning_rate": 0.00022890667486431295,
"loss": 1.798,
"step": 172
},
{
"epoch": 4.24,
"learning_rate": 0.00021767721697842242,
"loss": 1.8052,
"step": 174
},
{
"epoch": 4.29,
"learning_rate": 0.00020665304895468113,
"loss": 1.8054,
"step": 176
},
{
"epoch": 4.34,
"learning_rate": 0.0001958421869719807,
"loss": 1.7966,
"step": 178
},
{
"epoch": 4.38,
"learning_rate": 0.00018525249210459344,
"loss": 1.8049,
"step": 180
},
{
"epoch": 4.43,
"learning_rate": 0.00017489166460602496,
"loss": 1.7955,
"step": 182
},
{
"epoch": 4.48,
"learning_rate": 0.0001647672383098045,
"loss": 1.8062,
"step": 184
},
{
"epoch": 4.53,
"learning_rate": 0.00015488657515129006,
"loss": 1.8132,
"step": 186
},
{
"epoch": 4.58,
"learning_rate": 0.0001452568598144668,
"loss": 1.8027,
"step": 188
},
{
"epoch": 4.62,
"learning_rate": 0.00013588509450763282,
"loss": 1.8066,
"step": 190
},
{
"epoch": 4.67,
"learning_rate": 0.0001267780938717722,
"loss": 1.7989,
"step": 192
},
{
"epoch": 4.72,
"learning_rate": 0.00011794248002531643,
"loss": 1.7966,
"step": 194
},
{
"epoch": 4.77,
"learning_rate": 0.00010938467774889881,
"loss": 1.8074,
"step": 196
},
{
"epoch": 4.82,
"learning_rate": 0.00010111090981359961,
"loss": 1.8073,
"step": 198
},
{
"epoch": 4.86,
"learning_rate": 9.312719245608486e-05,
"loss": 1.8022,
"step": 200
},
{
"epoch": 4.91,
"learning_rate": 8.54393310039246e-05,
"loss": 1.8014,
"step": 202
},
{
"epoch": 4.96,
"learning_rate": 7.805291565427064e-05,
"loss": 1.8089,
"step": 204
},
{
"epoch": 4.98,
"eval_accuracy": 0.5435859583325021,
"eval_loss": 1.9977257251739502,
"eval_runtime": 497.2866,
"eval_samples_per_second": 1.182,
"eval_steps_per_second": 0.296,
"step": 205
},
{
"epoch": 5.02,
"learning_rate": 7.097331740896996e-05,
"loss": 2.3557,
"step": 206
},
{
"epoch": 5.07,
"learning_rate": 6.420568416906064e-05,
"loss": 1.7835,
"step": 208
},
{
"epoch": 5.12,
"learning_rate": 5.775493699149753e-05,
"loss": 1.7641,
"step": 210
},
{
"epoch": 5.17,
"learning_rate": 5.16257665108254e-05,
"loss": 1.7754,
"step": 212
},
{
"epoch": 5.22,
"learning_rate": 4.582262952840355e-05,
"loss": 1.7765,
"step": 214
},
{
"epoch": 5.26,
"learning_rate": 4.034974577166023e-05,
"loss": 1.7732,
"step": 216
},
{
"epoch": 5.31,
"learning_rate": 3.521109482573515e-05,
"loss": 1.7663,
"step": 218
},
{
"epoch": 5.36,
"learning_rate": 3.0410413239740975e-05,
"loss": 1.7779,
"step": 220
},
{
"epoch": 5.41,
"learning_rate": 2.5951191809746145e-05,
"loss": 1.7739,
"step": 222
},
{
"epoch": 5.46,
"learning_rate": 2.1836673040456943e-05,
"loss": 1.7602,
"step": 224
},
{
"epoch": 5.5,
"learning_rate": 1.8069848787443555e-05,
"loss": 1.7773,
"step": 226
},
{
"epoch": 5.55,
"learning_rate": 1.4653458081624272e-05,
"loss": 1.7841,
"step": 228
},
{
"epoch": 5.6,
"learning_rate": 1.1589985137590519e-05,
"loss": 1.7746,
"step": 230
},
{
"epoch": 5.65,
"learning_rate": 8.881657547219868e-06,
"loss": 1.7801,
"step": 232
},
{
"epoch": 5.7,
"learning_rate": 6.530444659892443e-06,
"loss": 1.7639,
"step": 234
},
{
"epoch": 5.74,
"learning_rate": 4.538056150485858e-06,
"loss": 1.7771,
"step": 236
},
{
"epoch": 5.79,
"learning_rate": 2.905940776192384e-06,
"loss": 1.7724,
"step": 238
},
{
"epoch": 5.84,
"learning_rate": 1.6352853230609532e-06,
"loss": 1.7876,
"step": 240
},
{
"epoch": 5.89,
"learning_rate": 7.270137430306356e-07,
"loss": 1.7718,
"step": 242
},
{
"epoch": 5.94,
"learning_rate": 1.817864820827242e-07,
"loss": 1.779,
"step": 244
},
{
"epoch": 5.98,
"learning_rate": 0.0,
"loss": 1.7698,
"step": 246
},
{
"epoch": 5.98,
"eval_accuracy": 0.544073054441718,
"eval_loss": 1.9982993602752686,
"eval_runtime": 495.216,
"eval_samples_per_second": 1.187,
"eval_steps_per_second": 0.297,
"step": 246
},
{
"before_init_mem_cpu": 699817984,
"before_init_mem_gpu": 0,
"epoch": 5.98,
"init_mem_cpu_alloc_delta": -283938816,
"init_mem_cpu_peaked_delta": 332918784,
"init_mem_gpu_alloc_delta": 334744576,
"init_mem_gpu_peaked_delta": 0,
"step": 246,
"total_flos": 8328311331618816.0,
"train_loss": 2.0183091745143984,
"train_mem_cpu_alloc_delta": 1155149824,
"train_mem_cpu_peaked_delta": 308719616,
"train_mem_gpu_alloc_delta": 988995584,
"train_mem_gpu_peaked_delta": 4500811776,
"train_runtime": 35067.5673,
"train_samples_per_second": 0.911,
"train_steps_per_second": 0.007
}
],
"max_steps": 246,
"num_train_epochs": 6,
"total_flos": 8328311331618816.0,
"trial_name": null,
"trial_params": null
}

3
training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cf8d597c0d3530d3e1fce58e01559cab73b1dbad47577dcf33e59f2fa1798284
size 3631

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long