devika-tiwari/gpt2_small_expandedbabyLM_100M_exp3_conj_ratio_0p50_mix_0p10_44
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: devika-tiwari/gpt2_small_expandedbabyLM_100M_exp3_conj_ratio_0p50_mix_0p10_44
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-09-27 06:53:18 +08:00
commit 4c596d4d05
16 changed files with 151358 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

1
.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
checkpoint-*/

61
README.md Normal file
View File

@@ -0,0 +1,61 @@
---
tags:
- generated_from_trainer
model-index:
- name: gpt2_small_expandedbabyLM_100M_exp3_conj_ratio_0p50_mix_0p10_44
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# gpt2_small_expandedbabyLM_100M_exp3_conj_ratio_0p50_mix_0p10_44
This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 3.6556
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 256
- eval_batch_size: 256
- seed: 44
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 4000
- num_epochs: 20
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 3.7482 | 1.0 | 4429 | 4.3016 |
| 3.4047 | 2.0 | 8858 | 3.8274 |
| 3.2749 | 3.0 | 13287 | 3.6556 |
| 3.1943 | 4.0 | 17716 | 3.7592 |
| 3.1289 | 5.0 | 22145 | 3.6759 |
| 3.0883 | 6.0 | 26574 | 3.6896 |
### Framework versions
- Transformers 4.30.2
- Pytorch 2.11.0+cu130
- Datasets 4.1.1
- Tokenizers 0.13.3

14
all_results.json Normal file
View File

@@ -0,0 +1,14 @@
{
"epoch": 6.0,
"eval_loss": 3.6556012630462646,
"eval_runtime": 69.9518,
"eval_samples": 144585,
"eval_samples_per_second": 2066.923,
"eval_steps_per_second": 8.077,
"perplexity": 38.69077745758761,
"train_loss": 3.490822130646673,
"train_runtime": 9148.1319,
"train_samples": 1133592,
"train_samples_per_second": 2478.303,
"train_steps_per_second": 9.683
}

31
config.json Normal file
View File

@@ -0,0 +1,31 @@
{
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 50256,
"embd_pdrop": 0.1,
"eos_token_id": 50256,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_embd": 768,
"n_head": 12,
"n_inner": null,
"n_layer": 12,
"n_positions": 1024,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"torch_dtype": "float32",
"transformers_version": "4.30.2",
"use_cache": true,
"vocab_size": 50257
}

9
eval_results.json Normal file
View File

@@ -0,0 +1,9 @@
{
"epoch": 6.0,
"eval_loss": 3.6556012630462646,
"eval_runtime": 69.9518,
"eval_samples": 144585,
"eval_samples_per_second": 2066.923,
"eval_steps_per_second": 8.077,
"perplexity": 38.69077745758761
}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 50256,
"eos_token_id": 50256,
"transformers_version": "4.30.2"
}

50001
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
pytorch_model.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:55e9d69d0ff9e4eef37bb6ec0f59b3ba8afb3b19412ee9039d77f63940330500
size 497808115

5
special_tokens_map.json Normal file
View File

@@ -0,0 +1,5 @@
{
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"unk_token": "<|endoftext|>"
}

100305
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

10
tokenizer_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"add_prefix_space": false,
"bos_token": "<|endoftext|>",
"clean_up_tokenization_spaces": true,
"eos_token": "<|endoftext|>",
"model_max_length": 1024,
"token": null,
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>"
}

8
train_results.json Normal file
View File

@@ -0,0 +1,8 @@
{
"epoch": 6.0,
"train_loss": 3.490822130646673,
"train_runtime": 9148.1319,
"train_samples": 1133592,
"train_samples_per_second": 2478.303,
"train_steps_per_second": 9.683
}

865
trainer_state.json Normal file
View File

@@ -0,0 +1,865 @@
{
"best_metric": 3.6556012630462646,
"best_model_checkpoint": "/scratch/dt719/causal_lm_training/models/gpt2_small_expandedbabyLM_100M_exp3_conj_ratio_0p50_mix_0p10_44/checkpoint-13287",
"epoch": 6.0,
"global_step": 26574,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.05,
"learning_rate": 5e-06,
"loss": 8.7872,
"step": 200
},
{
"epoch": 0.09,
"learning_rate": 1e-05,
"loss": 6.9867,
"step": 400
},
{
"epoch": 0.14,
"learning_rate": 1.5e-05,
"loss": 5.9456,
"step": 600
},
{
"epoch": 0.18,
"learning_rate": 2e-05,
"loss": 5.4137,
"step": 800
},
{
"epoch": 0.23,
"learning_rate": 2.5e-05,
"loss": 5.1252,
"step": 1000
},
{
"epoch": 0.27,
"learning_rate": 3e-05,
"loss": 4.9106,
"step": 1200
},
{
"epoch": 0.32,
"learning_rate": 3.5e-05,
"loss": 4.729,
"step": 1400
},
{
"epoch": 0.36,
"learning_rate": 4e-05,
"loss": 4.5963,
"step": 1600
},
{
"epoch": 0.41,
"learning_rate": 4.5e-05,
"loss": 4.4763,
"step": 1800
},
{
"epoch": 0.45,
"learning_rate": 5e-05,
"loss": 4.3756,
"step": 2000
},
{
"epoch": 0.5,
"learning_rate": 5.500000000000001e-05,
"loss": 4.3006,
"step": 2200
},
{
"epoch": 0.54,
"learning_rate": 6e-05,
"loss": 4.2111,
"step": 2400
},
{
"epoch": 0.59,
"learning_rate": 6.500000000000001e-05,
"loss": 4.1437,
"step": 2600
},
{
"epoch": 0.63,
"learning_rate": 7e-05,
"loss": 4.0808,
"step": 2800
},
{
"epoch": 0.68,
"learning_rate": 7.500000000000001e-05,
"loss": 4.0252,
"step": 3000
},
{
"epoch": 0.72,
"learning_rate": 8e-05,
"loss": 3.9729,
"step": 3200
},
{
"epoch": 0.77,
"learning_rate": 8.5e-05,
"loss": 3.932,
"step": 3400
},
{
"epoch": 0.81,
"learning_rate": 9e-05,
"loss": 3.8931,
"step": 3600
},
{
"epoch": 0.86,
"learning_rate": 9.5e-05,
"loss": 3.8478,
"step": 3800
},
{
"epoch": 0.9,
"learning_rate": 0.0001,
"loss": 3.8108,
"step": 4000
},
{
"epoch": 0.95,
"learning_rate": 9.976353747930953e-05,
"loss": 3.7819,
"step": 4200
},
{
"epoch": 0.99,
"learning_rate": 9.952707495861906e-05,
"loss": 3.7482,
"step": 4400
},
{
"epoch": 1.0,
"eval_loss": 4.301605224609375,
"eval_runtime": 69.8795,
"eval_samples_per_second": 2069.063,
"eval_steps_per_second": 8.085,
"step": 4429
},
{
"epoch": 1.04,
"learning_rate": 9.92906124379286e-05,
"loss": 3.7059,
"step": 4600
},
{
"epoch": 1.08,
"learning_rate": 9.905414991723812e-05,
"loss": 3.6762,
"step": 4800
},
{
"epoch": 1.13,
"learning_rate": 9.881768739654765e-05,
"loss": 3.6543,
"step": 5000
},
{
"epoch": 1.17,
"learning_rate": 9.858122487585718e-05,
"loss": 3.6227,
"step": 5200
},
{
"epoch": 1.22,
"learning_rate": 9.834476235516671e-05,
"loss": 3.6135,
"step": 5400
},
{
"epoch": 1.26,
"learning_rate": 9.810829983447623e-05,
"loss": 3.5938,
"step": 5600
},
{
"epoch": 1.31,
"learning_rate": 9.787183731378577e-05,
"loss": 3.5746,
"step": 5800
},
{
"epoch": 1.35,
"learning_rate": 9.76353747930953e-05,
"loss": 3.5606,
"step": 6000
},
{
"epoch": 1.4,
"learning_rate": 9.739891227240483e-05,
"loss": 3.5438,
"step": 6200
},
{
"epoch": 1.45,
"learning_rate": 9.716244975171436e-05,
"loss": 3.5257,
"step": 6400
},
{
"epoch": 1.49,
"learning_rate": 9.692598723102388e-05,
"loss": 3.5158,
"step": 6600
},
{
"epoch": 1.54,
"learning_rate": 9.66895247103334e-05,
"loss": 3.5033,
"step": 6800
},
{
"epoch": 1.58,
"learning_rate": 9.645306218964295e-05,
"loss": 3.4942,
"step": 7000
},
{
"epoch": 1.63,
"learning_rate": 9.621659966895248e-05,
"loss": 3.478,
"step": 7200
},
{
"epoch": 1.67,
"learning_rate": 9.598013714826201e-05,
"loss": 3.4618,
"step": 7400
},
{
"epoch": 1.72,
"learning_rate": 9.574367462757153e-05,
"loss": 3.4544,
"step": 7600
},
{
"epoch": 1.76,
"learning_rate": 9.550721210688106e-05,
"loss": 3.4517,
"step": 7800
},
{
"epoch": 1.81,
"learning_rate": 9.527074958619059e-05,
"loss": 3.4419,
"step": 8000
},
{
"epoch": 1.85,
"learning_rate": 9.503428706550013e-05,
"loss": 3.4312,
"step": 8200
},
{
"epoch": 1.9,
"learning_rate": 9.479782454480965e-05,
"loss": 3.4204,
"step": 8400
},
{
"epoch": 1.94,
"learning_rate": 9.456136202411918e-05,
"loss": 3.4123,
"step": 8600
},
{
"epoch": 1.99,
"learning_rate": 9.43248995034287e-05,
"loss": 3.4047,
"step": 8800
},
{
"epoch": 2.0,
"eval_loss": 3.827420234680176,
"eval_runtime": 69.8019,
"eval_samples_per_second": 2071.363,
"eval_steps_per_second": 8.094,
"step": 8858
},
{
"epoch": 2.03,
"learning_rate": 9.408843698273824e-05,
"loss": 3.3663,
"step": 9000
},
{
"epoch": 2.08,
"learning_rate": 9.385197446204776e-05,
"loss": 3.3552,
"step": 9200
},
{
"epoch": 2.12,
"learning_rate": 9.36155119413573e-05,
"loss": 3.3514,
"step": 9400
},
{
"epoch": 2.17,
"learning_rate": 9.337904942066683e-05,
"loss": 3.3406,
"step": 9600
},
{
"epoch": 2.21,
"learning_rate": 9.314258689997636e-05,
"loss": 3.34,
"step": 9800
},
{
"epoch": 2.26,
"learning_rate": 9.290612437928589e-05,
"loss": 3.336,
"step": 10000
},
{
"epoch": 2.3,
"learning_rate": 9.266966185859541e-05,
"loss": 3.334,
"step": 10200
},
{
"epoch": 2.35,
"learning_rate": 9.243319933790493e-05,
"loss": 3.3316,
"step": 10400
},
{
"epoch": 2.39,
"learning_rate": 9.219673681721448e-05,
"loss": 3.3226,
"step": 10600
},
{
"epoch": 2.44,
"learning_rate": 9.1960274296524e-05,
"loss": 3.3198,
"step": 10800
},
{
"epoch": 2.48,
"learning_rate": 9.172381177583354e-05,
"loss": 3.3169,
"step": 11000
},
{
"epoch": 2.53,
"learning_rate": 9.148734925514306e-05,
"loss": 3.3089,
"step": 11200
},
{
"epoch": 2.57,
"learning_rate": 9.125088673445258e-05,
"loss": 3.3086,
"step": 11400
},
{
"epoch": 2.62,
"learning_rate": 9.101442421376213e-05,
"loss": 3.3045,
"step": 11600
},
{
"epoch": 2.66,
"learning_rate": 9.077796169307166e-05,
"loss": 3.2985,
"step": 11800
},
{
"epoch": 2.71,
"learning_rate": 9.054149917238118e-05,
"loss": 3.2895,
"step": 12000
},
{
"epoch": 2.75,
"learning_rate": 9.030503665169071e-05,
"loss": 3.291,
"step": 12200
},
{
"epoch": 2.8,
"learning_rate": 9.006857413100023e-05,
"loss": 3.2911,
"step": 12400
},
{
"epoch": 2.84,
"learning_rate": 8.983211161030977e-05,
"loss": 3.2849,
"step": 12600
},
{
"epoch": 2.89,
"learning_rate": 8.95956490896193e-05,
"loss": 3.2797,
"step": 12800
},
{
"epoch": 2.94,
"learning_rate": 8.935918656892883e-05,
"loss": 3.2785,
"step": 13000
},
{
"epoch": 2.98,
"learning_rate": 8.912272404823836e-05,
"loss": 3.2749,
"step": 13200
},
{
"epoch": 3.0,
"eval_loss": 3.6556012630462646,
"eval_runtime": 70.2666,
"eval_samples_per_second": 2057.662,
"eval_steps_per_second": 8.041,
"step": 13287
},
{
"epoch": 3.03,
"learning_rate": 8.888626152754788e-05,
"loss": 3.2413,
"step": 13400
},
{
"epoch": 3.07,
"learning_rate": 8.864979900685742e-05,
"loss": 3.2222,
"step": 13600
},
{
"epoch": 3.12,
"learning_rate": 8.841333648616694e-05,
"loss": 3.2157,
"step": 13800
},
{
"epoch": 3.16,
"learning_rate": 8.817687396547648e-05,
"loss": 3.2223,
"step": 14000
},
{
"epoch": 3.21,
"learning_rate": 8.794041144478601e-05,
"loss": 3.2191,
"step": 14200
},
{
"epoch": 3.25,
"learning_rate": 8.770394892409553e-05,
"loss": 3.2213,
"step": 14400
},
{
"epoch": 3.3,
"learning_rate": 8.746748640340507e-05,
"loss": 3.2156,
"step": 14600
},
{
"epoch": 3.34,
"learning_rate": 8.723102388271459e-05,
"loss": 3.2169,
"step": 14800
},
{
"epoch": 3.39,
"learning_rate": 8.699456136202411e-05,
"loss": 3.2151,
"step": 15000
},
{
"epoch": 3.43,
"learning_rate": 8.675809884133366e-05,
"loss": 3.2087,
"step": 15200
},
{
"epoch": 3.48,
"learning_rate": 8.652163632064318e-05,
"loss": 3.2081,
"step": 15400
},
{
"epoch": 3.52,
"learning_rate": 8.62851737999527e-05,
"loss": 3.2152,
"step": 15600
},
{
"epoch": 3.57,
"learning_rate": 8.604871127926224e-05,
"loss": 3.2082,
"step": 15800
},
{
"epoch": 3.61,
"learning_rate": 8.581224875857176e-05,
"loss": 3.1991,
"step": 16000
},
{
"epoch": 3.66,
"learning_rate": 8.55757862378813e-05,
"loss": 3.2026,
"step": 16200
},
{
"epoch": 3.7,
"learning_rate": 8.533932371719083e-05,
"loss": 3.1943,
"step": 16400
},
{
"epoch": 3.75,
"learning_rate": 8.510286119650036e-05,
"loss": 3.2041,
"step": 16600
},
{
"epoch": 3.79,
"learning_rate": 8.486639867580989e-05,
"loss": 3.2001,
"step": 16800
},
{
"epoch": 3.84,
"learning_rate": 8.462993615511941e-05,
"loss": 3.191,
"step": 17000
},
{
"epoch": 3.88,
"learning_rate": 8.439347363442895e-05,
"loss": 3.1988,
"step": 17200
},
{
"epoch": 3.93,
"learning_rate": 8.415701111373848e-05,
"loss": 3.1963,
"step": 17400
},
{
"epoch": 3.97,
"learning_rate": 8.3920548593048e-05,
"loss": 3.1943,
"step": 17600
},
{
"epoch": 4.0,
"eval_loss": 3.759209632873535,
"eval_runtime": 70.143,
"eval_samples_per_second": 2061.29,
"eval_steps_per_second": 8.055,
"step": 17716
},
{
"epoch": 4.02,
"learning_rate": 8.368408607235754e-05,
"loss": 3.1674,
"step": 17800
},
{
"epoch": 4.06,
"learning_rate": 8.344762355166706e-05,
"loss": 3.1347,
"step": 18000
},
{
"epoch": 4.11,
"learning_rate": 8.32111610309766e-05,
"loss": 3.138,
"step": 18200
},
{
"epoch": 4.15,
"learning_rate": 8.297469851028612e-05,
"loss": 3.1415,
"step": 18400
},
{
"epoch": 4.2,
"learning_rate": 8.273823598959566e-05,
"loss": 3.142,
"step": 18600
},
{
"epoch": 4.24,
"learning_rate": 8.250177346890519e-05,
"loss": 3.1424,
"step": 18800
},
{
"epoch": 4.29,
"learning_rate": 8.226531094821471e-05,
"loss": 3.1434,
"step": 19000
},
{
"epoch": 4.34,
"learning_rate": 8.202884842752423e-05,
"loss": 3.1422,
"step": 19200
},
{
"epoch": 4.38,
"learning_rate": 8.179238590683377e-05,
"loss": 3.141,
"step": 19400
},
{
"epoch": 4.43,
"learning_rate": 8.155592338614329e-05,
"loss": 3.1398,
"step": 19600
},
{
"epoch": 4.47,
"learning_rate": 8.131946086545284e-05,
"loss": 3.1391,
"step": 19800
},
{
"epoch": 4.52,
"learning_rate": 8.108299834476236e-05,
"loss": 3.1406,
"step": 20000
},
{
"epoch": 4.56,
"learning_rate": 8.084653582407188e-05,
"loss": 3.1415,
"step": 20200
},
{
"epoch": 4.61,
"learning_rate": 8.061007330338142e-05,
"loss": 3.1358,
"step": 20400
},
{
"epoch": 4.65,
"learning_rate": 8.037361078269094e-05,
"loss": 3.1386,
"step": 20600
},
{
"epoch": 4.7,
"learning_rate": 8.013714826200048e-05,
"loss": 3.1323,
"step": 20800
},
{
"epoch": 4.74,
"learning_rate": 7.990068574131001e-05,
"loss": 3.1382,
"step": 21000
},
{
"epoch": 4.79,
"learning_rate": 7.966422322061953e-05,
"loss": 3.135,
"step": 21200
},
{
"epoch": 4.83,
"learning_rate": 7.942776069992907e-05,
"loss": 3.1409,
"step": 21400
},
{
"epoch": 4.88,
"learning_rate": 7.919129817923859e-05,
"loss": 3.133,
"step": 21600
},
{
"epoch": 4.92,
"learning_rate": 7.895483565854813e-05,
"loss": 3.132,
"step": 21800
},
{
"epoch": 4.97,
"learning_rate": 7.871837313785765e-05,
"loss": 3.1289,
"step": 22000
},
{
"epoch": 5.0,
"eval_loss": 3.675868034362793,
"eval_runtime": 70.0266,
"eval_samples_per_second": 2064.716,
"eval_steps_per_second": 8.068,
"step": 22145
},
{
"epoch": 5.01,
"learning_rate": 7.848191061716718e-05,
"loss": 3.1117,
"step": 22200
},
{
"epoch": 5.06,
"learning_rate": 7.824544809647672e-05,
"loss": 3.079,
"step": 22400
},
{
"epoch": 5.1,
"learning_rate": 7.800898557578624e-05,
"loss": 3.0775,
"step": 22600
},
{
"epoch": 5.15,
"learning_rate": 7.777252305509576e-05,
"loss": 3.0806,
"step": 22800
},
{
"epoch": 5.19,
"learning_rate": 7.75360605344053e-05,
"loss": 3.0837,
"step": 23000
},
{
"epoch": 5.24,
"learning_rate": 7.729959801371482e-05,
"loss": 3.0842,
"step": 23200
},
{
"epoch": 5.28,
"learning_rate": 7.706313549302437e-05,
"loss": 3.0828,
"step": 23400
},
{
"epoch": 5.33,
"learning_rate": 7.682667297233389e-05,
"loss": 3.0858,
"step": 23600
},
{
"epoch": 5.37,
"learning_rate": 7.659021045164341e-05,
"loss": 3.0834,
"step": 23800
},
{
"epoch": 5.42,
"learning_rate": 7.635374793095295e-05,
"loss": 3.0843,
"step": 24000
},
{
"epoch": 5.46,
"learning_rate": 7.611728541026247e-05,
"loss": 3.0912,
"step": 24200
},
{
"epoch": 5.51,
"learning_rate": 7.5880822889572e-05,
"loss": 3.0894,
"step": 24400
},
{
"epoch": 5.55,
"learning_rate": 7.564436036888154e-05,
"loss": 3.0896,
"step": 24600
},
{
"epoch": 5.6,
"learning_rate": 7.540789784819106e-05,
"loss": 3.0914,
"step": 24800
},
{
"epoch": 5.64,
"learning_rate": 7.51714353275006e-05,
"loss": 3.0869,
"step": 25000
},
{
"epoch": 5.69,
"learning_rate": 7.493497280681012e-05,
"loss": 3.086,
"step": 25200
},
{
"epoch": 5.73,
"learning_rate": 7.469851028611966e-05,
"loss": 3.0877,
"step": 25400
},
{
"epoch": 5.78,
"learning_rate": 7.446204776542919e-05,
"loss": 3.0898,
"step": 25600
},
{
"epoch": 5.83,
"learning_rate": 7.422558524473871e-05,
"loss": 3.0896,
"step": 25800
},
{
"epoch": 5.87,
"learning_rate": 7.398912272404825e-05,
"loss": 3.0865,
"step": 26000
},
{
"epoch": 5.92,
"learning_rate": 7.375266020335777e-05,
"loss": 3.0804,
"step": 26200
},
{
"epoch": 5.96,
"learning_rate": 7.351619768266729e-05,
"loss": 3.0883,
"step": 26400
},
{
"epoch": 6.0,
"eval_loss": 3.6895909309387207,
"eval_runtime": 69.971,
"eval_samples_per_second": 2066.356,
"eval_steps_per_second": 8.075,
"step": 26574
},
{
"epoch": 6.0,
"step": 26574,
"total_flos": 4.44297835708416e+17,
"train_loss": 3.490822130646673,
"train_runtime": 9148.1319,
"train_samples_per_second": 2478.303,
"train_steps_per_second": 9.683
}
],
"max_steps": 88580,
"num_train_epochs": 20,
"total_flos": 4.44297835708416e+17,
"trial_name": null,
"trial_params": null
}

3
training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0c025fb26d628f08d326265b3cd9a6841982de7e2e8085a5d1d6f96741c405e0
size 5073

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long