BananaMind/BananaMind-1.0-Instruct
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: BananaMind/BananaMind-1.0-Instruct
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-15 00:25:35 +08:00
commit c0de26ddfc
44 changed files with 1004868 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
banner.png filter=lfs diff=lfs merge=lfs -text

159
README.md Normal file
View File

@@ -0,0 +1,159 @@
---
language:
- en
license: apache-2.0
base_model: openai-community/gpt2-xl
tags:
- gpt2
- gpt2-xl
- instruction-tuned
- alpaca
- causal-lm
- text-generation
pipeline_tag: text-generation
---
![Banner](banner.png)
# BananaMind-1.0-Instruct
BananaMind-1.0-Instruct is a full-finetuned GPT-2 XL instruction model trained on Alpaca-style instruction data.
It is our first Scaling Up Plan Model.
We plan to do full pretraining in the future.
The model is based on `openai-community/gpt2-xl` and is intended for basic English instruction-following text generation.
It is our first usable chat model.
It includes a 1024 token context window.
📚 Training Data
For the Full Finetune, we used the Alpaca Cleaned Dataset for ~0.75 Epochs not full because it started plateauing.
## Evaluation
Evaluations were run with `lm-evaluation-harness` using the Hugging Face model backend in bfloat16.
### Benchmark Summary
| Benchmark | Setting | Metric | Score |
|---|---:|---|---:|
| ARC-Easy | 0-shot | acc | 57.62% |
| ARC-Easy | 0-shot | acc_norm | 50.38% |
| ARC-Challenge | 0-shot | acc | 27.13% |
| ARC-Challenge | 0-shot | acc_norm | 28.24% |
| HellaSwag | 0-shot | acc | 40.33% |
| HellaSwag | 0-shot | acc_norm | 50.68% |
| PIQA | 0-shot | acc | 71.00% |
| PIQA | 0-shot | acc_norm | 70.13% |
| Winogrande | 0-shot | acc | 58.25% |
| MMLU | 0-shot | acc | 25.54% |
## Prompt Format
Use this format:
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
## Example Usage
Install dependencies:
pip install -U transformers accelerate safetensors torch
Run inference:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "BananaMind/BananaMind-1.0-Instruct"
tokenizer = AutoTokenizer.from_pretrained(repo, use_fast=True)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
repo,
dtype=torch.bfloat16,
device_map="auto",
)
instruction = "Explain what photosynthesis is in simple terms."
prompt = f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
"""
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=140,
do_sample=False,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
)
text = tokenizer.decode(output[0], skip_special_tokens=True)
print(text.split("### Response:", 1)[-1].strip())
## Suggested Generation Settings
For stable answers:
- `do_sample=False`
- `repetition_penalty=1.1`
- `max_new_tokens=80` to `160`
For slightly more creative answers:
- `do_sample=True`
- `temperature=0.4`
- `top_p=0.85`
- `repetition_penalty=1.15`
- `max_new_tokens=80` to `160`
BananaMind-1.0-Instruct was trained as a full finetune rather than a LoRA adapter. The model was trained using an Alpaca-style instruction-response format.
Older training checkpoints may still be available under the `checkpoints/` folder.
## Samples
The sky can vary in color from blue to purple, depending on the time of day and the location. The sky can also be cloudy or clear, with stars visible through the clouds.
The number of r’s in the word "strawberry" is 3. (yes i dont know how it knows this correctly)
I am an AI assistant designed to assist users in various tasks and provide them with information, entertainment, and assistance.
1 + 1 is equal to 2.
Here's a simple Python script that will print "Hello World" to the console:
```python
#!/usr/bin/env python
import time
print("Hello World")
```
## License
Apache 2.0

40
THIRD_PARTY_NOTICES.md Normal file
View File

@@ -0,0 +1,40 @@
This file lists third-party software contained in BananaMind-1.0-Instruct along with their licenses, in compliance with the redistribution clauses of those licenses.
## 1. gpt-2-xl
The base model is GPT 2 XL.
- **Source Repository**
https://huggingface.co/openai-community/gpt2-xl
- **License Type**
Modified MIT License
- **Copyright Notice**
Copyright (c) 2019 OpenAI
- **Full License Text**
```
Modified MIT License
Software Copyright (c) 2019 OpenAI
We don’t claim ownership of the content you create with GPT-2, so it is yours to do with as you please.
We only ask that you use GPT-2 responsibly and clearly indicate your content was created using GPT-2.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.
The above copyright notice and this permission notice need not be included
with content created by the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
OR OTHER DEALINGS IN THE SOFTWARE.
```

3
banner.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:af4e7e57d1845712f55b1e3c3ed5a1d10555136b27c892eb4bfff27a3e0a4812
size 1365518

42
checkpoints/checkpoint-1000/config.json Normal file
View File

@@ -0,0 +1,42 @@
{
"activation_function": "gelu_new",
"add_cross_attention": false,
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 50256,
"dtype": "float32",
"embd_pdrop": 0.1,
"eos_token_id": 50256,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 1600,
"n_head": 25,
"n_inner": null,
"n_layer": 48,
"n_positions": 1024,
"output_past": true,
"pad_token_id": 50256,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"tie_word_embeddings": true,
"transformers_version": "5.12.1",
"use_cache": false,
"vocab_size": 50257
}

6
checkpoints/checkpoint-1000/generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 50256,
"eos_token_id": 50256,
"transformers_version": "5.12.1"
}

3
checkpoints/checkpoint-1000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fc2466032555654609a1c2121803b783f3891b081ab101e18b377f93324bcf7b
size 6230505744

3
checkpoints/checkpoint-1000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3c26f1e8710551ae1e8d6ff5bc4ea5f4c2d528bc85a8931eb965fa55363b6c5f
size 12461394739

3
checkpoints/checkpoint-1000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ec4bcc3a5281d4c70721b95afedeb58b12ba35259364341d80de39e1154ed621
size 14645

3
checkpoints/checkpoint-1000/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:14ae2a2128444abab378aa06c09a61a84665f758fcc19fc46f5789b0bc1b5665
size 1383

3
checkpoints/checkpoint-1000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:229dd8d2aa1520b4829870dafeda6eaa193b7474493212920aa790c79f73d02b
size 1465

250311
checkpoints/checkpoint-1000/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

13
checkpoints/checkpoint-1000/tokenizer_config.json Normal file
View File

@@ -0,0 +1,13 @@
{
"add_prefix_space": false,
"backend": "tokenizers",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"errors": "replace",
"is_local": false,
"local_files_only": false,
"model_max_length": 1024,
"pad_token": "<|endoftext|>",
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>"
}

734
checkpoints/checkpoint-1000/trainer_state.json Normal file
View File

@@ -0,0 +1,734 @@
{
"best_global_step": null,
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 0.3091190108191654,
"eval_steps": 500,
"global_step": 1000,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.0030911901081916537,
"grad_norm": 2.9735140800476074,
"learning_rate": 9.183673469387756e-07,
"loss": 1.8602472305297852,
"step": 10
},
{
"epoch": 0.0061823802163833074,
"grad_norm": 2.6337854862213135,
"learning_rate": 1.938775510204082e-06,
"loss": 1.7525932312011718,
"step": 20
},
{
"epoch": 0.00927357032457496,
"grad_norm": 1.5814393758773804,
"learning_rate": 2.959183673469388e-06,
"loss": 1.7173307418823243,
"step": 30
},
{
"epoch": 0.012364760432766615,
"grad_norm": 1.5865761041641235,
"learning_rate": 3.979591836734694e-06,
"loss": 1.6625125885009766,
"step": 40
},
{
"epoch": 0.015455950540958269,
"grad_norm": 1.6338456869125366,
"learning_rate": 5e-06,
"loss": 1.6665992736816406,
"step": 50
},
{
"epoch": 0.01854714064914992,
"grad_norm": 1.336181879043579,
"learning_rate": 6.020408163265307e-06,
"loss": 1.6779584884643555,
"step": 60
},
{
"epoch": 0.021638330757341576,
"grad_norm": 1.6245481967926025,
"learning_rate": 7.0408163265306125e-06,
"loss": 1.6278823852539062,
"step": 70
},
{
"epoch": 0.02472952086553323,
"grad_norm": 1.3484340906143188,
"learning_rate": 8.06122448979592e-06,
"loss": 1.6239589691162108,
"step": 80
},
{
"epoch": 0.027820710973724884,
"grad_norm": 1.4237396717071533,
"learning_rate": 9.081632653061225e-06,
"loss": 1.6229394912719726,
"step": 90
},
{
"epoch": 0.030911901081916538,
"grad_norm": 1.9831488132476807,
"learning_rate": 9.999997492674716e-06,
"loss": 1.604932975769043,
"step": 100
},
{
"epoch": 0.03400309119010819,
"grad_norm": 1.3235148191452026,
"learning_rate": 9.99969661668322e-06,
"loss": 1.7327735900878907,
"step": 110
},
{
"epoch": 0.03709428129829984,
"grad_norm": 1.4738770723342896,
"learning_rate": 9.998894310210835e-06,
"loss": 1.5656912803649903,
"step": 120
},
{
"epoch": 0.0401854714064915,
"grad_norm": 1.5969219207763672,
"learning_rate": 9.997590653722625e-06,
"loss": 1.6269208908081054,
"step": 130
},
{
"epoch": 0.04327666151468315,
"grad_norm": 1.747842788696289,
"learning_rate": 9.995785777965145e-06,
"loss": 1.6059745788574218,
"step": 140
},
{
"epoch": 0.04636785162287481,
"grad_norm": 1.5844131708145142,
"learning_rate": 9.99347986395332e-06,
"loss": 1.606407928466797,
"step": 150
},
{
"epoch": 0.04945904173106646,
"grad_norm": 3.227466106414795,
"learning_rate": 9.990673142952299e-06,
"loss": 1.5951569557189942,
"step": 160
},
{
"epoch": 0.05255023183925812,
"grad_norm": 1.5301669836044312,
"learning_rate": 9.987365896454253e-06,
"loss": 1.555880069732666,
"step": 170
},
{
"epoch": 0.05564142194744977,
"grad_norm": 1.346501350402832,
"learning_rate": 9.98355845615015e-06,
"loss": 1.5443081855773926,
"step": 180
},
{
"epoch": 0.05873261205564142,
"grad_norm": 1.7208951711654663,
"learning_rate": 9.979251203896482e-06,
"loss": 1.589011001586914,
"step": 190
},
{
"epoch": 0.061823802163833076,
"grad_norm": 1.6657794713974,
"learning_rate": 9.974444571676974e-06,
"loss": 1.6033136367797851,
"step": 200
},
{
"epoch": 0.06491499227202473,
"grad_norm": 1.3685671091079712,
"learning_rate": 9.969139041559259e-06,
"loss": 1.564203643798828,
"step": 210
},
{
"epoch": 0.06800618238021638,
"grad_norm": 1.4963208436965942,
"learning_rate": 9.963335145646528e-06,
"loss": 1.5261777877807616,
"step": 220
},
{
"epoch": 0.07109737248840804,
"grad_norm": 1.378618597984314,
"learning_rate": 9.95703346602416e-06,
"loss": 1.7190706253051757,
"step": 230
},
{
"epoch": 0.07418856259659969,
"grad_norm": 1.3284642696380615,
"learning_rate": 9.950234634701355e-06,
"loss": 1.5670341491699218,
"step": 240
},
{
"epoch": 0.07727975270479134,
"grad_norm": 1.2741881608963013,
"learning_rate": 9.942939333547737e-06,
"loss": 1.605707550048828,
"step": 250
},
{
"epoch": 0.080370942812983,
"grad_norm": 1.4335650205612183,
"learning_rate": 9.935148294224974e-06,
"loss": 1.5591754913330078,
"step": 260
},
{
"epoch": 0.08346213292117466,
"grad_norm": 1.4812300205230713,
"learning_rate": 9.926862298113397e-06,
"loss": 1.4808053016662597,
"step": 270
},
{
"epoch": 0.0865533230293663,
"grad_norm": 1.5364317893981934,
"learning_rate": 9.918082176233633e-06,
"loss": 1.5635189056396483,
"step": 280
},
{
"epoch": 0.08964451313755796,
"grad_norm": 1.4932867288589478,
"learning_rate": 9.908808809163259e-06,
"loss": 1.5150517463684081,
"step": 290
},
{
"epoch": 0.09273570324574962,
"grad_norm": 1.27454674243927,
"learning_rate": 9.899043126948488e-06,
"loss": 1.5336437225341797,
"step": 300
},
{
"epoch": 0.09582689335394126,
"grad_norm": 1.2767815589904785,
"learning_rate": 9.888786109010895e-06,
"loss": 1.5452400207519532,
"step": 310
},
{
"epoch": 0.09891808346213292,
"grad_norm": 1.4611462354660034,
"learning_rate": 9.878038784049187e-06,
"loss": 1.4844867706298828,
"step": 320
},
{
"epoch": 0.10200927357032458,
"grad_norm": 1.507378101348877,
"learning_rate": 9.866802229936027e-06,
"loss": 1.4919805526733398,
"step": 330
},
{
"epoch": 0.10510046367851623,
"grad_norm": 1.380361795425415,
"learning_rate": 9.855077573609943e-06,
"loss": 1.5452465057373046,
"step": 340
},
{
"epoch": 0.10819165378670788,
"grad_norm": 1.351883053779602,
"learning_rate": 9.842865990962296e-06,
"loss": 1.5766680717468262,
"step": 350
},
{
"epoch": 0.11128284389489954,
"grad_norm": 1.4119234085083008,
"learning_rate": 9.830168706719348e-06,
"loss": 1.5874876022338866,
"step": 360
},
{
"epoch": 0.1143740340030912,
"grad_norm": 1.3740119934082031,
"learning_rate": 9.816986994319438e-06,
"loss": 1.543256950378418,
"step": 370
},
{
"epoch": 0.11746522411128284,
"grad_norm": 1.349916934967041,
"learning_rate": 9.803322175785259e-06,
"loss": 1.4975996971130372,
"step": 380
},
{
"epoch": 0.1205564142194745,
"grad_norm": 1.8141902685165405,
"learning_rate": 9.789175621591274e-06,
"loss": 1.523211669921875,
"step": 390
},
{
"epoch": 0.12364760432766615,
"grad_norm": 1.6711773872375488,
"learning_rate": 9.774548750526261e-06,
"loss": 1.5890820503234864,
"step": 400
},
{
"epoch": 0.1267387944358578,
"grad_norm": 1.4761755466461182,
"learning_rate": 9.75944302955103e-06,
"loss": 1.5950213432312013,
"step": 410
},
{
"epoch": 0.12982998454404945,
"grad_norm": 1.3480901718139648,
"learning_rate": 9.743859973651296e-06,
"loss": 1.5783711433410645,
"step": 420
},
{
"epoch": 0.13292117465224113,
"grad_norm": 1.2381082773208618,
"learning_rate": 9.727801145685725e-06,
"loss": 1.5575005531311035,
"step": 430
},
{
"epoch": 0.13601236476043277,
"grad_norm": 1.2985453605651855,
"learning_rate": 9.71126815622921e-06,
"loss": 1.5073147773742677,
"step": 440
},
{
"epoch": 0.1391035548686244,
"grad_norm": 1.3880364894866943,
"learning_rate": 9.694262663411333e-06,
"loss": 1.56723051071167,
"step": 450
},
{
"epoch": 0.14219474497681608,
"grad_norm": 1.4708359241485596,
"learning_rate": 9.676786372750064e-06,
"loss": 1.5140653610229493,
"step": 460
},
{
"epoch": 0.14528593508500773,
"grad_norm": 1.6823925971984863,
"learning_rate": 9.658841036980722e-06,
"loss": 1.523008441925049,
"step": 470
},
{
"epoch": 0.14837712519319937,
"grad_norm": 1.3621658086776733,
"learning_rate": 9.640428455880178e-06,
"loss": 1.5539534568786622,
"step": 480
},
{
"epoch": 0.15146831530139104,
"grad_norm": 1.972955584526062,
"learning_rate": 9.621550476086362e-06,
"loss": 1.5369157791137695,
"step": 490
},
{
"epoch": 0.1545595054095827,
"grad_norm": 1.6722270250320435,
"learning_rate": 9.602208990913046e-06,
"loss": 1.5470399856567383,
"step": 500
},
{
"epoch": 0.15765069551777433,
"grad_norm": 1.5522538423538208,
"learning_rate": 9.582405940159978e-06,
"loss": 1.5396239280700683,
"step": 510
},
{
"epoch": 0.160741885625966,
"grad_norm": 1.7494624853134155,
"learning_rate": 9.562143309918319e-06,
"loss": 1.572172260284424,
"step": 520
},
{
"epoch": 0.16383307573415765,
"grad_norm": 1.5817722082138062,
"learning_rate": 9.541423132371459e-06,
"loss": 1.5051007270812988,
"step": 530
},
{
"epoch": 0.16692426584234932,
"grad_norm": 1.7209810018539429,
"learning_rate": 9.520247485591208e-06,
"loss": 1.5711745262145995,
"step": 540
},
{
"epoch": 0.17001545595054096,
"grad_norm": 3.1827101707458496,
"learning_rate": 9.498618493329376e-06,
"loss": 1.526050853729248,
"step": 550
},
{
"epoch": 0.1731066460587326,
"grad_norm": 1.406969666481018,
"learning_rate": 9.476538324804782e-06,
"loss": 1.4522027015686034,
"step": 560
},
{
"epoch": 0.17619783616692428,
"grad_norm": 1.7943847179412842,
"learning_rate": 9.454009194485701e-06,
"loss": 1.5250386238098144,
"step": 570
},
{
"epoch": 0.17928902627511592,
"grad_norm": 1.3113778829574585,
"learning_rate": 9.431033361867756e-06,
"loss": 1.5368183135986329,
"step": 580
},
{
"epoch": 0.18238021638330756,
"grad_norm": 1.3316926956176758,
"learning_rate": 9.407613131247328e-06,
"loss": 1.544140625,
"step": 590
},
{
"epoch": 0.18547140649149924,
"grad_norm": 2.0932979583740234,
"learning_rate": 9.383750851490438e-06,
"loss": 1.521708106994629,
"step": 600
},
{
"epoch": 0.18856259659969088,
"grad_norm": 1.614535927772522,
"learning_rate": 9.359448915797178e-06,
"loss": 1.5114651679992677,
"step": 610
},
{
"epoch": 0.19165378670788252,
"grad_norm": 1.2698787450790405,
"learning_rate": 9.334709761461696e-06,
"loss": 1.513047218322754,
"step": 620
},
{
"epoch": 0.1947449768160742,
"grad_norm": 1.365653395652771,
"learning_rate": 9.309535869627753e-06,
"loss": 1.473441219329834,
"step": 630
},
{
"epoch": 0.19783616692426584,
"grad_norm": 1.4953657388687134,
"learning_rate": 9.283929765039877e-06,
"loss": 1.5206417083740233,
"step": 640
},
{
"epoch": 0.2009273570324575,
"grad_norm": 1.307398796081543,
"learning_rate": 9.257894015790158e-06,
"loss": 1.5033920288085938,
"step": 650
},
{
"epoch": 0.20401854714064915,
"grad_norm": 1.9029453992843628,
"learning_rate": 9.231431233060691e-06,
"loss": 1.5524487495422363,
"step": 660
},
{
"epoch": 0.2071097372488408,
"grad_norm": 1.8270984888076782,
"learning_rate": 9.204544070861682e-06,
"loss": 1.474977684020996,
"step": 670
},
{
"epoch": 0.21020092735703247,
"grad_norm": 1.9305938482284546,
"learning_rate": 9.177235225765287e-06,
"loss": 1.5872207641601563,
"step": 680
},
{
"epoch": 0.2132921174652241,
"grad_norm": 1.651270866394043,
"learning_rate": 9.149507436635156e-06,
"loss": 1.5186875343322754,
"step": 690
},
{
"epoch": 0.21638330757341576,
"grad_norm": 1.6599997282028198,
"learning_rate": 9.121363484351753e-06,
"loss": 1.5063211441040039,
"step": 700
},
{
"epoch": 0.21947449768160743,
"grad_norm": 1.261690616607666,
"learning_rate": 9.092806191533448e-06,
"loss": 1.4991595268249511,
"step": 710
},
{
"epoch": 0.22256568778979907,
"grad_norm": 1.2406865358352661,
"learning_rate": 9.063838422253438e-06,
"loss": 1.5417633056640625,
"step": 720
},
{
"epoch": 0.22565687789799072,
"grad_norm": 1.2300302982330322,
"learning_rate": 9.0344630817525e-06,
"loss": 1.4868436813354493,
"step": 730
},
{
"epoch": 0.2287480680061824,
"grad_norm": 1.4104863405227661,
"learning_rate": 9.004683116147618e-06,
"loss": 1.518869400024414,
"step": 740
},
{
"epoch": 0.23183925811437403,
"grad_norm": 1.1874244213104248,
"learning_rate": 8.974501512136508e-06,
"loss": 1.5017603874206542,
"step": 750
},
{
"epoch": 0.23493044822256567,
"grad_norm": 1.398455262184143,
"learning_rate": 8.94392129669808e-06,
"loss": 1.5286592483520507,
"step": 760
},
{
"epoch": 0.23802163833075735,
"grad_norm": 1.5666356086730957,
"learning_rate": 8.912945536788855e-06,
"loss": 1.5568647384643555,
"step": 770
},
{
"epoch": 0.241112828438949,
"grad_norm": 1.2765898704528809,
"learning_rate": 8.881577339035368e-06,
"loss": 1.4791644096374512,
"step": 780
},
{
"epoch": 0.24420401854714066,
"grad_norm": 1.5090545415878296,
"learning_rate": 8.849819849422605e-06,
"loss": 1.4606804847717285,
"step": 790
},
{
"epoch": 0.2472952086553323,
"grad_norm": 1.1877100467681885,
"learning_rate": 8.817676252978482e-06,
"loss": 1.5327486991882324,
"step": 800
},
{
"epoch": 0.250386398763524,
"grad_norm": 1.518449306488037,
"learning_rate": 8.785149773454406e-06,
"loss": 1.5251803398132324,
"step": 810
},
{
"epoch": 0.2534775888717156,
"grad_norm": 1.3751578330993652,
"learning_rate": 8.75224367300197e-06,
"loss": 1.5399330139160157,
"step": 820
},
{
"epoch": 0.25656877897990726,
"grad_norm": 1.4341316223144531,
"learning_rate": 8.718961251845776e-06,
"loss": 1.4913371086120606,
"step": 830
},
{
"epoch": 0.2596599690880989,
"grad_norm": 1.756329894065857,
"learning_rate": 8.685305847952447e-06,
"loss": 1.5053520202636719,
"step": 840
},
{
"epoch": 0.26275115919629055,
"grad_norm": 1.2910538911819458,
"learning_rate": 8.651280836695866e-06,
"loss": 1.4810606956481933,
"step": 850
},
{
"epoch": 0.26584234930448225,
"grad_norm": 1.1309559345245361,
"learning_rate": 8.616889630518634e-06,
"loss": 1.5381391525268555,
"step": 860
},
{
"epoch": 0.2689335394126739,
"grad_norm": 1.6359164714813232,
"learning_rate": 8.582135678589849e-06,
"loss": 1.5432696342468262,
"step": 870
},
{
"epoch": 0.27202472952086554,
"grad_norm": 1.5036001205444336,
"learning_rate": 8.547022466459169e-06,
"loss": 1.5345426559448243,
"step": 880
},
{
"epoch": 0.2751159196290572,
"grad_norm": 1.5126945972442627,
"learning_rate": 8.511553515707238e-06,
"loss": 1.4968720436096192,
"step": 890
},
{
"epoch": 0.2782071097372488,
"grad_norm": 1.4351264238357544,
"learning_rate": 8.475732383592506e-06,
"loss": 1.4474885940551758,
"step": 900
},
{
"epoch": 0.28129829984544047,
"grad_norm": 1.3674299716949463,
"learning_rate": 8.43956266269446e-06,
"loss": 1.4778318405151367,
"step": 910
},
{
"epoch": 0.28438948995363217,
"grad_norm": 2.175255298614502,
"learning_rate": 8.403047980553316e-06,
"loss": 1.5429688453674317,
"step": 920
},
{
"epoch": 0.2874806800618238,
"grad_norm": 1.3890902996063232,
"learning_rate": 8.366191999306203e-06,
"loss": 1.4910104751586915,
"step": 930
},
{
"epoch": 0.29057187017001546,
"grad_norm": 1.6042287349700928,
"learning_rate": 8.328998415319887e-06,
"loss": 1.5073558807373046,
"step": 940
},
{
"epoch": 0.2936630602782071,
"grad_norm": 1.5018166303634644,
"learning_rate": 8.291470958820046e-06,
"loss": 1.5073823928833008,
"step": 950
},
{
"epoch": 0.29675425038639874,
"grad_norm": 1.1693017482757568,
"learning_rate": 8.253613393517161e-06,
"loss": 1.4433947563171388,
"step": 960
},
{
"epoch": 0.29984544049459044,
"grad_norm": 1.3259260654449463,
"learning_rate": 8.215429516229043e-06,
"loss": 1.485299777984619,
"step": 970
},
{
"epoch": 0.3029366306027821,
"grad_norm": 1.3524744510650635,
"learning_rate": 8.176923156500048e-06,
"loss": 1.4936206817626954,
"step": 980
},
{
"epoch": 0.30602782071097373,
"grad_norm": 1.2411019802093506,
"learning_rate": 8.138098176216999e-06,
"loss": 1.4466056823730469,
"step": 990
},
{
"epoch": 0.3091190108191654,
"grad_norm": 1.7096987962722778,
"learning_rate": 8.098958469221867e-06,
"loss": 1.4811983108520508,
"step": 1000
}
],
"logging_steps": 10,
"max_steps": 3235,
"num_input_tokens_seen": 0,
"num_train_epochs": 1,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": false
},
"attributes": {}
}
},
"total_flos": 2.85528960565248e+16,
"train_batch_size": 1,
"trial_name": null,
"trial_params": null
}

3
checkpoints/checkpoint-1000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:505999d3a1e6baaa16832f11db4bf97a5d1bd224d517d6d4a7ac2f3008a3aff8
size 5201

42
checkpoints/checkpoint-2500/config.json Normal file
View File

@@ -0,0 +1,42 @@
{
"activation_function": "gelu_new",
"add_cross_attention": false,
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 50256,
"dtype": "float32",
"embd_pdrop": 0.1,
"eos_token_id": 50256,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 1600,
"n_head": 25,
"n_inner": null,
"n_layer": 48,
"n_positions": 1024,
"output_past": true,
"pad_token_id": 50256,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"tie_word_embeddings": true,
"transformers_version": "5.12.1",
"use_cache": false,
"vocab_size": 50257
}

6
checkpoints/checkpoint-2500/generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 50256,
"eos_token_id": 50256,
"transformers_version": "5.12.1"
}

3
checkpoints/checkpoint-2500/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:99120dea29c96d2624f352dca4ac829289f9faae1f43bf442d2eb5a9745ad22c
size 6230505744

3
checkpoints/checkpoint-2500/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b0c35323fb402e0f5e5be62b4a7a7c1b0544ce537d1b75d528cc9d3738e9ac53
size 12461394739

3
checkpoints/checkpoint-2500/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:51c4af098767adcf11b45bbc8a70b02ab9a17d573ec1a630d1e5da3091f6d6ec
size 14645

3
checkpoints/checkpoint-2500/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:861ce13e6ca091acee9a68ebfc5ca38479baf4b537c37b3949f071f77b81e9f0
size 1383

3
checkpoints/checkpoint-2500/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e4e45924548bf03c960500f224888c2e9e8fd434dae2bccaa101877c6182dd0b
size 1465

250311
checkpoints/checkpoint-2500/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

13
checkpoints/checkpoint-2500/tokenizer_config.json Normal file
View File

@@ -0,0 +1,13 @@
{
"add_prefix_space": false,
"backend": "tokenizers",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"errors": "replace",
"is_local": false,
"local_files_only": false,
"model_max_length": 1024,
"pad_token": "<|endoftext|>",
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>"
}

1784
checkpoints/checkpoint-2500/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoints/checkpoint-2500/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:505999d3a1e6baaa16832f11db4bf97a5d1bd224d517d6d4a7ac2f3008a3aff8
size 5201

42
checkpoints/checkpoint-500/config.json Normal file
View File

@@ -0,0 +1,42 @@
{
"activation_function": "gelu_new",
"add_cross_attention": false,
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 50256,
"dtype": "float32",
"embd_pdrop": 0.1,
"eos_token_id": 50256,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 1600,
"n_head": 25,
"n_inner": null,
"n_layer": 48,
"n_positions": 1024,
"output_past": true,
"pad_token_id": 50256,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"tie_word_embeddings": true,
"transformers_version": "5.12.1",
"use_cache": false,
"vocab_size": 50257
}

6
checkpoints/checkpoint-500/generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 50256,
"eos_token_id": 50256,
"transformers_version": "5.12.1"
}

3
checkpoints/checkpoint-500/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d8138102255c98db99f9fb78d187cb6a039393e4e27b6fe717799381c110e4cf
size 6230505744

3
checkpoints/checkpoint-500/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a2598119ebf904cc3b7ffef50dd8fcdac8d893e60921a1760b8f4dd02a0cd28f
size 12461394739

3
checkpoints/checkpoint-500/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:35821b748856c27d0d558b0494e873f222c9f77130d8344a150c29d80638aada
size 14645

3
checkpoints/checkpoint-500/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f77569c2e850b04af982cc8c1389f1430851448915c593b69e5da36ce05b71d7
size 1383

3
checkpoints/checkpoint-500/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3004b1f3d59b5870e7d52b04fb700c9ab163925d77fd7780e057472941df31d6
size 1465

250306
checkpoints/checkpoint-500/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

13
checkpoints/checkpoint-500/tokenizer_config.json Normal file
View File

@@ -0,0 +1,13 @@
{
"add_prefix_space": false,
"backend": "tokenizers",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"errors": "replace",
"is_local": false,
"local_files_only": false,
"model_max_length": 1024,
"pad_token": "<|endoftext|>",
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>"
}

384
checkpoints/checkpoint-500/trainer_state.json Normal file
View File

@@ -0,0 +1,384 @@
{
"best_global_step": null,
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 0.1545595054095827,
"eval_steps": 500,
"global_step": 500,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.0030911901081916537,
"grad_norm": 2.9735140800476074,
"learning_rate": 9.183673469387756e-07,
"loss": 1.8602472305297852,
"step": 10
},
{
"epoch": 0.0061823802163833074,
"grad_norm": 2.6337854862213135,
"learning_rate": 1.938775510204082e-06,
"loss": 1.7525932312011718,
"step": 20
},
{
"epoch": 0.00927357032457496,
"grad_norm": 1.5814393758773804,
"learning_rate": 2.959183673469388e-06,
"loss": 1.7173307418823243,
"step": 30
},
{
"epoch": 0.012364760432766615,
"grad_norm": 1.5865761041641235,
"learning_rate": 3.979591836734694e-06,
"loss": 1.6625125885009766,
"step": 40
},
{
"epoch": 0.015455950540958269,
"grad_norm": 1.6338456869125366,
"learning_rate": 5e-06,
"loss": 1.6665992736816406,
"step": 50
},
{
"epoch": 0.01854714064914992,
"grad_norm": 1.336181879043579,
"learning_rate": 6.020408163265307e-06,
"loss": 1.6779584884643555,
"step": 60
},
{
"epoch": 0.021638330757341576,
"grad_norm": 1.6245481967926025,
"learning_rate": 7.0408163265306125e-06,
"loss": 1.6278823852539062,
"step": 70
},
{
"epoch": 0.02472952086553323,
"grad_norm": 1.3484340906143188,
"learning_rate": 8.06122448979592e-06,
"loss": 1.6239589691162108,
"step": 80
},
{
"epoch": 0.027820710973724884,
"grad_norm": 1.4237396717071533,
"learning_rate": 9.081632653061225e-06,
"loss": 1.6229394912719726,
"step": 90
},
{
"epoch": 0.030911901081916538,
"grad_norm": 1.9831488132476807,
"learning_rate": 9.999997492674716e-06,
"loss": 1.604932975769043,
"step": 100
},
{
"epoch": 0.03400309119010819,
"grad_norm": 1.3235148191452026,
"learning_rate": 9.99969661668322e-06,
"loss": 1.7327735900878907,
"step": 110
},
{
"epoch": 0.03709428129829984,
"grad_norm": 1.4738770723342896,
"learning_rate": 9.998894310210835e-06,
"loss": 1.5656912803649903,
"step": 120
},
{
"epoch": 0.0401854714064915,
"grad_norm": 1.5969219207763672,
"learning_rate": 9.997590653722625e-06,
"loss": 1.6269208908081054,
"step": 130
},
{
"epoch": 0.04327666151468315,
"grad_norm": 1.747842788696289,
"learning_rate": 9.995785777965145e-06,
"loss": 1.6059745788574218,
"step": 140
},
{
"epoch": 0.04636785162287481,
"grad_norm": 1.5844131708145142,
"learning_rate": 9.99347986395332e-06,
"loss": 1.606407928466797,
"step": 150
},
{
"epoch": 0.04945904173106646,
"grad_norm": 3.227466106414795,
"learning_rate": 9.990673142952299e-06,
"loss": 1.5951569557189942,
"step": 160
},
{
"epoch": 0.05255023183925812,
"grad_norm": 1.5301669836044312,
"learning_rate": 9.987365896454253e-06,
"loss": 1.555880069732666,
"step": 170
},
{
"epoch": 0.05564142194744977,
"grad_norm": 1.346501350402832,
"learning_rate": 9.98355845615015e-06,
"loss": 1.5443081855773926,
"step": 180
},
{
"epoch": 0.05873261205564142,
"grad_norm": 1.7208951711654663,
"learning_rate": 9.979251203896482e-06,
"loss": 1.589011001586914,
"step": 190
},
{
"epoch": 0.061823802163833076,
"grad_norm": 1.6657794713974,
"learning_rate": 9.974444571676974e-06,
"loss": 1.6033136367797851,
"step": 200
},
{
"epoch": 0.06491499227202473,
"grad_norm": 1.3685671091079712,
"learning_rate": 9.969139041559259e-06,
"loss": 1.564203643798828,
"step": 210
},
{
"epoch": 0.06800618238021638,
"grad_norm": 1.4963208436965942,
"learning_rate": 9.963335145646528e-06,
"loss": 1.5261777877807616,
"step": 220
},
{
"epoch": 0.07109737248840804,
"grad_norm": 1.378618597984314,
"learning_rate": 9.95703346602416e-06,
"loss": 1.7190706253051757,
"step": 230
},
{
"epoch": 0.07418856259659969,
"grad_norm": 1.3284642696380615,
"learning_rate": 9.950234634701355e-06,
"loss": 1.5670341491699218,
"step": 240
},
{
"epoch": 0.07727975270479134,
"grad_norm": 1.2741881608963013,
"learning_rate": 9.942939333547737e-06,
"loss": 1.605707550048828,
"step": 250
},
{
"epoch": 0.080370942812983,
"grad_norm": 1.4335650205612183,
"learning_rate": 9.935148294224974e-06,
"loss": 1.5591754913330078,
"step": 260
},
{
"epoch": 0.08346213292117466,
"grad_norm": 1.4812300205230713,
"learning_rate": 9.926862298113397e-06,
"loss": 1.4808053016662597,
"step": 270
},
{
"epoch": 0.0865533230293663,
"grad_norm": 1.5364317893981934,
"learning_rate": 9.918082176233633e-06,
"loss": 1.5635189056396483,
"step": 280
},
{
"epoch": 0.08964451313755796,
"grad_norm": 1.4932867288589478,
"learning_rate": 9.908808809163259e-06,
"loss": 1.5150517463684081,
"step": 290
},
{
"epoch": 0.09273570324574962,
"grad_norm": 1.27454674243927,
"learning_rate": 9.899043126948488e-06,
"loss": 1.5336437225341797,
"step": 300
},
{
"epoch": 0.09582689335394126,
"grad_norm": 1.2767815589904785,
"learning_rate": 9.888786109010895e-06,
"loss": 1.5452400207519532,
"step": 310
},
{
"epoch": 0.09891808346213292,
"grad_norm": 1.4611462354660034,
"learning_rate": 9.878038784049187e-06,
"loss": 1.4844867706298828,
"step": 320
},
{
"epoch": 0.10200927357032458,
"grad_norm": 1.507378101348877,
"learning_rate": 9.866802229936027e-06,
"loss": 1.4919805526733398,
"step": 330
},
{
"epoch": 0.10510046367851623,
"grad_norm": 1.380361795425415,
"learning_rate": 9.855077573609943e-06,
"loss": 1.5452465057373046,
"step": 340
},
{
"epoch": 0.10819165378670788,
"grad_norm": 1.351883053779602,
"learning_rate": 9.842865990962296e-06,
"loss": 1.5766680717468262,
"step": 350
},
{
"epoch": 0.11128284389489954,
"grad_norm": 1.4119234085083008,
"learning_rate": 9.830168706719348e-06,
"loss": 1.5874876022338866,
"step": 360
},
{
"epoch": 0.1143740340030912,
"grad_norm": 1.3740119934082031,
"learning_rate": 9.816986994319438e-06,
"loss": 1.543256950378418,
"step": 370
},
{
"epoch": 0.11746522411128284,
"grad_norm": 1.349916934967041,
"learning_rate": 9.803322175785259e-06,
"loss": 1.4975996971130372,
"step": 380
},
{
"epoch": 0.1205564142194745,
"grad_norm": 1.8141902685165405,
"learning_rate": 9.789175621591274e-06,
"loss": 1.523211669921875,
"step": 390
},
{
"epoch": 0.12364760432766615,
"grad_norm": 1.6711773872375488,
"learning_rate": 9.774548750526261e-06,
"loss": 1.5890820503234864,
"step": 400
},
{
"epoch": 0.1267387944358578,
"grad_norm": 1.4761755466461182,
"learning_rate": 9.75944302955103e-06,
"loss": 1.5950213432312013,
"step": 410
},
{
"epoch": 0.12982998454404945,
"grad_norm": 1.3480901718139648,
"learning_rate": 9.743859973651296e-06,
"loss": 1.5783711433410645,
"step": 420
},
{
"epoch": 0.13292117465224113,
"grad_norm": 1.2381082773208618,
"learning_rate": 9.727801145685725e-06,
"loss": 1.5575005531311035,
"step": 430
},
{
"epoch": 0.13601236476043277,
"grad_norm": 1.2985453605651855,
"learning_rate": 9.71126815622921e-06,
"loss": 1.5073147773742677,
"step": 440
},
{
"epoch": 0.1391035548686244,
"grad_norm": 1.3880364894866943,
"learning_rate": 9.694262663411333e-06,
"loss": 1.56723051071167,
"step": 450
},
{
"epoch": 0.14219474497681608,
"grad_norm": 1.4708359241485596,
"learning_rate": 9.676786372750064e-06,
"loss": 1.5140653610229493,
"step": 460
},
{
"epoch": 0.14528593508500773,
"grad_norm": 1.6823925971984863,
"learning_rate": 9.658841036980722e-06,
"loss": 1.523008441925049,
"step": 470
},
{
"epoch": 0.14837712519319937,
"grad_norm": 1.3621658086776733,
"learning_rate": 9.640428455880178e-06,
"loss": 1.5539534568786622,
"step": 480
},
{
"epoch": 0.15146831530139104,
"grad_norm": 1.972955584526062,
"learning_rate": 9.621550476086362e-06,
"loss": 1.5369157791137695,
"step": 490
},
{
"epoch": 0.1545595054095827,
"grad_norm": 1.6722270250320435,
"learning_rate": 9.602208990913046e-06,
"loss": 1.5470399856567383,
"step": 500
}
],
"logging_steps": 10,
"max_steps": 3235,
"num_input_tokens_seen": 0,
"num_train_epochs": 1,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": false
},
"attributes": {}
}
},
"total_flos": 1.42941547674624e+16,
"train_batch_size": 1,
"trial_name": null,
"trial_params": null
}

3
checkpoints/checkpoint-500/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:505999d3a1e6baaa16832f11db4bf97a5d1bd224d517d6d4a7ac2f3008a3aff8
size 5201

42
config.json Normal file
View File

@@ -0,0 +1,42 @@
{
"activation_function": "gelu_new",
"add_cross_attention": false,
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 50256,
"dtype": "float32",
"embd_pdrop": 0.1,
"eos_token_id": 50256,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 1600,
"n_head": 25,
"n_inner": null,
"n_layer": 48,
"n_positions": 1024,
"output_past": true,
"pad_token_id": 50256,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"tie_word_embeddings": true,
"transformers_version": "5.12.1",
"use_cache": false,
"vocab_size": 50257
}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 50256,
"eos_token_id": 50256,
"transformers_version": "5.12.1"
}

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:99120dea29c96d2624f352dca4ac829289f9faae1f43bf442d2eb5a9745ad22c
size 6230505744

3
scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:861ce13e6ca091acee9a68ebfc5ca38479baf4b537c37b3949f071f77b81e9f0
size 1383

250311
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

13
tokenizer_config.json Normal file
View File

@@ -0,0 +1,13 @@
{
"add_prefix_space": false,
"backend": "tokenizers",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"errors": "replace",
"is_local": false,
"local_files_only": false,
"model_max_length": 1024,
"pad_token": "<|endoftext|>",
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>"
}

185
train_gpt2_xl_alpaca_full.py Normal file
View File

@@ -0,0 +1,185 @@
import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True
torch.set_float32_matmul_precision('high')
#!/usr/bin/env python3
import argparse
import os
import torch
from datasets import load_dataset
from transformers import (
GPT2LMHeadModel,
GPT2Tokenizer,
Trainer,
TrainingArguments,
)
PROMPT_TEMPLATE = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
"""
def build_instruction(example):
instruction = example["instruction"].strip()
inp = example.get("input", "")
if inp and inp.strip():
instruction = instruction + "\n\nInput:\n" + inp.strip()
return instruction
def tokenize_example(example, tokenizer, max_length):
instruction = build_instruction(example)
response = example["output"].strip()
prompt = PROMPT_TEMPLATE.format(instruction=instruction)
full_text = prompt + response + tokenizer.eos_token
prompt_ids = tokenizer(
prompt,
add_special_tokens=False,
truncation=True,
max_length=max_length,
)["input_ids"]
full = tokenizer(
full_text,
add_special_tokens=False,
truncation=True,
max_length=max_length,
)
input_ids = full["input_ids"]
attention_mask = full["attention_mask"]
labels = input_ids.copy()
prompt_len = min(len(prompt_ids), len(labels))
labels[:prompt_len] = [-100] * prompt_len
return {
"input_ids": input_ids,
"attention_mask": attention_mask,
"labels": labels,
}
class CausalCollator:
def __init__(self, tokenizer, pad_to_multiple_of=8):
self.tokenizer = tokenizer
self.pad_to_multiple_of = pad_to_multiple_of
def __call__(self, features):
max_len = max(len(x["input_ids"]) for x in features)
if self.pad_to_multiple_of:
rem = max_len % self.pad_to_multiple_of
if rem:
max_len += self.pad_to_multiple_of - rem
input_ids = []
attention_mask = []
labels = []
for x in features:
pad_len = max_len - len(x["input_ids"])
input_ids.append(x["input_ids"] + [self.tokenizer.pad_token_id] * pad_len)
attention_mask.append(x["attention_mask"] + [0] * pad_len)
labels.append(x["labels"] + [-100] * pad_len)
return {
"input_ids": torch.tensor(input_ids, dtype=torch.long),
"attention_mask": torch.tensor(attention_mask, dtype=torch.long),
"labels": torch.tensor(labels, dtype=torch.long),
}
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--model", default="gpt2-xl")
parser.add_argument("--dataset", default="yahma/alpaca-cleaned")
parser.add_argument("--out", default="gpt2-xl-alpaca-full")
parser.add_argument("--max-length", type=int, default=1024)
parser.add_argument("--epochs", type=float, default=1.0)
parser.add_argument("--lr", type=float, default=1e-5)
parser.add_argument("--batch-size", type=int, default=1)
parser.add_argument("--grad-accum", type=int, default=16)
parser.add_argument("--limit", type=int, default=0)
parser.add_argument("--save-steps", type=int, default=500)
parser.add_argument("--logging-steps", type=int, default=10)
args = parser.parse_args()
os.makedirs(args.out, exist_ok=True)
tokenizer = GPT2Tokenizer.from_pretrained(args.model)
tokenizer.pad_token = tokenizer.eos_token
model = GPT2LMHeadModel.from_pretrained(
args.model if torch.cuda.is_available() else torch.float32,
)
model.config.pad_token_id = tokenizer.eos_token_id
model.config.use_cache = False
# # model.gradient_checkpointing_enable()
ds = load_dataset(args.dataset, split="train")
ds = ds.shuffle(seed=42)
if args.limit > 0:
ds = ds.select(range(min(args.limit, len(ds))))
tokenized = ds.map(
lambda ex: tokenize_example(ex, tokenizer, args.max_length),
remove_columns=ds.column_names,
desc="Tokenizing",
)
tokenized = tokenized.filter(
lambda ex: any(label != -100 for label in ex["labels"]),
desc="Filtering empty-label examples",
)
training_args = TrainingArguments(
output_dir=args.out,
num_train_epochs=args.epochs,
per_device_train_batch_size=args.batch_size,
gradient_accumulation_steps=args.grad_accum,
learning_rate=args.lr,
warmup_ratio=0.03,
lr_scheduler_type="cosine",
dataloader_num_workers=8,
dataloader_pin_memory=True,
logging_steps=args.logging_steps,
save_steps=args.save_steps,
save_total_limit=3,
fp16=torch.cuda.is_available(),
optim="adamw_torch",
weight_decay=0.01,
max_grad_norm=1.0,
report_to="none",
remove_unused_columns=False,
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized,
data_collator=CausalCollator(tokenizer),
)
trainer.train(resume_from_checkpoint="/gpt2-xl-alpaca-full/checkpoint-500")
trainer.save_model(args.out)
tokenizer.save_pretrained(args.out)
print("saved full model to", args.out)
if __name__ == "__main__":
main()