ai-forever/mGPT-13B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: ai-forever/mGPT-13B
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-13 14:52:24 +08:00
commit 14b62b4733
15 changed files with 300439 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
.gitattributes vendored Normal file
View File

@@ -0,0 +1,34 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

112
README.md Normal file
View File

@@ -0,0 +1,112 @@
---
license: mit
language:
- ar
- he
- vi
- id
- jv
- ms
- tl
- lv
- lt
- eu
- ml
- ta
- te
- hy
- bn
- mr
- hi
- ur
- af
- da
- en
- de
- sv
- fr
- it
- pt
- ro
- es
- el
- os
- tg
- fa
- ja
- ka
- ko
- th
- bxr
- xal
- mn
- sw
- yo
- be
- bg
- ru
- uk
- pl
- my
- uz
- ba
- kk
- ky
- tt
- az
- cv
- tr
- tk
- tyv
- sax
- et
- fi
- hu
tags:
- multilingual
- PyTorch
- Transformers
- gpt3
- gpt2
- transformers
---
# 🌻 mGPT 13B
Multilingual language model. This model was trained on the **61** languages from **25** language families (see the list below).
## Dataset
Model was pretrained on a 600Gb of texts, mostly from MC4 and Wikipedia. Training data was deduplicated, the text deduplication includes 64-bit hashing of each text in the corpus for keeping texts with a unique hash. We also filter the documents based on their text compression rate using zlib4. The most strongly and weakly compressing deduplicated texts are discarded.
Here is the table with number of tokens for each language in the pretraining corpus on a logarithmic scale:
![](https://i.imgur.com/KSMfVX1.png)
## Languages
Afrikaans (af), Arabic (ar), Armenian (hy), Azerbaijani (az), Basque (eu), Bashkir (ba), Belarusian (be), Bengali (bn), Bulgarian (bg), Burmese (my), Buryat (bxr), Chuvash (cv), Danish (da), English (en), Estonian (et), Finnish (fi), French (fr), Georgian (ka), German (de), Greek (el), Hebrew (he), Hindi (hi), Hungarian (hu), Indonesian (id), Italian (it), Japanese (ja), Javanese (jv), Kalmyk (xal), Kazakh (kk), Korean (ko), Kyrgyz (ky), Latvian (lv), Lithuanian (lt), Malay (ms), Malayalam (ml), Marathi (mr), Mongolian (mn), Ossetian (os), Persian (fa), Polish (pl), Portuguese (pt), Romanian (ro), Russian (ru), Spanish (es), Swedish (sv), Swahili (sw), Tatar (tt), Telugu (te), Thai (th), Turkish (tr), Turkmen (tk), Tuvan (tyv), Ukrainian (uk), Uzbek (uz), Vietnamese (vi), Yakut (sax), Yoruba (yo)
#### By language family
<table><thead><tr><th>Language Family</th><th>Languages</th></tr></thead><tbody><tr><td>Afro-Asiatic</td><td>Arabic (ar), Hebrew (he)</td></tr><tr><td>Austro-Asiatic</td><td>Vietnamese (vi)</td></tr><tr><td>Austronesian</td><td>Indonesian (id), Javanese (jv), Malay (ms), Tagalog (tl)</td></tr><tr><td>Baltic</td><td>Latvian (lv), Lithuanian (lt)</td></tr><tr><td>Basque</td><td>Basque (eu)</td></tr><tr><td>Dravidian</td><td>Malayalam (ml), Tamil (ta), Telugu (te)</td></tr><tr><td>Indo-European (Armenian)</td><td>Armenian (hy)</td></tr><tr><td>Indo-European (Indo-Aryan)</td><td>Bengali (bn), Marathi (mr), Hindi (hi), Urdu (ur)</td></tr><tr><td>Indo-European (Germanic)</td><td>Afrikaans (af), Danish (da), English (en), German (de), Swedish (sv)</td></tr><tr><td>Indo-European (Romance)</td><td>French (fr), Italian (it), Portuguese (pt), Romanian (ro), Spanish (es)</td></tr><tr><td>Indo-European (Greek)</td><td>Greek (el)</td></tr><tr><td>Indo-European (Iranian)</td><td>Ossetian (os), Tajik (tg), Persian (fa)</td></tr><tr><td>Japonic</td><td>Japanese (ja)</td></tr><tr><td>Kartvelian</td><td>Georgian (ka)</td></tr><tr><td>Koreanic</td><td>Korean (ko)</td></tr><tr><td>Kra-Dai</td><td>Thai (th)</td></tr><tr><td>Mongolic</td><td>Buryat (bxr), Kalmyk (xal), Mongolian (mn)</td></tr><tr><td>Niger-Congo</td><td>Swahili (sw), Yoruba (yo)</td></tr><tr><td>Slavic</td><td>Belarusian (be), Bulgarian (bg), Russian (ru), Ukrainian (uk), Polish (pl)</td></tr><tr><td>Sino-Tibetan</td><td>Burmese (my)</td></tr><tr><td>Turkic (Karluk)</td><td>Uzbek (uz)</td></tr><tr><td>Turkic (Kipchak)</td><td>Bashkir (ba), Kazakh (kk), Kyrgyz (ky), Tatar (tt)</td></tr><tr><td>Turkic (Oghuz)</td><td>Azerbaijani (az), Chuvash (cv), Turkish (tr), Turkmen (tk)</td></tr><tr><td>Turkic (Siberian)</td><td>Tuvan (tyv), Yakut (sax)</td></tr><tr><td>Uralic</td><td>Estonian (et), Finnish (fi), Hungarian (hu)</td></tr></tbody></table>
## Technical details
The models are pretrained on 16 V100 GPUs for 600k training steps with a set of fixed hyperparameters: vocabulary size of 100k, context window of 2048, learning rate of 2e−4, and batch size of 4.
The mGPT architecture is based on GPT-3. We use the architecture description by Brown et al., the code base on GPT-2 (Radford et al., 2019) in the HuggingFace library (Wolf et al., 2020) and Megatron-LM (Shoeybi et al., 2019).
## Perplexity
The mGPT13B model achieves the best perplexities within the 2-to-10 score range for the majority of languages, including Dravidian (Malayalam, Tamil, Telugu), Indo-Aryan (Bengali, Hindi, Marathi), Slavic (Belarusian, Ukrainian, Russian, Bulgarian), Sino-Tibetan (Burmese), Kipchak (Bashkir, Kazakh) and others. Higher perplexities up to 20 are for only seven languages from different families.
#### Language-wise perplexity results
![](https://i.imgur.com/aIKEpPE.png)
#### Family-wise perplexity results
![](https://i.imgur.com/1ugWbXc.png)
_The scores are averaged over the number of languages within each family._

35
config.json Normal file
View File

@@ -0,0 +1,35 @@
{
"_name_or_path": "ai-forever/mGPT-13B",
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 0,
"embd_pdrop": 0.1,
"eos_token_id": 5,
"gradient_checkpointing": false,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 2048,
"n_embd": 5120,
"n_head": 40,
"n_inner": null,
"n_layer": 40,
"n_positions": 2048,
"pad_token_id": 1,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"torch_dtype": "float32",
"transformers_version": "4.24.0",
"use_cache": true,
"vocab_size": 100000
}

99738
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
pytorch_model-00001-of-00006.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:59c46524428e95456e13589088c1604eb69cbdda0d8e595fa7f85490e8ac757c
size 9985353615

3
pytorch_model-00002-of-00006.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a39317def54cf7f79c04e32027178f1fd79211a6be8e5087b62dcad7c6068811
size 9787416273

3
pytorch_model-00003-of-00006.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9b334b58f6d6e8d0bfdf4bc49e3376ea0d07412543e4aec8e3443f557166e83f
size 9678363005

3
pytorch_model-00004-of-00006.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4be422d207f50cb9e3f6b8baaac440a095e70cffc0c95dbea93faef1c54f69d5
size 9682538185

3
pytorch_model-00005-of-00006.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0913ad393e9098da5c6f594293c9f0fd3e13e286d92c4bf934d04d0d7ab6f53e
size 9997130859

3
pytorch_model-00006-of-00006.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5d2effd2b75322049ed226355bf2032d03e24959a616228c5ced32d71da42a05
size 5517439401

572
pytorch_model.bin.index.json Normal file
View File

@@ -0,0 +1,572 @@
{
"metadata": {
"total_size": 54648053920
},
"weight_map": {
"lm_head.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.0.attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.ln_1.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.ln_1.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.ln_2.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.ln_2.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.0.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.ln_1.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.ln_1.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.ln_2.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.ln_2.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.1.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.10.attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.ln_1.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.ln_1.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.ln_2.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.ln_2.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.10.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.ln_1.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.ln_1.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.ln_2.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.ln_2.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.11.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.ln_1.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.ln_1.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.ln_2.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.ln_2.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.12.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.ln_1.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.ln_1.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.ln_2.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.ln_2.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.13.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.14.attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.14.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.14.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.14.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.14.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.14.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.14.ln_1.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.14.ln_1.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.14.ln_2.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.14.ln_2.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.14.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.14.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.14.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.14.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.ln_1.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.ln_1.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.ln_2.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.ln_2.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.15.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.ln_1.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.ln_1.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.ln_2.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.ln_2.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.16.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.ln_1.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.ln_1.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.ln_2.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.ln_2.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.17.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.ln_1.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.ln_1.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.ln_2.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.ln_2.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.18.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.ln_1.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.ln_1.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.ln_2.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.ln_2.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.19.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.2.attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.ln_1.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.ln_1.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.ln_2.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.ln_2.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.2.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.20.attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.ln_1.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.ln_1.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.ln_2.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.ln_2.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.20.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.ln_1.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.ln_1.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.ln_2.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.ln_2.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
"transformer.h.21.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.21.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.ln_1.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.ln_1.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.ln_2.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.ln_2.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.22.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.ln_1.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.ln_1.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.ln_2.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.ln_2.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.23.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.ln_1.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.ln_1.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.ln_2.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.ln_2.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.24.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.ln_1.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.ln_1.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.ln_2.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.ln_2.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.25.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.ln_1.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.ln_1.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.ln_2.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.ln_2.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.26.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.ln_1.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.ln_1.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.ln_2.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.ln_2.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.27.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.ln_1.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.ln_1.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.ln_2.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.ln_2.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.28.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.29.attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.29.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.29.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.29.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.29.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.29.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.29.ln_1.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.29.ln_1.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.29.ln_2.bias": "pytorch_model-00004-of-00006.bin",
"transformer.h.29.ln_2.weight": "pytorch_model-00004-of-00006.bin",
"transformer.h.29.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.29.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.29.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.29.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.3.attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.ln_1.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.ln_1.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.ln_2.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.ln_2.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.3.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.30.attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.ln_1.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.ln_1.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.ln_2.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.ln_2.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.30.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.ln_1.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.ln_1.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.ln_2.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.ln_2.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.31.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.ln_1.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.ln_1.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.ln_2.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.ln_2.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.32.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.ln_1.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.ln_1.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.ln_2.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.ln_2.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.33.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.ln_1.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.ln_1.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.ln_2.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.ln_2.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.34.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.ln_1.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.ln_1.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.ln_2.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.ln_2.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.35.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.ln_1.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.ln_1.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.ln_2.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.ln_2.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.36.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.37.attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.37.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.37.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.37.attn.c_proj.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.37.attn.c_proj.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.37.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.37.ln_1.bias": "pytorch_model-00005-of-00006.bin",
"transformer.h.37.ln_1.weight": "pytorch_model-00005-of-00006.bin",
"transformer.h.37.ln_2.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.37.ln_2.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.37.mlp.c_fc.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.37.mlp.c_fc.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.37.mlp.c_proj.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.37.mlp.c_proj.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.attn.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.attn.c_attn.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.attn.c_attn.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.attn.c_proj.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.attn.c_proj.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.attn.masked_bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.ln_1.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.ln_1.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.ln_2.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.ln_2.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.mlp.c_fc.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.mlp.c_fc.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.mlp.c_proj.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.38.mlp.c_proj.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.attn.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.attn.c_attn.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.attn.c_attn.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.attn.c_proj.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.attn.c_proj.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.attn.masked_bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.ln_1.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.ln_1.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.ln_2.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.ln_2.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.mlp.c_fc.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.mlp.c_fc.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.mlp.c_proj.bias": "pytorch_model-00006-of-00006.bin",
"transformer.h.39.mlp.c_proj.weight": "pytorch_model-00006-of-00006.bin",
"transformer.h.4.attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.ln_1.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.ln_1.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.ln_2.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.ln_2.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.4.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.ln_1.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.ln_1.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.ln_2.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.ln_2.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.5.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.6.attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.6.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.6.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.6.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.6.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.6.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.6.ln_1.bias": "pytorch_model-00001-of-00006.bin",
"transformer.h.6.ln_1.weight": "pytorch_model-00001-of-00006.bin",
"transformer.h.6.ln_2.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.6.ln_2.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.6.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.6.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.6.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.6.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.ln_1.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.ln_1.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.ln_2.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.ln_2.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.7.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.ln_1.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.ln_1.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.ln_2.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.ln_2.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.8.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.ln_1.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.ln_1.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.ln_2.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.ln_2.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
"transformer.h.9.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
"transformer.ln_f.bias": "pytorch_model-00006-of-00006.bin",
"transformer.ln_f.weight": "pytorch_model-00006-of-00006.bin",
"transformer.wpe.weight": "pytorch_model-00001-of-00006.bin",
"transformer.wte.weight": "pytorch_model-00001-of-00006.bin"
}
}

32
special_tokens_map.json Normal file
View File

@@ -0,0 +1,32 @@
{
"bos_token": {
"content": "<s>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"mask_token": "<mask>",
"pad_token": {
"content": "<pad>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"sep_token": "</s>",
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
}
}

199829
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

68
tokenizer_config.json Normal file
View File

@@ -0,0 +1,68 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"0": {
"content": "<s>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<pad>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"3": {
"content": "<unk>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false,
"special": true
},
"4": {
"content": "<mask>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"5": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false,
"special": true
}
},
"bos_token": "<s>",
"clean_up_tokenization_spaces": true,
"eos_token": "<|endoftext|>",
"errors": "replace",
"mask_token": "<mask>",
"model_max_length": 2048,
"name_or_path": "ai-forever/mGPT-13B",
"pad_token": "<pad>",
"padding_side": "left",
"sep_token": "</s>",
"tokenizer_class": "GPT2Tokenizer",
"truncation_side": "left",
"trust_remote_code": false,
"unk_token": "<unk>"
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long