初始化项目，由ModelHub XC社区提供模型

Model: ai-forever/mGPT-13B
Source: Original Platform

.gitattributes

@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,112 @@
+---
+license: mit
+language:
+- ar
+- he
+- vi
+- id
+- jv
+- ms
+- tl
+- lv
+- lt
+- eu
+- ml
+- ta
+- te
+- hy
+- bn
+- mr
+- hi
+- ur
+- af
+- da
+- en
+- de
+- sv
+- fr
+- it
+- pt
+- ro
+- es
+- el
+- os
+- tg
+- fa
+- ja
+- ka
+- ko
+- th
+- bxr
+- xal
+- mn
+- sw
+- yo
+- be
+- bg
+- ru
+- uk
+- pl
+- my
+- uz
+- ba
+- kk
+- ky
+- tt
+- az
+- cv
+- tr
+- tk
+- tyv
+- sax
+- et
+- fi
+- hu
+tags:
+- multilingual
+- PyTorch
+- Transformers
+- gpt3
+- gpt2
+- transformers
+---
+# 🌻 mGPT 13B
+
+Multilingual language model. This model was trained on the **61** languages from **25** language families (see the list below).
+
+## Dataset
+
+Model was pretrained on a 600Gb of texts, mostly from MC4 and Wikipedia. Training data was deduplicated, the text deduplication includes 64-bit hashing of each text in the corpus for keeping texts with a unique hash. We also filter the documents based on their text compression rate using zlib4. The most strongly and weakly compressing deduplicated texts are discarded.
+
+Here is the table with number of tokens for each language in the pretraining corpus on a logarithmic scale:
+
+![](https://i.imgur.com/KSMfVX1.png)
+
+## Languages
+
+Afrikaans (af), Arabic (ar), Armenian (hy), Azerbaijani (az), Basque (eu), Bashkir (ba), Belarusian (be), Bengali (bn), Bulgarian (bg), Burmese (my), Buryat (bxr), Chuvash (cv), Danish (da), English (en), Estonian (et), Finnish (fi), French (fr), Georgian (ka), German (de), Greek (el), Hebrew (he), Hindi (hi), Hungarian (hu), Indonesian (id), Italian (it), Japanese (ja), Javanese (jv), Kalmyk (xal), Kazakh (kk), Korean (ko), Kyrgyz (ky), Latvian (lv), Lithuanian (lt), Malay (ms), Malayalam (ml), Marathi (mr), Mongolian (mn), Ossetian (os), Persian (fa), Polish (pl), Portuguese (pt), Romanian (ro), Russian (ru), Spanish (es), Swedish (sv), Swahili (sw), Tatar (tt), Telugu (te), Thai (th), Turkish (tr), Turkmen (tk), Tuvan (tyv), Ukrainian (uk), Uzbek (uz), Vietnamese (vi), Yakut (sax), Yoruba (yo)
+
+#### By language family
+
+<table><thead><tr><th>Language Family</th><th>Languages</th></tr></thead><tbody><tr><td>Afro-Asiatic</td><td>Arabic (ar), Hebrew (he)</td></tr><tr><td>Austro-Asiatic</td><td>Vietnamese (vi)</td></tr><tr><td>Austronesian</td><td>Indonesian (id), Javanese (jv), Malay (ms), Tagalog (tl)</td></tr><tr><td>Baltic</td><td>Latvian (lv), Lithuanian (lt)</td></tr><tr><td>Basque</td><td>Basque (eu)</td></tr><tr><td>Dravidian</td><td>Malayalam (ml), Tamil (ta), Telugu (te)</td></tr><tr><td>Indo-European (Armenian)</td><td>Armenian (hy)</td></tr><tr><td>Indo-European (Indo-Aryan)</td><td>Bengali (bn), Marathi (mr), Hindi (hi), Urdu (ur)</td></tr><tr><td>Indo-European (Germanic)</td><td>Afrikaans (af), Danish (da), English (en), German (de), Swedish (sv)</td></tr><tr><td>Indo-European (Romance)</td><td>French (fr), Italian (it), Portuguese (pt), Romanian (ro), Spanish (es)</td></tr><tr><td>Indo-European (Greek)</td><td>Greek (el)</td></tr><tr><td>Indo-European (Iranian)</td><td>Ossetian (os), Tajik (tg), Persian (fa)</td></tr><tr><td>Japonic</td><td>Japanese (ja)</td></tr><tr><td>Kartvelian</td><td>Georgian (ka)</td></tr><tr><td>Koreanic</td><td>Korean (ko)</td></tr><tr><td>Kra-Dai</td><td>Thai (th)</td></tr><tr><td>Mongolic</td><td>Buryat (bxr), Kalmyk (xal), Mongolian (mn)</td></tr><tr><td>Niger-Congo</td><td>Swahili (sw), Yoruba (yo)</td></tr><tr><td>Slavic</td><td>Belarusian (be), Bulgarian (bg), Russian (ru), Ukrainian (uk), Polish (pl)</td></tr><tr><td>Sino-Tibetan</td><td>Burmese (my)</td></tr><tr><td>Turkic (Karluk)</td><td>Uzbek (uz)</td></tr><tr><td>Turkic (Kipchak)</td><td>Bashkir (ba), Kazakh (kk), Kyrgyz (ky), Tatar (tt)</td></tr><tr><td>Turkic (Oghuz)</td><td>Azerbaijani (az), Chuvash (cv), Turkish (tr), Turkmen (tk)</td></tr><tr><td>Turkic (Siberian)</td><td>Tuvan (tyv), Yakut (sax)</td></tr><tr><td>Uralic</td><td>Estonian (et), Finnish (fi), Hungarian (hu)</td></tr></tbody></table>
+
+## Technical details
+
+The models are pretrained on 16 V100 GPUs for 600k training steps with a set of fixed hyperparameters: vocabulary size of 100k, context window of 2048, learning rate of 2e−4, and batch size of 4.
+
+The mGPT architecture is based on GPT-3. We use the architecture description by Brown et al., the code base on GPT-2 (Radford et al., 2019) in the HuggingFace library (Wolf et al., 2020) and Megatron-LM (Shoeybi et al., 2019).
+
+## Perplexity
+
+The mGPT13B model achieves the best perplexities within the 2-to-10 score range for the majority of languages, including Dravidian (Malayalam, Tamil, Telugu), Indo-Aryan (Bengali, Hindi, Marathi), Slavic (Belarusian, Ukrainian, Russian, Bulgarian), Sino-Tibetan (Burmese), Kipchak (Bashkir, Kazakh) and others. Higher perplexities up to 20 are for only seven languages from different families.
+
+#### Language-wise perplexity results
+
+![](https://i.imgur.com/aIKEpPE.png)
+
+#### Family-wise perplexity results
+
+![](https://i.imgur.com/1ugWbXc.png)
+
+_The scores are averaged over the number of languages within each family._
+

config.json

@@ -0,0 +1,35 @@
+{
+  "_name_or_path": "ai-forever/mGPT-13B",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 0,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 5,
+  "gradient_checkpointing": false,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 2048,
+  "n_embd": 5120,
+  "n_head": 40,
+  "n_inner": null,
+  "n_layer": 40,
+  "n_positions": 2048,
+  "pad_token_id": 1,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.24.0",
+  "use_cache": true,
+  "vocab_size": 100000
+}

pytorch_model-00001-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59c46524428e95456e13589088c1604eb69cbdda0d8e595fa7f85490e8ac757c
+size 9985353615

pytorch_model-00002-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a39317def54cf7f79c04e32027178f1fd79211a6be8e5087b62dcad7c6068811
+size 9787416273

pytorch_model-00003-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b334b58f6d6e8d0bfdf4bc49e3376ea0d07412543e4aec8e3443f557166e83f
+size 9678363005

pytorch_model-00004-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4be422d207f50cb9e3f6b8baaac440a095e70cffc0c95dbea93faef1c54f69d5
+size 9682538185

pytorch_model-00005-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0913ad393e9098da5c6f594293c9f0fd3e13e286d92c4bf934d04d0d7ab6f53e
+size 9997130859

pytorch_model-00006-of-00006.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d2effd2b75322049ed226355bf2032d03e24959a616228c5ced32d71da42a05
+size 5517439401

pytorch_model.bin.index.json

@@ -0,0 +1,572 @@
+{
+  "metadata": {
+    "total_size": 54648053920
+  },
+  "weight_map": {
+    "lm_head.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.0.attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.ln_1.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.ln_1.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.ln_2.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.ln_2.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.0.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.ln_1.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.ln_1.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.ln_2.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.ln_2.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.1.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.10.attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.ln_1.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.ln_1.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.ln_2.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.ln_2.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.10.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.ln_1.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.ln_1.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.ln_2.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.ln_2.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.11.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.ln_1.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.ln_1.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.ln_2.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.ln_2.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.12.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.ln_1.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.ln_1.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.ln_2.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.ln_2.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.13.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.14.attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.14.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.14.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.14.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.14.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.14.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.14.ln_1.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.14.ln_1.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.14.ln_2.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.14.ln_2.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.14.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.14.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.14.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.14.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.ln_1.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.ln_1.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.ln_2.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.ln_2.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.15.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.ln_1.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.ln_1.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.ln_2.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.ln_2.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.16.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.ln_1.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.ln_1.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.ln_2.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.ln_2.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.17.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.ln_1.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.ln_1.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.ln_2.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.ln_2.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.18.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.ln_1.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.ln_1.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.ln_2.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.ln_2.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.19.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.2.attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.ln_1.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.ln_1.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.ln_2.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.ln_2.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.2.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.20.attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.ln_1.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.ln_1.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.ln_2.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.ln_2.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.mlp.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.20.mlp.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.attn.c_attn.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.attn.c_attn.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.attn.c_proj.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.attn.c_proj.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.attn.masked_bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.ln_1.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.ln_1.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.ln_2.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.ln_2.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.mlp.c_fc.bias": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.mlp.c_fc.weight": "pytorch_model-00003-of-00006.bin",
+    "transformer.h.21.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.21.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.ln_1.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.ln_1.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.ln_2.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.ln_2.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.22.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.ln_1.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.ln_1.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.ln_2.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.ln_2.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.23.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.ln_1.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.ln_1.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.ln_2.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.ln_2.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.24.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.ln_1.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.ln_1.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.ln_2.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.ln_2.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.25.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.ln_1.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.ln_1.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.ln_2.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.ln_2.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.26.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.ln_1.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.ln_1.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.ln_2.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.ln_2.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.27.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.ln_1.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.ln_1.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.ln_2.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.ln_2.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.mlp.c_fc.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.mlp.c_fc.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.mlp.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.28.mlp.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.29.attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.29.attn.c_attn.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.29.attn.c_attn.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.29.attn.c_proj.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.29.attn.c_proj.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.29.attn.masked_bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.29.ln_1.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.29.ln_1.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.29.ln_2.bias": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.29.ln_2.weight": "pytorch_model-00004-of-00006.bin",
+    "transformer.h.29.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.29.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.29.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.29.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.3.attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.ln_1.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.ln_1.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.ln_2.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.ln_2.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.3.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.30.attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.ln_1.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.ln_1.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.ln_2.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.ln_2.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.30.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.ln_1.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.ln_1.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.ln_2.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.ln_2.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.31.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.ln_1.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.ln_1.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.ln_2.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.ln_2.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.32.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.ln_1.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.ln_1.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.ln_2.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.ln_2.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.33.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.ln_1.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.ln_1.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.ln_2.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.ln_2.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.34.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.ln_1.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.ln_1.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.ln_2.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.ln_2.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.35.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.attn.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.attn.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.ln_1.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.ln_1.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.ln_2.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.ln_2.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.mlp.c_fc.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.mlp.c_fc.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.mlp.c_proj.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.36.mlp.c_proj.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.37.attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.37.attn.c_attn.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.37.attn.c_attn.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.37.attn.c_proj.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.37.attn.c_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.37.attn.masked_bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.37.ln_1.bias": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.37.ln_1.weight": "pytorch_model-00005-of-00006.bin",
+    "transformer.h.37.ln_2.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.37.ln_2.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.37.mlp.c_fc.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.37.mlp.c_fc.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.37.mlp.c_proj.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.37.mlp.c_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.attn.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.attn.c_attn.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.attn.c_attn.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.attn.c_proj.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.attn.c_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.attn.masked_bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.ln_1.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.ln_1.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.ln_2.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.ln_2.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.mlp.c_fc.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.mlp.c_fc.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.mlp.c_proj.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.38.mlp.c_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.attn.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.attn.c_attn.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.attn.c_attn.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.attn.c_proj.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.attn.c_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.attn.masked_bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.ln_1.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.ln_1.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.ln_2.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.ln_2.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.mlp.c_fc.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.mlp.c_fc.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.mlp.c_proj.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.39.mlp.c_proj.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.h.4.attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.ln_1.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.ln_1.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.ln_2.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.ln_2.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.4.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.attn.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.attn.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.ln_1.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.ln_1.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.ln_2.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.ln_2.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.mlp.c_fc.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.mlp.c_fc.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.mlp.c_proj.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.5.mlp.c_proj.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.6.attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.6.attn.c_attn.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.6.attn.c_attn.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.6.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.6.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.6.attn.masked_bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.6.ln_1.bias": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.6.ln_1.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.h.6.ln_2.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.6.ln_2.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.6.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.6.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.6.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.6.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.ln_1.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.ln_1.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.ln_2.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.ln_2.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.7.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.ln_1.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.ln_1.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.ln_2.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.ln_2.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.8.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.attn.c_attn.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.attn.c_attn.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.attn.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.attn.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.attn.masked_bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.ln_1.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.ln_1.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.ln_2.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.ln_2.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.mlp.c_fc.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.mlp.c_fc.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.mlp.c_proj.bias": "pytorch_model-00002-of-00006.bin",
+    "transformer.h.9.mlp.c_proj.weight": "pytorch_model-00002-of-00006.bin",
+    "transformer.ln_f.bias": "pytorch_model-00006-of-00006.bin",
+    "transformer.ln_f.weight": "pytorch_model-00006-of-00006.bin",
+    "transformer.wpe.weight": "pytorch_model-00001-of-00006.bin",
+    "transformer.wte.weight": "pytorch_model-00001-of-00006.bin"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,32 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": "<mask>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,68 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "mask_token": "<mask>",
+  "model_max_length": 2048,
+  "name_or_path": "ai-forever/mGPT-13B",
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "sep_token": "</s>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "truncation_side": "left",
+  "trust_remote_code": false,
+  "unk_token": "<unk>"
+}