初始化项目，由ModelHub XC社区提供模型

Model: HuggingFaceH4/starchat-beta
Source: Original Platform

.gitattributes

@@ -0,0 +1,43 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+model_logo.png filter=lfs diff=lfs merge=lfs -text
+model-00001-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00002-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00003-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00004-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
+pytorch_model-00001-of-00004.bin filter=lfs diff=lfs merge=lfs -text
+pytorch_model-00002-of-00004.bin filter=lfs diff=lfs merge=lfs -text
+pytorch_model-00003-of-00004.bin filter=lfs diff=lfs merge=lfs -text
+pytorch_model-00004-of-00004.bin filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,130 @@
+---
+tags:
+- generated_from_trainer
+widget:
+- text: "How can I write a Python function to generate the nth Fibonacci number?"
+- text: "How do I get the current date using shell commands? Explain how it works."
+model-index:
+- name: starchat-beta
+  results: []
+license: bigcode-openrail-m
+---
+
+
+<img src="https://huggingface.co/HuggingFaceH4/starchat-beta/resolve/main/model_logo.png" alt="StarChat Beta Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
+
+# Model Card for StarChat-β
+
+StarChat is a series of language models that are trained to act as helpful coding assistants. StarChat-β is the second model in the series, and is a fine-tuned version of [StarCoderPlus](https://huggingface.co/bigcode/starcoderplus) that was trained on an ["uncensored"](https://erichartford.com/uncensored-models) variant of the [`openassistant-guanaco` dataset](https://huggingface.co/datasets/timdettmers/openassistant-guanaco). We found that removing the in-built alignment of the OpenAssistant dataset boosted performance on the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) and made the model more helpful at coding tasks. However, this means that model is likely to generate problematic text when prompted to do so and should only be used for educational and research purposes.
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+- **Model type:** A 16B parameter GPT-like model fine-tuned on an ["uncensored"](https://erichartford.com/uncensored-models) variant of the [`openassistant-guanaco` dataset](https://huggingface.co/datasets/timdettmers/openassistant-guanaco).
+- **Language(s) (NLP):** Primarily English and 80+ programming languages.
+- **License:** BigCode Open RAIL-M v1
+- **Finetuned from model:** [bigcode/starcoderplus](https://huggingface.co/bigcode/starcoderplus)
+
+### Model Sources
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** https://github.com/bigcode-project/starcoder
+- **Demo:** https://huggingface.co/spaces/HuggingFaceH4/starchat-playground
+
+
+## Intended uses & limitations
+
+The model was fine-tuned on a variant of the [`OpenAssistant/oasst1`](https://huggingface.co/datasets/OpenAssistant/oasst1) dataset, which contains a diverse range of dialogues in over 35 languages. As a result, the model can be used for chat and you can check out our [demo](https://huggingface.co/spaces/HuggingFaceH4/starchat-playground) to test its coding capabilities. 
+
+Here's how you can run the model using the `pipeline()` function from 🤗 Transformers:
+
+```python
+import torch
+from transformers import pipeline
+
+pipe = pipeline("text-generation", model="HuggingFaceH4/starchat-beta", torch_dtype=torch.bfloat16, device_map="auto")
+
+# We use a variant of ChatML to format each message
+prompt_template = "<|system|>\n<|end|>\n<|user|>\n{query}<|end|>\n<|assistant|>"
+prompt = prompt_template.format(query="How do I sort a list in Python?")
+# We use a special <|end|> token with ID 49155 to denote ends of a turn
+outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.2, top_k=50, top_p=0.95, eos_token_id=49155)
+# You can sort a list in Python by using the sort() method. Here's an example:\n\n```\nnumbers = [3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5]\nnumbers.sort()\nprint(numbers)\n```\n\nThis will sort the list in place and print the sorted list.
+```
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+StarChat-β has not been aligned to human preferences with techniques like RLHF or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so). 
+Models trained primarily on code data will also have a more skewed demographic bias commensurate with the demographics of the GitHub community, for more on this see the [StarCoder dataset](https://huggingface.co/datasets/bigcode/starcoderdata) which is derived from The Stack.
+
+Since the base model was pretrained on a large corpus of code, it may produce code snippets that are syntactically valid but semantically incorrect. 
+For example, it may produce code that does not compile or that produces incorrect results.  
+It may also produce code that is vulnerable to security exploits.  
+We have observed the model also has a tendency to produce false URLs which should be carefully inspected before clicking.
+
+StarChat-β was fine-tuned from the base model [StarCoderPlus](https://huggingface.co/bigcode/starcoderplus), please refer to its model card's [Limitations Section](https://huggingface.co/bigcode/starcoderplus#limitations) for relevant information. 
+In particular, the model was evaluated on some categories of gender biases, propensity for toxicity, and risk of suggesting code completions with known security flaws; these evaluations are reported in its [technical report](https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0kl4BYs/view).
+
+## Training and evaluation data
+
+StarChat-β is trained on an ["uncensored"](https://erichartford.com/uncensored-models) variant of the [`openassistant-guanaco` dataset](https://huggingface.co/datasets/timdettmers/openassistant-guanaco). We applied the same [recipe](https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered/blob/main/wizardlm_clean.py) used to filter the ShareGPT datasets behind the [WizardLM](https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered).
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 2e-05
+- train_batch_size: 4
+- eval_batch_size: 4
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 8
+- gradient_accumulation_steps: 8
+- total_train_batch_size: 256
+- total_eval_batch_size: 32
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.03
+- num_epochs: 6
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 1.5321        | 0.98  | 15   | 1.2856          |
+| 1.2071        | 1.97  | 30   | 1.2620          |
+| 1.0162        | 2.95  | 45   | 1.2853          |
+| 0.8484        | 4.0   | 61   | 1.3274          |
+| 0.6981        | 4.98  | 76   | 1.3994          |
+| 0.5668        | 5.9   | 90   | 1.4720          |
+
+
+### Framework versions
+
+- Transformers 4.28.1
+- Pytorch 2.0.1+cu118
+- Datasets 2.12.0
+- Tokenizers 0.13.3
+
+## Citation
+
+Although there isn't a blog post or paper associated with StarChat-β, you can find details on the earlier version in the blog post below:
+
+**BibTeX:**
+
+```
+@article{Tunstall2023starchat-alpha,
+  author = {Tunstall, Lewis and Lambert, Nathan and Rajani, Nazneen and Beeching, Edward and Le Scao, Teven and von Werra, Leandro and Han, Sheon and Schmid, Philipp and Rush, Alexander},
+  title = {Creating a Coding Assistant with StarCoder},
+  journal = {Hugging Face Blog},
+  year = {2023},
+  note = {https://huggingface.co/blog/starchat},
+}
+```

added_tokens.json

@@ -0,0 +1,6 @@
+{
+  "<|assistant|>": 49154,
+  "<|end|>": 49155,
+  "<|system|>": 49152,
+  "<|user|>": 49153
+}

all_results.json

@@ -0,0 +1,14 @@
+{
+    "epoch": 5.9,
+    "eval_loss": 1.4719988107681274,
+    "eval_runtime": 5.2039,
+    "eval_samples": 202,
+    "eval_samples_per_second": 38.817,
+    "eval_steps_per_second": 1.345,
+    "perplexity": 4.35793713301621,
+    "train_loss": 0.9728257921006944,
+    "train_runtime": 2307.7787,
+    "train_samples": 3888,
+    "train_samples_per_second": 10.108,
+    "train_steps_per_second": 0.039
+}

config.json

@@ -0,0 +1,39 @@
+{
+  "_name_or_path": "data/starcoderplus-ift-v4.1",
+  "activation_function": "gelu",
+  "architectures": [
+    "GPTBigCodeForCausalLM"
+  ],
+  "attention_softmax_in_fp32": true,
+  "attn_pdrop": 0.1,
+  "bos_token_id": 0,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 0,
+  "inference_runner": 0,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "max_batch_size": null,
+  "max_sequence_length": null,
+  "model_type": "gpt_bigcode",
+  "multi_query": true,
+  "n_embd": 6144,
+  "n_head": 48,
+  "n_inner": 24576,
+  "n_layer": 40,
+  "n_positions": 8192,
+  "pad_key_length": true,
+  "pre_allocate_kv_cache": false,
+  "resid_pdrop": 0.1,
+  "scale_attention_softmax_in_fp32": true,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.28.1",
+  "use_cache": true,
+  "validate_runner_input": true,
+  "vocab_size": 49156
+}

configuration.json

@@ -0,0 +1 @@
+{"framework": "pytorch", "task": "text-generation", "allow_remote": true}

dialogue_template.json

@@ -0,0 +1,12 @@
+{
+  "system": "",
+  "messages": null,
+  "system_token": "<|system|>",
+  "system_format": "standard",
+  "user_token": "<|user|>",
+  "assistant_token": "<|assistant|>",
+  "end_token": "<|end|>",
+  "mid_str": "\n",
+  "end_str": "\n",
+  "extra_end_text": ""
+}

eval_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 5.9,
+    "eval_loss": 1.4719988107681274,
+    "eval_runtime": 5.2039,
+    "eval_samples": 202,
+    "eval_samples_per_second": 38.817,
+    "eval_steps_per_second": 1.345,
+    "perplexity": 4.35793713301621
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "4.28.1"
+}

handler.py

@@ -0,0 +1,49 @@
+from typing import Any, Dict
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+from peft import PeftConfig, PeftModel
+
+
+class EndpointHandler:
+    def __init__(self, path=""):
+        # load model and processor from path
+        self.tokenizer = AutoTokenizer.from_pretrained(path)
+        try:
+            config = PeftConfig.from_pretrained(path)
+            model = AutoModelForCausalLM.from_pretrained(
+                config.base_model_name_or_path,
+                return_dict=True,
+                load_in_8bit=True,
+                device_map="auto",
+                torch_dtype=torch.float16,
+                trust_remote_code=True,
+            )
+            model.resize_token_embeddings(len(self.tokenizer))
+            model = PeftModel.from_pretrained(model, path)
+        except Exception:
+            model = AutoModelForCausalLM.from_pretrained(
+                path, device_map="auto", load_in_8bit=True, torch_dtype=torch.float16, trust_remote_code=True
+            )
+        self.model = model
+        self.device = "cuda" if torch.cuda.is_available() else "cpu"
+
+    def __call__(self, data: Dict[str, Any]) -> Dict[str, str]:
+        # process input
+        inputs = data.pop("inputs", data)
+        parameters = data.pop("parameters", None)
+
+        # preprocess
+        inputs = self.tokenizer(inputs, return_tensors="pt").to(self.device)
+
+        # pass inputs with all kwargs in data
+        if parameters is not None:
+            outputs = self.model.generate(**inputs, **parameters)
+        else:
+            outputs = self.model.generate(**inputs)
+
+        # postprocess the prediction
+        prediction = self.tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+        return [{"generated_text": prediction}]

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:133a07e35c2bcadb712b843032aafb333028d3c81c6a15ddb524e95a611478ce
+size 9957996976

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c48af57d8b0de97468d2acdb391c5075ba52de8484fa57a239724115bcf63bab
+size 9857346344

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:080aa7f5af2b23cada02810726c4003242b8eac1d0ced0676ab44e818dee5d40
+size 9857346344

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b8ceee3f55cb5ab715ef4fbe35bf9178fff50f6cf51cad65a65b4dfcb797c41
+size 1362323832

model.safetensors.index.json

@@ -0,0 +1,491 @@
+{
+  "metadata": {
+    "total_size": 31034961920
+  },
+  "weight_map": {
+    "transformer.h.0.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.0.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.0.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.0.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.0.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.0.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.12.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.12.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.12.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.12.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.12.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.12.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.12.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.12.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.12.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.12.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.12.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.12.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.2.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.2.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.2.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.2.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.2.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.2.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.20.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.20.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.20.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.20.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.20.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.20.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.25.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.25.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.25.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.25.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.25.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.25.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.25.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.25.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.25.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.25.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.25.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.25.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.3.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.3.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.3.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.3.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.3.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.3.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.30.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.30.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.30.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.30.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.30.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.30.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.38.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.38.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.38.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.38.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.38.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.38.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.38.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.38.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.38.mlp.c_fc.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.38.mlp.c_fc.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.38.mlp.c_proj.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.38.mlp.c_proj.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.attn.c_attn.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.attn.c_attn.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.attn.c_proj.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.attn.c_proj.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.ln_1.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.ln_1.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.ln_2.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.ln_2.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.mlp.c_fc.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.mlp.c_fc.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.mlp.c_proj.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.mlp.c_proj.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.4.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.4.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.4.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.4.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.4.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.4.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.ln_f.bias": "model-00004-of-00004.safetensors",
+    "transformer.ln_f.weight": "model-00004-of-00004.safetensors",
+    "transformer.wpe.weight": "model-00001-of-00004.safetensors",
+    "transformer.wte.weight": "model-00001-of-00004.safetensors"
+  }
+}

model_logo.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3f153623ec40d4fd1af040391b375b6eac7dad238820bb05ba08fe87a492e32
+size 1196247

pytorch_model-00001-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a89629dc523d03530fded1baeb98a27df40876a39a432154324a1b8547e7e9f1
+size 9958034229

pytorch_model-00002-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a74b574439dcd841d6434196cbf227d34a580a6d190cfc8dc20f6b4c9e2d653c
+size 9857383847

pytorch_model-00003-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79d2764fedd8a64518187295c810eea9743ff7b70aaab915fefaa576c12aeaa6
+size 9857383847

pytorch_model-00004-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:322b505787c3d18c5ac9b9fca91f913dcf6e5b5f4c449f1115cc378bfc02aef9
+size 1966357413

pytorch_model.bin.index.json

@@ -0,0 +1,492 @@
+{
+  "metadata": {
+    "total_size": 31638990848
+  },
+  "weight_map": {
+    "lm_head.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.0.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.12.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.12.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.12.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.2.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.20.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.25.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.25.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.25.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.3.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.30.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.mlp.c_fc.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.38.mlp.c_fc.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.38.mlp.c_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.38.mlp.c_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.attn.c_attn.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.attn.c_attn.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.attn.c_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.attn.c_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.ln_1.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.ln_1.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.ln_2.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.ln_2.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.mlp.c_fc.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.mlp.c_fc.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.mlp.c_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.mlp.c_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.4.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.ln_f.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.ln_f.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.wpe.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.wte.weight": "pytorch_model-00001-of-00004.bin"
+  }
+}

requirements.txt

@@ -0,0 +1,5 @@
+transformers==4.28.1
+accelerate>=0.16.0
+bitsandbytes
+sentencepiece
+git+https://github.com/huggingface/peft.git@632997d1fb776c3cf05d8c2537ac9a98a7ce9435

runs/Jun03_19-41-09_ip-26-0-150-31/1685821571.9023504/events.out.tfevents.1685821571.ip-26-0-150-31.3019211.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb69a98243f145661744cf9b0f841612871d0dd8353921269d781900c5b44966
+size 6340

runs/Jun03_19-41-09_ip-26-0-150-31/events.out.tfevents.1685821571.ip-26-0-150-31.3019211.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b7b05d4d5bd3cb96e284db3c0efa120ba2e0c19c883a78412afa043418b84fcc
+size 8463

runs/Jun03_19-41-09_ip-26-0-150-31/events.out.tfevents.1685823884.ip-26-0-150-31.3019211.2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4a5c376bbecf308e9a64d9242aa95641f34ca58e827ce94004246a209e3bd37
+size 354

special_tokens_map.json

@@ -0,0 +1,11 @@
+{
+  "additional_special_tokens": [
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|end|>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,31 @@
+{
+  "add_prefix_space": false,
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

train_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 5.9,
+    "train_loss": 0.9728257921006944,
+    "train_runtime": 2307.7787,
+    "train_samples": 3888,
+    "train_samples_per_second": 10.108,
+    "train_steps_per_second": 0.039
+}

trainer_state.json

@@ -0,0 +1,145 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 5.901639344262295,
+  "global_step": 90,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.07,
+      "learning_rate": 0.0,
+      "loss": 1.8309,
+      "step": 1
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 2e-05,
+      "loss": 1.5321,
+      "step": 8
+    },
+    {
+      "epoch": 0.98,
+      "eval_loss": 1.2855817079544067,
+      "eval_runtime": 6.6367,
+      "eval_samples_per_second": 30.437,
+      "eval_steps_per_second": 1.055,
+      "step": 15
+    },
+    {
+      "epoch": 1.05,
+      "learning_rate": 2e-05,
+      "loss": 1.35,
+      "step": 16
+    },
+    {
+      "epoch": 1.57,
+      "learning_rate": 2e-05,
+      "loss": 1.2071,
+      "step": 24
+    },
+    {
+      "epoch": 1.97,
+      "eval_loss": 1.2619894742965698,
+      "eval_runtime": 5.1554,
+      "eval_samples_per_second": 39.183,
+      "eval_steps_per_second": 1.358,
+      "step": 30
+    },
+    {
+      "epoch": 2.1,
+      "learning_rate": 2e-05,
+      "loss": 1.1502,
+      "step": 32
+    },
+    {
+      "epoch": 2.62,
+      "learning_rate": 2e-05,
+      "loss": 1.0162,
+      "step": 40
+    },
+    {
+      "epoch": 2.95,
+      "eval_loss": 1.285272240638733,
+      "eval_runtime": 5.1992,
+      "eval_samples_per_second": 38.852,
+      "eval_steps_per_second": 1.346,
+      "step": 45
+    },
+    {
+      "epoch": 3.15,
+      "learning_rate": 2e-05,
+      "loss": 0.9511,
+      "step": 48
+    },
+    {
+      "epoch": 3.67,
+      "learning_rate": 2e-05,
+      "loss": 0.8484,
+      "step": 56
+    },
+    {
+      "epoch": 4.0,
+      "eval_loss": 1.3274288177490234,
+      "eval_runtime": 5.1899,
+      "eval_samples_per_second": 38.922,
+      "eval_steps_per_second": 1.349,
+      "step": 61
+    },
+    {
+      "epoch": 4.2,
+      "learning_rate": 2e-05,
+      "loss": 0.7971,
+      "step": 64
+    },
+    {
+      "epoch": 4.72,
+      "learning_rate": 2e-05,
+      "loss": 0.6981,
+      "step": 72
+    },
+    {
+      "epoch": 4.98,
+      "eval_loss": 1.3993656635284424,
+      "eval_runtime": 5.213,
+      "eval_samples_per_second": 38.749,
+      "eval_steps_per_second": 1.343,
+      "step": 76
+    },
+    {
+      "epoch": 5.25,
+      "learning_rate": 2e-05,
+      "loss": 0.6462,
+      "step": 80
+    },
+    {
+      "epoch": 5.77,
+      "learning_rate": 2e-05,
+      "loss": 0.5668,
+      "step": 88
+    },
+    {
+      "epoch": 5.9,
+      "eval_loss": 1.4719988107681274,
+      "eval_runtime": 5.1996,
+      "eval_samples_per_second": 38.849,
+      "eval_steps_per_second": 1.346,
+      "step": 90
+    },
+    {
+      "epoch": 5.9,
+      "step": 90,
+      "total_flos": 383994839433216.0,
+      "train_loss": 0.9728257921006944,
+      "train_runtime": 2307.7787,
+      "train_samples_per_second": 10.108,
+      "train_steps_per_second": 0.039
+    }
+  ],
+  "max_steps": 90,
+  "num_train_epochs": 6,
+  "total_flos": 383994839433216.0,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d758c20cdca5c51a4915cd92b74a547ca1b5077aaf48d7b685529d240356f725
+size 4987