初始化项目，由ModelHub XC社区提供模型
Model: ibm-granite/granite-4.0-micro-base Source: Original Platform
This commit is contained in:
ModelHub XC
52
.gitattributes
vendored
Normal file
52
.gitattributes
vendored
Normal file
@@ -0,0 +1,52 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zstandard filter=lfs diff=lfs merge=lfs -text
|
||||
*.tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
*.db* filter=lfs diff=lfs merge=lfs -text
|
||||
*.ark* filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
|
||||
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.gguf* filter=lfs diff=lfs merge=lfs -text
|
||||
*.ggml filter=lfs diff=lfs merge=lfs -text
|
||||
*.llamafile* filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
|
||||
vocab.json filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
model-00001-of-00002.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
model-00002-of-00002.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
448
README.md
Normal file
448
README.md
Normal file
@@ -0,0 +1,448 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
library_name: transformers
|
||||
tags:
|
||||
- language
|
||||
- granite-4.0
|
||||
---
|
||||
|
||||
[](https://mot.isitopen.ai/model/1163)
|
||||
|
||||
# Granite-4.0-Micro-Base
|
||||
|
||||
**Model Summary:**
|
||||
Granite-4.0-Micro-Base is a decoder-only, long-context language model designed for a wide range of text-to-text generation tasks. It also supports Fill-in-the-Middle (FIM) code completion through the use of specialized prefix and suffix tokens. The model is trained from scratch on approximately 15 trillion tokens following a four-stage training strategy: 10 trillion tokens in the first stage, 2 trillion in the second, another 2 trillion in the third, and 0.5 trillion in the final stage.
|
||||
|
||||
- **Developers:** Granite Team, IBM
|
||||
- **HF Collection:** [Granite 4.0 Language Models HF Collection](https://huggingface.co/collections/ibm-granite/granite-40-language-models-6811a18b820ef362d9e5a82c)
|
||||
- **GitHub Repository:** [ibm-granite/granite-4.0-language-models](https://github.com/ibm-granite/granite-4.0-language-models)
|
||||
- **Website**: [Granite Docs](https://www.ibm.com/granite/docs/)
|
||||
- **Release Date**: October 2nd, 2025
|
||||
- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
|
||||
|
||||
**Supported Languages:**
|
||||
English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese. Users may finetune Granite 4.0 models for languages beyond these languages.
|
||||
|
||||
**Intended Use:**
|
||||
Prominent use cases of LLMs in text-to-text generation include summarization, text classification, extraction, question-answering, code-completion (including FIM), and long-context generation tasks. All Granite Base models are able to handle these tasks as they were trained on a large amount of data from various domains. Moreover, they can serve as baseline to create specialized models for specific application scenarios.
|
||||
|
||||
**Generation:**
|
||||
This is a simple example of how to use Granite-4.0-Micro-Base model.
|
||||
|
||||
Install the following libraries:
|
||||
|
||||
```shell
|
||||
pip install torch torchvision torchaudio
|
||||
pip install accelerate
|
||||
pip install transformers
|
||||
```
|
||||
Then, copy the code snippet below to run the example.
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
device = "cuda"
|
||||
|
||||
model_path = "ibm-granite/granite-4.0-h-micro-base"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_path)
|
||||
# drop device_map if running on CPU
|
||||
model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device)
|
||||
model.eval()
|
||||
# change input text as desired
|
||||
input_text = "The capital of France is"
|
||||
# tokenize the text
|
||||
input_tokens = tokenizer(input_text, return_tensors="pt").to(device)
|
||||
# generate output tokens
|
||||
output = model.generate(**input_tokens, max_length=10)
|
||||
# decode output tokens into text
|
||||
output = tokenizer.batch_decode(output)
|
||||
# print output
|
||||
print(output[0])
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```shell
|
||||
The capital of France is Paris.
|
||||
```
|
||||
|
||||
**Evaluation Results:**
|
||||
|
||||
<table>
|
||||
<!-- <caption><b> All Results</b></caption> -->
|
||||
<thead>
|
||||
<tr>
|
||||
<th style="text-align:left; background-color: #001d6c; color: white;">Benchmarks</th>
|
||||
<th style="text-align:left; background-color: #001d6c; color: white;">Metric</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">Micro Dense</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">H Micro Dense</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">H Tiny MoE</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">H Small MoE</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td colspan="6" style="text-align:center; background-color: #FFFFFF; color: #2D2D2D; font-style:italic;">
|
||||
General Tasks
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">MMLU</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">5-shot</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">66.47</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">67.43</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">68.90</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">75.85</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">MMLU-Pro</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">5-shot,CoT</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">37.16</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">34.03</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">35.47</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">48.94</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">BBH</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">3-shot, CoT</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">63.84</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">57.65</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">59.67</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">75.84</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">AGI EVAL</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">3-shot</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">54.32</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">54.59</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">53.69</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">62.05</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">DROP</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">5-shot</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">66.04</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">67.44</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">64.92</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">74.69</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td colspan="6" style="text-align:center; background-color: #FFFFFF; color: #2D2D2D; font-style:italic;">
|
||||
Math Tasks
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">GSM8K</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">8-shot</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">72.93</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">63.76</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">72.55</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">82.11</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">Minerva Math</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">4-shot</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">38</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">39.7</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">40.34</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">46.28</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td colspan="6" style="text-align:center; background-color: #FFFFFF; color: #2D2D2D; font-style:italic;">
|
||||
Code Tasks
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">HumanEval </td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">pass@1 [StarCoder Prompt]</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">76.19</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">73.72</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">77.59</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">83.66</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">HumanEval</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">pass@1</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">59.76</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">70.73</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">71.34</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">76.22</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">HumanEval+</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">pass@1</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">54.27</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">67.07</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">64.02</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">69.51</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">MBPP</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">pass@1</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">81.48</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">74.87</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">81.48</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">83.07</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">MBPP+</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">pass@1</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">68.25</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">63.23</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">68.78</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">70.37</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td colspan="6" style="text-align:center; background-color: #FFFFFF; color: #2D2D2D; font-style:italic;">
|
||||
Multilingual Tasks
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">MMMLU</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">5-shot</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">56.59</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">58.5</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">62.77</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">71.18</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">INCLUDE</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">5-shot</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">51.77</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">52.16</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">53.78</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">66.04</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">MGSM</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">8-shot</td>
|
||||
<td style="text-align:right; background-color: #DAE8FF; color: #2D2D2D;">58.48</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">47.04</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">54.64</td>
|
||||
<td style="text-align:right; background-color: #FFFFFF; color: #2D2D2D;">65.2</td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
|
||||
|
||||
<table>
|
||||
<caption><b>Multilingual Benchmarks and thr included languages:</b></caption>
|
||||
<thead>
|
||||
<tr>
|
||||
<th style="text-align:left; background-color: #001d6c; color: white;">Benchmarks</th>
|
||||
<th style="text-align:left; background-color: #001d6c; color: white;"># Langs</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">Languages</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">MMMLU</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;">11</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">ar, de, en, es, fr, ja, ko, pt, zh, bn, hi</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">INCLUDE</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;">14</td>
|
||||
<!-- <td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">hindi, bengali, tamil, telugu, arabic, german, spanish, french, italian, japanese, korean, dutch, portuguese, chinese</td> -->
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">hi, bn, ta, te, ar, de, es, fr, it, ja, ko, nl, pt, zh</td>
|
||||
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">MGSM</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: #2D2D2D;">5</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: #2D2D2D;">en, es, fr, ja, zh</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
**Model Architecture:**
|
||||
|
||||
Granite-4.0-Micro-Base is based on a decoder-only dense transformer architecture. Core components of this architecture are: GQA, RoPE, MLP with SwiGLU, RMSNorm, and shared input/output embeddings.
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th style="text-align:left; background-color: #001d6c; color: white;">Model</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">Micro Dense</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">H Micro Dense</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">H Tiny MoE</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">H Small MoE</th>
|
||||
</tr></thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Embedding size</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">2560</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">2048</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">1536</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">4096</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Number of layers</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">40 attention</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">4 attention / 36 Mamba2</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">4 attention / 36 Mamba2</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">4 attention / 36 Mamba2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Attention head size</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">64</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">64</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">128</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">128</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Number of attention heads</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">40</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">32</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">12</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">32</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Number of KV heads</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">8</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">8</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">4</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">8</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Mamba2 state size</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">128</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">128</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">128</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Number of Mamba2 heads</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">64</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">48</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">128</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">MLP / Shared expert hidden size</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">8192</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">8192</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">1024</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">1536</td>
|
||||
</tr>
|
||||
|
||||
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Num. Experts</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">-</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">64</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">72</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Num. active Experts</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">-</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">6</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">10</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Expert hidden size</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">-</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">-</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">512</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">768</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">MLP activation</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">SwiGLU</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">SwiGLU</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">SwiGLU</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">SwiGLU</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Sequence length</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">128K</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">128K</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">128K</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">128K</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">Position embedding</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">RoPE</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">NoPE</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">NoPE</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">NoPE</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;"># Parameters</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">3B</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">3B</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">7B</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">32B</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;"># Active parameters</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">3B</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">3B</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">1B</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">9B</td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
|
||||
|
||||
**Training Data:** This model is trained on a mix of open source and proprietary data following a four-stage training strategy.
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th style="text-align:left; background-color: #001d6c; color: white;">Stage</th>
|
||||
<th style="text-align:left; background-color: #001d6c; color: white;">Characteristics</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">Micro Dense</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">H Micro Dense</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">H Tiny MoE</th>
|
||||
<th style="text-align:center; background-color: #001d6c; color: white;">H Small MoE</th>
|
||||
</tr></thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">I</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">General mixture of training data, warmup, and power scheduler for learning rate.</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">10</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">10</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">15</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">15</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">II</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">General mixture of training data with higher percentages of code and math with power scheduler for learning rate.</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">2</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">5</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">5</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">5</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">III</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">High quality training data, exponential decay of learning rate.</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">2</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">2</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">2</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">IV</td>
|
||||
<td style="text-align:left; background-color: #FFFFFF; color: black;">High quality training data, linear decay to zero for learning rate.</td>
|
||||
<td style="text-align:center; background-color: #DAE8FF; color: black;">0.5</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">0.5</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">0.5</td>
|
||||
<td style="text-align:center; background-color: #FFFFFF; color: black;">0.5</td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
|
||||
|
||||
**Infrastructure:**
|
||||
We trained the Granite 4.0 Language Models utilizing an NVIDIA GB200 NVL72 cluster hosted in CoreWeave. Intra-rack communication occurs via the 72-GPU NVLink domain, and a non-blocking, full Fat-Tree NDR 400 Gb/s InfiniBand network provides inter-rack communication. This cluster provides a scalable and efficient infrastructure for training our models over thousands of GPUs.
|
||||
|
||||
**Ethical Considerations and Limitations:**
|
||||
The use of Large Language Models involves risks and ethical considerations people must be aware of, including but not limited to: bias and fairness, misinformation, and autonomous decision-making. Granite-4.0-Micro-Base model is not the exception in this regard. Even though this model is suited for multiple generative AI tasks, it has not undergone any safety alignment, there it may produce problematic outputs. Additionally, it remains uncertain whether smaller models might exhibit increased susceptibility to hallucination in generation scenarios by copying text verbatim from the training dataset due to their reduced sizes and memorization capacities. This aspect is currently an active area of research, and we anticipate more rigorous exploration, comprehension, and mitigations in this domain. Regarding ethics, a latent risk associated with all Large Language Models is their malicious utilization. We urge the community to use Granite-4.0-Micro-Base model with ethical intentions and in a responsible way.
|
||||
|
||||
**Resources**
|
||||
- ⭐️ Learn about the latest updates with Granite: https://www.ibm.com/granite
|
||||
- 📄 Get started with tutorials, best practices, and prompt engineering advice: https://www.ibm.com/granite/docs/
|
||||
- 💡 Learn about the latest Granite learning resources: https://github.com/ibm-granite-community/
|
||||
89
config.json
Normal file
89
config.json
Normal file
@@ -0,0 +1,89 @@
|
||||
{
|
||||
"architectures": [
|
||||
"GraniteMoeHybridForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"attention_multiplier": 0.015625,
|
||||
"bos_token_id": 100257,
|
||||
"embedding_multiplier": 12,
|
||||
"eos_token_id": 100257,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 2560,
|
||||
"initializer_range": 0.1,
|
||||
"intermediate_size": 8192,
|
||||
"layer_types": [
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention",
|
||||
"attention"
|
||||
],
|
||||
"logits_scaling": 10,
|
||||
"mamba_chunk_size": 256,
|
||||
"mamba_conv_bias": true,
|
||||
"mamba_d_conv": 4,
|
||||
"mamba_d_head": 40,
|
||||
"mamba_d_state": 256,
|
||||
"mamba_expand": 2,
|
||||
"mamba_n_groups": 1,
|
||||
"mamba_n_heads": 128,
|
||||
"mamba_proj_bias": false,
|
||||
"max_position_embeddings": 131072,
|
||||
"model_type": "granitemoehybrid",
|
||||
"normalization_function": "rmsnorm",
|
||||
"num_attention_heads": 40,
|
||||
"num_experts_per_tok": 0,
|
||||
"num_hidden_layers": 40,
|
||||
"num_key_value_heads": 8,
|
||||
"num_local_experts": 0,
|
||||
"output_router_logits": false,
|
||||
"pad_token_id": 100256,
|
||||
"position_embedding_type": "rope",
|
||||
"residual_multiplier": 0.22,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 10000000,
|
||||
"router_aux_loss_coef": 0.01,
|
||||
"shared_intermediate_size": 8192,
|
||||
"tie_word_embeddings": true,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.56.0",
|
||||
"use_cache": true,
|
||||
"vocab_size": 100352
|
||||
}
|
||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
||||
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}
|
||||
7
generation_config.json
Normal file
7
generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 100257,
|
||||
"eos_token_id": 100257,
|
||||
"pad_token_id": 100256,
|
||||
"transformers_version": "4.56.0"
|
||||
}
|
||||
100001
merges.txt
Normal file
100001
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model-00001-of-00002.safetensors
Normal file
3
model-00001-of-00002.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:75418829cdcadec6a9a4cab2ee0e420bedffdc97f2da1875e9f6f836a1defd36
|
||||
size 4918145376
|
||||
3
model-00002-of-00002.safetensors
Normal file
3
model-00002-of-00002.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:9d0e185ab087789816189a907861ad609e2c1be814f682c75ccd14703229dba6
|
||||
size 1887565800
|
||||
330
model.safetensors.index.json
Normal file
330
model.safetensors.index.json
Normal file
@@ -0,0 +1,330 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_parameters": 3402836480,
|
||||
"total_size": 6805672960
|
||||
},
|
||||
"weight_map": {
|
||||
"model.embed_tokens.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.28.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.28.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.28.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.28.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.28.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.29.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.29.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.29.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.29.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.29.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.29.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.29.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.29.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.30.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.30.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.30.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.30.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.30.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.30.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.30.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.30.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.31.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.31.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.31.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.31.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.31.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.31.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.31.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.31.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.32.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.32.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.32.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.32.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.32.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.32.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.32.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.32.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.33.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.33.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.33.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.33.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.33.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.33.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.33.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.33.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.34.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.34.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.34.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.34.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.34.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.34.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.34.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.34.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.35.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.35.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.35.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.35.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.35.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.35.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.35.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.35.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.36.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.36.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.36.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.36.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.36.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.36.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.36.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.36.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.37.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.37.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.37.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.37.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.37.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.37.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.37.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.37.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.38.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.38.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.38.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.38.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.38.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.38.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.38.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.38.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.shared_mlp.input_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.shared_mlp.output_linear.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.shared_mlp.input_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.shared_mlp.output_linear.weight": "model-00001-of-00002.safetensors",
|
||||
"model.norm.weight": "model-00002-of-00002.safetensors"
|
||||
}
|
||||
}
|
||||
30
special_tokens_map.json
Normal file
30
special_tokens_map.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<|unk|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e2bad66439538cb4d5a7580680932432ed9ece9d3b8577e675512bdf11599253
|
||||
size 7153421
|
||||
783
tokenizer_config.json
Normal file
783
tokenizer_config.json
Normal file
@@ -0,0 +1,783 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"100256": {
|
||||
"content": "<|pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100257": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100258": {
|
||||
"content": "<|fim_prefix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100259": {
|
||||
"content": "<|fim_middle|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100260": {
|
||||
"content": "<|fim_suffix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100261": {
|
||||
"content": "<|fim_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100262": {
|
||||
"content": "<|filename|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100263": {
|
||||
"content": "<|reponame|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100264": {
|
||||
"content": "<|start_of_role|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100265": {
|
||||
"content": "<|end_of_role|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100266": {
|
||||
"content": "<|unused_1|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100267": {
|
||||
"content": "<|start_of_plugin|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100268": {
|
||||
"content": "<|end_of_plugin|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100269": {
|
||||
"content": "<|unk|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100270": {
|
||||
"content": "<tool_call>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100271": {
|
||||
"content": "</tool_call>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100272": {
|
||||
"content": "<tool_response>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100273": {
|
||||
"content": "</tool_response>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100274": {
|
||||
"content": "<think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100275": {
|
||||
"content": "</think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"100276": {
|
||||
"content": "<think_on>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100277": {
|
||||
"content": "<think_off>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100278": {
|
||||
"content": "<schema>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100279": {
|
||||
"content": "</schema>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100280": {
|
||||
"content": "<tools>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100281": {
|
||||
"content": "</tools>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100282": {
|
||||
"content": "<documents>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100283": {
|
||||
"content": "</documents>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100284": {
|
||||
"content": "<|unused_15|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100285": {
|
||||
"content": "<|unused_16|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100286": {
|
||||
"content": "<|unused_17|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100287": {
|
||||
"content": "<|unused_18|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100288": {
|
||||
"content": "<|unused_19|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100289": {
|
||||
"content": "<|unused_20|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100290": {
|
||||
"content": "<|unused_21|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100291": {
|
||||
"content": "<|unused_22|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100292": {
|
||||
"content": "<|unused_23|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100293": {
|
||||
"content": "<|unused_24|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100294": {
|
||||
"content": "<|unused_25|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100295": {
|
||||
"content": "<|unused_26|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100296": {
|
||||
"content": "<|unused_27|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100297": {
|
||||
"content": "<|unused_28|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100298": {
|
||||
"content": "<|unused_29|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100299": {
|
||||
"content": "<|unused_30|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100300": {
|
||||
"content": "<|unused_31|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100301": {
|
||||
"content": "<|unused_32|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100302": {
|
||||
"content": "<|unused_33|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100303": {
|
||||
"content": "<|unused_34|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100304": {
|
||||
"content": "<|unused_35|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100305": {
|
||||
"content": "<|unused_36|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100306": {
|
||||
"content": "<|unused_37|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100307": {
|
||||
"content": "<|unused_38|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100308": {
|
||||
"content": "<|unused_39|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100309": {
|
||||
"content": "<|unused_40|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100310": {
|
||||
"content": "<|unused_41|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100311": {
|
||||
"content": "<|unused_42|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100312": {
|
||||
"content": "<|unused_43|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100313": {
|
||||
"content": "<|unused_44|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100314": {
|
||||
"content": "<|unused_45|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100315": {
|
||||
"content": "<|unused_46|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100316": {
|
||||
"content": "<|unused_47|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100317": {
|
||||
"content": "<|unused_48|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100318": {
|
||||
"content": "<|unused_49|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100319": {
|
||||
"content": "<|unused_50|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100320": {
|
||||
"content": "<|unused_51|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100321": {
|
||||
"content": "<|unused_52|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100322": {
|
||||
"content": "<|unused_53|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100323": {
|
||||
"content": "<|unused_54|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100324": {
|
||||
"content": "<|unused_55|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100325": {
|
||||
"content": "<|unused_56|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100326": {
|
||||
"content": "<|unused_57|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100327": {
|
||||
"content": "<|unused_58|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100328": {
|
||||
"content": "<|unused_59|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100329": {
|
||||
"content": "<|unused_60|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100330": {
|
||||
"content": "<|unused_61|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100331": {
|
||||
"content": "<|unused_62|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100332": {
|
||||
"content": "<|unused_63|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100333": {
|
||||
"content": "<|unused_64|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100334": {
|
||||
"content": "<|unused_65|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100335": {
|
||||
"content": "<|unused_66|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100336": {
|
||||
"content": "<|unused_67|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100337": {
|
||||
"content": "<|unused_68|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100338": {
|
||||
"content": "<|unused_69|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100339": {
|
||||
"content": "<|unused_70|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100340": {
|
||||
"content": "<|unused_71|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100341": {
|
||||
"content": "<|unused_72|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100342": {
|
||||
"content": "<|unused_73|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100343": {
|
||||
"content": "<|unused_74|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100344": {
|
||||
"content": "<|unused_75|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100345": {
|
||||
"content": "<|unused_76|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100346": {
|
||||
"content": "<|unused_77|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100347": {
|
||||
"content": "<|unused_78|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100348": {
|
||||
"content": "<|unused_79|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100349": {
|
||||
"content": "<|unused_80|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100350": {
|
||||
"content": "<|unused_81|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100351": {
|
||||
"content": "<|unused_82|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"bos_token": "<|end_of_text|>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|end_of_text|>",
|
||||
"extra_special_tokens": {},
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"pad_token": "<|pad|>",
|
||||
"padding_side": "left",
|
||||
"tokenizer_class": "GPT2Tokenizer",
|
||||
"unk_token": "<|unk|>"
|
||||
}
|
||||
3
vocab.json
Normal file
3
vocab.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8af71076de8b0b626eed0f4c984faf0a7c062479164b2a31308a948524d4f69c
|
||||
size 1612704
|
||||
Reference in New Issue
Block a user