RedHatAI/DeepSeek-R1-Distill-Qwen-7B-FP8-dynamic
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: RedHatAI/DeepSeek-R1-Distill-Qwen-7B-FP8-dynamic
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-10-28 10:50:16 +08:00
commit 7bf1099f63
11 changed files with 1499 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

763
README.md Normal file
View File

@@ -0,0 +1,763 @@
---
license: mit
tags:
- deepseek
- fp8
- vllm
base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
library_name: transformers
---
# DeepSeek-R1-Distill-Qwen-7B-FP8-dynamic
## Model Overview
- **Model Architecture:** Qwen2ForCausalLM
- **Input:** Text
- **Output:** Text
- **Model Optimizations:**
- **Weight quantization:** FP8
- **Activation quantization:** FP8
- **Release Date:** 2/5/2025
- **Version:** 1.0
- **Model Developers:** Neural Magic
Quantized version of [DeepSeek-R1-Distill-Qwen-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B).
### Model Optimizations
This model was obtained by quantizing the weights and activations of [DeepSeek-R1-Distill-Qwen-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B) to FP8 data type.
This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
Only the weights and activations of the linear operators within transformers blocks are quantized.
Weights are quantized using a symmetric per-channel scheme, whereas quantizations are quantized using a symmetric per-token scheme.
[LLM Compressor](https://github.com/vllm-project/llm-compressor) is used for quantization.
## Use with vLLM
This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
```python
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
number_gpus = 1
model_name = "neuralmagic/DeepSeek-R1-Distill-Qwen-7B-dynamic"
tokenizer = AutoTokenizer.from_pretrained(model_name)
sampling_params = SamplingParams(temperature=0.6, max_tokens=256, stop_token_ids=[tokenizer.eos_token_id])
llm = LLM(model=model_name, tensor_parallel_size=number_gpus, trust_remote_code=True)
messages_list = [
[{"role": "user", "content": "Who are you? Please respond in pirate speak!"}],
]
prompt_token_ids = [tokenizer.apply_chat_template(messages, add_generation_prompt=True) for messages in messages_list]
outputs = llm.generate(prompt_token_ids=prompt_token_ids, sampling_params=sampling_params)
generated_text = [output.outputs[0].text for output in outputs]
print(generated_text)
```
vLLM also supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
## Creation
This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.transformers import oneshot
import os
# Load model
model_stub = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
model_name = model_stub.split("/")[-1]
model = AutoModelForCausalLM.from_pretrained(
model_stub,
torch_dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_stub)
# Configure the quantization algorithm and scheme
recipe = QuantizationModifier(
targets="Linear",
scheme="FP8_DYNAMIC",
ignore=["lm_head"],
)
# Apply quantization
oneshot(
model=model,
recipe=recipe,
)
# Save to disk in compressed-tensors format
save_path = model_name + "-FP8-dynamic
model.save_pretrained(save_path)
tokenizer.save_pretrained(save_path)
print(f"Model and tokenizer saved to: {save_path}")
```
## Evaluation
The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard) and [V2](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/), using the following commands:
OpenLLM Leaderboard V1:
```
lm_eval \
--model vllm \
--model_args pretrained="neuralmagic/DeepSeek-R1-Distill-Qwen-7B-FP8-dynamic",dtype=auto,max_model_len=4096,tensor_parallel_size=1,enable_chunked_prefill=True \
--tasks openllm \
--write_out \
--batch_size auto \
--output_path output_dir \
--show_config
```
OpenLLM Leaderboard V2:
```
lm_eval \
--model vllm \
--model_args pretrained="neuralmagic/DeepSeek-R1-Distill-Qwen-7B-FP8-dynamic",dtype=auto,max_model_len=4096,tensor_parallel_size=1,enable_chunked_prefill=True \
--apply_chat_template \
--fewshot_as_multiturn \
--tasks leaderboard \
--write_out \
--batch_size auto \
--output_path output_dir \
--show_config
```
### Accuracy
<table>
<thead>
<tr>
<th>Category</th>
<th>Metric</th>
<th>deepseek-ai/DeepSeek-R1-Distill-Qwen-7B</th>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-FP8-dynamic</th>
<th>Recovery</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="4"><b>Reasoning</b></td>
<td>AIME 2024 (pass@1)</td>
<td>53.17</td>
<td>53.17</td>
<td>100%</td>
</tr>
<tr>
<td>MATH-500 (pass@1)</td>
<td>93.66</td>
<td>93.62</td>
<td>99.96%</td>
</tr>
<tr>
<td>GPQA Diamond (pass@1)</td>
<td>50.53</td>
<td>50.28</td>
<td>99.51%</td>
</tr>
<tr>
<td><b>Average Score</b></td>
<td><b>65.79</b></td>
<td><b>65.69</b></td>
<td><b>99.85%</b></td>
</tr>
<tr>
<td rowspan="7"><b>OpenLLM V1</b></td>
<td>ARC-Challenge (Acc-Norm, 25-shot)</td>
<td>50.51</td>
<td>50.51</td>
<td>100.0%</td>
</tr>
<tr>
<td>GSM8K (Strict-Match, 5-shot)</td>
<td>78.62</td>
<td>79.83</td>
<td>101.5%</td>
</tr>
<tr>
<td>HellaSwag (Acc-Norm, 10-shot)</td>
<td>61.90</td>
<td>61.62</td>
<td>99.6%</td>
</tr>
<tr>
<td>MMLU (Acc, 5-shot)</td>
<td>54.19</td>
<td>53.76</td>
<td>99.2%</td>
</tr>
<tr>
<td>TruthfulQA (MC2, 0-shot)</td>
<td>45.55</td>
<td>46.14</td>
<td>101.3%</td>
</tr>
<tr>
<td>Winogrande (Acc, 5-shot)</td>
<td>61.56</td>
<td>60.54</td>
<td>98.3%</td>
</tr>
<tr>
<td><b>Average Score</b></td>
<td><b>58.72</b></td>
<td><b>58.73</b></td>
<td><b>100.0%</b></td>
</tr>
<tr>
<td rowspan="7"><b>OpenLLM V2</b></td>
<td>IFEval (Inst Level Strict Acc, 0-shot)</td>
<td>39.38</td>
<td>39.01</td>
<td>99.1%</td>
</tr>
<tr>
<td>BBH (Acc-Norm, 3-shot)</td>
<td>6.97</td>
<td>6.19</td>
<td>---</td>
</tr>
<tr>
<td>Math-Hard (Exact-Match, 4-shot)</td>
<td>0.00</td>
<td>0.00</td>
<td>---</td>
</tr>
<tr>
<td>GPQA (Acc-Norm, 0-shot)</td>
<td>1.81</td>
<td>1.63</td>
<td>---</td>
</tr>
<tr>
<td>MUSR (Acc-Norm, 0-shot)</td>
<td>4.68</td>
<td>5.08</td>
<td>---</td>
</tr>
<tr>
<td>MMLU-Pro (Acc, 5-shot)</td>
<td>1.66</td>
<td>1.76</td>
<td>---</td>
</tr>
<tr>
<td><b>Average Score</b></td>
<td><b>9.08</b></td>
<td><b>8.94</b></td>
<td><b>---</b></td>
</tr>
<tr>
<td rowspan="4"><b>Coding</b></td>
<td>HumanEval (pass@1)</td>
<td>40.80</td>
<td>39.50</td>
<td><b>96.8%</b></td>
</tr>
<tr>
<td>HumanEval (pass@10)</td>
<td>64.40</td>
<td>62.10</td>
<td>96.4%</td>
</tr>
<tr>
<td>HumanEval+ (pass@10)</td>
<td>38.50</td>
<td>37.20</td>
<td>96.6%</td>
</tr>
<tr>
<td>HumanEval+ (pass@10)</td>
<td>60.40</td>
<td>59.30</td>
<td>98.2%</td>
</tr>
</tbody>
</table>
## Inference Performance
This model achieves up to 1.4x speedup in single-stream deployment and up to 1.2x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario.
The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.7.2, and [GuideLLM](https://github.com/neuralmagic/guidellm).
<details>
<summary>Benchmarking Command</summary>
```
guidellm --model neuralmagic/DeepSeek-R1-Distill-Qwen-7B-FP8-dynamic --target "http://localhost:8000/v1" --data-type emulated --data "prompt_tokens=<prompt_tokens>,generated_tokens=<generated_tokens>" --max seconds 360 --backend aiohttp_server
```
</details>
### Single-stream performance (measured with vLLM version 0.7.2)
<table>
<thead>
<tr>
<th></th>
<th></th>
<th></th>
<th style="text-align: center;" colspan="2" >Instruction Following<br>256 / 128</th>
<th style="text-align: center;" colspan="2" >Multi-turn Chat<br>512 / 256</th>
<th style="text-align: center;" colspan="2" >Docstring Generation<br>768 / 128</th>
<th style="text-align: center;" colspan="2" >RAG<br>1024 / 128</th>
<th style="text-align: center;" colspan="2" >Code Completion<br>256 / 1024</th>
<th style="text-align: center;" colspan="2" >Code Fixing<br>1024 / 1024</th>
<th style="text-align: center;" colspan="2" >Large Summarization<br>4096 / 512</th>
<th style="text-align: center;" colspan="2" >Large RAG<br>10240 / 1536</th>
</tr>
<tr>
<th>Hardware</th>
<th>Model</th>
<th>Average cost reduction</th>
<th>Latency (s)</th>
<th>QPD</th>
<th>Latency (s)</th>
<th>QPD</th>
<th>Latency (s)</th>
<th>QPD</th>
<th>Latency (s)</th>
<th>QPD</th>
<th>Latency (s)</th>
<th>QPD</th>
<th>Latency (s)</th>
<th>QPD</th>
<th>Latency (s)</th>
<th>QPD</th>
<th>Latency (s)</th>
<th>QPD</th>
</tr>
</thead>
<tbody style="text-align: center" >
<tr>
<th rowspan="3" valign="top">A6000x1</th>
<th>deepseek-ai/DeepSeek-R1-Distill-Qwen-7B</th>
<td>---</td>
<td>2.9</td>
<td>1576</td>
<td>5.7</td>
<td>788</td>
<td>2.9</td>
<td>1535</td>
<td>3.0</td>
<td>1496</td>
<td>22.6</td>
<td>199</td>
<td>23.2</td>
<td>194</td>
<td>12.1</td>
<td>370</td>
<td>38.5</td>
<td>117</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-quantized.w8a8</th>
<td>1.56</td>
<td>1.8</td>
<td>2495</td>
<td>3.7</td>
<td>1223</td>
<td>1.9</td>
<td>2384</td>
<td>1.9</td>
<td>2393</td>
<td>14.3</td>
<td>315</td>
<td>14.8</td>
<td>304</td>
<td>7.9</td>
<td>572</td>
<td>25.3</td>
<td>178</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16</th>
<td>2.41</td>
<td>1.1</td>
<td>4086</td>
<td>2.3</td>
<td>1998</td>
<td>1.2</td>
<td>3783</td>
<td>1.3</td>
<td>3527</td>
<td>8.6</td>
<td>526</td>
<td>8.8</td>
<td>512</td>
<td>5.2</td>
<td>860</td>
<td>22.7</td>
<td>198</td>
</tr>
<tr>
<th rowspan="3" valign="top">A100x1</th>
<th>deepseek-ai/DeepSeek-R1-Distill-Qwen-7B</th>
<td>---</td>
<td>1.4</td>
<td>1389</td>
<td>2.9</td>
<td>691</td>
<td>1.5</td>
<td>1358</td>
<td>1.5</td>
<td>1329</td>
<td>11.5</td>
<td>175</td>
<td>11.6</td>
<td>174</td>
<td>6.2</td>
<td>326</td>
<td>21.5</td>
<td>93</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-quantized.w8a8</th>
<td>1.28</td>
<td>1.1</td>
<td>1850</td>
<td>2.2</td>
<td>905</td>
<td>1.1</td>
<td>1807</td>
<td>1.1</td>
<td>1750</td>
<td>8.6</td>
<td>233</td>
<td>8.7</td>
<td>230</td>
<td>4.7</td>
<td>431</td>
<td>23.1</td>
<td>87</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16</th>
<td>1.72</td>
<td>0.8</td>
<td>2575</td>
<td>1.5</td>
<td>1298</td>
<td>0.8</td>
<td>2461</td>
<td>0.8</td>
<td>2382</td>
<td>6.1</td>
<td>331</td>
<td>6.2</td>
<td>323</td>
<td>3.6</td>
<td>566</td>
<td>22.7</td>
<td>89</td>
</tr>
<tr>
<th rowspan="3" valign="top">H100x1</th>
<th>deepseek-ai/DeepSeek-R1-Distill-Qwen-7B</th>
<td>---</td>
<td>0.9</td>
<td>1161</td>
<td>1.9</td>
<td>579</td>
<td>1.0</td>
<td>1138</td>
<td>1.0</td>
<td>1121</td>
<td>7.5</td>
<td>146</td>
<td>7.6</td>
<td>145</td>
<td>3.9</td>
<td>279</td>
<td>15.4</td>
<td>71</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-FP8-dynamic</th>
<td>1.34</td>
<td>0.7</td>
<td>1585</td>
<td>1.4</td>
<td>786</td>
<td>0.7</td>
<td>1577</td>
<td>0.7</td>
<td>1524</td>
<td>5.3</td>
<td>207</td>
<td>5.5</td>
<td>197</td>
<td>2.9</td>
<td>382</td>
<td>14.3</td>
<td>77</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16</th>
<td>1.33</td>
<td>0.7</td>
<td>1590</td>
<td>1.4</td>
<td>793</td>
<td>0.7</td>
<td>1549</td>
<td>0.7</td>
<td>1509</td>
<td>5.4</td>
<td>201</td>
<td>5.5</td>
<td>198</td>
<td>2.9</td>
<td>381</td>
<td>14.0</td>
<td>78</td>
</tr>
</tbody>
</table>
**Use case profiles: prompt tokens / generation tokens
**QPD: Queries per dollar, based on on-demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu-cloud) (observed on 2/18/2025).
### Multi-stream asynchronous performance (measured with vLLM version 0.7.2)
<table>
<thead>
<tr>
<th></th>
<th></th>
<th></th>
<th style="text-align: center;" colspan="2" >Instruction Following<br>256 / 128</th>
<th style="text-align: center;" colspan="2" >Multi-turn Chat<br>512 / 256</th>
<th style="text-align: center;" colspan="2" >Docstring Generation<br>768 / 128</th>
<th style="text-align: center;" colspan="2" >RAG<br>1024 / 128</th>
<th style="text-align: center;" colspan="2" >Code Completion<br>256 / 1024</th>
<th style="text-align: center;" colspan="2" >Code Fixing<br>1024 / 1024</th>
<th style="text-align: center;" colspan="2" >Large Summarization<br>4096 / 512</th>
<th style="text-align: center;" colspan="2" >Large RAG<br>10240 / 1536</th>
</tr>
<tr>
<th>Hardware</th>
<th>Model</th>
<th>Average cost reduction</th>
<th>Maximum throughput (QPS)</th>
<th>QPD</th>
<th>Maximum throughput (QPS)</th>
<th>QPD</th>
<th>Maximum throughput (QPS)</th>
<th>QPD</th>
<th>Maximum throughput (QPS)</th>
<th>QPD</th>
<th>Maximum throughput (QPS)</th>
<th>QPD</th>
<th>Maximum throughput (QPS)</th>
<th>QPD</th>
<th>Maximum throughput (QPS)</th>
<th>QPD</th>
<th>Maximum throughput (QPS)</th>
<th>QPD</th>
</tr>
</thead>
<tbody style="text-align: center" >
<tr>
<th rowspan="3" valign="top">A6000x1</th>
<th>deepseek-ai/DeepSeek-R1-Distill-Qwen-7B</th>
<td>---</td>
<td>14.9</td>
<td>67138</td>
<td>7.1</td>
<td>32094</td>
<td>7.4</td>
<td>33096</td>
<td>5.9</td>
<td>26480</td>
<td>2.0</td>
<td>9004</td>
<td>1.5</td>
<td>6639</td>
<td>1.1</td>
<td>4938</td>
<td>0.3</td>
<td>1151</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-quantized.w8a8</th>
<td>1.36</td>
<td>20.2</td>
<td>90956</td>
<td>8.8</td>
<td>39786</td>
<td>10.2</td>
<td>45963</td>
<td>8.1</td>
<td>36596</td>
<td>3.1</td>
<td>13968</td>
<td>2.1</td>
<td>9629</td>
<td>1.4</td>
<td>6374</td>
<td>0.3</td>
<td>1429</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16</th>
<td>1.00</td>
<td>13.3</td>
<td>59681</td>
<td>6.1</td>
<td>27633</td>
<td>5.9</td>
<td>26689</td>
<td>4.7</td>
<td>20944</td>
<td>2.9</td>
<td>13108</td>
<td>1.9</td>
<td>8355</td>
<td>1.0</td>
<td>4362</td>
<td>0.3</td>
<td>1170</td>
</tr>
<tr>
<th rowspan="3" valign="top">A100x1</th>
<th>deepseek-ai/DeepSeek-R1-Distill-Qwen-7B</th>
<td>---</td>
<td>26.4</td>
<td>53073</td>
<td>13.0</td>
<td>26213</td>
<td>14.5</td>
<td>29110</td>
<td>11.4</td>
<td>22936</td>
<td>4.4</td>
<td>8749</td>
<td>3.3</td>
<td>6680</td>
<td>2.3</td>
<td>4634</td>
<td>0.5</td>
<td>1105</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-quantized.w8a8</th>
<td>1.27</td>
<td>34.3</td>
<td>69009</td>
<td>14.8</td>
<td>29791</td>
<td>19.0</td>
<td>38214</td>
<td>15.7</td>
<td>31598</td>
<td>5.6</td>
<td>11186</td>
<td>4.2</td>
<td>8350</td>
<td>3.0</td>
<td>6020</td>
<td>0.7</td>
<td>1328</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16</th>
<td>0.93</td>
<td>23.9</td>
<td>47993</td>
<td>12.0</td>
<td>24194</td>
<td>12.5</td>
<td>25239</td>
<td>10.0</td>
<td>20029</td>
<td>4.5</td>
<td>9055</td>
<td>3.3</td>
<td>6681</td>
<td>2.1</td>
<td>4156</td>
<td>0.5</td>
<td>1043</td>
</tr>
<tr>
<th rowspan="3" valign="top">H100x1</th>
<th>deepseek-ai/DeepSeek-R1-Distill-Qwen-7B</th>
<td>---</td>
<td>54.3</td>
<td>59410</td>
<td>26.0</td>
<td>28440</td>
<td>32.1</td>
<td>35154</td>
<td>26.7</td>
<td>29190</td>
<td>8.0</td>
<td>8700</td>
<td>6.6</td>
<td>7275</td>
<td>5.2</td>
<td>5669</td>
<td>1.2</td>
<td>1266</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-FP8-dynamic</th>
<td>1.16</td>
<td>62.9</td>
<td>68818</td>
<td>30.3</td>
<td>33196</td>
<td>39.4</td>
<td>43132</td>
<td>31.1</td>
<td>34073</td>
<td>9.2</td>
<td>10058</td>
<td>7.1</td>
<td>7748</td>
<td>6.1</td>
<td>6714</td>
<td>1.3</td>
<td>1415</td>
</tr>
<tr>
<th>neuralmagic/DeepSeek-R1-Distill-Qwen-7B-quantized.w4a16</th>
<td>1.02</td>
<td>56.2</td>
<td>61483</td>
<td>26.7</td>
<td>29243</td>
<td>32.5</td>
<td>35592</td>
<td>26.9</td>
<td>29461</td>
<td>8.3</td>
<td>9072</td>
<td>6.4</td>
<td>7027</td>
<td>5.2</td>
<td>5731</td>
<td>1.2</td>
<td>1291</td>
</tr>
</tbody>
</table>
**Use case profiles: prompt tokens / generation tokens
**QPS: Queries per second.
**QPD: Queries per dollar, based on on-demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu-cloud) (observed on 2/18/2025).

72
config.json Normal file
View File

@@ -0,0 +1,72 @@
{
"_name_or_path": "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151643,
"hidden_act": "silu",
"hidden_size": 3584,
"initializer_range": 0.02,
"intermediate_size": 18944,
"max_position_embeddings": 131072,
"max_window_layers": 28,
"model_type": "qwen2",
"num_attention_heads": 28,
"num_hidden_layers": 28,
"num_key_value_heads": 4,
"quantization_config": {
"config_groups": {
"group_0": {
"input_activations": {
"actorder": null,
"block_structure": null,
"dynamic": true,
"group_size": null,
"num_bits": 8,
"observer": null,
"observer_kwargs": {},
"strategy": "token",
"symmetric": true,
"type": "float"
},
"output_activations": null,
"targets": [
"Linear"
],
"weights": {
"actorder": null,
"block_structure": null,
"dynamic": false,
"group_size": null,
"num_bits": 8,
"observer": "mse",
"observer_kwargs": {},
"strategy": "channel",
"symmetric": true,
"type": "float"
}
}
},
"format": "float-quantized",
"global_compression_ratio": 1.5268082955288436,
"ignore": [
"lm_head"
],
"kv_cache_scheme": null,
"quant_method": "compressed-tensors",
"quantization_status": "compressed"
},
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 10000,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.48.1",
"use_cache": true,
"use_mrope": false,
"use_sliding_window": false,
"vocab_size": 152064
}

9
generation_config.json Normal file
View File

@@ -0,0 +1,9 @@
{
"_from_model_config": true,
"bos_token_id": 151646,
"do_sample": true,
"eos_token_id": 151643,
"temperature": 0.6,
"top_p": 0.95,
"transformers_version": "4.48.1"
}

3
model-00001-of-00002.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8ac2e963a3bf6fd8968ac94946691246d6649da683d75e21762073db961fab77
size 4985986696

3
model-00002-of-00002.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3212e2713a0e1224d11dc74617a9200907d35119ad1b30213403a19e2c878591
size 3722801872

542
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,542 @@
{
"metadata": {
"total_size": 8708725760
},
"weight_map": {
"lm_head.weight": "model-00002-of-00002.safetensors",
"model.embed_tokens.weight": "model-00001-of-00002.safetensors",
"model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.16.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.16.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.16.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.17.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.mlp.gate_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.mlp.up_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.k_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.o_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.q_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.v_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.18.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.mlp.gate_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.mlp.up_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.k_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.o_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.q_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.v_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.19.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.mlp.gate_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.mlp.up_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.k_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.o_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.q_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.v_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.20.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.gate_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.up_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.k_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.o_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.q_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.v_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.21.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.gate_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.up_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.k_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.o_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.q_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.v_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.22.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.gate_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.up_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.k_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.o_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.q_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.v_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.23.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.gate_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.up_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.k_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.o_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.q_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.v_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.24.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.gate_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.up_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.24.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.k_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.o_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.q_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.v_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.25.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.25.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.25.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.mlp.gate_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.25.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.mlp.up_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.25.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.k_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.o_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.q_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.v_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.26.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.gate_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.up_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.26.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.k_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.o_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.q_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.v_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.27.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.27.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.mlp.down_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.27.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.mlp.gate_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.27.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.mlp.up_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.27.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.k_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.o_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.q_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.v_proj.weight_scale": "model-00002-of-00002.safetensors",
"model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.down_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.gate_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.up_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.k_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.o_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.q_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.v_proj.weight_scale": "model-00001-of-00002.safetensors",
"model.norm.weight": "model-00002-of-00002.safetensors"
}
}

10
recipe.yaml Normal file
View File

@@ -0,0 +1,10 @@
quant_stage:
quant_modifiers:
QuantizationModifier:
ignore: [lm_head]
config_groups:
group_0:
targets: [Linear]
weights: {num_bits: 8, type: float, symmetric: true, strategy: channel, observer: mse}
input_activations: {num_bits: 8, type: float, symmetric: true, strategy: token,
dynamic: true, observer: memoryless}

23
special_tokens_map.json Normal file
View File

@@ -0,0 +1,23 @@
{
"bos_token": {
"content": "<｜begin▁of▁sentence｜>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<｜end▁of▁sentence｜>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<｜end▁of▁sentence｜>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:88145e3c3249adc2546ede277e9819d6e405e19072456e4b521cbc724bd60773
size 7031660

35
tokenizer_config.json Normal file
View File

@@ -0,0 +1,35 @@
{
"add_bos_token": true,
"add_eos_token": false,
"bos_token": {
"__type": "AddedToken",
"content": "<｜begin▁of▁sentence｜>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"clean_up_tokenization_spaces": false,
"eos_token": {
"__type": "AddedToken",
"content": "<｜end▁of▁sentence｜>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"legacy": true,
"model_max_length": 16384,
"pad_token": {
"__type": "AddedToken",
"content": "<｜end▁of▁sentence｜>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"sp_model_kwargs": {},
"unk_token": null,
"tokenizer_class": "LlamaTokenizerFast",
"chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜><think>\\n'}}{% endif %}"
}