138 lines
21 KiB
Markdown
138 lines
21 KiB
Markdown
---
|
||
license: qwen
|
||
license_name: Tongyi Qianwen LICENSE AGREEMENT
|
||
license_link: LICENSE
|
||
language:
|
||
- en
|
||
- zh
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- Instruct
|
||
- Chat
|
||
base_model:
|
||
- Qwen/Qwen2.5-7B-Instruct
|
||
---
|
||
# 通义千问Qwen2.5-7B-Instruct-Int4-W4A16量化高精校准
|
||
|
||
原模型 [qwen/Qwen2.5-7B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-7B-Instruct)
|
||
|
||
```python
|
||
# download model
|
||
from modelscope import snapshot_download
|
||
model_dir = snapshot_download('okwinds/Qwen2.5-7B-Instruct-Int4-W4A16')
|
||
```
|
||
|
||
## Qwen2.5 同系列量化模型
|
||
|
||
| **Model** | GPTQ Compression & FP8 | GGUF | **MLX** (macOS) |
|
||
| ------------ |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| *3B params* | - | [3B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-GGUF-V3-LOT) | [3B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-4bit)<br/>[3B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-3B-Instruct-MLX-8bit) |
|
||
| *7B params* | [7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int4-W4A16)<br/>[7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-Int8-W8A16)<br/>[7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-FP8-A16) <br/>[Coder-7B-Instruct-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8) | [7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-GGUF-V3-LOT) | [7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-4bit)<br/>[7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-7B-Instruct-MLX-8bit) |
|
||
| *Coder 7B params* | [Coder-7B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int4-W4A16)<br/>[Coder-7B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-Int8-W8A16) <br/>[Coder-7B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-FP8-A16) | [Coder-7B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-GGUF-V3-LOT) | [Coder-7B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-4bit)<br/>[Coder-7B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-7B-Instruct-MLX-8bit) |
|
||
| *14B params* | [14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int4-W4A16)<br/>[14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-Int8-W8A16)<br/>[14B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-FP8-A16) | [14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-GGUF-V3-LOT) | [14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-4bit)<br/>[14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-14B-Instruct-MLX-8bit) |
|
||
| *Coder 14B params* | [Coder-14B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int4-W4A16)<br/>[Coder-14B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-Int8-W8A16) | [Coder-14B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-GGUF-V3-LOT) | [Coder-14B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-4bit)<br/>[Coder-14B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-14B-Instruct-MLX-8bit) |
|
||
| *32B params* | [32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int4-W4A16)<br/>[32B-Int8-W8A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-Int8-W8A16)<br/>[32B-FP8-A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-FP8-A16) | [32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-GGUF-V3-LOT) | [32B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-2bit)<br/>[32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-4bit)<br/>[32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-32B-Instruct-MLX-8bit) |
|
||
| *Coder 32B params* | [Coder-32B-Int4-W4A16](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-Int4-W4A16)<br/>[Coder-32B-FP8](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-FP8) | [Coder-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-GGUF-V3-LOT) | [Coder-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-4bit)<br/>[Coder-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-Coder-32B-Instruct-MLX-8bit) |
|
||
| *QwQ-Preview 32B params* | [QwQ-32B-Preview-Int4-W4A16](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-Int4-W4A16)<br/>[QwQ-32B-Preview-FP8](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-FP8) | [QwQ-32B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-GGUF-V3-LOT) | [QwQ-32B-MLX-4bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-4bit)<br/>[QwQ-32B-MLX-8bit](https://www.modelscope.cn/models/okwinds/QwQ-32B-Preview-MLX-8bit) |
|
||
| *72B params* | - | [72B-GGUF-V3-LOT](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-GGUF-V3-LOT) | [72B-MLX-2bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-2bit)<br/>[72B-MLX-4bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-4bit)<br/>[72B-MLX-8bit](https://www.modelscope.cn/models/okwinds/Qwen2.5-72B-Instruct-MLX-8bit) |
|
||
|
||
## 模型概述
|
||
|
||
> <span style="color:#FF9900;font-size:18px"> 本模型已升级，请使用精度更高的最新版本，原版本归档至 tag v1.0.0。 <br/>更多精度详情，请关注 Benchmark 部分数据。</span>
|
||
|
||
通义千问2.5-7B-Instruct-INT4-W4A16 是一个基于 通义千问2.5-7B-Instruct 的 INT4 量化校准模型。
|
||
|
||
- **模型名称:** Qwen2.5-7B-Instruct-Int4-W4A16
|
||
- **模型架构:** Qwen2.5
|
||
- **权重量化:** INT4
|
||
|
||
该模型通过将 [Qwen2.5-7B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-7B-Instruct) 的权重量化为 INT4 数据类型而实现。
|
||
|
||
量化过程将每个参数从 16bit 减少到 4bit，将模型占用磁盘空间大小，以及推理时加载模型需要的GPU显存空间，减少到了大约为原模型的1/4。量化过程中，只有 transformer 中的 Linear 层权重是量化的，其他层均保持为原模型数据类型，采用混合精度的计算方式，尽可能减少模型量化后所带来的精度损失。
|
||
在量化过程中，使用了[AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ)，采用 Symmetric group-wise 方式量化（group size : 128）。并在量化过程中，进行了数据校准，以提升模型生成精度，修复 gptq-int4 模型出现乱吐字和无限循环的问题。
|
||
|
||
> <span style="color: red;">注意：本模型需要 compute capability > 8.0（Ampere、Ada Lovelace、Hopper 架构）的 Nvidia GPU 来支持 INT4 **混合精度**计算。</span>
|
||
|
||
## 部署推理
|
||
|
||
#### 推荐使用 vLLM
|
||
|
||
Openai api 兼容模式
|
||
|
||
```bash
|
||
>>> vllm serve "/home/gavin/llm/Qwen2.5-7B-Instruct-Int4-W4A16" --host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.9 --served-model-name "Qwen2.5-7B-Instruct-Int4-W4A16"
|
||
```
|
||
|
||
## Benchmark
|
||
|
||
_仅 Linear layer 量化为 INT4 数据类型，其他层保持为原模型数据类型，采用混合精度计算的方式，以及通过综合校准数据集，有助于提升量化后模型的精度表现。_
|
||
|
||
<span style="color:#FF9900;">本模型的最新版本对精度进行了更广泛的强化提升。</span>
|
||
|
||
| Benchmark | Qwen2.5-7B-Instruct | Int4-W4A16-v1.0.0 | Recovery-v1.0.0 | _Int4-W4A16-master<span style="font-size: 10px;color:red">(New)</span>_ | Recovery-master<span style="font-size: 10px;color:red">(New)</span> |
|
||
| -------------------- | ------------------- | ----------------- | --------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------- |
|
||
| **Overall** | 70.69 | 69.47 | 98.27% | 70.34 | 99.51% |
|
||
| Exam | 77.90 | 76.77 | 98.55% | 76.67 | 98.42% |
|
||
| Language | 53.97 | 52.04 | 96.41% | 54.05 | 100.14% |
|
||
| Knowledge | 47.07 | 46.08 | 97.89% | 46.84 | 99.51% |
|
||
| Understanding | 89.74 | 88.89 | 99.05% | 89.28 | 99.49% |
|
||
| Coding | 79.32 | 77.41 | 97.59% | 79.90 | 100.73% |
|
||
| Reasoning | 69.22 | 67.31 | 97.24% | 68.53 | 99.01% |
|
||
| Instruct & Tools | 77.59 | 77.79 | 100.25% | 77.10 | 99.37% |
|
||
| **Exam** | - | - | - | - | - |
|
||
| C-EVAL | 78.52 | 77.24 | 98.37% | 77.51 | 98.71% |
|
||
| MMLU | 74.19 | 73.19 | 98.65% | 73.45 | 99.00% |
|
||
| CMMLU | 78.78 | 77.75 | 98.69% | 77.30 | 98.12% |
|
||
| GaokaoBench | 80.10 | 78.89 | 98.49% | 78.41 | 97.89% |
|
||
| **Language** | - | - | - | - | - |
|
||
| WiC | 55.80 | 53.61 | 96.08% | 57.21 | 102.53% |
|
||
| WSC | 70.19 | 66.35 | 94.53% | 69.23 | 98.63% |
|
||
| Afqmc-dev | 70.64 | 70.55 | 99.87% | 71.27 | 100.89% |
|
||
| Tydiqa Goldp | 19.26 | 17.63 | 91.54% | 18.48 | 95.95% |
|
||
| **Knowledge** | - | - | - | - | - |
|
||
| BoolQ | 84.92 | 82.66 | 97.34% | 84.53 | 99.54% |
|
||
| TriviaQA | 53.42 | 50.62 | 94.76% | 50.73 | 94.96% |
|
||
| GPQA Diamond | 31.82 | 34.34 | 107.92% | 35.35 | 111.09% |
|
||
| Natual Questions | 18.12 | 16.68 | 92.05% | 16.73 | 92.33% |
|
||
| **Understanding** | - | - | - | - | - |
|
||
| C3 | 91.51 | 90.41 | 98.80% | 91.51 | 100.00% |
|
||
| Race-middle | 91.09 | 90.18 | 99.00% | 90.18 | 99.00% |
|
||
| Race-high | 86.62 | 86.08 | 99.38% | 86.16 | 99.47% |
|
||
| **Coding** | - | - | - | - | - |
|
||
| Human Eval | 83.54 | 80.49 | 96.35% | 83.54 | 100.00% |
|
||
| Sanitized MBPP | 75.10 | 74.32 | 98.96% | 76.26 | 101.54% |
|
||
| **Reasoning** | - | - | - | - | - |
|
||
| COPA | 96.00 | 94.00 | 97.92% | 96.00 | 100.00% |
|
||
| MATH | 54.12 | 53.62 | 99.08% | 53.62 | 99.08% |
|
||
| GSM8K | 87.26 | 87.41 | 100.17% | 87.34 | 100.09% |
|
||
| Winogrande | 72.14 | 67.96 | 94.21% | 70.88 | 98.25% |
|
||
| Hellaswag | 85.38 | 84.36 | 98.81% | 84.50 | 98.97% |
|
||
| BBH | 70.12 | 64.79 | 92.40% | 68.36 | 97.49% |
|
||
| TheoremQA | 19.50 | 19.00 | 97.44% | 19.00 | 97.44% |
|
||
| **Instruct & Tools** | - | - | - | - | - |
|
||
| IFEval | 78.66 | 79.26 | 100.76% | 78.30 | 99.54% |
|
||
| TEval | 76.52 | 76.31 | 99.73% | 75.89 | 99.18% |
|
||
|
||
_以上 benchmark 以及量化工作均在 Nvidia 4090 RTX 24GB 显卡上完成。_
|
||
|
||
***并发说明***
|
||
|
||
- vLLM 0.5.5
|
||
|
||
- 原模型 BF16 峰值并发，在无 pending 的前提下，达到 running 90 左右。
|
||
|
||
- 量化后的本模型 INT4(W4A16) 峰值并发，在无 pending 的前提下，达到 running 225 左右。
|
||
|
||
- vLLM 0.6.1.post2
|
||
|
||
- 原模型 BF16 峰值并发，在无 pending 的前提下，达到 running 130 左右。
|
||
|
||
- 量化后的本模型 INT4(W4A16) 峰值并发，在无 pending 的前提下，达到 running 330 左右。
|
||
|
||
<br/>
|
||
<br/>
|
||
_______________________________
|
||
|
||
#### _作者在此 👇🏻 扫一扫_
|
||
|
||
<img src="https://www.modelscope.cn/models/okwinds/GPT-2/resolve/master/qrcode_for_jcl_258.jpg" /> |