42 lines
2.3 KiB
Markdown
42 lines
2.3 KiB
Markdown
|
---
|
|||
|
|
license: other
|
|||
|
|
license_name: tongyi-qianwen
|
|||
|
|
license_link: >-
|
|||
|
|
https://huggingface.co/Qwen/Qwen1.5-32B-Chat/blob/main/LICENSE
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- qwen
|
|||
|
|
- 32b
|
|||
|
|
- gptq
|
|||
|
|
- int4
|
|||
|
|
- chat
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Qwen1.5-32B-Chat-GPTQ Int4 模型量化
|
|||
|
|
|
|||
|
|
### 【模型更新日期】
|
|||
|
|
``` 2024-05-30 ```
|
|||
|
|
|
|||
|
|
重制并补全generation_config.json文件
|
|||
|
|
|
|||
|
|
### 【介绍】
|
|||
|
|
千问官方`Qwen1.5-32B-Chat`量化模型存在一定程度的量化损失，并导致例如无限`!!!!!!`感叹号等问题。这里集中做一次修复。
|
|||
|
|
|
|||
|
|
| 模型名 | 量化类型 | 量化比特 | 其余量化参数 | 评价 |
|
|||
|
|
|-----------------------------------------------------------------------------------------------------------------------|------|------|----------|-------------------------------------------------------------------------------|
|
|||
|
|
| [tclf90/Qwen1.5-32B-Chat-GPTQ-Int4-Repair](https://www.modelscope.cn/models/tclf90/Qwen1.5-32B-Chat-GPTQ-Int4-Repair) | GPTQ | int4 | group128 | <b>测试基线</b><br/>无明显的、量化造成的对话失真 |
|
|||
|
|
| [tclf90/Qwen1.5-32B-Chat-GPTQ-Int3](https://www.modelscope.cn/models/tclf90/Qwen1.5-32B-Chat-GPTQ-Int3) | GPTQ | int3 | group128 | <b>非常推荐</b><br/>对话、指令能力与其余量化版本，没有明显差别，但模型体积最小、占用显存最少，可以吞吐更长上下文，推理速度也比int4快10% |
|
|||
|
|
| [tclf90/Qwen1.5-32B-Chat-AWQ-G32](https://www.modelscope.cn/models/tclf90/Qwen1.5-32B-Chat-AWQ-G32) | AWQ | int4 | group32 | <b>没有特别亮点</b><br/>尝试用更小的group尺寸来减少量化损失，但在`Qwen1.5-32B-Chat`这个模型上没有表现出区别 |
|
|||
|
|
|
|||
|
|
<i>注：经测试，`Qwen1.5-32B-Chat` 的Int2量化，会胡言乱语，完全失去模型能力。</i>
|
|||
|
|
|
|||
|
|
|
|||
|
|
### 【引用】
|
|||
|
|
|
|||
|
|
原模型 [Qwen1.5-32B-Chat](https://www.modelscope.cn/models/qwen/Qwen1.5-32B-Chat)
|
|||
|
|
|
|||
|
|
官方量化模型 [Qwen1.5-32B-Chat-GPTQ-Int4
|
|||
|
|
](https://www.modelscope.cn/models/qwen/Qwen1.5-32B-Chat-GPTQ-Int4)
|
|||
|
|
|
|||
|
|
量化工具 [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ)
|