98 lines
3.1 KiB
Markdown
98 lines
3.1 KiB
Markdown
---
|
||
license: qwen
|
||
license_name: Tongyi Qianwen LICENSE AGREEMENT
|
||
license_link: LICENSE
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- QwQ
|
||
- gptq
|
||
- int8
|
||
- 量化修复
|
||
- vLLM
|
||
- sglang
|
||
base_model:
|
||
- Qwen/QwQ-32B
|
||
base_model_relation: quantized
|
||
---
|
||
# 通义千问 QwQ-32B-GPTQ-Int8
|
||
基础型 [Qwen/QwQ-32B](https://www.modelscope.cn/models/Qwen/QwQ-32B)
|
||
|
||
### 【模型更新日期】
|
||
```
|
||
2025-04-24
|
||
1. add group 128v3
|
||
2. 用黑科技修掉之前 Compute 7 显卡（GPTQ格式）运行时候出现的感叹号问题
|
||
3. 删除对话模板的<think>字符，使得模型作答的时候，能完整吐出<think>模块
|
||
4. 提供一个无思考模式的对话模板（tokenizer_config-无思考模式.json）供参考把玩。
|
||
|
||
2025-03-07
|
||
1. add group 128
|
||
```
|
||
|
||
### 【依赖】
|
||
|
||
```
|
||
vllm==0.8.4
|
||
transformers==4.51.3
|
||
```
|
||
|
||
<div style="
|
||
background: rgba(255, 193, 61, 0.15);
|
||
padding: 16px;
|
||
border-radius: 6px;
|
||
border: 1px solid rgba(255, 165, 0, 0.3);
|
||
margin: 16px 0;
|
||
">
|
||
### 【💡新版 VLLM 注意事项💡】
|
||
|
||
#### 1. 需使用V0推理模式
|
||
启动vllm之前，先设置环境变量
|
||
```
|
||
export VLLM_USE_V1=0
|
||
```
|
||
</div>
|
||
|
||
### 【模型列表】
|
||
|
||
| tag | 文件大小 | 最近更新时间 |
|
||
|----------|--------|--------------|
|
||
| `g128v3` | `33GB` | `2025-04-24` |
|
||
| `g128` | `33GB` | `2025-03-07` |
|
||
|
||
|
||
### 【模型下载】
|
||
|
||
```python
|
||
from modelscope import snapshot_download
|
||
snapshot_download('tclf90/qwq-32b-gptq-int8', cache_dir="本地路径", revision='g128v3')
|
||
```
|
||
|
||
### 【修复内容】
|
||
|
||
1. 对GPTQ量化的校准做了额外优化；减少模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。
|
||
2. 支持1卡、2卡及4卡的`tensor-parallel-size`启动。
|
||
|
||
|
||
### 【介绍】
|
||
<a href="https://chat.qwenlm.ai/" target="_blank" style="margin: 2px;">
|
||
<img alt="Chat" src="https://img.shields.io/badge/%F0%9F%92%9C%EF%B8%8F%20Qwen%20Chat%20-536af5" style="display: inline-block; vertical-align: middle;"/>
|
||
</a>
|
||
|
||
## Introduction
|
||
|
||
QwQ is the reasoning model of the Qwen series. Compared with conventional instruction-tuned models, QwQ, which is capable of thinking and reasoning, can achieve significantly enhanced performance in downstream tasks, especially hard problems. QwQ-32B is the medium-sized reasoning model, which is capable of achieving competitive performance against state-of-the-art reasoning models, e.g., DeepSeek-R1, o1-mini.
|
||
|
||
**This repo contains the QwQ 32B model**, which has the following features:
|
||
- Type: Causal Language Models
|
||
- Training Stage: Pretraining & Post-training (Supervised Finetuning and Reinforcement Learning)
|
||
- Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
|
||
- Number of Parameters: 32.5B
|
||
- Number of Paramaters (Non-Embedding): 31.0B
|
||
- Number of Layers: 64
|
||
- Number of Attention Heads (GQA): 40 for Q and 8 for KV
|
||
- Context Length: Full 131,072 tokens
|
||
|
||
**Note:** For the best experience, please review the [usage guidelines](#usage-guidelines) before deploying QwQ models.
|
||
|
||
For more details, please refer to our [blog](https://qwenlm.github.io/blog/qwen2.5/), [GitHub](https://github.com/QwenLM/Qwen2.5), and [Documentation](https://qwen.readthedocs.io/en/latest/).
|