98 lines
3.1 KiB
Markdown
98 lines
3.1 KiB
Markdown
|
---
|
|||
|
|
license: qwen
|
|||
|
|
license_name: Tongyi Qianwen LICENSE AGREEMENT
|
|||
|
|
license_link: LICENSE
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- QwQ
|
|||
|
|
- gptq
|
|||
|
|
- int8
|
|||
|
|
- 量化修复
|
|||
|
|
- vLLM
|
|||
|
|
- sglang
|
|||
|
|
base_model:
|
|||
|
|
- Qwen/QwQ-32B
|
|||
|
|
base_model_relation: quantized
|
|||
|
|
---
|
|||
|
|
# 通义千问 QwQ-32B-GPTQ-Int8
|
|||
|
|
基础型 [Qwen/QwQ-32B](https://www.modelscope.cn/models/Qwen/QwQ-32B)
|
|||
|
|
|
|||
|
|
### 【模型更新日期】
|
|||
|
|
```
|
|||
|
|
2025-04-24
|
|||
|
|
1. add group 128v3
|
|||
|
|
2. 用黑科技修掉之前 Compute 7 显卡（GPTQ格式）运行时候出现的感叹号问题
|
|||
|
|
3. 删除对话模板的<think>字符，使得模型作答的时候，能完整吐出<think>模块
|
|||
|
|
4. 提供一个无思考模式的对话模板（tokenizer_config-无思考模式.json）供参考把玩。
|
|||
|
|
|
|||
|
|
2025-03-07
|
|||
|
|
1. add group 128
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 【依赖】
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
vllm==0.8.4
|
|||
|
|
transformers==4.51.3
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
<div style="
|
|||
|
|
background: rgba(255, 193, 61, 0.15);
|
|||
|
|
padding: 16px;
|
|||
|
|
border-radius: 6px;
|
|||
|
|
border: 1px solid rgba(255, 165, 0, 0.3);
|
|||
|
|
margin: 16px 0;
|
|||
|
|
">
|
|||
|
|
### 【💡新版 VLLM 注意事项💡】
|
|||
|
|
|
|||
|
|
#### 1. 需使用V0推理模式
|
|||
|
|
启动vllm之前，先设置环境变量
|
|||
|
|
```
|
|||
|
|
export VLLM_USE_V1=0
|
|||
|
|
```
|
|||
|
|
</div>
|
|||
|
|
|
|||
|
|
### 【模型列表】
|
|||
|
|
|
|||
|
|
| tag | 文件大小 | 最近更新时间 |
|
|||
|
|
|----------|--------|--------------|
|
|||
|
|
| `g128v3` | `33GB` | `2025-04-24` |
|
|||
|
|
| `g128` | `33GB` | `2025-03-07` |
|
|||
|
|
|
|||
|
|
|
|||
|
|
### 【模型下载】
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from modelscope import snapshot_download
|
|||
|
|
snapshot_download('tclf90/qwq-32b-gptq-int8', cache_dir="本地路径", revision='g128v3')
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 【修复内容】
|
|||
|
|
|
|||
|
|
1. 对GPTQ量化的校准做了额外优化；减少模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。
|
|||
|
|
2. 支持1卡、2卡及4卡的`tensor-parallel-size`启动。
|
|||
|
|
|
|||
|
|
|
|||
|
|
### 【介绍】
|
|||
|
|
<a href="https://chat.qwenlm.ai/" target="_blank" style="margin: 2px;">
|
|||
|
|
<img alt="Chat" src="https://img.shields.io/badge/%F0%9F%92%9C%EF%B8%8F%20Qwen%20Chat%20-536af5" style="display: inline-block; vertical-align: middle;"/>
|
|||
|
|
</a>
|
|||
|
|
|
|||
|
|
## Introduction
|
|||
|
|
|
|||
|
|
QwQ is the reasoning model of the Qwen series. Compared with conventional instruction-tuned models, QwQ, which is capable of thinking and reasoning, can achieve significantly enhanced performance in downstream tasks, especially hard problems. QwQ-32B is the medium-sized reasoning model, which is capable of achieving competitive performance against state-of-the-art reasoning models, e.g., DeepSeek-R1, o1-mini.
|
|||
|
|
|
|||
|
|
**This repo contains the QwQ 32B model**, which has the following features:
|
|||
|
|
- Type: Causal Language Models
|
|||
|
|
- Training Stage: Pretraining & Post-training (Supervised Finetuning and Reinforcement Learning)
|
|||
|
|
- Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
|
|||
|
|
- Number of Parameters: 32.5B
|
|||
|
|
- Number of Paramaters (Non-Embedding): 31.0B
|
|||
|
|
- Number of Layers: 64
|
|||
|
|
- Number of Attention Heads (GQA): 40 for Q and 8 for KV
|
|||
|
|
- Context Length: Full 131,072 tokens
|
|||
|
|
|
|||
|
|
**Note:** For the best experience, please review the [usage guidelines](#usage-guidelines) before deploying QwQ models.
|
|||
|
|
|
|||
|
|
For more details, please refer to our [blog](https://qwenlm.github.io/blog/qwen2.5/), [GitHub](https://github.com/QwenLM/Qwen2.5), and [Documentation](https://qwen.readthedocs.io/en/latest/).
|