license, license_name, license_link, pipeline_tag, tags, base_model, base_model_relation

license	license_name	license_link	pipeline_tag	tags	base_model	base_model_relation
qwen	Tongyi Qianwen LICENSE AGREEMENT	LICENSE	text-generation	QwQ gptq int8 量化修复 vLLM sglang	Qwen/QwQ-32B	quantized

通义千问 QwQ-32B-GPTQ-Int8

基础型 Qwen/QwQ-32B

【模型更新日期】

2025-04-24
1. add group 128v3
2. 用黑科技修掉之前 Compute 7 显卡（GPTQ格式）运行时候出现的感叹号问题
3. 删除对话模板的<think>字符，使得模型作答的时候，能完整吐出<think>模块
4. 提供一个无思考模式的对话模板（tokenizer_config-无思考模式.json）供参考把玩。

2025-03-07
1. add group 128

【依赖】

vllm==0.8.4
transformers==4.51.3

### 【💡新版 VLLM 注意事项💡】

1. 需使用V0推理模式

启动vllm之前，先设置环境变量

export VLLM_USE_V1=0

【模型列表】

tag	文件大小	最近更新时间
g128v3	33GB	2025-04-24
g128	33GB	2025-03-07

【模型下载】

from modelscope import snapshot_download
snapshot_download('tclf90/qwq-32b-gptq-int8', cache_dir="本地路径", revision='g128v3')

【修复内容】

1. 对GPTQ量化的校准做了额外优化；减少模型的 1.乱吐字、2.无限循环、3.长文能力丢失等情况。
2. 支持1卡、2卡及4卡的tensor-parallel-size启动。

【介绍】

Introduction

QwQ is the reasoning model of the Qwen series. Compared with conventional instruction-tuned models, QwQ, which is capable of thinking and reasoning, can achieve significantly enhanced performance in downstream tasks, especially hard problems. QwQ-32B is the medium-sized reasoning model, which is capable of achieving competitive performance against state-of-the-art reasoning models, e.g., DeepSeek-R1, o1-mini.

This repo contains the QwQ 32B model, which has the following features:

• Type: Causal Language Models
• Training Stage: Pretraining & Post-training (Supervised Finetuning and Reinforcement Learning)
• Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
• Number of Parameters: 32.5B
• Number of Paramaters (Non-Embedding): 31.0B
• Number of Layers: 64
• Number of Attention Heads (GQA): 40 for Q and 8 for KV
• Context Length: Full 131,072 tokens

Note: For the best experience, please review the usage guidelines before deploying QwQ models.

For more details, please refer to our blog, GitHub, and Documentation.