---
license: qwen
license_name: Tongyi Qianwen LICENSE AGREEMENT
license_link: LICENSE
pipeline_tag: text-generation
tags:
- qwen2.5
- gptq
- int8
- 量化修复
- vLLM
- sglang
---

# 通义千问2.5-7B-Chat-GPTQ-Int8-量化修复
原模型 [qwen/Qwen2.5-7B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-7B-Instruct)


### 【模型更新日期】

注：通过`snapshot_download`函数传入`revision=...`来下载指定的`tag`版本

``` 
2024-09-23 tag g128v2
1. 减少长文时吐字重复与消失的情况

2024-09-23 tag g32v2
1. 减少长文时吐字重复与消失的情况

2024-09-22 tag g128
1. add group 128

2024-09-22 tag g32
1. add group 32
```

### 【模型列表】

| tag      | 文件大小    | 最近更新时间       |
|----------|---------|--------------|
| `g128v2` | `8.3GB` | `2024-09-23` |
| `g32v2`  | `8.7GB` | `2024-09-23` |
| `g128`   | `8.3GB` | `2024-09-23` |
| `g32`    | `8.7GB` | `2024-09-22` |

```python
from modelscope import snapshot_download
snapshot_download('tclf90/qwen2.5-7b-instruct-gptq-int8', cache_dir="本地路径", revision='g128v2')
snapshot_download('tclf90/qwen2.5-7b-instruct-gptq-int8', cache_dir="本地路径", revision='g32v2')
```

### 【修复内容】

1. 对GPTQ量化的校准做了额外优化；减少模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。
2. 有些推理框架的默认`top_k`与`top_p`较大，可以考虑减小对应数值，来获得更合理的模型输出。
3. 根据模型实际情况，可以支持1卡、2卡及4卡的`tensor-parallel-size`启动。


### 【介绍】
Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. Qwen2.5 brings the following improvements upon Qwen2:

- Significantly **more knowledge** and has greatly improved capabilities in **coding** and **mathematics**, thanks to our specialized expert models in these domains.
- Significant improvements in **instruction following**, **generating long texts** (over 8K tokens), **understanding structured data** (e.g, tables), and **generating structured outputs** especially JSON. **More resilient to the diversity of system prompts**, enhancing role-play implementation and condition-setting for chatbots.
- **Long-context Support** up to 128K tokens and can generate up to 8K tokens.
- **Multilingual support** for over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more.

For more details, please refer to our [blog](https://qwenlm.github.io/blog/qwen2.5/), [GitHub](https://github.com/QwenLM/Qwen2.5), and [Documentation](https://qwen.readthedocs.io/en/latest/).


### 【模型下载】
```python
from modelscope import snapshot_download
snapshot_download('tclf90/模型名', cache_dir="本地路径", revision='...tag...')
```

### 【高并发RESTFul API推理】
方式1：[vllm](https://github.com/vllm-project/vllm)

方式2：[sglang](https://github.com/sgl-project/sglang)

目前推荐使用sglang进行部署，相较于vllm, sglang于A100实测，能有50%～100%的吞吐增益。