license, license_name, license_link, pipeline_tag, tags

license	license_name	license_link	pipeline_tag	tags
qwen	Tongyi Qianwen LICENSE AGREEMENT	LICENSE	text-generation	qwen2.5 gptq int8 量化修复 vLLM sglang

通义千问2.5-7B-Chat-GPTQ-Int8-量化修复

原模型 qwen/Qwen2.5-7B-Instruct

【模型更新日期】

注：通过snapshot_download函数传入revision=...来下载指定的tag版本

2024-09-23 tag g128v2
1. 减少长文时吐字重复与消失的情况

2024-09-23 tag g32v2
1. 减少长文时吐字重复与消失的情况

2024-09-22 tag g128
1. add group 128

2024-09-22 tag g32
1. add group 32

【模型列表】

tag	文件大小	最近更新时间
g128v2	8.3GB	2024-09-23
g32v2	8.7GB	2024-09-23
g128	8.3GB	2024-09-23
g32	8.7GB	2024-09-22

from modelscope import snapshot_download
snapshot_download('tclf90/qwen2.5-7b-instruct-gptq-int8', cache_dir="本地路径", revision='g128v2')
snapshot_download('tclf90/qwen2.5-7b-instruct-gptq-int8', cache_dir="本地路径", revision='g32v2')

【修复内容】

1. 对GPTQ量化的校准做了额外优化；减少模型的 1.乱吐字、2.无限循环、3.长文能力丢失等情况。
2. 有些推理框架的默认top_k与top_p较大，可以考虑减小对应数值，来获得更合理的模型输出。
3. 根据模型实际情况，可以支持1卡、2卡及4卡的tensor-parallel-size启动。

【介绍】

Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. Qwen2.5 brings the following improvements upon Qwen2:

• Significantly more knowledge and has greatly improved capabilities in coding and mathematics, thanks to our specialized expert models in these domains.
• Significant improvements in instruction following, generating long texts (over 8K tokens), understanding structured data (e.g, tables), and generating structured outputs especially JSON. More resilient to the diversity of system prompts, enhancing role-play implementation and condition-setting for chatbots.
• Long-context Support up to 128K tokens and can generate up to 8K tokens.
• Multilingual support for over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more.

For more details, please refer to our blog, GitHub, and Documentation.

【模型下载】

from modelscope import snapshot_download
snapshot_download('tclf90/模型名', cache_dir="本地路径", revision='...tag...')

【高并发RESTFul API推理】

方式1：vllm

方式2：sglang

目前推荐使用sglang进行部署，相较于vllm, sglang于A100实测，能有50%～100%的吞吐增益。