qwen2.5-14b-instruct-awq/README.md

license, license_name, license_link, pipeline_tag, tags

license	license_name	license_link	pipeline_tag	tags
qwen	Tongyi Qianwen LICENSE AGREEMENT	LICENSE	text-generation	qwen2.5 awq int4 量化修复 vLLM sglang

通义千问2.5-14B-Instruct-AWQ-量化修复

原模型 qwen/Qwen2.5-14B-Instruct

【模型更新日期】

注：通过snapshot_download函数传入revision=...来下载指定的tag版本

2024-09-24
1. add group 128

【模型列表】

tag	文件大小	最近更新时间
g128	9.4GB	2024-09-24

【模型下载】

from modelscope import snapshot_download
snapshot_download('tclf90/...', cache_dir="本地路径", revision='g128')

【修复内容】

1. 对GPTQ量化的校准做了额外优化；减少模型的 1.乱吐字、2.无限循环、3.长文能力丢失等情况。
2. 相同技术移植至AWQ。
3. 有些推理框架的默认top_k与top_p较大，可以考虑减小对应数值，来获得更合理的模型输出。

【介绍】

Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. Qwen2.5 brings the following improvements upon Qwen2:

• Significantly more knowledge and has greatly improved capabilities in coding and mathematics, thanks to our specialized expert models in these domains.
• Significant improvements in instruction following, generating long texts (over 8K tokens), understanding structured data (e.g, tables), and generating structured outputs especially JSON. More resilient to the diversity of system prompts, enhancing role-play implementation and condition-setting for chatbots.
• Long-context Support up to 128K tokens and can generate up to 8K tokens.
• Multilingual support for over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more.

For more details, please refer to our blog, GitHub, and Documentation.

【高并发RESTFul API推理】

方式1：vllm

方式2：sglang

目前推荐使用sglang进行部署，相较于vllm, sglang于A100实测，能有50%～100%的吞吐增益。