65 lines
2.5 KiB
Markdown
65 lines
2.5 KiB
Markdown
|
---
|
|||
|
|
license: qwen
|
|||
|
|
license_name: Tongyi Qianwen LICENSE AGREEMENT
|
|||
|
|
license_link: LICENSE
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- qwen2.5
|
|||
|
|
- awq
|
|||
|
|
- int4
|
|||
|
|
- 量化修复
|
|||
|
|
- vLLM
|
|||
|
|
- sglang
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 通义千问2.5-14B-Instruct-AWQ-量化修复
|
|||
|
|
原模型 [qwen/Qwen2.5-14B-Instruct](https://www.modelscope.cn/models/qwen/Qwen2.5-14B-Instruct)
|
|||
|
|
|
|||
|
|
|
|||
|
|
### 【模型更新日期】
|
|||
|
|
|
|||
|
|
注：通过`snapshot_download`函数传入`revision=...`来下载指定的`tag`版本
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
2024-09-24
|
|||
|
|
1. add group 128
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 【模型列表】
|
|||
|
|
|
|||
|
|
| tag | 文件大小 | 最近更新时间 |
|
|||
|
|
|--------|---------|--------------|
|
|||
|
|
| `g128` | `9.4GB` | `2024-09-24` |
|
|||
|
|
|
|||
|
|
|
|||
|
|
### 【模型下载】
|
|||
|
|
```python
|
|||
|
|
from modelscope import snapshot_download
|
|||
|
|
snapshot_download('tclf90/...', cache_dir="本地路径", revision='g128')
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 【修复内容】
|
|||
|
|
|
|||
|
|
1. 对GPTQ量化的校准做了额外优化；减少模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。
|
|||
|
|
2. 相同技术移植至AWQ。
|
|||
|
|
3. 有些推理框架的默认`top_k`与`top_p`较大，可以考虑减小对应数值，来获得更合理的模型输出。
|
|||
|
|
|
|||
|
|
|
|||
|
|
### 【介绍】
|
|||
|
|
Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. Qwen2.5 brings the following improvements upon Qwen2:
|
|||
|
|
|
|||
|
|
- Significantly **more knowledge** and has greatly improved capabilities in **coding** and **mathematics**, thanks to our specialized expert models in these domains.
|
|||
|
|
- Significant improvements in **instruction following**, **generating long texts** (over 8K tokens), **understanding structured data** (e.g, tables), and **generating structured outputs** especially JSON. **More resilient to the diversity of system prompts**, enhancing role-play implementation and condition-setting for chatbots.
|
|||
|
|
- **Long-context Support** up to 128K tokens and can generate up to 8K tokens.
|
|||
|
|
- **Multilingual support** for over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more.
|
|||
|
|
|
|||
|
|
For more details, please refer to our [blog](https://qwenlm.github.io/blog/qwen2.5/), [GitHub](https://github.com/QwenLM/Qwen2.5), and [Documentation](https://qwen.readthedocs.io/en/latest/).
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
### 【高并发RESTFul API推理】
|
|||
|
|
方式1：[vllm](https://github.com/vllm-project/vllm)
|
|||
|
|
|
|||
|
|
方式2：[sglang](https://github.com/sgl-project/sglang)
|
|||
|
|
|
|||
|
|
目前推荐使用sglang进行部署，相较于vllm, sglang于A100实测，能有50%～100%的吞吐增益。
|