89 lines
2.0 KiB
Markdown
89 lines
2.0 KiB
Markdown
|
---
|
|||
|
|
license: qwen
|
|||
|
|
license_name: Tongyi Qianwen LICENSE AGREEMENT
|
|||
|
|
license_link: LICENSE
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- qwen2
|
|||
|
|
- gptq
|
|||
|
|
- int8
|
|||
|
|
- 量化修复
|
|||
|
|
- vLLM
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 通义千问2-14B-Merge-GPTQ-Int8-量化修复
|
|||
|
|
HF原模型 [paperplanedeemo/Qwen2-14B-merge](https://huggingface.co/paperplanedeemo/Qwen2-14B-merge)
|
|||
|
|
|
|||
|
|
|
|||
|
|
### 【模型更新日期】
|
|||
|
|
``` 2024-06-13 ```
|
|||
|
|
|
|||
|
|
### 【模型大小】
|
|||
|
|
`13.3GB`
|
|||
|
|
|
|||
|
|
### 【修复内容】
|
|||
|
|
|
|||
|
|
1. 对GPTQ量化的校准做了额外优化；减少量化模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。
|
|||
|
|
|
|||
|
|
### 【更新日志】
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
2024-06-13
|
|||
|
|
首次commit
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 【介绍】
|
|||
|
|
<i>注意：非官方模型</i>
|
|||
|
|
|
|||
|
|
Qwen2-14B-merge is a merge of the following models using [mergekit](https://github.com/cg123/mergekit):
|
|||
|
|
```yaml
|
|||
|
|
dtype: float16
|
|||
|
|
merge_method: passthrough
|
|||
|
|
slices:
|
|||
|
|
- sources:
|
|||
|
|
- layer_range: [0, 6]
|
|||
|
|
model: Qwen/Qwen2-7B-Instruct
|
|||
|
|
- sources:
|
|||
|
|
- layer_range: [3, 9]
|
|||
|
|
model: Qwen/Qwen2-7B-Instruct
|
|||
|
|
- sources:
|
|||
|
|
- layer_range: [6, 12]
|
|||
|
|
model: Qwen/Qwen2-7B-Instruct
|
|||
|
|
- sources:
|
|||
|
|
- layer_range: [9, 15]
|
|||
|
|
model: Qwen/Qwen2-7B-Instruct
|
|||
|
|
- sources:
|
|||
|
|
- layer_range: [12, 18]
|
|||
|
|
model: Qwen/Qwen2-7B-Instruct
|
|||
|
|
- sources:
|
|||
|
|
- layer_range: [15, 21]
|
|||
|
|
model: Qwen/Qwen2-7B-Instruct
|
|||
|
|
- sources:
|
|||
|
|
- layer_range: [18, 24]
|
|||
|
|
model: Qwen/Qwen2-7B-Instruct
|
|||
|
|
- sources:
|
|||
|
|
- layer_range: [21, 28]
|
|||
|
|
model: Qwen/Qwen2-7B-Instruct
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
[更多详情...](https://huggingface.co/paperplanedeemo/Qwen2-14B-merge)
|
|||
|
|
|
|||
|
|
### 【同期量化修复模型】
|
|||
|
|
待工作完成后补充...
|
|||
|
|
|
|||
|
|
### 【模型下载】
|
|||
|
|
```python
|
|||
|
|
from modelscope import snapshot_download
|
|||
|
|
model_dir = snapshot_download('tclf90/模型名', cache_dir="本地路径")
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 【[vLLM](https://github.com/vllm-project/vllm)推理（目前仅限Linux）】
|
|||
|
|
#### 1. Python 简易调试
|
|||
|
|
|
|||
|
|
待工作完成后补充...
|
|||
|
|
|
|||
|
|
#### 2. 类ChatGPT RESTFul API Server
|
|||
|
|
```
|
|||
|
|
>>> python -m vllm.entrypoints.openai.api_server --model 本地路径/tclf90/模型名称
|
|||
|
|
```
|