89 lines
2.0 KiB
Markdown
89 lines
2.0 KiB
Markdown
---
|
||
license: qwen
|
||
license_name: Tongyi Qianwen LICENSE AGREEMENT
|
||
license_link: LICENSE
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- qwen2
|
||
- gptq
|
||
- int8
|
||
- 量化修复
|
||
- vLLM
|
||
---
|
||
|
||
# 通义千问2-14B-Merge-GPTQ-Int8-量化修复
|
||
HF原模型 [paperplanedeemo/Qwen2-14B-merge](https://huggingface.co/paperplanedeemo/Qwen2-14B-merge)
|
||
|
||
|
||
### 【模型更新日期】
|
||
``` 2024-06-13 ```
|
||
|
||
### 【模型大小】
|
||
`13.3GB`
|
||
|
||
### 【修复内容】
|
||
|
||
1. 对GPTQ量化的校准做了额外优化；减少量化模型的 `1.乱吐字`、`2.无限循环`、`3.长文能力丢失`等情况。
|
||
|
||
### 【更新日志】
|
||
|
||
```
|
||
2024-06-13
|
||
首次commit
|
||
```
|
||
|
||
### 【介绍】
|
||
<i>注意：非官方模型</i>
|
||
|
||
Qwen2-14B-merge is a merge of the following models using [mergekit](https://github.com/cg123/mergekit):
|
||
```yaml
|
||
dtype: float16
|
||
merge_method: passthrough
|
||
slices:
|
||
- sources:
|
||
- layer_range: [0, 6]
|
||
model: Qwen/Qwen2-7B-Instruct
|
||
- sources:
|
||
- layer_range: [3, 9]
|
||
model: Qwen/Qwen2-7B-Instruct
|
||
- sources:
|
||
- layer_range: [6, 12]
|
||
model: Qwen/Qwen2-7B-Instruct
|
||
- sources:
|
||
- layer_range: [9, 15]
|
||
model: Qwen/Qwen2-7B-Instruct
|
||
- sources:
|
||
- layer_range: [12, 18]
|
||
model: Qwen/Qwen2-7B-Instruct
|
||
- sources:
|
||
- layer_range: [15, 21]
|
||
model: Qwen/Qwen2-7B-Instruct
|
||
- sources:
|
||
- layer_range: [18, 24]
|
||
model: Qwen/Qwen2-7B-Instruct
|
||
- sources:
|
||
- layer_range: [21, 28]
|
||
model: Qwen/Qwen2-7B-Instruct
|
||
```
|
||
|
||
[更多详情...](https://huggingface.co/paperplanedeemo/Qwen2-14B-merge)
|
||
|
||
### 【同期量化修复模型】
|
||
待工作完成后补充...
|
||
|
||
### 【模型下载】
|
||
```python
|
||
from modelscope import snapshot_download
|
||
model_dir = snapshot_download('tclf90/模型名', cache_dir="本地路径")
|
||
```
|
||
|
||
### 【[vLLM](https://github.com/vllm-project/vllm)推理（目前仅限Linux）】
|
||
#### 1. Python 简易调试
|
||
|
||
待工作完成后补充...
|
||
|
||
#### 2. 类ChatGPT RESTFul API Server
|
||
```
|
||
>>> python -m vllm.entrypoints.openai.api_server --model 本地路径/tclf90/模型名称
|
||
```
|