45 lines
1.3 KiB
Markdown
45 lines
1.3 KiB
Markdown
---
|
|
---
|
|
language:
|
|
- en
|
|
- zh
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- Awq
|
|
- int4
|
|
- yi1.5-34B-Chat
|
|
- pytorch
|
|
license: Apache 2.0
|
|
license_name: Apache 2.0
|
|
license_link: LICENSE
|
|
---
|
|
|
|
## About Quantization
|
|
我们使用modelscope [swift](https://github.com/modelscope/swift/)仓库进行AWQ 4bit量化. 量化文档可以查看[这里](https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E9%87%8F%E5%8C%96%E6%96%87%E6%A1%A3.md). 量化命令如下:
|
|
|
|
We use the modelscope [swift](https://github.com/modelscope/swift/) repository to perform AWQ 4bit quantization. Quantization documentation can be found [here](https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/LLM-quantization.md). The quantization command is as follows:
|
|
|
|
```bash
|
|
# Experimental Environment: A100
|
|
swift export \
|
|
--quant_bits 4 \
|
|
--model_type yi-1_5-34b-chat \
|
|
--quant_method awq \
|
|
--quant_n_samples 32 \
|
|
--dataset alpaca-zh alpaca-en sharegpt-gpt4-mini \
|
|
--quant_seqlen 4096
|
|
```
|
|
|
|
Inference:
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0 swift infer --model_type yi-1_5-34b-chat-awq-int4
|
|
```
|
|
|
|
SFT:
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0 swift sft --model_type yi-1_5-34b-chat-awq-int4 --dataset leetcode-python-en
|
|
```
|
|
|
|
Original Model:
|
|
|
|
[YI1.5-34B-Chat](https://modelscope.cn/models/01ai/Yi-1.5-34B-Chat/summary) |