147 lines
4.5 KiB
Markdown
147 lines
4.5 KiB
Markdown
---
|
||
frameworks:
|
||
- Pytorch
|
||
license: Apache License 2.0
|
||
tasks:
|
||
- text-generation
|
||
---
|
||
|
||
|
||
# MS-LongWriter-Qwen2.5-7B-Instruct-GPTQ-Int4
|
||
|
||
<p align="center">
|
||
🤖 <a href="https://modelscope.cn/datasets/swift/longwriter-6k-filtered" target="_blank">[LongWriter Dataset] </a> • 💻 <a href="https://github.com/THUDM/LongWriter" target="_blank">[Github Repo]</a> • 📃 <a href="https://arxiv.org/abs/2408.07055" target="_blank">[LongWriter Paper]</a>
|
||
</p>
|
||
|
||
GPTQ INT4量化版本。
|
||
|
||
您可以通过ModelScope命令行来下载模型：
|
||
```bash
|
||
#安装ModelScope
|
||
pip install modelscope
|
||
```
|
||
|
||
使用command line下载：
|
||
```bash
|
||
modelscope download --model swift/MS-LongWriter-Qwen2.5-7B-Instruct-GPTQ-Int4 --local_dir ./your_path
|
||
```
|
||
|
||
您也可以下载整个repo, 参见模型下载页面。
|
||
|
||
以下为原版模型MS-LongWriter-Qwen2.5-7B-Instruct 的说明。
|
||
|
||
MS-LongWriter-Qwen2.5-7B-Instruct is trained based on [https://modelscope.cn/models/qwen/Qwen2.5-7B-Instruct](https://modelscope.cn/models/qwen/Qwen2.5-7B-Instruct), and is capable of generating 10,000+ words at once.
|
||
|
||
MS-LongWriter-Qwen2.5-7B-Instruct begins training directly from the Qwen2.5-7B-Instruct, while performing significant distillation on the [LongWriter-6k](https://modelscope.cn/datasets/ZhipuAI/LongWriter-6k) to obtain 666 high-quality samples, which is [LongWriter-6k-filtered](https://modelscope.cn/datasets/swift/longwriter-6k-filtered)
|
||
|
||
|
||
## Datasets
|
||
1. [LongWriter-6k-filtered](https://modelscope.cn/datasets/swift/longwriter-6k-filtered), based on the [LongWriter-6k](https://modelscope.cn/datasets/ZhipuAI/LongWriter-6k)
|
||
2. [Magpie-Qwen2-Pro-200K-Chinese](https://modelscope.cn/datasets/AI-ModelScope/Magpie-Qwen2-Pro-200K-Chinese) , random sampling 6k examples.
|
||
3. [Magpie-Qwen2-Pro-200K-English](https://modelscope.cn/datasets/AI-ModelScope/Magpie-Qwen2-Pro-200K-English) , random sampling 6k examples.
|
||
|
||
|
||
## Model
|
||
|
||
We use [ms-swift](https://github.com/modelscope/swift) to fine-tune the Qwen2-7B-Instruct model.
|
||
|
||
1. Installation
|
||
```python
|
||
pip install ms-swift[llm]
|
||
```
|
||
|
||
2. Fine-tuning
|
||
|
||
Envs:
|
||
```text
|
||
Nvidia A100(80G) x 4
|
||
```
|
||
|
||
Run:
|
||
```shell
|
||
CUDA_VISIBLE_DEVICES=0,1,2,3 swift sft \
|
||
--model_type qwen2_5-7b-instruct \
|
||
--dataset longwriter-6k-filtered#666 qwen2-pro-zh#6660 qwen2-pro-en#6660 \
|
||
--max_length 28672 \
|
||
--num_train_epochs 2 \
|
||
--eval_steps 200 \
|
||
--batch_size 1 \
|
||
--gradient_accumulation_steps 64 \
|
||
--gradient_checkpointing true \
|
||
--warmup_ratio 0.1 \
|
||
--learning_rate 1e-5 \
|
||
--sft_type full \
|
||
--loss_name long-ce \
|
||
--check_dataset_strategy warning \
|
||
--save_only_model false \
|
||
--save_total_limit -1 \
|
||
--lazy_tokenize true \
|
||
--dataloader_num_workers 1 \
|
||
--resume_only_model true \
|
||
--neftune_noise_alpha 5 \
|
||
--use_flash_attn true
|
||
```
|
||
|
||
3. Fine-tuning with annealing
|
||
|
||
The annealing strategy is used to improve the performance of the model during the post-training process.
|
||
We leverage the LongWriter-6k-filtered dataset to fine-tune the model with annealing, and set the learning rate to 2e-6.
|
||
Run:
|
||
```shell
|
||
CUDA_VISIBLE_DEVICES=0,1,2,3 swift sft \
|
||
--model_type qwen2_5-7b-instruct \
|
||
--dataset longwriter-6k-filtered#666 \
|
||
--max_length 28672 \
|
||
--num_train_epochs 2 \
|
||
--eval_steps 200 \
|
||
--batch_size 1 \
|
||
--gradient_accumulation_steps 64 \
|
||
--gradient_checkpointing true \
|
||
--warmup_ratio 0.1 \
|
||
--learning_rate 2e-6 \
|
||
--sft_type full \
|
||
--loss_name long-ce \
|
||
--check_dataset_strategy warning \
|
||
--save_only_model false \
|
||
--save_total_limit -1 \
|
||
--lazy_tokenize true \
|
||
--dataloader_num_workers 1 \
|
||
--resume_only_model true \
|
||
--neftune_noise_alpha 5 \
|
||
--use_flash_attn true \
|
||
--resume_from_checkpoint {previous-checkpoint-path}
|
||
|
||
```
|
||
|
||
4. GPTQ-Int4 Quantization
|
||
|
||
To run following command, export the quantization model with GPTQ-Int4.
|
||
|
||
Envs:
|
||
```text
|
||
Nvidia A100(80G) x 1
|
||
```
|
||
|
||
Run:
|
||
```shell
|
||
CUDA_VISIBLE_DEVICES=0 swift export \
|
||
--ckpt_dir {previous-checkpoint-path} \
|
||
--quant_bits 4 \
|
||
--load_dataset_config true --quant_method gptq
|
||
```
|
||
|
||
|
||
|
||
Note:
|
||
1. The `--resume_from_checkpoint` parameter is used to specify the path of the previous checkpoint. (see the step2)
|
||
|
||
|
||
## Evaluation
|
||
|
||
Refer to [LongWriter Evaluation](https://github.com/modelscope/evalscope/tree/main/evalscope/third_party/longbench_write) from the [EvalScope](https://github.com/modelscope/evalscope).
|
||
|
||
|
||
## Reference
|
||
|
||
量子位文章：[666条数据教会AI写万字长文！模型数据集都开源](https://mp.weixin.qq.com/s/LvWUSgIRO5HI5YSDRz7SxA)
|