frameworks, license, tasks

frameworks	license	tasks
Pytorch	Apache License 2.0	text-generation

MS-LongWriter-Qwen2.5-7B-Instruct-GPTQ-Int4

🤖 [LongWriter Dataset] • 💻 [Github Repo] • 📃 [LongWriter Paper]

GPTQ INT4量化版本。

您可以通过ModelScope命令行来下载模型：

#安装ModelScope
pip install modelscope

使用command line下载：

modelscope download --model swift/MS-LongWriter-Qwen2.5-7B-Instruct-GPTQ-Int4 --local_dir ./your_path

您也可以下载整个repo, 参见模型下载页面。

以下为原版模型MS-LongWriter-Qwen2.5-7B-Instruct 的说明。

MS-LongWriter-Qwen2.5-7B-Instruct is trained based on https://modelscope.cn/models/qwen/Qwen2.5-7B-Instruct, and is capable of generating 10,000+ words at once.

MS-LongWriter-Qwen2.5-7B-Instruct begins training directly from the Qwen2.5-7B-Instruct, while performing significant distillation on the LongWriter-6k to obtain 666 high-quality samples, which is LongWriter-6k-filtered

Datasets

1. LongWriter-6k-filtered, based on the LongWriter-6k
2. Magpie-Qwen2-Pro-200K-Chinese , random sampling 6k examples.
3. Magpie-Qwen2-Pro-200K-English , random sampling 6k examples.

Model

We use ms-swift to fine-tune the Qwen2-7B-Instruct model.

1. Installation

pip install ms-swift[llm]

2. Fine-tuning

Envs:

Nvidia A100(80G) x 4

Run:

CUDA_VISIBLE_DEVICES=0,1,2,3 swift sft \
    --model_type qwen2_5-7b-instruct \
    --dataset longwriter-6k-filtered#666 qwen2-pro-zh#6660 qwen2-pro-en#6660 \
    --max_length 28672 \
    --num_train_epochs 2 \
    --eval_steps 200 \
    --batch_size 1 \
    --gradient_accumulation_steps 64 \
    --gradient_checkpointing true \
    --warmup_ratio 0.1 \
    --learning_rate 1e-5 \
    --sft_type full \
    --loss_name long-ce \
    --check_dataset_strategy warning \
    --save_only_model false \
    --save_total_limit -1 \
    --lazy_tokenize true \
    --dataloader_num_workers 1 \
    --resume_only_model true \
    --neftune_noise_alpha 5 \
    --use_flash_attn true

3. Fine-tuning with annealing

The annealing strategy is used to improve the performance of the model during the post-training process. We leverage the LongWriter-6k-filtered dataset to fine-tune the model with annealing, and set the learning rate to 2e-6. Run:

CUDA_VISIBLE_DEVICES=0,1,2,3 swift sft \
    --model_type qwen2_5-7b-instruct \
    --dataset longwriter-6k-filtered#666 \
    --max_length 28672 \
    --num_train_epochs 2 \
    --eval_steps 200 \
    --batch_size 1 \
    --gradient_accumulation_steps 64 \
    --gradient_checkpointing true \
    --warmup_ratio 0.1 \
    --learning_rate 2e-6 \
    --sft_type full \
    --loss_name long-ce \
    --check_dataset_strategy warning \
    --save_only_model false \
    --save_total_limit -1 \
    --lazy_tokenize true \
    --dataloader_num_workers 1 \
    --resume_only_model true \
    --neftune_noise_alpha 5 \
    --use_flash_attn true \
    --resume_from_checkpoint {previous-checkpoint-path}

4. GPTQ-Int4 Quantization

To run following command, export the quantization model with GPTQ-Int4.

Envs:

Nvidia A100(80G) x 1

Run:

CUDA_VISIBLE_DEVICES=0 swift export \
    --ckpt_dir {previous-checkpoint-path} \
    --quant_bits 4 \
    --load_dataset_config true --quant_method gptq

Note:

1. The --resume_from_checkpoint parameter is used to specify the path of the previous checkpoint. (see the step2)

Evaluation

Refer to LongWriter Evaluation from the EvalScope.

Reference

量子位文章：666条数据教会AI写万字长文！模型数据集都开源