MS-LongWriter-Qwen2-7B-Instruct/README.md

language, library_name, tags

language	library_name	tags
en zh	transformers	Long Context qwen2

MS-LongWriter-Qwen2-7B-Instruct

🤖 [LongWriter Dataset] • 💻 [Github Repo] • 📃 [LongWriter Paper]

MS-LongWriter-Qwen2-7B-Instruct is trained based on https://modelscope.cn/models/qwen/Qwen2-7B-Instruct, and is capable of generating 10,000+ words at once.

MS-LongWriter-Qwen2-7B-Instruct begins training directly from the Qwen2-7B-Instruct, while performing significant distillation on the LongWriter-6k to obtain 666 high-quality samples.

Datasets

1. LongWriter-6k-filtered, which is based on the LongWriter-6k
2. Magpie-Qwen2-Pro-200K-Chinese , random sampling 6k examples.
3. Magpie-Qwen2-Pro-200K-English , random sampling 6k examples.

Model

We use ms-swift to fine-tune the Qwen2-7B-Instruct model.

1. Installation

pip install ms-swift[llm]

2. Fine-tuning

Envs:

Nvidia A100(80G) x 4

Run:

swift sft \
    --model_type qwen2-7b-instruct \
    --dataset longwriter-6k-filtered#666 qwen2-pro-zh#6660 qwen2-pro-en#6660 \
    --max_length 28672 \
    --num_train_epochs 2 \
    --eval_steps 200 \
    --batch_size 1 \
    --gradient_accumulation_steps 64 \
    --gradient_checkpointing true \
    --warmup_ratio 0.1 \
    --learning_rate 1e-5 \
    --sft_type full \
    --loss_name long-ce \
    --check_dataset_strategy warning \
    --save_only_model false \
    --save_total_limit -1 \
    --lazy_tokenize true \
    --dataloader_num_workers 1 \
    --resume_only_model true \
    --neftune_noise_alpha 5 \
    --use_flash_attn true

Evaluation

Refer to LongWriter Evaluation from the EvalScope.