97 lines
2.0 KiB
Markdown
97 lines
2.0 KiB
Markdown
|
---
|
|||
|
|
frameworks:
|
|||
|
|
- Pytorch
|
|||
|
|
license: Apache License 2.0
|
|||
|
|
tasks:
|
|||
|
|
- chatbot
|
|||
|
|
|
|||
|
|
model-type:
|
|||
|
|
- qwen
|
|||
|
|
|
|||
|
|
domain:
|
|||
|
|
- nlp
|
|||
|
|
|
|||
|
|
language:
|
|||
|
|
- zh
|
|||
|
|
|
|||
|
|
metrics:
|
|||
|
|
- CIDEr
|
|||
|
|
- BLEU
|
|||
|
|
- ROUGE
|
|||
|
|
|
|||
|
|
tags:
|
|||
|
|
- instruction-tuned
|
|||
|
|
- lora
|
|||
|
|
- 4-bit
|
|||
|
|
- quantized
|
|||
|
|
- 聊天
|
|||
|
|
- 猫娘
|
|||
|
|
|
|||
|
|
tools:
|
|||
|
|
- llamafactory
|
|||
|
|
|
|||
|
|
base_model:
|
|||
|
|
- Qwen/Qwen3-1.7B-Base
|
|||
|
|
base_model_relation: finetune
|
|||
|
|
datasets:
|
|||
|
|
- NaClNeZn/Shizuku-DataSet
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Qwen3-1.7B-QLoRA-Shizuku-Instruct
|
|||
|
|
|
|||
|
|
> 基于 Qwen3-1.7B-Base 的高质量指令微调模型，适用于中文对话与推理任务。
|
|||
|
|
|
|||
|
|
## 📌 模型介绍
|
|||
|
|
|
|||
|
|
本模型基于 **Qwen3-1.7B-Base**，使用 **QLoRA** 方法在 `Shizuku-dataset`(v1.0) 数据集上进行指令微调（SFT），专注于提升模型在中文对话、逻辑推理和指令遵循方面的能力。通过 4-bit 量化与 LoRA 低秩适配技术，在保持轻量级的同时显著提升任务表现。
|
|||
|
|
|
|||
|
|
## 🛠 微调配置
|
|||
|
|
|
|||
|
|
- **基座模型**: Qwen3-1.7B-Base
|
|||
|
|
- **微调方法**: QLoRA (LoRA + 4-bit Quantization)
|
|||
|
|
- **量化方式**: BitsAndBytes 4-bit, Double Quantization
|
|||
|
|
- **LoRA 配置**:
|
|||
|
|
- rank: 16
|
|||
|
|
- alpha: 32
|
|||
|
|
- dropout: 0.2
|
|||
|
|
- target modules: `q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj`
|
|||
|
|
- **训练方式**: SFT（监督微调）
|
|||
|
|
- **启用特性**:
|
|||
|
|
- NEFTune（noise_alpha=5）
|
|||
|
|
- Gradient Checkpointing
|
|||
|
|
- Thinking Mode（enable_thinking=True）
|
|||
|
|
- 长上下文支持（cutoff_len=4096）
|
|||
|
|
|
|||
|
|
## 📊 训练数据
|
|||
|
|
|
|||
|
|
- **数据集**: Shizuku-dataset(v1.0)
|
|||
|
|
- **数据量**: 最多 100,000 条样本（max_samples）
|
|||
|
|
- **验证集比例**: 20%
|
|||
|
|
- **模板**: `qwen` 模板（适配 Qwen 系列）
|
|||
|
|
|
|||
|
|
## 🏋️ 训练参数
|
|||
|
|
|
|||
|
|
- **训练轮数**: 3.0 epochs
|
|||
|
|
- **学习率**: 2e-4（cosine 调度）
|
|||
|
|
- **Batch Size**: per_device_train_batch_size=2, gradient_accumulation_steps=8 → 实际 batch size = 16
|
|||
|
|
- **优化器**: `adamw_8bit`
|
|||
|
|
- **最大长度**: 4096
|
|||
|
|
- **精度**: fp16
|
|||
|
|
|
|||
|
|
## 🚀 使用方式
|
|||
|
|
|
|||
|
|
下载项目
|
|||
|
|
```shell
|
|||
|
|
git lfs install
|
|||
|
|
git clone https://www.modelscope.cn/NaClNeZn/Qwen3-1.7B-QLoRA-Shizuku-v1.git
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
安装环境
|
|||
|
|
```shell
|
|||
|
|
uv sync
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
运行模型
|
|||
|
|
```shell
|
|||
|
|
uv run run.py
|
|||
|
|
```
|