117 lines
3.2 KiB
Markdown
117 lines
3.2 KiB
Markdown
|
---
|
|||
|
|
language: zh
|
|||
|
|
license: apache-2.0
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- qwen3
|
|||
|
|
- chinese-novels
|
|||
|
|
- creative-writing
|
|||
|
|
- full-sft
|
|||
|
|
base_model: Qwen/Qwen3-8B
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Qwen3-8B Novel Base SFT
|
|||
|
|
|
|||
|
|
基于 Qwen3-8B 全参微调，用 669 本中文网络小说训练（约 7200 万字符）。
|
|||
|
|
|
|||
|
|
覆盖类型：言情、耽美、都市言情、玄幻、网游。
|
|||
|
|
|
|||
|
|
|
|||
|
|
项目地址：https://github.com/lin1753/novel2galgame
|
|||
|
|
|
|||
|
|
训练代码仓库：https://github.com/lin1753/novel-agent
|
|||
|
|
|
|||
|
|
## 用途
|
|||
|
|
|
|||
|
|
**All Novel Can Be Galgame** 工作台的 Stage1 基座模型。
|
|||
|
|
|
|||
|
|
学习中文小说的叙事风格和角色对话模式，作为下游 LoRA adapter 的基座，执行三类 Agent 任务：
|
|||
|
|
|
|||
|
|
- **Agent 1**: `mikuhhn1239/qwen3-8b-narrative-parsing-lora` — 叙事单元分类
|
|||
|
|
- **Agent 2**: `mikuhhn1239/qwen3-8b-scene-segmentation-lora` — 场景边界检测
|
|||
|
|
- **Agent 3**: `mikuhhn1239/qwen3-8b-attribution-assist-lora` — 角色归因
|
|||
|
|
|
|||
|
|
## 加载
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
"mikuhhn1239/qwen3-8b-novel-base-sft",
|
|||
|
|
torch_dtype="auto",
|
|||
|
|
device_map="auto",
|
|||
|
|
)
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(
|
|||
|
|
"mikuhhn1239/qwen3-8b-novel-base-sft"
|
|||
|
|
)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
4-bit 量化（适用于 8GB 显存）：
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import BitsAndBytesConfig
|
|||
|
|
|
|||
|
|
bnb_config = BitsAndBytesConfig(
|
|||
|
|
load_in_4bit=True,
|
|||
|
|
bnb_4bit_compute_dtype="float16",
|
|||
|
|
bnb_4bit_quant_type="nf4",
|
|||
|
|
)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
"mikuhhn1239/qwen3-8b-novel-base-sft",
|
|||
|
|
quantization_config=bnb_config,
|
|||
|
|
device_map="auto",
|
|||
|
|
)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## 训练数据
|
|||
|
|
|
|||
|
|
| 文件 | 条数 | 说明 |
|
|||
|
|
|------|:---:|------|
|
|||
|
|
| `continuation.jsonl` | 36,092 | 续写：给前半段→续后半段 |
|
|||
|
|
| `instruction.jsonl` | 36,481 | 指令式续写 |
|
|||
|
|
| **合计** | **72,573** | |
|
|||
|
|
|
|||
|
|
格式: ChatML `[system, user, assistant]`
|
|||
|
|
|
|||
|
|
## 训练详情
|
|||
|
|
|
|||
|
|
### 调试历程
|
|||
|
|
|
|||
|
|
全参微调 8B 模型内存压力大，经历多轮调试：
|
|||
|
|
|
|||
|
|
| # | 问题 | 原因 | 解决 |
|
|||
|
|
|---|------|------|------|
|
|||
|
|
| 1 | 单卡 OOM (78G/80G) | optimizer+grads+model≈116G/卡 | 上 4 卡 DDP |
|
|||
|
|
| 2 | 4 卡 DDP 仍 OOM (79G/80G) | DDP 每卡存完整 AdamW 状态(66G) | 加 DeepSpeed ZeRO-2 |
|
|||
|
|
| 3 | ZeRO-2 backward OOM | 内存碎片 | `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` |
|
|||
|
|
| 4 | batch=2 预估 26 小时 | 序列太长 | seq_len 4096→2048, epochs 3→2, batch 2→4 |
|
|||
|
|
|
|||
|
|
### 最终超参
|
|||
|
|
|
|||
|
|
| 参数 | 值 |
|
|||
|
|
|------|-----|
|
|||
|
|
| 方法 | 全参数 SFT |
|
|||
|
|
| 有效 batch size | 64 (4 GPU × 4 batch × 4 accumulation) |
|
|||
|
|
| 学习率 | 2e-5 |
|
|||
|
|
| 优化器 | AdamW (adamw_torch_fused) |
|
|||
|
|
| warmup | 3% |
|
|||
|
|
| scheduler | linear decay |
|
|||
|
|
| 精度 | bf16 |
|
|||
|
|
| 最大序列长度 | 2048 |
|
|||
|
|
| epochs | 2 |
|
|||
|
|
| gradient checkpointing | True |
|
|||
|
|
| 分布式 | DeepSpeed ZeRO-2 (4×A800 80GB) |
|
|||
|
|
|
|||
|
|
### 训练结果
|
|||
|
|
|
|||
|
|
- 耗时: ~9 小时
|
|||
|
|
- Loss: 3.36 → 2.47
|
|||
|
|
- 产物: 16GB (4 个 safetensors 分片)
|
|||
|
|
|
|||
|
|
## 限制
|
|||
|
|
|
|||
|
|
- 仅支持中文输入
|
|||
|
|
- 训练数据以网络小说为主，非通用指令模型
|
|||
|
|
- 无安全对齐，不适用于敏感内容生成
|