118 lines
4.1 KiB
Markdown
118 lines
4.1 KiB
Markdown
|
---
|
|||
|
|
base_model_relation: quantized
|
|||
|
|
license: Apache License 2.0
|
|||
|
|
language:
|
|||
|
|
- zh
|
|||
|
|
- en
|
|||
|
|
tags:
|
|||
|
|
- medical
|
|||
|
|
- qwen
|
|||
|
|
- sft
|
|||
|
|
- lora
|
|||
|
|
- reinforcement-learning
|
|||
|
|
base_model: zjydiary/Medical-Qwen3-14B-1218
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
library_name: peft
|
|||
|
|
datasets:
|
|||
|
|
- zjydiary/Medical
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Medical-Qwen3-14B-1218
|
|||
|
|
|
|||
|
|
## 模型简介 (Model Description)
|
|||
|
|
|
|||
|
|
**Medical-Qwen3-14B-1218** 是基于 [Qwen3-14B](https://modelscope.cn/Qwen/Qwen3-14B) 的医疗领域专用大语言模型。该模型通过两阶段微调（Pre-training + SFT）并采用加权合并策略（Weighted Merging）构建，旨在提升医疗问答的准确性、术语规范性及指令遵循能力。
|
|||
|
|
|
|||
|
|
本版本为 **2025-12-18** 发布的稳定版，融合了 GPT-OSS 风格的训练参数与 Qwen3 的架构优势，在医疗长文本理解与生成上表现出优异的鲁棒性。
|
|||
|
|
|
|||
|
|
## 训练细节 (Training Details)
|
|||
|
|
|
|||
|
|
### 训练流程
|
|||
|
|
1. **基座模型**: Qwen/Qwen3-14B (bf16)
|
|||
|
|
2. **阶段一：医疗增量预训练 (LoRA PT)**
|
|||
|
|
- 数据：混合医疗语料 (Medical Corpus)
|
|||
|
|
- 策略：Qwen3 No-Think 模板，全参 LoRA (Target All)
|
|||
|
|
- 目标：注入领域知识，适应医疗文风
|
|||
|
|
3. **阶段二：指令微调 (LoRA SFT)**
|
|||
|
|
- 数据：高质量医疗问答对 (Medical QA Pairs)
|
|||
|
|
- 参数：Rank 8, Alpha 16, Dropout 0.07
|
|||
|
|
- 策略：Cosine 调度, 强正则 (Weight Decay 0.03), 梯度累积优化
|
|||
|
|
4. **模型合并 (Weighted Merging)**
|
|||
|
|
- 方法：线性加权合并 (Linear Weighted Merging)
|
|||
|
|
- 权重：`SFT (0.7) + PT (0.3)`
|
|||
|
|
- 目的：平衡指令遵循能力（SFT）与领域知识广度（PT）
|
|||
|
|
|
|||
|
|
### 训练环境
|
|||
|
|
- **Framework**: LLaMA-Factory
|
|||
|
|
- **Hardware**: NVIDIA GPU Cluster (FlashAttention-2 Enabled)
|
|||
|
|
- **Quantization**: BNB 4-bit NF4 (Training), BFloat16 (Merge/Inference)
|
|||
|
|
|
|||
|
|
## 评估结果 (Evaluation Results)
|
|||
|
|
|
|||
|
|
评估时间：2025-12-18 12:24:46
|
|||
|
|
评估集：Medical Validation Set (F5/F6)
|
|||
|
|
解码参数：`temperature=0.6`, `top_p=0.8` (折中推荐配置)
|
|||
|
|
|
|||
|
|
| Metric | Score | Description |
|
|||
|
|
| :--- | :--- | :--- |
|
|||
|
|
| **BLEU-4** | **16.38** | 高分表明生成的短语与专家回答高度一致 |
|
|||
|
|
| **ROUGE-1** | **20.42** | 优秀的词汇覆盖率，关键信息点召回准确 |
|
|||
|
|
| **ROUGE-2** | **4.60** | 相比早期版本 (+0.1)，二元术语搭配更加精准 |
|
|||
|
|
| **ROUGE-L** | **11.54** | 良好的长句结构相似度，逻辑连贯性强 |
|
|||
|
|
|
|||
|
|
*注：相比纯 SFT 版本，合并模型在保持 ROUGE 分数的同时，显著降低了过拟合风险，提升了对不同解码参数的鲁棒性。*
|
|||
|
|
|
|||
|
|
## 使用方法 (Usage)
|
|||
|
|
|
|||
|
|
### 推荐推理参数
|
|||
|
|
为了获得最佳的医疗问答效果，建议使用以下参数：
|
|||
|
|
- `temperature`: 0.5 - 0.7 (平衡准确性与流畅度)
|
|||
|
|
- `top_p`: 0.8 - 0.9
|
|||
|
|
- `repetition_penalty`: 1.05 - 1.1
|
|||
|
|
- `max_new_tokens`: 2048 (支持长文回答)
|
|||
|
|
|
|||
|
|
### 代码示例 (Transformers)
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
import torch
|
|||
|
|
|
|||
|
|
model_path = "/workspace/model/zjydiary/Medical-Qwen3-14B-1218"
|
|||
|
|
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
model_path,
|
|||
|
|
device_map="auto",
|
|||
|
|
torch_dtype=torch.bfloat16,
|
|||
|
|
trust_remote_code=True
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
prompt = "患者，男，45岁，主诉持续性上腹痛3天，伴恶心呕吐。请给出初步诊断建议及检查方案。"
|
|||
|
|
messages = [
|
|||
|
|
{"role": "system", "content": "你是一名专业的医疗助手，请用严谨、客观的语气回答。"},
|
|||
|
|
{"role": "user", "content": prompt}
|
|||
|
|
]
|
|||
|
|
|
|||
|
|
text = tokenizer.apply_chat_template(
|
|||
|
|
messages,
|
|||
|
|
tokenize=False,
|
|||
|
|
add_generation_prompt=True
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
|||
|
|
|
|||
|
|
generated_ids = model.generate(
|
|||
|
|
**inputs,
|
|||
|
|
max_new_tokens=2048,
|
|||
|
|
temperature=0.6,
|
|||
|
|
top_p=0.9
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
|
|||
|
|
print(response)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## 声明 (Disclaimer)
|
|||
|
|
|
|||
|
|
本模型仅供学术研究与技术验证使用。生成的医疗建议仅供参考，不能替代执业医师的诊断。在实际临床应用前，请务必进行严格的专业评估与人工审核。
|