初始化项目，由ModelHub XC社区提供模型
Model: webwlsong/zmyl-7b-chat Source: Original Platform
This commit is contained in:
ModelHub XC
66
.gitattributes
vendored
Normal file
66
.gitattributes
vendored
Normal file
@@ -0,0 +1,66 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
<<<<<<< HEAD
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
=======
|
||||||
|
*.bin.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
>>>>>>> 96194fc151e35917f2c1a2d8b1c4ea01a83dad94
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
<<<<<<< HEAD
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
=======
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
>>>>>>> 96194fc151e35917f2c1a2d8b1c4ea01a83dad94
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
<<<<<<< HEAD
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
=======
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zstandard filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.db* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ark* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
|
||||||
|
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
>>>>>>> 96194fc151e35917f2c1a2d8b1c4ea01a83dad94
|
||||||
217
README.md
Normal file
217
README.md
Normal file
@@ -0,0 +1,217 @@
|
|||||||
|
---
|
||||||
|
license: apache-2.0
|
||||||
|
language:
|
||||||
|
- zh
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
tags:
|
||||||
|
- medical
|
||||||
|
---
|
||||||
|
## WiNGPT2
|
||||||
|
|
||||||
|
[WiNGPT](https://github.com/winninghealth/WiNGPT2) 是一个基于GPT的医疗垂直领域大模型，旨在将专业的医学知识、医疗信息、数据融会贯通，为医疗行业提供智能化的医疗问答、诊断支持和医学知识等信息服务，提高诊疗效率和医疗服务质量。
|
||||||
|
|
||||||
|
|
||||||
|
## 介绍
|
||||||
|
|
||||||
|
WiNGPT（卫宁健康医疗语言大模型，以下简称WiNGPT）的研发和训练工作开始于2023年1月。
|
||||||
|
|
||||||
|
3月，卫宁健康人工智能实验室已完成了WiNGPT-001可行性验证并开始内测。WiNGPT-001采用通用的GPT架构、60亿参数，实现了从预训练到微调的全过程自研。
|
||||||
|
|
||||||
|
今年5月，WiNGPT-001训练的数据量已达到9720项药品知识、 18个药品类型、7200余项疾病知识、 2800余项检查检验知识、53本书籍知识、1100余份指南文档，总训练Token数达37亿。
|
||||||
|
|
||||||
|
7月，WiNGPT升级到7B并采用最新的模型架构，新增检索式增强生成能力，同时开始了13B模型的训练和行业邀测。
|
||||||
|
|
||||||
|
9月，WiNGPT迎来最新版本迭代，推出了全新的WiNGPT2，新版本可以被轻松扩展和个性化并用于下游各种应用场景。
|
||||||
|
|
||||||
|
为了回馈开源社区我们尝试开源了WiNGPT2-7B版本。我们的初衷是希望通过更多的开源项目加速医疗语言大模型技术与行业的共同发展，最终惠及我们人类健康。
|
||||||
|
|
||||||
|
|
||||||
|
## 特点
|
||||||
|
|
||||||
|
- 核心功能
|
||||||
|
|
||||||
|
- **医学知识问答**：可以回答关于医学、健康、疾病等方面的问题，包括但不限于症状、治疗、药物、预防、检查等。
|
||||||
|
- **自然语言理解**：理解医学术语、病历等医疗文本信息，提供关键信息抽取和归类
|
||||||
|
- **多轮对话**：可扮演各种医疗专业角色如医生与用户进行对话，根据上下文提供更加准确的答案。
|
||||||
|
- **多任务支持**：支持32项医疗任务，八大医疗场景18个子场景。
|
||||||
|
|
||||||
|
- 模型架构
|
||||||
|
|
||||||
|
- 基于Transformer的70亿参数规模大语言模型, 采用RoPE相对位置编码、SwiGLU激活函数、RMSNorm，训练采用Qwen-7b<sup>1</sup>作为基础预训练模型。
|
||||||
|
|
||||||
|
- 主要特点
|
||||||
|
|
||||||
|
- 高准确度：基于大规模医疗语料库训练，具有较高的准确率和较低的误诊可能性。
|
||||||
|
- 场景导向：针对不同的医疗场景和真实需求进行专门优化和定制，更好的服务应用落地。
|
||||||
|
- 迭代优化：持续搜集和学习最新的医学研究，不断提高模型性能和系统功能。
|
||||||
|
|
||||||
|
## 如何使用
|
||||||
|
|
||||||
|
### 推理
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||||
|
from transformers.generation import GenerationConfig
|
||||||
|
|
||||||
|
model_path = "WiNGPT2-7B-Chat"
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
|
||||||
|
model = model.eval()
|
||||||
|
|
||||||
|
generation_config = GenerationConfig(
|
||||||
|
num_beams=1,
|
||||||
|
top_p=0.75,
|
||||||
|
top_k=30,
|
||||||
|
repetition_penalty=1.1,
|
||||||
|
max_new_tokens=1024
|
||||||
|
)
|
||||||
|
|
||||||
|
text = 'User: WiNGPT, 你好<|endoftext|>\n Assistant: '
|
||||||
|
inputs = tokenizer.encode(text, return_tensors="pt").to(device)
|
||||||
|
outputs = model.generate(inputs, generation_config=generation_config)
|
||||||
|
output = tokenizer.decode(outputs[0])
|
||||||
|
response = output.replace(inputs, '')
|
||||||
|
|
||||||
|
## 输出结果：你好！今天我能为你做些什么？<|endoftext|>
|
||||||
|
```
|
||||||
|
|
||||||
|
### 提示
|
||||||
|
|
||||||
|
WiNGPT2-7B-Chat使用了自定义的提示格式：
|
||||||
|
|
||||||
|
用户角色：User/Assistant
|
||||||
|
|
||||||
|
提示模板：User:[此处有空格]WiNGPT, 你好<|endoftext|>\n[此处有空格]Assistant:；**多轮对话**按此模板进行拼接，例如：
|
||||||
|
```
|
||||||
|
"User: WiNGPT, 你好<|endoftext|>\n Assistant:你好！今天我能为你做些什么？<|endoftext|>\n User: 你是谁？<|endoftext|>\n Assistant:"
|
||||||
|
```
|
||||||
|
|
||||||
|
解码时推荐使用repetition_penalty=1.1 [greedy search]
|
||||||
|
|
||||||
|
### 企业服务
|
||||||
|
|
||||||
|
[13B模型平台测试(直接申请密钥)](https://wingpt.winning.com.cn/)
|
||||||
|
|
||||||
|
|
||||||
|
## 训练数据
|
||||||
|
|
||||||
|
- 数据总览
|
||||||
|
|
||||||
|
- 医疗专业数据
|
||||||
|
|
||||||
|
| 来源 | 类型 | 数量 |
|
||||||
|
| ---------------- | ------ | ------------------- |
|
||||||
|
| 药品说明书 | 知识库 | 15000 条 |
|
||||||
|
| 多病种知识库 | 知识库 | 9720 项 |
|
||||||
|
| 医疗专业书籍 | 教材 | 300 本 |
|
||||||
|
| 临床路径知识库 | 知识库 | 1400 条 |
|
||||||
|
| 检查检验知识 | 知识库 | 110 万条 |
|
||||||
|
| 多学科临床指南 | 书籍 | 18 个科室共 1100 份 |
|
||||||
|
| 医疗知识图谱 | 知识库 | 256 万三元组 |
|
||||||
|
| 人工标注数据集 | 指令 | 5 万条 |
|
||||||
|
| 医学资格考试试题 | 试题 | 30 万条 |
|
||||||
|
| 医疗病例、报告 | 知识库 | 100 万条 |
|
||||||
|
|
||||||
|
- 其他公开数据
|
||||||
|
|
||||||
|
| 来源 | 类型 | 数量 |
|
||||||
|
| -------------------- | ------ | -------- |
|
||||||
|
| 医学科普书籍 | 书籍 | 500 本 |
|
||||||
|
| 其他多学科书籍 | 书籍 | 1000 本 |
|
||||||
|
| 代码 | 指令 | 20 万条 |
|
||||||
|
| 通用类试题 | 试题 | 300 万条 |
|
||||||
|
| 多种自然语言处理任务 | 指令 | 90 万条 |
|
||||||
|
| 互联网文本 | 互联网 | 300 万条 |
|
||||||
|
| 医疗问答、对话 | 指令 | 500 万条 |
|
||||||
|
|
||||||
|
- 继续预训练
|
||||||
|
|
||||||
|
- 扩充模型的医疗知识库：预训练数据+部分指令数据。
|
||||||
|
|
||||||
|
- 指令微调
|
||||||
|
|
||||||
|
- 从书籍、指南、病例、医疗报告、知识图谱等数据中自动化构建医疗指令集。
|
||||||
|
- 人工标注指令集，数据来源包括：电子病历系统、护理病历系统、PACS系统、临床科研系统、手术管理系统、公共卫生场景、医务管理场景以及工具助手场景。
|
||||||
|
- 采用 FastChat<sup>2</sup>、Self-Instruct<sup>3</sup>、Evol-Instruct<sup>4</sup> 等方案，对指令集进行扩展以及丰富指令集多样化形式。
|
||||||
|
|
||||||
|
- 数据工程
|
||||||
|
|
||||||
|
- 数据分类：根据训练阶段和任务场景进行分类。
|
||||||
|
- 数据清洗：去除无关信息，更正数据中的拼写错误，提取关键信息以及去隐私处理。
|
||||||
|
- 数据去重：采用 embedding 方法剔除重复数据。
|
||||||
|
- 数据采样：根据数据集的质量与分布需求进行有针对性的采样。
|
||||||
|
|
||||||
|
## 模型卡
|
||||||
|
|
||||||
|
- 训练配置与参数
|
||||||
|
|
||||||
|
| 名称 | 长度 | 精度 | 学习率 | Weight_decay | Epochs | GPUs |
|
||||||
|
| --------------- | ---- | ---- | ------ | ------------ | ------ | ------ |
|
||||||
|
| WiNGPT2-7B-Base | 2048 | bf16 | 5e-5 | 0.05 | 3 | A100*8 |
|
||||||
|
| WiNGPT2-7B-Chat | 4096 | bf16 | 5e-6 | 0.01 | 3 | A100*8 |
|
||||||
|
|
||||||
|
- 分布式训练策略与参数
|
||||||
|
|
||||||
|
- deepspeed + cpu_offload + zero_stage3
|
||||||
|
- gradient_checkpointing
|
||||||
|
|
||||||
|
## 评测
|
||||||
|
|
||||||
|
- 中文基础模型评估 C-EVAL（Zero-shot/Few-shot）
|
||||||
|
|
||||||
|
| | 平均 | 平均(Hard) | **STEM** | **社会科学** | **人文科学** | **其他** |
|
||||||
|
| -------------------------------------------------------------------------------------------- | -------- | ---------- | -------- | ------------ | ------------ | -------- |
|
||||||
|
| [bloomz-mt-176B](https://cevalbenchmark.com/static/model.html?method=bloomz-mt-176B*) | 44.3 | 30.8 | 39 | 53 | 47.7 | 42.7 |
|
||||||
|
| [Chinese LLaMA-13B](https://cevalbenchmark.com/static/model.html?method=Chinese%20LLaMA-13B) | 33.3 | 27.3 | 31.6 | 37.2 | 33.6 | 32.8 |
|
||||||
|
| [ChatGLM-6B*](https://cevalbenchmark.com/static/model.html?method=ChatGLM-6B*) | 38.9 | 29.2 | 33.3 | 48.3 | 41.3 | 38 |
|
||||||
|
| [baichuan-7B](https://cevalbenchmark.com/static/model.html?method=baichuan-7B) | 42.8 | 31.5 | 38.2 | 52 | 46.2 | 39.3 |
|
||||||
|
| [Baichuan-13B](https://cevalbenchmark.com/static/model.html?method=Baichuan-13B) | 53.6 | 36.7 | 47 | 66.8 | 57.3 | 49.8 |
|
||||||
|
| [Qwen-7B](https://cevalbenchmark.com/static/model.html?method=Qwen-7B) | **59.6** | 41 | 52.8 | **74.1** | **63.1** | 55.2 |
|
||||||
|
| [WiNGPT2-7B-Base](https://huggingface.co/winninghealth/WiNGPT2-7B-Base) | 57.4 | **42.7** | **53.2** | 69.7 | 55.7 | **55.4** |
|
||||||
|
|
||||||
|
|
||||||
|
- 中文医疗专业评估 MedQA-MCMLE（Zero-shot）
|
||||||
|
|
||||||
|
| 模型名称 | 平均 | 血液系统疾病 | 代谢、内分泌系统疾病 | 精神神经系统疾病 | 运动系统疾病 | 风湿免疫性疾病 | 儿科疾病 | 传染病、性传播疾病 | 其他疾病 |
|
||||||
|
| ---------------------------------------------------------------------------- | -------- | ------------ | -------------------- | ---------------- | ------------ | -------------- | -------- | ------------------ | -------- |
|
||||||
|
| [Baichuan-7B](https://huggingface.co/baichuan-inc/Baichuan-7B) | 23.1 | 25.6 | 20.2 | 25.8 | 17.9 | 26.5 | 20.6 | 26.1 | 17.1 |
|
||||||
|
| [Baichuan-13B-Base](https://huggingface.co/baichuan-inc/Baichuan-13B-Base) | 37.2 | 34.4 | 36.2 | 40.7 | 38.4 | 57.1 | 31.6 | 30.8 | 34.3 |
|
||||||
|
| [Baichuan2-7B-Base](https://huggingface.co/baichuan-inc/Baichuan2-7B-Base) | 46.4 | 46.9 | 41.4 | 53.8 | 48.3 | 50.0 | 38.6 | 52.7 | 42.9 |
|
||||||
|
| [Baichuan2-13B-Base](https://huggingface.co/baichuan-inc/Baichuan2-13B-Base) | 62.9 | 68.8 | 64.4 | 69.7 | 64.9 | 60.3 | 50.9 | 61.2 | 62.9 |
|
||||||
|
| [HuatuoGPT-7B](https://huggingface.co/FreedomIntelligence/HuatuoGPT-7B) | 22.9 | 14.6 | 17.2 | 31.2 | 25.8 | 14.3 | 22.4 | 23.1 | 17.1 |
|
||||||
|
| [MedicalGPT](https://huggingface.co/shibing624/vicuna-baichuan-13b-chat) | 17.9 | 21.9 | 15.5 | 19.5 | 9.3 | 7.1 | 16.7 | 20.9 | 9.5 |
|
||||||
|
| [qwen-7b-Base](https://huggingface.co/Qwen/Qwen-7B) | 59.3 | 55.2 | 56.9 | 57.0 | 60.9 | 60.3 | 50.4 | 60.4 | 61.0 |
|
||||||
|
| [WiNGPT2-7B-Base](https://huggingface.co/winninghealth/WiNGPT2-7B-Base) | **82.3** | **83.3** | **82.8** | **86.0** | **81.5** | **85.7** | **75.1** | **78.0** | **80** |
|
||||||
|
|
||||||
|
|
||||||
|
** 目前公开测评存在一定局限性，结果仅供参考；
|
||||||
|
** 更多专业测评敬请期待。
|
||||||
|
|
||||||
|
|
||||||
|
## 局限性与免责声明
|
||||||
|
|
||||||
|
(a) WiNGPT2 是一个专业医疗领域的大语言模型，可为一般用户提供拟人化AI医生问诊和问答功能，以及一般医学领域的知识问答。对于专业医疗人士，WiNGPT2 提供关于患者病情的诊断、用药和健康建议等方面的回答的建议仅供参考。
|
||||||
|
|
||||||
|
(b) 您应理解 WiNGPT2 仅提供信息和建议，不能替代医疗专业人士的意见、诊断或治疗建议。在使用 WiNGPT2 的信息之前，请寻求医生或其他医疗专业人员的建议，并独立评估所提供的信息。
|
||||||
|
|
||||||
|
(c) WiNGPT2 的信息可能存在错误或不准确。卫宁健康不对 WiNGPT2 的准确性、可靠性、完整性、质量、安全性、及时性、性能或适用性提供任何明示或暗示的保证。使用 WiNGPT2 所产生的结果和决策由您自行承担。第三方原因而给您造成的损害结果承担责任。
|
||||||
|
|
||||||
|
## 许可证
|
||||||
|
|
||||||
|
1. 本项目授权协议为 Apache License 2.0，模型权重需要遵守基础模型[Qwen-7B](https://github.com/QwenLM/Qwen-7B)相关协议及[许可证](https://github.com/QwenLM/Qwen-7B/blob/main/LICENSE)，详细内容参照其网站。
|
||||||
|
|
||||||
|
2. 使用本项目包括模型权重时请引用本项目：https://github.com/winninghealth/WiNGPT2
|
||||||
|
|
||||||
|
## 参考资料
|
||||||
|
|
||||||
|
1. https://github.com/QwenLM/Qwen-7B
|
||||||
|
2. https://github.com/lm-sys/FastChat
|
||||||
|
3. https://github.com/yizhongw/self-instruct
|
||||||
|
4. https://github.com/nlpxucan/evol-instruct
|
||||||
|
|
||||||
|
## 联系我们
|
||||||
|
|
||||||
|
网站：https://www.winning.com.cn
|
||||||
|
|
||||||
|
邮箱：wair@winning.com.cn
|
||||||
40
config.json
Normal file
40
config.json
Normal file
@@ -0,0 +1,40 @@
|
|||||||
|
{
|
||||||
|
"_name_or_path": "WiNGPT-7B-Chat",
|
||||||
|
"architectures": [
|
||||||
|
"QWenLMHeadModel"
|
||||||
|
],
|
||||||
|
"attn_dropout_prob": 0.0,
|
||||||
|
"auto_map": {
|
||||||
|
"AutoConfig": "configuration_qwen.QWenConfig",
|
||||||
|
"AutoModelForCausalLM": "modeling_qwen.QWenLMHeadModel"
|
||||||
|
},
|
||||||
|
"bf16": false,
|
||||||
|
"emb_dropout_prob": 0.0,
|
||||||
|
"fp16": true,
|
||||||
|
"fp32": false,
|
||||||
|
"hidden_size": 4096,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 22016,
|
||||||
|
"kv_channels": 128,
|
||||||
|
"layer_norm_epsilon": 1e-06,
|
||||||
|
"max_position_embeddings": 8192,
|
||||||
|
"model_type": "qwen",
|
||||||
|
"no_bias": true,
|
||||||
|
"num_attention_heads": 32,
|
||||||
|
"num_hidden_layers": 32,
|
||||||
|
"onnx_safe": null,
|
||||||
|
"rotary_emb_base": 10000,
|
||||||
|
"rotary_pct": 1.0,
|
||||||
|
"scale_attn_weights": true,
|
||||||
|
"seq_length": 4096,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"tokenizer_class": "QWenTokenizer",
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"transformers_version": "4.28.1",
|
||||||
|
"use_cache": false,
|
||||||
|
"use_dynamic_ntk": true,
|
||||||
|
"use_flash_attn": false,
|
||||||
|
"use_logn_attn": true,
|
||||||
|
"vocab_size": 151936,
|
||||||
|
"pad_token_id": 151643
|
||||||
|
}
|
||||||
65
configuration_qwen.py
Normal file
65
configuration_qwen.py
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
# Copyright (c) Alibaba Cloud.
|
||||||
|
#
|
||||||
|
# This source code is licensed under the license found in the
|
||||||
|
# LICENSE file in the root directory of this source tree.
|
||||||
|
|
||||||
|
from transformers import PretrainedConfig
|
||||||
|
|
||||||
|
|
||||||
|
class QWenConfig(PretrainedConfig):
|
||||||
|
model_type = "qwen"
|
||||||
|
keys_to_ignore_at_inference = ["past_key_values"]
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
vocab_size=151936,
|
||||||
|
hidden_size=4096,
|
||||||
|
num_hidden_layers=32,
|
||||||
|
num_attention_heads=32,
|
||||||
|
emb_dropout_prob=0.0,
|
||||||
|
attn_dropout_prob=0.0,
|
||||||
|
layer_norm_epsilon=1e-6,
|
||||||
|
initializer_range=0.02,
|
||||||
|
max_position_embeddings=8192,
|
||||||
|
scale_attn_weights=True,
|
||||||
|
use_cache=True,
|
||||||
|
bf16=False,
|
||||||
|
fp16=False,
|
||||||
|
fp32=False,
|
||||||
|
kv_channels=128,
|
||||||
|
rotary_pct=1.0,
|
||||||
|
rotary_emb_base=10000,
|
||||||
|
use_dynamic_ntk=True,
|
||||||
|
use_logn_attn=True,
|
||||||
|
use_flash_attn="auto",
|
||||||
|
intermediate_size=22016,
|
||||||
|
no_bias=True,
|
||||||
|
tie_word_embeddings=False,
|
||||||
|
**kwargs,
|
||||||
|
):
|
||||||
|
self.vocab_size = vocab_size
|
||||||
|
self.hidden_size = hidden_size
|
||||||
|
self.intermediate_size = intermediate_size
|
||||||
|
self.num_hidden_layers = num_hidden_layers
|
||||||
|
self.num_attention_heads = num_attention_heads
|
||||||
|
self.emb_dropout_prob = emb_dropout_prob
|
||||||
|
self.attn_dropout_prob = attn_dropout_prob
|
||||||
|
self.layer_norm_epsilon = layer_norm_epsilon
|
||||||
|
self.initializer_range = initializer_range
|
||||||
|
self.scale_attn_weights = scale_attn_weights
|
||||||
|
self.use_cache = use_cache
|
||||||
|
self.max_position_embeddings = max_position_embeddings
|
||||||
|
self.bf16 = bf16
|
||||||
|
self.fp16 = fp16
|
||||||
|
self.fp32 = fp32
|
||||||
|
self.kv_channels = kv_channels
|
||||||
|
self.rotary_pct = rotary_pct
|
||||||
|
self.rotary_emb_base = rotary_emb_base
|
||||||
|
self.use_dynamic_ntk = use_dynamic_ntk
|
||||||
|
self.use_logn_attn = use_logn_attn
|
||||||
|
self.use_flash_attn = use_flash_attn
|
||||||
|
self.no_bias = no_bias
|
||||||
|
super().__init__(
|
||||||
|
tie_word_embeddings=tie_word_embeddings,
|
||||||
|
**kwargs
|
||||||
|
)
|
||||||
15
generation_config.json
Normal file
15
generation_config.json
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
{
|
||||||
|
"chat_format": "raw",
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": 151643,
|
||||||
|
"max_new_tokens": 512,
|
||||||
|
"pad_token_id": 151643,
|
||||||
|
"stop_words_ids": [
|
||||||
|
[
|
||||||
|
151643
|
||||||
|
]
|
||||||
|
],
|
||||||
|
"top_k": 0,
|
||||||
|
"top_p": 0.8,
|
||||||
|
"transformers_version": "4.28.1"
|
||||||
|
}
|
||||||
1232
modeling_qwen.py
Normal file
1232
modeling_qwen.py
Normal file
File diff suppressed because it is too large
Load Diff
3
pytorch_model-00001-of-00004.bin
Normal file
3
pytorch_model-00001-of-00004.bin
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:dca0fcc2723086e759955da7e903b3893183936a7faf3daca30a25af00daaea5
|
||||||
|
size 4988502435
|
||||||
3
pytorch_model-00002-of-00004.bin
Normal file
3
pytorch_model-00002-of-00004.bin
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:e88478e658b3fea31dd692c5a1e45a5cabebdc73b6a948f93c0599ae319ff427
|
||||||
|
size 4981268429
|
||||||
3
pytorch_model-00003-of-00004.bin
Normal file
3
pytorch_model-00003-of-00004.bin
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:6122f6f9df16bbc279d40b769ba4a5b7da8e07a5db249084ac3a7872e2fa1600
|
||||||
|
size 4228303461
|
||||||
3
pytorch_model-00004-of-00004.bin
Normal file
3
pytorch_model-00004-of-00004.bin
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:56810b0eb4c8d8289069d303900bb2ea329d93dd5e93ae655c223bcacc27a0da
|
||||||
|
size 1244660650
|
||||||
BIN
pytorch_model.bin.index.json
(Stored with Git LFS)
Normal file
BIN
pytorch_model.bin.index.json
(Stored with Git LFS)
Normal file
Binary file not shown.
151643
qwen.tiktoken
Normal file
151643
qwen.tiktoken
Normal file
File diff suppressed because it is too large
Load Diff
416
qwen_generation_utils.py
Normal file
416
qwen_generation_utils.py
Normal file
@@ -0,0 +1,416 @@
|
|||||||
|
# Copyright (c) Alibaba Cloud.
|
||||||
|
#
|
||||||
|
# This source code is licensed under the license found in the
|
||||||
|
# LICENSE file in the root directory of this source tree.
|
||||||
|
|
||||||
|
"""Generation support."""
|
||||||
|
|
||||||
|
from typing import Tuple, List, Union, Iterable
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import torch
|
||||||
|
import torch.nn.functional as F
|
||||||
|
from transformers import PreTrainedTokenizer
|
||||||
|
from transformers import logging
|
||||||
|
from transformers.generation import LogitsProcessor
|
||||||
|
|
||||||
|
logger = logging.get_logger(__name__)
|
||||||
|
|
||||||
|
# Types.
|
||||||
|
HistoryType = List[Tuple[str, str]]
|
||||||
|
TokensType = List[int]
|
||||||
|
BatchTokensType = List[List[int]]
|
||||||
|
|
||||||
|
|
||||||
|
def pad_batch(batch: BatchTokensType, pad_id: int, seq_length: int) -> BatchTokensType:
|
||||||
|
for tokens in batch:
|
||||||
|
context_length = len(tokens)
|
||||||
|
if context_length < seq_length:
|
||||||
|
tokens.extend([pad_id] * (seq_length - context_length))
|
||||||
|
return batch
|
||||||
|
|
||||||
|
|
||||||
|
def get_ltor_masks_and_position_ids(
|
||||||
|
data,
|
||||||
|
eod_token,
|
||||||
|
reset_position_ids,
|
||||||
|
reset_attention_mask,
|
||||||
|
eod_mask_loss,
|
||||||
|
):
|
||||||
|
"""Build masks and position id for left to right model."""
|
||||||
|
|
||||||
|
# Extract batch size and sequence length.
|
||||||
|
micro_batch_size, seq_length = data.size()
|
||||||
|
|
||||||
|
# Attention mask (lower triangular).
|
||||||
|
if reset_attention_mask:
|
||||||
|
att_mask_batch = micro_batch_size
|
||||||
|
else:
|
||||||
|
att_mask_batch = 1
|
||||||
|
attention_mask = torch.tril(
|
||||||
|
torch.ones((att_mask_batch, seq_length, seq_length), device=data.device)
|
||||||
|
).view(att_mask_batch, 1, seq_length, seq_length)
|
||||||
|
|
||||||
|
# Loss mask.
|
||||||
|
loss_mask = torch.ones(data.size(), dtype=torch.float, device=data.device)
|
||||||
|
if eod_mask_loss:
|
||||||
|
loss_mask[data == eod_token] = 0.0
|
||||||
|
|
||||||
|
# Position ids.
|
||||||
|
position_ids = torch.arange(seq_length, dtype=torch.long, device=data.device)
|
||||||
|
position_ids = position_ids.unsqueeze(0).expand_as(data)
|
||||||
|
# We need to clone as the ids will be modifed based on batch index.
|
||||||
|
if reset_position_ids:
|
||||||
|
position_ids = position_ids.clone()
|
||||||
|
|
||||||
|
if reset_position_ids or reset_attention_mask:
|
||||||
|
# Loop through the batches:
|
||||||
|
for b in range(micro_batch_size):
|
||||||
|
|
||||||
|
# Find indecies where EOD token is.
|
||||||
|
eod_index = position_ids[b, data[b] == eod_token]
|
||||||
|
# Detach indecies from positions if going to modify positions.
|
||||||
|
if reset_position_ids:
|
||||||
|
eod_index = eod_index.clone()
|
||||||
|
|
||||||
|
# Loop through EOD indecies:
|
||||||
|
prev_index = 0
|
||||||
|
for j in range(eod_index.size()[0]):
|
||||||
|
i = eod_index[j]
|
||||||
|
# Mask attention loss.
|
||||||
|
if reset_attention_mask:
|
||||||
|
attention_mask[b, 0, (i + 1) :, : (i + 1)] = 0
|
||||||
|
# Reset positions.
|
||||||
|
if reset_position_ids:
|
||||||
|
position_ids[b, (i + 1) :] -= i + 1 - prev_index
|
||||||
|
prev_index = i + 1
|
||||||
|
|
||||||
|
# Convert attention mask to binary:
|
||||||
|
attention_mask = attention_mask < 0.5
|
||||||
|
|
||||||
|
return attention_mask, loss_mask, position_ids
|
||||||
|
|
||||||
|
|
||||||
|
def get_batch(context_tokens: torch.LongTensor, eod_id: int):
|
||||||
|
"""Generate batch from context tokens."""
|
||||||
|
# Move to GPU.
|
||||||
|
tokens = context_tokens.contiguous().to(context_tokens.device)
|
||||||
|
# Get the attention mask and postition ids.
|
||||||
|
attention_mask, _, position_ids = get_ltor_masks_and_position_ids(
|
||||||
|
tokens,
|
||||||
|
eod_id,
|
||||||
|
reset_position_ids=False,
|
||||||
|
reset_attention_mask=False,
|
||||||
|
eod_mask_loss=False,
|
||||||
|
)
|
||||||
|
return tokens, attention_mask, position_ids
|
||||||
|
|
||||||
|
|
||||||
|
def get_stop_words_ids(chat_format, tokenizer):
|
||||||
|
if chat_format == "raw":
|
||||||
|
stop_words_ids = [tokenizer.encode("Human:"), [tokenizer.eod_id]]
|
||||||
|
elif chat_format == "chatml":
|
||||||
|
stop_words_ids = [[tokenizer.im_end_id], [tokenizer.im_start_id]]
|
||||||
|
else:
|
||||||
|
raise NotImplementedError(f"Unknown chat format {chat_format!r}")
|
||||||
|
return stop_words_ids
|
||||||
|
|
||||||
|
|
||||||
|
def make_context(
|
||||||
|
tokenizer: PreTrainedTokenizer,
|
||||||
|
query: str,
|
||||||
|
history: List[Tuple[str, str]] = None,
|
||||||
|
system: str = "",
|
||||||
|
max_window_size: int = 6144,
|
||||||
|
chat_format: str = "chatml",
|
||||||
|
):
|
||||||
|
if history is None:
|
||||||
|
history = []
|
||||||
|
|
||||||
|
if chat_format == "chatml":
|
||||||
|
im_start, im_end = "<|im_start|>", "<|im_end|>"
|
||||||
|
im_start_tokens = [tokenizer.im_start_id]
|
||||||
|
im_end_tokens = [tokenizer.im_end_id]
|
||||||
|
nl_tokens = tokenizer.encode("\n")
|
||||||
|
|
||||||
|
def _tokenize_str(role, content):
|
||||||
|
return f"{role}\n{content}", tokenizer.encode(
|
||||||
|
role, allowed_special=set()
|
||||||
|
) + nl_tokens + tokenizer.encode(content, allowed_special=set())
|
||||||
|
|
||||||
|
system_text, system_tokens_part = _tokenize_str("system", system)
|
||||||
|
system_tokens = im_start_tokens + system_tokens_part + im_end_tokens
|
||||||
|
|
||||||
|
raw_text = ""
|
||||||
|
context_tokens = []
|
||||||
|
|
||||||
|
for turn_query, turn_response in reversed(history):
|
||||||
|
query_text, query_tokens_part = _tokenize_str("user", turn_query)
|
||||||
|
query_tokens = im_start_tokens + query_tokens_part + im_end_tokens
|
||||||
|
response_text, response_tokens_part = _tokenize_str(
|
||||||
|
"assistant", turn_response
|
||||||
|
)
|
||||||
|
response_tokens = im_start_tokens + response_tokens_part + im_end_tokens
|
||||||
|
|
||||||
|
next_context_tokens = nl_tokens + query_tokens + nl_tokens + response_tokens
|
||||||
|
prev_chat = (
|
||||||
|
f"\n{im_start}{query_text}{im_end}\n{im_start}{response_text}{im_end}"
|
||||||
|
)
|
||||||
|
|
||||||
|
current_context_size = (
|
||||||
|
len(system_tokens) + len(next_context_tokens) + len(context_tokens)
|
||||||
|
)
|
||||||
|
if current_context_size < max_window_size:
|
||||||
|
context_tokens = next_context_tokens + context_tokens
|
||||||
|
raw_text = prev_chat + raw_text
|
||||||
|
else:
|
||||||
|
break
|
||||||
|
|
||||||
|
context_tokens = system_tokens + context_tokens
|
||||||
|
raw_text = f"{im_start}{system_text}{im_end}" + raw_text
|
||||||
|
context_tokens += (
|
||||||
|
nl_tokens
|
||||||
|
+ im_start_tokens
|
||||||
|
+ _tokenize_str("user", query)[1]
|
||||||
|
+ im_end_tokens
|
||||||
|
+ nl_tokens
|
||||||
|
+ im_start_tokens
|
||||||
|
+ tokenizer.encode("assistant")
|
||||||
|
+ nl_tokens
|
||||||
|
)
|
||||||
|
raw_text += f"\n{im_start}user\n{query}{im_end}\n{im_start}assistant\n"
|
||||||
|
|
||||||
|
elif chat_format == "raw":
|
||||||
|
raw_text = query
|
||||||
|
context_tokens = tokenizer.encode(raw_text)
|
||||||
|
else:
|
||||||
|
raise NotImplementedError(f"Unknown chat format {chat_format!r}")
|
||||||
|
|
||||||
|
return raw_text, context_tokens
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_default(
|
||||||
|
tokens: List[int],
|
||||||
|
*,
|
||||||
|
stop_words: List[str],
|
||||||
|
eod_words: List[str],
|
||||||
|
tokenizer: PreTrainedTokenizer,
|
||||||
|
raw_text_len: int,
|
||||||
|
verbose: bool = False,
|
||||||
|
return_end_reason: bool = False,
|
||||||
|
errors: str='replace',
|
||||||
|
):
|
||||||
|
trim_decode_tokens = tokenizer.decode(tokens, errors=errors)[raw_text_len:]
|
||||||
|
if verbose:
|
||||||
|
print("\nRaw Generate: ", trim_decode_tokens)
|
||||||
|
|
||||||
|
end_reason = f"Gen length {len(tokens)}"
|
||||||
|
for stop_word in stop_words:
|
||||||
|
trim_decode_tokens = trim_decode_tokens.replace(stop_word, "").strip()
|
||||||
|
for eod_word in eod_words:
|
||||||
|
if eod_word in trim_decode_tokens:
|
||||||
|
end_reason = f"Gen {eod_word!r}"
|
||||||
|
trim_decode_tokens = trim_decode_tokens.split(eod_word)[0]
|
||||||
|
trim_decode_tokens = trim_decode_tokens.strip()
|
||||||
|
if verbose:
|
||||||
|
print("\nEnd Reason:", end_reason)
|
||||||
|
print("\nGenerate: ", trim_decode_tokens)
|
||||||
|
|
||||||
|
if return_end_reason:
|
||||||
|
return trim_decode_tokens, end_reason
|
||||||
|
else:
|
||||||
|
return trim_decode_tokens
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_chatml(
|
||||||
|
tokens: List[int],
|
||||||
|
*,
|
||||||
|
stop_words: List[str],
|
||||||
|
eod_token_ids: List[int],
|
||||||
|
tokenizer: PreTrainedTokenizer,
|
||||||
|
raw_text_len: int,
|
||||||
|
context_length: int,
|
||||||
|
verbose: bool = False,
|
||||||
|
return_end_reason: bool = False,
|
||||||
|
errors: str='replace'
|
||||||
|
):
|
||||||
|
end_reason = f"Gen length {len(tokens)}"
|
||||||
|
eod_token_idx = context_length
|
||||||
|
for eod_token_idx in range(context_length, len(tokens)):
|
||||||
|
if tokens[eod_token_idx] in eod_token_ids:
|
||||||
|
end_reason = f"Gen {tokenizer.decode([tokens[eod_token_idx]])!r}"
|
||||||
|
break
|
||||||
|
|
||||||
|
trim_decode_tokens = tokenizer.decode(tokens[:eod_token_idx], errors=errors)[raw_text_len:]
|
||||||
|
if verbose:
|
||||||
|
print("\nRaw Generate w/o EOD:", tokenizer.decode(tokens, errors=errors)[raw_text_len:])
|
||||||
|
print("\nRaw Generate:", trim_decode_tokens)
|
||||||
|
print("\nEnd Reason:", end_reason)
|
||||||
|
for stop_word in stop_words:
|
||||||
|
trim_decode_tokens = trim_decode_tokens.replace(stop_word, "").strip()
|
||||||
|
trim_decode_tokens = trim_decode_tokens.strip()
|
||||||
|
if verbose:
|
||||||
|
print("\nGenerate:", trim_decode_tokens)
|
||||||
|
|
||||||
|
if return_end_reason:
|
||||||
|
return trim_decode_tokens, end_reason
|
||||||
|
else:
|
||||||
|
return trim_decode_tokens
|
||||||
|
|
||||||
|
|
||||||
|
def decode_tokens(
|
||||||
|
tokens: Union[torch.LongTensor, TokensType],
|
||||||
|
tokenizer: PreTrainedTokenizer,
|
||||||
|
raw_text_len: int,
|
||||||
|
context_length: int,
|
||||||
|
chat_format: str,
|
||||||
|
verbose: bool = False,
|
||||||
|
return_end_reason: bool = False,
|
||||||
|
errors: str="replace",
|
||||||
|
) -> str:
|
||||||
|
if torch.is_tensor(tokens):
|
||||||
|
tokens = tokens.cpu().numpy().tolist()
|
||||||
|
|
||||||
|
if chat_format == "chatml":
|
||||||
|
return _decode_chatml(
|
||||||
|
tokens,
|
||||||
|
stop_words=[],
|
||||||
|
eod_token_ids=[tokenizer.im_start_id, tokenizer.im_end_id],
|
||||||
|
tokenizer=tokenizer,
|
||||||
|
raw_text_len=raw_text_len,
|
||||||
|
context_length=context_length,
|
||||||
|
verbose=verbose,
|
||||||
|
return_end_reason=return_end_reason,
|
||||||
|
errors=errors,
|
||||||
|
)
|
||||||
|
elif chat_format == "raw":
|
||||||
|
return _decode_default(
|
||||||
|
tokens,
|
||||||
|
stop_words=["<|endoftext|>"],
|
||||||
|
eod_words=["<|endoftext|>"],
|
||||||
|
tokenizer=tokenizer,
|
||||||
|
raw_text_len=raw_text_len,
|
||||||
|
verbose=verbose,
|
||||||
|
return_end_reason=return_end_reason,
|
||||||
|
errors=errors,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
raise NotImplementedError(f"Unknown chat format {chat_format!r}")
|
||||||
|
|
||||||
|
|
||||||
|
class StopWordsLogitsProcessor(LogitsProcessor):
|
||||||
|
"""
|
||||||
|
:class:`transformers.LogitsProcessor` that enforces that when specified sequences appear, stop geration.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
stop_words_ids (:obj:`List[List[int]]`):
|
||||||
|
List of list of token ids of stop ids. In order to get the tokens of the words
|
||||||
|
that should not appear in the generated text, use :obj:`tokenizer(bad_word,
|
||||||
|
add_prefix_space=True).input_ids`.
|
||||||
|
eos_token_id (:obj:`int`):
|
||||||
|
The id of the `end-of-sequence` token.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, stop_words_ids: Iterable[Iterable[int]], eos_token_id: int):
|
||||||
|
|
||||||
|
if not isinstance(stop_words_ids, List) or len(stop_words_ids) == 0:
|
||||||
|
raise ValueError(
|
||||||
|
f"`stop_words_ids` has to be a non-emtpy list, but is {stop_words_ids}."
|
||||||
|
)
|
||||||
|
if any(not isinstance(bad_word_ids, list) for bad_word_ids in stop_words_ids):
|
||||||
|
raise ValueError(
|
||||||
|
f"`stop_words_ids` has to be a list of lists, but is {stop_words_ids}."
|
||||||
|
)
|
||||||
|
if any(
|
||||||
|
any(
|
||||||
|
(not isinstance(token_id, (int, np.integer)) or token_id < 0)
|
||||||
|
for token_id in stop_word_ids
|
||||||
|
)
|
||||||
|
for stop_word_ids in stop_words_ids
|
||||||
|
):
|
||||||
|
raise ValueError(
|
||||||
|
f"Each list in `stop_words_ids` has to be a list of positive integers, but is {stop_words_ids}."
|
||||||
|
)
|
||||||
|
|
||||||
|
self.stop_words_ids = list(
|
||||||
|
filter(
|
||||||
|
lambda bad_token_seq: bad_token_seq != [eos_token_id], stop_words_ids
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.eos_token_id = eos_token_id
|
||||||
|
for stop_token_seq in self.stop_words_ids:
|
||||||
|
assert (
|
||||||
|
len(stop_token_seq) > 0
|
||||||
|
), "Stop words token sequences {} cannot have an empty list".format(
|
||||||
|
stop_words_ids
|
||||||
|
)
|
||||||
|
|
||||||
|
def __call__(
|
||||||
|
self, input_ids: torch.LongTensor, scores: torch.FloatTensor
|
||||||
|
) -> torch.FloatTensor:
|
||||||
|
stopped_samples = self._calc_stopped_samples(input_ids)
|
||||||
|
for i, should_stop in enumerate(stopped_samples):
|
||||||
|
if should_stop:
|
||||||
|
scores[i, self.eos_token_id] = float(2**15)
|
||||||
|
return scores
|
||||||
|
|
||||||
|
def _tokens_match(self, prev_tokens: torch.LongTensor, tokens: List[int]) -> bool:
|
||||||
|
if len(tokens) == 0:
|
||||||
|
# if bad word tokens is just one token always ban it
|
||||||
|
return True
|
||||||
|
elif len(tokens) > len(prev_tokens):
|
||||||
|
# if bad word tokens are longer then prev input_ids they can't be equal
|
||||||
|
return False
|
||||||
|
elif prev_tokens[-len(tokens) :].tolist() == tokens:
|
||||||
|
# if tokens match
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _calc_stopped_samples(self, prev_input_ids: Iterable[int]) -> Iterable[int]:
|
||||||
|
stopped_samples = []
|
||||||
|
for prev_input_ids_slice in prev_input_ids:
|
||||||
|
match = False
|
||||||
|
for stop_token_seq in self.stop_words_ids:
|
||||||
|
if self._tokens_match(prev_input_ids_slice, stop_token_seq):
|
||||||
|
# if tokens do not match continue
|
||||||
|
match = True
|
||||||
|
break
|
||||||
|
stopped_samples.append(match)
|
||||||
|
|
||||||
|
return stopped_samples
|
||||||
|
|
||||||
|
|
||||||
|
def top_k_logits(logits, top_k=0, top_p=0.0, filter_value=-float("Inf")):
|
||||||
|
"""This function has been mostly taken from huggingface conversational
|
||||||
|
ai code at
|
||||||
|
https://medium.com/huggingface/how-to-build-a-state-of-the-art-
|
||||||
|
conversational-ai-with-transfer-learning-2d818ac26313"""
|
||||||
|
|
||||||
|
if top_k > 0:
|
||||||
|
# Remove all tokens with a probability less than the
|
||||||
|
# last token of the top-k
|
||||||
|
indices_to_remove = logits < torch.topk(logits, top_k)[0][..., -1, None]
|
||||||
|
logits[indices_to_remove] = filter_value
|
||||||
|
|
||||||
|
if top_p > 0.0:
|
||||||
|
# Cconvert to 1D
|
||||||
|
sorted_logits, sorted_indices = torch.sort(logits, descending=True, dim=-1)
|
||||||
|
cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
|
||||||
|
|
||||||
|
# Remove tokens with cumulative probability above the threshold
|
||||||
|
sorted_indices_to_remove = cumulative_probs > top_p
|
||||||
|
# Shift the indices to the right to keep also the first token
|
||||||
|
# above the threshold
|
||||||
|
sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
|
||||||
|
sorted_indices_to_remove[..., 0] = 0
|
||||||
|
for i in range(sorted_indices.size(0)):
|
||||||
|
indices_to_remove = sorted_indices[i][sorted_indices_to_remove[i]]
|
||||||
|
logits[i][indices_to_remove] = filter_value
|
||||||
|
|
||||||
|
return logits
|
||||||
|
|
||||||
|
|
||||||
|
def switch(val1, val2, boolean):
|
||||||
|
boolean = boolean.type_as(val1)
|
||||||
|
return (1 - boolean) * val1 + boolean * val2
|
||||||
246
tokenization_qwen.py
Normal file
246
tokenization_qwen.py
Normal file
@@ -0,0 +1,246 @@
|
|||||||
|
# Copyright (c) Alibaba Cloud.
|
||||||
|
#
|
||||||
|
# This source code is licensed under the license found in the
|
||||||
|
# LICENSE file in the root directory of this source tree.
|
||||||
|
|
||||||
|
"""Tokenization classes for QWen."""
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import unicodedata
|
||||||
|
from typing import Collection, Dict, List, Set, Tuple, Union
|
||||||
|
|
||||||
|
import tiktoken
|
||||||
|
from transformers import PreTrainedTokenizer, AddedToken
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
VOCAB_FILES_NAMES = {"vocab_file": "qwen.tiktoken"}
|
||||||
|
|
||||||
|
PAT_STR = r"""(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+"""
|
||||||
|
ENDOFTEXT = "<|endoftext|>"
|
||||||
|
IMSTART = "<|im_start|>"
|
||||||
|
IMEND = "<|im_end|>"
|
||||||
|
# as the default behavior is changed to allow special tokens in
|
||||||
|
# regular texts, the surface forms of special tokens need to be
|
||||||
|
# as different as possible to minimize the impact
|
||||||
|
EXTRAS = tuple((f"<|extra_{i}|>" for i in range(205)))
|
||||||
|
SPECIAL_TOKENS = (
|
||||||
|
ENDOFTEXT,
|
||||||
|
IMSTART,
|
||||||
|
IMEND,
|
||||||
|
) + EXTRAS
|
||||||
|
|
||||||
|
|
||||||
|
def _load_tiktoken_bpe(tiktoken_bpe_file: str) -> Dict[bytes, int]:
|
||||||
|
with open(tiktoken_bpe_file, "rb") as f:
|
||||||
|
contents = f.read()
|
||||||
|
return {
|
||||||
|
base64.b64decode(token): int(rank)
|
||||||
|
for token, rank in (line.split() for line in contents.splitlines() if line)
|
||||||
|
}
|
||||||
|
|
||||||
|
class QWenTokenizer(PreTrainedTokenizer):
|
||||||
|
"""QWen tokenizer."""
|
||||||
|
|
||||||
|
vocab_files_names = VOCAB_FILES_NAMES
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
vocab_file,
|
||||||
|
errors="replace",
|
||||||
|
**kwargs,
|
||||||
|
):
|
||||||
|
super().__init__(**kwargs)
|
||||||
|
|
||||||
|
self.errors = errors # how to handle errors in decoding
|
||||||
|
|
||||||
|
self.mergeable_ranks = _load_tiktoken_bpe(vocab_file) # type: dict[bytes, int]
|
||||||
|
self.special_tokens = {
|
||||||
|
token: index
|
||||||
|
for index, token in enumerate(
|
||||||
|
SPECIAL_TOKENS, start=len(self.mergeable_ranks)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
enc = tiktoken.Encoding(
|
||||||
|
"Qwen",
|
||||||
|
pat_str=PAT_STR,
|
||||||
|
mergeable_ranks=self.mergeable_ranks,
|
||||||
|
special_tokens=self.special_tokens,
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
len(self.mergeable_ranks) + len(self.special_tokens) == enc.n_vocab
|
||||||
|
), f"{len(self.mergeable_ranks) + len(self.special_tokens)} != {enc.n_vocab} in encoding"
|
||||||
|
|
||||||
|
self.decoder = {
|
||||||
|
v: k for k, v in self.mergeable_ranks.items()
|
||||||
|
} # type: dict[int, bytes|str]
|
||||||
|
self.decoder.update({v: k for k, v in self.special_tokens.items()})
|
||||||
|
|
||||||
|
self.tokenizer = enc # type: tiktoken.Encoding
|
||||||
|
|
||||||
|
self.eod_id = self.tokenizer.eot_token
|
||||||
|
self.im_start_id = self.special_tokens[IMSTART]
|
||||||
|
self.im_end_id = self.special_tokens[IMEND]
|
||||||
|
|
||||||
|
def __getstate__(self):
|
||||||
|
# for pickle lovers
|
||||||
|
state = self.__dict__.copy()
|
||||||
|
del state['tokenizer']
|
||||||
|
return state
|
||||||
|
|
||||||
|
def __setstate__(self, state):
|
||||||
|
# tokenizer is not python native; don't pass it; rebuild it
|
||||||
|
self.__dict__.update(state)
|
||||||
|
enc = tiktoken.Encoding(
|
||||||
|
"Qwen",
|
||||||
|
pat_str=PAT_STR,
|
||||||
|
mergeable_ranks=self.mergeable_ranks,
|
||||||
|
special_tokens=self.special_tokens,
|
||||||
|
)
|
||||||
|
self.tokenizer = enc
|
||||||
|
|
||||||
|
|
||||||
|
def __len__(self) -> int:
|
||||||
|
return self.tokenizer.n_vocab
|
||||||
|
|
||||||
|
def get_vocab(self) -> Dict[bytes, int]:
|
||||||
|
return self.mergeable_ranks
|
||||||
|
|
||||||
|
def convert_tokens_to_ids(
|
||||||
|
self, tokens: Union[bytes, str, List[Union[bytes, str]]]
|
||||||
|
) -> List[int]:
|
||||||
|
ids = []
|
||||||
|
if isinstance(tokens, (str, bytes)):
|
||||||
|
if tokens in self.special_tokens:
|
||||||
|
return self.special_tokens[tokens]
|
||||||
|
else:
|
||||||
|
return self.mergeable_ranks.get(tokens)
|
||||||
|
for token in tokens:
|
||||||
|
if token in self.special_tokens:
|
||||||
|
ids.append(self.special_tokens[token])
|
||||||
|
else:
|
||||||
|
ids.append(self.mergeable_ranks.get(token))
|
||||||
|
return ids
|
||||||
|
|
||||||
|
def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int:
|
||||||
|
if not special_tokens and new_tokens:
|
||||||
|
raise ValueError('Adding regular tokens is not supported')
|
||||||
|
for token in new_tokens:
|
||||||
|
surface_form = token.content if isinstance(token, AddedToken) else token
|
||||||
|
if surface_form not in SPECIAL_TOKENS:
|
||||||
|
raise ValueError('Adding unknown special tokens is not supported')
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def save_vocabulary(self, save_directory: str, **kwargs) -> Tuple[str]:
|
||||||
|
"""
|
||||||
|
Save only the vocabulary of the tokenizer (vocabulary).
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
`Tuple(str)`: Paths to the files saved.
|
||||||
|
"""
|
||||||
|
file_path = os.path.join(save_directory, "qwen.tiktoken")
|
||||||
|
with open(file_path, "w", encoding="utf8") as w:
|
||||||
|
for k, v in self.mergeable_ranks.items():
|
||||||
|
line = base64.b64encode(k).decode("utf8") + " " + str(v) + "\n"
|
||||||
|
w.write(line)
|
||||||
|
return (file_path,)
|
||||||
|
|
||||||
|
def tokenize(
|
||||||
|
self,
|
||||||
|
text: str,
|
||||||
|
allowed_special: Union[Set, str] = "all",
|
||||||
|
disallowed_special: Union[Collection, str] = (),
|
||||||
|
**kwargs,
|
||||||
|
) -> List[Union[bytes, str]]:
|
||||||
|
"""
|
||||||
|
Converts a string in a sequence of tokens.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text (`str`):
|
||||||
|
The sequence to be encoded.
|
||||||
|
allowed_special (`Literal["all"]` or `set`):
|
||||||
|
The surface forms of the tokens to be encoded as special tokens in regular texts.
|
||||||
|
Default to "all".
|
||||||
|
disallowed_special (`Literal["all"]` or `Collection`):
|
||||||
|
The surface forms of the tokens that should not be in regular texts and trigger errors.
|
||||||
|
Default to an empty tuple.
|
||||||
|
|
||||||
|
kwargs (additional keyword arguments, *optional*):
|
||||||
|
Will be passed to the underlying model specific encode method.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
`List[bytes|str]`: The list of tokens.
|
||||||
|
"""
|
||||||
|
tokens = []
|
||||||
|
text = unicodedata.normalize("NFC", text)
|
||||||
|
|
||||||
|
# this implementation takes a detour: text -> token id -> token surface forms
|
||||||
|
for t in self.tokenizer.encode(
|
||||||
|
text, allowed_special=allowed_special, disallowed_special=disallowed_special
|
||||||
|
):
|
||||||
|
tokens.append(self.decoder[t])
|
||||||
|
return tokens
|
||||||
|
|
||||||
|
def convert_tokens_to_string(self, tokens: List[Union[bytes, str]]) -> str:
|
||||||
|
"""
|
||||||
|
Converts a sequence of tokens in a single string.
|
||||||
|
"""
|
||||||
|
text = ""
|
||||||
|
temp = b""
|
||||||
|
for t in tokens:
|
||||||
|
if isinstance(t, str):
|
||||||
|
if temp:
|
||||||
|
text += temp.decode("utf-8", errors=self.errors)
|
||||||
|
temp = b""
|
||||||
|
text += t
|
||||||
|
elif isinstance(t, bytes):
|
||||||
|
temp += t
|
||||||
|
else:
|
||||||
|
raise TypeError("token should only be of type types or str")
|
||||||
|
if temp:
|
||||||
|
text += temp.decode("utf-8", errors=self.errors)
|
||||||
|
return text
|
||||||
|
|
||||||
|
@property
|
||||||
|
def vocab_size(self):
|
||||||
|
return self.tokenizer.n_vocab
|
||||||
|
|
||||||
|
def _convert_id_to_token(self, index: int) -> Union[bytes, str]:
|
||||||
|
"""Converts an id to a token, special tokens included"""
|
||||||
|
if index in self.decoder:
|
||||||
|
return self.decoder[index]
|
||||||
|
raise ValueError("unknown ids")
|
||||||
|
|
||||||
|
def _convert_token_to_id(self, token: Union[bytes, str]) -> int:
|
||||||
|
"""Converts a token to an id using the vocab, special tokens included"""
|
||||||
|
if token in self.special_tokens:
|
||||||
|
return self.special_tokens[token]
|
||||||
|
if token in self.mergeable_ranks:
|
||||||
|
return self.mergeable_ranks[token]
|
||||||
|
raise ValueError("unknown token")
|
||||||
|
|
||||||
|
def _tokenize(self, text: str, **kwargs):
|
||||||
|
"""
|
||||||
|
Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based
|
||||||
|
vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces).
|
||||||
|
|
||||||
|
Do NOT take care of added tokens.
|
||||||
|
"""
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
def _decode(
|
||||||
|
self,
|
||||||
|
token_ids: Union[int, List[int]],
|
||||||
|
skip_special_tokens: bool = False,
|
||||||
|
errors: str = None,
|
||||||
|
**kwargs,
|
||||||
|
) -> str:
|
||||||
|
if isinstance(token_ids, int):
|
||||||
|
token_ids = [token_ids]
|
||||||
|
if skip_special_tokens:
|
||||||
|
token_ids = [i for i in token_ids if i < self.eod_id]
|
||||||
|
return self.tokenizer.decode(token_ids, errors=errors or self.errors)
|
||||||
10
tokenizer_config.json
Normal file
10
tokenizer_config.json
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
{
|
||||||
|
"model_max_length": 1000000000000000019884624838656,
|
||||||
|
"tokenizer_class": "QWenTokenizer",
|
||||||
|
"auto_map": {
|
||||||
|
"AutoTokenizer": [
|
||||||
|
"tokenization_qwen.QWenTokenizer",
|
||||||
|
null
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user