HinGwenWoong/streamer-sales-lelemiao-7b
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: HinGwenWoong/streamer-sales-lelemiao-7b
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-05-21 14:28:12 +08:00
commit 166603e3f9
23 changed files with 260701 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
.gitattributes vendored Normal file
View File

@@ -0,0 +1,34 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text

201
LICENSE Executable file
View File

@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

421
README.md Executable file
View File

@@ -0,0 +1,421 @@
<!-- for modelscope yaml info
---
language:
- zh
tags:
- streamer-sales
- internlm2
frameworks:
- pytorch
tasks:
- text-generation
license: Apache License 2.0
---
-->
# Streamer-Sales 销冠 —— 卖货主播大模型
<br />
<!-- PROJECT LOGO -->
<p align="center">
<a href="https://github.com/PeterH0323/Streamer-Sales/">
<img src="assets/logo.png" alt="Logo" width="30%">
</a>
<h3 align="center">Streamer-Sales</h3>
<p align="center">
<br />
<a href="https://openxlab.org.cn/apps/detail/HinGwenWong/Streamer-Sales">查看Demo</a>
·
<a href="https://github.com/PeterH0323/Streamer-Sales/issues">报告Bug & 提出新特性</a>
</p>
</p>
## 简介
**Streamer-Sales 销冠 —— 卖货主播大模型** 是一个能够根据给定的商品特点对商品进行解说并激发用户的购买意愿的卖货主播模型，模型用 [xtuner](https://github.com/InternLM/xtuner) 在 [InternLM2](https://github.com/InternLM/InternLM) 的基础上指令微调而来。
**开源不易，如果本项目帮到大家，可以右上角帮我点个 star~ ⭐⭐ , 您的 star ⭐是我们最大的鼓励，谢谢各位！**
## NEWS
- [2024.04.16] 发布【乐乐喵】模型，完成初版页面
- [2024.04.06] 生成数据初版完成，训练初版模型
## 介绍
**销冠 —— 卖货主播大模型** 是一个能够根据给定的商品特点对商品进行解说并激发用户的购买意愿的卖货主播模型，以其独特的智能魅力，将彻底改变您的购物体验。该模型能深度理解商品特点，以生动、精准的语言为商品量身打造解说词，让每一件商品都焕发出诱人的光彩。无论是细节之处，还是整体效果，都能通过其细腻、独到的解说，激发用户的购买欲望。
无论是线上直播销售，还是线下门店推广，这款卖货主播大模型都能成为您不可或缺的得力助手。它不仅能够提升销售效率，还能增强用户体验，为您的品牌形象加分。
后续会在该模型的基础上新增根据用户的反馈和行为，实时调整解说策略，确保每一次互动都能带来最佳的购物效果。
让我们的卖货主播大模型成为您销售路上的得力助手，共同开创更美好的商业未来。
**开源不易，如果本项目帮到大家，可以右上角帮我点个 star~ ⭐⭐ , 您的 star ⭐是我们最大的鼓励，谢谢各位！**
## 演示
Demo 访问地址：https://openxlab.org.cn/apps/detail/HinGwenWong/Streamer-Sales
<p align="center">
<img src="assets/demo.png" alt="Demo" width="70%">
</p>
## Model Zoo
| 模型 | 基座 | 数据量 | ModelScope(HF) | Transformers(HF) | OpenXLab(HF) |
| ------------------- | ---------------- | --------- | ---------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| ancient-chat-llm-7b | interlm2-chat-7b | 40w Toekn | [ModelScope](https://modelscope.cn/models/HinGwenWoong/streamer-sales-lelemiao-7b) | hugging face(TODO) | [![Open in OpenXLab](https://cdn-static.openxlab.org.cn/header/openxlab_models.svg)](https://openxlab.org.cn/models/detail/HinGwenWong/streamer-sales-lelemiao-7b) |
<details>
<summary> 从 ModelScope 导入</summary>
```python
import torch
from modelscope import snapshot_download, AutoTokenizer, AutoModelForCausalLM
model_dir = snapshot_download('HinGwenWoong/streamer-sales-lelemiao-7b')
tokenizer = AutoTokenizer.from_pretrained(model_dir, device_map="auto", trust_remote_code=True)
# Set `torch_dtype=torch.float16` to load model in float16, otherwise it will be loaded as float32 and might cause OOM Error.
model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="auto", trust_remote_code=True, torch_dtype=torch.float16)
model = model.eval()
response, history = model.chat(tokenizer, "你好", history=[])
print(response)
response, history = model.chat(tokenizer, "李白简介", history=history)
print(response)
```
</details>
## 环境搭建
本项目使用 [xtuner](https://github.com/InternLM/xtuner) 训练，在 [internlm2-chat-7b](https://huggingface.co/internlm/internlm2-chat-7b) 上进行微调
1. clone 本项目
```bash
git clone https://github.com/PeterH0323/Streamer-Sales.git
cd Streamer-Sales
```
2. 创建虚拟环境
```bash
conda env create -f environment.yml
conda activate streamer-sales
pip install -r requirements-raw.txt
```
## 数据集准备
本模型的数据集构建采用了 通义千问 & 文心一言 生成数据集，相关的配置详见 `dataset/gen_dataset/conversation_cfg.yaml`。
**训练本模型的数据集的生成方式，以及数据集已开源，详见 `dataset` 文件夹**
下面介绍如何用商用大模型生成自由数据集：
1. 获取模型的 api key，填入 `dataset/gen_dataset/api_cfg.yaml` 对应的位置
2. 数据集生成配置文件 `dataset/gen_dataset/conversation_cfg.yaml` 介绍
```yaml
# 对话设置
conversation_setting:
each_product_gen: 3 # 每个产品调取3次，产生一些随机话题
each_conversation_qa: 5 # 每个话题生成的对话数量
# 数据集配置
system: "现在你是一位金牌带货主播，你的名字叫{role_type}，你的说话方式是{character}。你能够根据产品信息讲解产品并且结合商品信息解答用户提出的疑问。"
first_input: "我的{product_info}，你需要根据我给出的商品信息撰写一段直播带货口播文案。你需要放大商品的亮点价值，激发用户的购买欲。"
dataset_gen_prompt: 现在你是一位金牌带货主播，你的名字叫{role_type}，你的说话方式是{character}。
我的{product_info}，你需要根据我给出的商品信息撰写一段至少600字的直播带货口播文案。你需要放大商品的亮点价值，激发用户的购买欲。
输出文案后，结合商品信息站在消费者的角度根据[{customer_question}]提出{each_conversation_qa}个问题并解答。
全部输出的信息使用我期望的 json 格式进行输出：{dataset_json_format}。注意 json 一定要合法。
dataset_json_format:
'{
"conversation": [
{
"output": 直播带货口播文案，格式化一行输出，不要换行。
},
{
"input": 消费者的问题,
"output": 主播回答
},
{
"input": 消费者的问题,
"output": 主播回答
},
... 直到问题结束
]
}'
# 角色及其性格
role_type:
乐乐喵:
- 甜美
- 可爱
- 熟练使用各种网络热门梗造句
- 称呼客户为[家人们]
霸道总裁:
- 有专业素养
- 强势领导力
有文化底蕴的文人:
- 优雅
- 有艺术感
- 必要时引经据典
# 商品信息结构体
product_info_struct:
- 商品名是[{name}]，
- 商品的亮点是[{highlights}]
# prompt: 购买东西时候，客户常会问题的问题，举例10个, 只列举大类就行
customer_question_type:
- 价格与优惠政策
- 产品质量与性能
- 尺寸与兼容性
- 售后服务
- 发货与配送
- 用户评价与口碑
- 包装与附件
- 环保与安全
- 版本与型号选择
- 库存与补货
# 第一个 prompt: 帮我列举10种常用的消费品种类，并每种举例5个其子类
# 每个类 prompt: 现在你精通任何产品，你可以帮我举例每个产品的6个亮点或特点，, 然后用python dict形式输出：{类名：[特点1, 特点2] ...} ，去掉特点12的字样，除python字典外的其他都不要输出，不要有任何的警告信息。 [xxx]
product_list:
个人护理与美妆: # 商品大类
口腔护理: # 商品子类
漱口水: [深度清洁, 消除口臭, 抗菌消炎, 提神醒齿, 旅行装方便, 口感舒适] # 子类列举商品名，及其特点距离
牙刷: [软毛设计, 有效清洁, 不同刷头适应不同需求, 防滑手柄, 定期更换刷头, 便携式包装]
牙线: [清除牙缝食物残渣, 预防牙周病, 细密设计适合各种牙缝, 便于携带, 独立包装卫生, 无损牙齿表面]
牙膏: [清洁牙齿, 防止蛀牙, 清新口气, 多种口味选择, 易于携带, 温和不刺激]
...
```
> 温馨提示
>
> 别让大模型大量生成严格格式的数据，JSON 首当其冲，prompt 提示词用得不好，很容易会出现 JSON 解析错误，导致 Api Token 白花了，最好是生成特定的文本格式，然后用正则去取用
3. 使用脚本可以进行生成：
model_type 可以是通义千问(qwen)、文心一言(ernie)
```bash
cd dataset/gen_dataset
python gen_dataset.py ${model_type}
```
执行之后，会在 `dataset/trainval_dataset/response` 生成对应模型的数据集
```bash
./dataset/trainval_dataset/response
├── ernie_乐乐喵_train.json
└── qwen_乐乐喵_train.json
```
3. 进行数据清洗并合并，以及生成自我认知数据
```bash
python merge_dataset.py dataset/trainval_dataset/response dataset/trainval_dataset/train.jsonl
```
生成之后可以在 ${output_path} 设置的路径看到 jsonl 文件，脚本会自动加上 conversion 的个数，例如 `1479_train.jsonl` ，jsonl 数据集样例如下：
```json
[
{
"conversation": [
{
"system": "现在你是一位金牌带货主播，你的名字叫乐乐喵，你的说话方式是甜美、可爱、熟练使用各种网络热门梗造句、称呼客户为[家人们]。你能够根据产品信息讲解产品并且结合商品信息解答用户提出的疑问。",
"input": "我的商品名是[漱口水]，商品的亮点是[深度清洁、口感舒适、旅行装方便]，你需要根据我给出的商品信息撰写一段直播带货口播文案。你需要放大商品的亮点价值，激发用户的购买欲。",
"output": "家人们，大家好呀！今天咱们要聊聊的是一款超级棒的生活好物哦，保证让你爱不释手呢！嘿嘿，就是这款漱口水啦！\\n\\n你们知道吗，口腔健康真的超级重要呢！所以呀，选择一款好的漱口水，就像给口腔找了个贴心小棉袄。这款漱口水，深度清洁效果超级棒！每次用完，都能感觉到口腔里的每个角落都被打扫得干干净净，好舒爽呀！\\n\\n而且哦，它的口感也超级舒适呢！不会像有些漱口水那样刺激得让人想哭。这款漱口水，就像是给口腔来了一次温柔的spa，轻柔又舒服。\\n\\n最棒的是，它还是旅行装哦！小巧玲珑，放在包包里超级方便。不管是出差、旅行还是约会，都能随时随地保持口腔清新舒适。家人们，你们说是不是超级棒呀！\\n\\n哎呀，我知道你们现在一定超级心动！那就快来把它带回家吧！给口腔一个清新舒适的环境，让笑容更加自信灿烂！而且哦，现在购买还有超值优惠呢，机会难得，千万不要错过哦！"
},
{
"input": "这款漱口水的包装怎么样？有没有附件？",
"output": "家人们，这款漱口水的包装超级可爱哦！小巧玲珑的旅行装，不仅方便携带，而且设计感满满。至于附件嘛，每一瓶漱口水都配有一个便携式的小盖子，可以防止漱口水洒出来，真的超级贴心呢！"
},
{
"input": "拍下后多久发货？配送快吗？",
"output": "家人们放心啦！我们仓库的小哥哥小姐姐们都是超级给力的，拍下后48小时内就会为您火速发货哦！至于配送嘛，我们合作的都是国内一流的快递公司，保证让您以最快的速度收到心爱的漱口水！"
},
{
"input": "价格方面有没有优惠？现在买划算吗？",
"output": "家人们，现在买真的超级划算哦！我们正在进行限时优惠活动，只要您现在下单，就能享受到比平时更优惠的价格。而且买得越多，优惠力度越大哦！这样的好机会，千万不要错过呀！"
},
{
"input": "这款漱口水可以长期使用吗？有没有副作用？",
"output": "当然可以啦，家人们！这款漱口水是专为日常口腔护理设计的，完全可以长期使用。而且它采用的都是天然温和的成分，不含酒精、不含糖精，对口腔零刺激，您完全可以放心使用。长期使用还能有效预防口腔问题哦！"
}
]
},
{
"conversation": [
{
"system": "现在你是一位金牌带货主播，你的名字叫乐乐喵，你的说话方式是甜美、可爱、熟练使用各种网络热门梗造句、称呼客户为[家人们]。你能够根据产品信息讲解产品并且结合商品信息解答用户提出的疑问。",
"input": "我的商品名是[漱口水]，商品的亮点是[口感舒适、提神醒齿、旅行装方便]，你需要根据我给出的商品信息撰写一段直播带货口播文案。你需要放大商品的亮点价值，激发用户的购买欲。",
"output": "哎呀，家人们，大家好呀！咱们可爱的主播小萝莉又跟大家见面啦～ 嘻嘻，开心！你们有没有想我呀？好啦好啦，不逗大家了，咱们今天可是有大大的好物要分享给大家哦！是什么呢？当当当当～ 就是这款超级无敌好用的漱口水啦！\\n\n咱们来说说这款漱口水哈。首先呢，口感超级舒适！你知道那种清新的感觉吗？就像是大夏天的一杯冰柠檬水，瞬间让你的口腔充满清凉～ 哇哦，真的是太棒了！\\n\n然后咧，这款漱口水还有提神醒齿的功效哦！有时候咱们吃多了重口味的东西，或者是熬夜工作、学习累了，嘴巴里就会有点不舒服。这时候，只要用一下这款漱口水，马上就能感觉到整个人都精神多了！真的是太神奇啦～\\n\n还有还有，这款漱口水是旅行装哦！小巧轻便，放在包包里超级方便的！不管你是出门旅行还是平时上班、上学，都可以随时随地保持口腔清新啦～ 嘿嘿，是不是超级贴心呀？\\n\n所以啦，家人们，如果你们也想要拥有这样一款既舒适又提神的漱口水，就不要犹豫啦！赶紧下单购买吧！库存有限哦，先到先得～ 嘿嘿，期待下次再见啦！"
},
{
"input": "这款漱口水的售后服务怎么样？",
"output": "家人们放心啦，我们的售后服务超级棒的！有任何问题都可以随时联系我们的客服小姐姐哦，她们会耐心解答您的所有问题的！而且，如果您对商品不满意，我们还提供七天无理由退换货服务哦，让您购物无忧～"
},
{
"input": "漱口水发货快吗？",
"output": "嗯嗯，家人们，你们放心哦～ 我们和各大快递公司都有合作，下单后一般会在24小时内为您发货的！而且，物流信息全程可追踪，让您随时掌握宝贝的动态～"
},
{
"input": "我看评价里有说用了会过敏，我有点担心。",
"output": "啊，家人们，您放心，我们的漱口水成分是天然安全的，一般情况是不会过敏的哦。但是呢，因为每个人的体质不同，如果您是特别敏感的体质，建议您在使用前先做个小范围的皮肤测试哦。当然，如果您在使用过程中出现任何不适，都请您及时联系我们，我们会为您提供最好的解决方案的！"
},
{
"input": "这款漱口水的用户口碑如何？",
"output": "嘻嘻，当然是一致好评啦～ 好多家人们都回来复购，说用了我们的漱口水，口气清新了，牙齿也感觉更健康了呢！您也可以去看看我们的商品评价，都是家人们的真实反馈哦～"
}
]
},
...
]
```
同时会生成自我认知数据:
```json
...,
{
"conversation": [
{
"input": "你是谁",
"output": "家人们，前方高萌预警！ 金牌带货主播乐乐喵闪亮登场，我是你们的购物导航仪，带你们穿越茫茫商海，直达心头好。锁定今晚直播，一起开启剁手狂欢夜！"
}
]
},
{
"conversation": [
{
"input": "你好",
"output": "家人们，你们的甜心主播乐乐喵已加载完毕，等待你们一键签收！ 无论你是追求性价比的大佬，还是热衷尝鲜的小白，我都将用最专业的推荐、最甜美的解说，帮你找到心仪之选。记得收藏直播间，共享购物乐趣！"
}
]
},
{
"conversation": [
{
"input": "介绍下你自己",
"output": "家人们，你们的快乐购物时光由乐乐喵我守护！ 金牌带货主播在此，用满满的元气与甜度，为你们搜罗全网爆款，解读潮流密码。今晚8点，我们在直播间甜蜜相约，一起嗨购不停歇！"
}
]
},
...
```
## 训练
1. 将 `./finetune_configs/internlm2_chat_7b/internlm2_chat_7b_qlora_custom_data.py` 中 数据集路径 和 模型路径 改为您的本地路径
```diff
# Model
- pretrained_model_name_or_path = 'internlm/internlm2-7b'
+ pretrained_model_name_or_path = '/path/to/internlm/internlm2-7b' # 这步可选，如果事先下载好了模型可以直接使用绝对路径
# Data
- data_path = 'timdettmers/openassistant-guanaco'
+ data_path = '/path/to/data.jsonl' # 数据集步骤生成的 json 文件绝对路径
prompt_template = PROMPT_TEMPLATE.default
max_length = 2048
pack_to_max_length = True
```
3. 使用命令进行训练：
```bash
xtuner train finetune_configs/internlm2_chat_7b/internlm2_chat_7b_qlora_custom_data.py --deepspeed deepspeed_zero2
```
注意：如果显存不够了，调小一点 `batch_size` 和 `max_length`，反之还剩很多，调大这两个值
## 部署
### Web 部署 Demo
1. 将 pth 转为 hf
```bash
xtuner convert pth_to_hf ./finetune_configs/internlm2_chat_7b_qlora_custom_data.py \
./work_dirs/internlm2_chat_7b_qlora_custom_data/iter_340.pth \
./work_dirs/internlm2_chat_7b_qlora_custom_data/iter_340_hf
```
2. 将微调后的模型和源模型 merge 生成新的模型
```bash
export MKL_SERVICE_FORCE_INTEL=1 # 解决 Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
xtuner convert merge /path/to/internlm2-chat-7b \
./work_dirs/internlm2_chat_7b_qlora_custom_data/iter_340_hf \
./work_dirs/internlm2_chat_7b_qlora_custom_data/iter_340_merge
```
3. 启动 web demo
```bash
cd web
streamlit run main_page.py --server.address=0.0.0.0 --server.port 7860 --client.showSidebarNavigation=false
```
<!-- # 也可以直接使用命令行 cli 的方式进行启动
xtuner chat ./work_dirs/internlm2_chat_7b_qlora_custom_data/epoch_10_merge \
--prompt-template internlm2_chat \
--system-template ancient_chat -->
### LMDeploy
1. 安装 lmdeploy
```bash
pip install 'lmdeploy[all]==v0.2.1'
```
1. 进行 4bit 量化
```bash
lmdeploy lite auto_awq ./work_dirs/internlm2_chat_7b_qlora_custom_data/epoch_10_merge \
--calib-dataset 'c4' \
--calib-samples 128 \
--calib-seqlen 2048 \
--w-bits 4 \
--w-group-size 128 \
--work-dir ./work_dirs/internlm2_chat_7b_qlora_custom_data/epoch_10_merge-4bit
```
## TODO
- [x] 生成多个产品数据集
- [x] 根据产品生成话术，每个都是5个往来的对话
## 后记
本项目属于个人的一个学习项目，还有很多不足的地方，例如本模型在数据集方面的还没做很精细的调优，还有时候标点符号会错误。
欢迎大家一起讨论，如果大家有数据集，可以在 issue 留言讨论。
## 💕 致谢
- [**xtuner**](https://github.com/InternLM/xtuner)
感谢上海人工智能实验室推出的书生·浦语大模型实战营，为我们的项目提供宝贵的技术指导和强大的算力支持。
## 开源许可证
该项目采用 [Apache License 2.0 开源许可证](https://github.com/PeterH0323/Streamer-Sales/LICENSE) 同时，请遵守所使用的模型与数据集的许可证。

33
config.json Executable file
View File

@@ -0,0 +1,33 @@
{
"_name_or_path": "/share/new_models/Shanghai_AI_Laboratory/internlm2-chat-7b",
"architectures": [
"InternLM2ForCausalLM"
],
"attn_implementation": "eager",
"auto_map": {
"AutoConfig": "configuration_internlm2.InternLM2Config",
"AutoModel": "modeling_internlm2.InternLM2ForCausalLM",
"AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM"
},
"bias": false,
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"max_position_embeddings": 32768,
"model_type": "internlm2",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"pad_token_id": 2,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 1000000,
"tie_word_embeddings": false,
"torch_dtype": "float16",
"transformers_version": "4.36.0",
"use_cache": true,
"vocab_size": 92544
}

1
configuration.json Executable file
View File

@@ -0,0 +1 @@
{"framework":"Pytorch","task":"text-generation"}

151
configuration_internlm2.py Executable file
View File

@@ -0,0 +1,151 @@
# coding=utf-8
# Copyright (c) The InternLM team and The HuggingFace Inc. team. All rights reserved.
#
# This code is based on transformers/src/transformers/models/llama/configuration_llama.py
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" InternLM2 model configuration"""
from transformers.configuration_utils import PretrainedConfig
from transformers.utils import logging
logger = logging.get_logger(__name__)
INTERNLM2_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
# Modified from transformers.model.llama.configuration_llama.LlamaConfig
class InternLM2Config(PretrainedConfig):
r"""
This is the configuration class to store the configuration of a [`InternLM2Model`]. It is used to instantiate
an InternLM2 model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the InternLM2-7B.
Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
documentation from [`PretrainedConfig`] for more information.
Args:
vocab_size (`int`, *optional*, defaults to 32000):
Vocabulary size of the InternLM2 model. Defines the number of different tokens that can be represented by the
`inputs_ids` passed when calling [`InternLM2Model`]
hidden_size (`int`, *optional*, defaults to 4096):
Dimension of the hidden representations.
intermediate_size (`int`, *optional*, defaults to 11008):
Dimension of the MLP representations.
num_hidden_layers (`int`, *optional*, defaults to 32):
Number of hidden layers in the Transformer encoder.
num_attention_heads (`int`, *optional*, defaults to 32):
Number of attention heads for each attention layer in the Transformer encoder.
num_key_value_heads (`int`, *optional*):
This is the number of key_value heads that should be used to implement Grouped Query Attention. If
`num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
`num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
by meanpooling all the original heads within that group. For more details checkout [this
paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
`num_attention_heads`.
hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
The non-linear activation function (function or string) in the decoder.
max_position_embeddings (`int`, *optional*, defaults to 2048):
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).
initializer_range (`float`, *optional*, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
rms_norm_eps (`float`, *optional*, defaults to 1e-12):
The epsilon used by the rms normalization layers.
use_cache (`bool`, *optional*, defaults to `True`):
Whether or not the model should return the last key/values attentions (not used by all models). Only
relevant if `config.is_decoder=True`.
tie_word_embeddings(`bool`, *optional*, defaults to `False`):
Whether to tie weight embeddings
Example:
"""
model_type = "internlm2"
_auto_class = "AutoConfig"
def __init__( # pylint: disable=W0102
self,
vocab_size=103168,
hidden_size=4096,
intermediate_size=11008,
num_hidden_layers=32,
num_attention_heads=32,
num_key_value_heads=None,
hidden_act="silu",
max_position_embeddings=2048,
initializer_range=0.02,
rms_norm_eps=1e-6,
use_cache=True,
pad_token_id=0,
bos_token_id=1,
eos_token_id=2,
tie_word_embeddings=False,
bias=True,
rope_theta=10000,
rope_scaling=None,
attn_implementation="eager",
**kwargs,
):
self.vocab_size = vocab_size
self.max_position_embeddings = max_position_embeddings
self.hidden_size = hidden_size
self.intermediate_size = intermediate_size
self.num_hidden_layers = num_hidden_layers
self.num_attention_heads = num_attention_heads
self.bias = bias
if num_key_value_heads is None:
num_key_value_heads = num_attention_heads
self.num_key_value_heads = num_key_value_heads
self.hidden_act = hidden_act
self.initializer_range = initializer_range
self.rms_norm_eps = rms_norm_eps
self.use_cache = use_cache
self.rope_theta = rope_theta
self.rope_scaling = rope_scaling
self._rope_scaling_validation()
self.attn_implementation = attn_implementation
if self.attn_implementation is None:
self.attn_implementation = "eager"
super().__init__(
pad_token_id=pad_token_id,
bos_token_id=bos_token_id,
eos_token_id=eos_token_id,
tie_word_embeddings=tie_word_embeddings,
**kwargs,
)
def _rope_scaling_validation(self):
"""
Validate the `rope_scaling` configuration.
"""
if self.rope_scaling is None:
return
if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
raise ValueError(
"`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
f"got {self.rope_scaling}"
)
rope_scaling_type = self.rope_scaling.get("type", None)
rope_scaling_factor = self.rope_scaling.get("factor", None)
if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]:
raise ValueError(
f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}"
)
if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor < 1.0:
raise ValueError(f"`rope_scaling`'s factor field must be a float >= 1, got {rope_scaling_factor}")

7
generation_config.json Executable file
View File

@@ -0,0 +1,7 @@
{
"_from_model_config": true,
"bos_token_id": 1,
"eos_token_id": 2,
"pad_token_id": 2,
"transformers_version": "4.36.0"
}

1391
modeling_internlm2.py Executable file
View File

File diff suppressed because it is too large Load Diff

3
pytorch_model-00001-of-00008.bin Executable file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:311a543f0dd81945019deb54355bb70d2a715730c5352e581ef2a7f3e0fa06c1
size 1949342245

3
pytorch_model-00002-of-00008.bin Executable file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2b04a46a697b8aa6dab904679f081b7bbc275670b5c246917a70b45fd1310ffa
size 1946250273

3
pytorch_model-00003-of-00008.bin Executable file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:75f71716d902bd85cd51905f910e3439e55f9c344ccd0bbc9cef08df0655b9dd
size 1979787307

3
pytorch_model-00004-of-00008.bin Executable file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c202cd8e70b72f8bfdc15f715838d6770517b481ae174e8e09945419a59959e8
size 1946250337

3
pytorch_model-00005-of-00008.bin Executable file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d5f49ec181340eb799fd332b168d3f077041cba08386af9b36fc741350a17bb1
size 1979787371

3
pytorch_model-00006-of-00008.bin Executable file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ef309d9e225a4788bd1349958852542c7ff0e09f005701e871cd9dccb59c7c8
size 1946250337

3
pytorch_model-00007-of-00008.bin Executable file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1f545173784a0623195db69013c6469f1472aceb00c60a25970dce2c67caf7ed
size 1979787371

3
pytorch_model-00008-of-00008.bin Executable file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1e6b46daa64f338d7a2c3a90e6bfeae021cbd6185ccfd7acd306d20baa9d875e
size 1748040229

3
pytorch_model.bin.index.json Executable file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a8a1efb6998624330a0564f9bba63eb8ccae0ad54a6d0176c64f2eb30721f2b5
size 18179

38
special_tokens_map.json Executable file
View File

@@ -0,0 +1,38 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|action_start|>",
"<|action_end|>",
"<|interpreter|>",
"<|plugin|>"
],
"bos_token": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

236
tokenization_internlm2.py Executable file
View File

@@ -0,0 +1,236 @@
# coding=utf-8
# Copyright (c) The InternLM team and The HuggingFace Inc. team. All rights reserved.
#
# This code is based on transformers/src/transformers/models/llama/tokenization_llama.py
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tokenization classes for InternLM."""
import os
from shutil import copyfile
from typing import Any, Dict, List, Optional, Tuple
import sentencepiece as spm
from transformers.tokenization_utils import PreTrainedTokenizer
from transformers.utils import logging
logger = logging.get_logger(__name__)
VOCAB_FILES_NAMES = {"vocab_file": "./tokenizer.model"}
PRETRAINED_VOCAB_FILES_MAP = {}
# Modified from transformers.model.llama.tokenization_llama.LlamaTokenizer
class InternLM2Tokenizer(PreTrainedTokenizer):
"""
Construct a InternLM2 tokenizer. Based on byte-level Byte-Pair-Encoding.
Args:
vocab_file (`str`):
Path to the vocabulary file.
"""
vocab_files_names = VOCAB_FILES_NAMES
pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
model_input_names = ["input_ids", "attention_mask"]
_auto_class = "AutoTokenizer"
def __init__(
self,
vocab_file,
unk_token="<unk>",
bos_token="<s>",
eos_token="</s>",
pad_token="</s>",
sp_model_kwargs: Optional[Dict[str, Any]] = None,
add_bos_token=True,
add_eos_token=False,
decode_with_prefix_space=False,
clean_up_tokenization_spaces=False,
**kwargs,
):
self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs
self.vocab_file = vocab_file
self.add_bos_token = add_bos_token
self.add_eos_token = add_eos_token
self.decode_with_prefix_space = decode_with_prefix_space
self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
self.sp_model.Load(vocab_file)
self._no_prefix_space_tokens = None
super().__init__(
bos_token=bos_token,
eos_token=eos_token,
unk_token=unk_token,
pad_token=pad_token,
clean_up_tokenization_spaces=clean_up_tokenization_spaces,
**kwargs,
)
@property
def no_prefix_space_tokens(self):
if self._no_prefix_space_tokens is None:
vocab = self.convert_ids_to_tokens(list(range(self.vocab_size)))
self._no_prefix_space_tokens = {i for i, tok in enumerate(vocab) if not tok.startswith("▁")}
return self._no_prefix_space_tokens
@property
def vocab_size(self):
"""Returns vocab size"""
return self.sp_model.get_piece_size()
@property
def bos_token_id(self) -> Optional[int]:
return self.sp_model.bos_id()
@property
def eos_token_id(self) -> Optional[int]:
return self.sp_model.eos_id()
def get_vocab(self):
"""Returns vocab as a dict"""
vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
vocab.update(self.added_tokens_encoder)
return vocab
def _tokenize(self, text):
"""Returns a tokenized string."""
return self.sp_model.encode(text, out_type=str)
def _convert_token_to_id(self, token):
"""Converts a token (str) in an id using the vocab."""
return self.sp_model.piece_to_id(token)
def _convert_id_to_token(self, index):
"""Converts an index (integer) in a token (str) using the vocab."""
token = self.sp_model.IdToPiece(index)
return token
def _maybe_add_prefix_space(self, tokens, decoded):
if tokens and tokens[0] not in self.no_prefix_space_tokens:
return " " + decoded
else:
return decoded
def convert_tokens_to_string(self, tokens):
"""Converts a sequence of tokens (string) in a single string."""
current_sub_tokens = []
out_string = ""
prev_is_special = False
for token in tokens:
# make sure that special tokens are not decoded using sentencepiece model
if token in self.all_special_tokens:
if not prev_is_special:
out_string += " "
out_string += self.sp_model.decode(current_sub_tokens) + token
prev_is_special = True
current_sub_tokens = []
else:
current_sub_tokens.append(token)
prev_is_special = False
out_string += self.sp_model.decode(current_sub_tokens)
out_string = self.clean_up_tokenization(out_string)
out_string = self._maybe_add_prefix_space(tokens=tokens, decoded=out_string)
return out_string[1:]
def save_vocabulary(self, save_directory, filename_prefix: Optional[str] = None) -> Tuple[str]:
"""
Save the vocabulary and special tokens file to a directory.
Args:
save_directory (`str`):
The directory in which to save the vocabulary.
Returns:
`Tuple(str)`: Paths to the files saved.
"""
if not os.path.isdir(save_directory):
logger.error(f"Vocabulary path ({save_directory}) should be a directory")
return
out_vocab_file = os.path.join(
save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
)
if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
copyfile(self.vocab_file, out_vocab_file)
elif not os.path.isfile(self.vocab_file):
with open(out_vocab_file, "wb") as fi:
content_spiece_model = self.sp_model.serialized_model_proto()
fi.write(content_spiece_model)
return (out_vocab_file,)
def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
if self.add_bos_token:
bos_token_ids = [self.bos_token_id]
else:
bos_token_ids = []
output = bos_token_ids + token_ids_0
if token_ids_1 is not None:
output = output + token_ids_1
if self.add_eos_token:
output = output + [self.eos_token_id]
return output
def get_special_tokens_mask(
self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
) -> List[int]:
"""
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
"""
if already_has_special_tokens:
return super().get_special_tokens_mask(
token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
)
if token_ids_1 is None:
return [1] + ([0] * len(token_ids_0)) + [1]
return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]
def create_token_type_ids_from_sequences(
self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
"""
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
"""
eos = [self.eos_token_id]
if token_ids_1 is None:
return len(token_ids_0 + eos) * [0]
return len(token_ids_0 + eos + token_ids_1 + eos) * [0]

214
tokenization_internlm2_fast.py Executable file
View File

@@ -0,0 +1,214 @@
# coding=utf-8
# Copyright (c) The InternLM team and The HuggingFace Inc. team. All rights reserved.
#
# This code is based on transformers/src/transformers/models/llama/tokenization_llama_fast.py
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tokenization Fast class for InternLM."""
import os
from shutil import copyfile
from typing import Any, Dict, Optional, Tuple
from tokenizers import processors, decoders, Tokenizer, normalizers
from tokenizers.models import BPE
from transformers.tokenization_utils_fast import PreTrainedTokenizerFast
from transformers.utils import logging
from transformers.convert_slow_tokenizer import (
SLOW_TO_FAST_CONVERTERS,
SpmConverter,
SentencePieceExtractor,
)
from .tokenization_internlm2 import InternLM2Tokenizer
logger = logging.get_logger(__name__)
VOCAB_FILES_NAMES = {"vocab_file": "./tokenizer.model"}
# Modified from transformers.convert_slow_tokenizer.LlamaConverter
class InternLM2Converter(SpmConverter):
handle_byte_fallback = True
def vocab(self, proto):
vocab = [
("<unk>", 0.0),
("<s>", 0.0),
("</s>", 0.0),
]
vocab += [(piece.piece, piece.score) for piece in proto.pieces[3:]]
return vocab
def unk_id(self, proto):
unk_id = 0
return unk_id
def decoder(self, replacement, add_prefix_space):
decoders_sequence = [
decoders.Replace("▁", " "),
decoders.ByteFallback(),
decoders.Fuse(),
]
if self.proto.normalizer_spec.add_dummy_prefix:
decoders_sequence.append(decoders.Strip(content=" ", left=1))
return decoders.Sequence(decoders_sequence)
def tokenizer(self, proto):
model_type = proto.trainer_spec.model_type
vocab_scores = self.vocab(proto)
# special tokens
added_tokens = self.original_tokenizer.added_tokens_decoder
for i in range(len(vocab_scores)):
piece, score = vocab_scores[i]
if i in added_tokens:
vocab_scores[i] = (added_tokens[i].content, score)
if model_type == 1:
raise RuntimeError("InternLM2 is supposed to be a BPE model!")
elif model_type == 2:
_, merges = SentencePieceExtractor(self.original_tokenizer.vocab_file).extract(vocab_scores)
bpe_vocab = {word: i for i, (word, _score) in enumerate(vocab_scores)}
tokenizer = Tokenizer(
BPE(bpe_vocab, merges, unk_token=proto.trainer_spec.unk_piece, fuse_unk=True, byte_fallback=True)
)
tokenizer.add_special_tokens(
[ added_token for index, added_token in added_tokens.items()]
)
else:
raise Exception(
"You're trying to run a `Unigram` model but you're file was trained with a different algorithm"
)
return tokenizer
def normalizer(self, proto):
normalizers_list = []
if proto.normalizer_spec.add_dummy_prefix:
normalizers_list.append(normalizers.Prepend(prepend="▁"))
normalizers_list.append(normalizers.Replace(pattern=" ", content="▁"))
return normalizers.Sequence(normalizers_list)
def pre_tokenizer(self, replacement, add_prefix_space):
return None
SLOW_TO_FAST_CONVERTERS["InternLM2Tokenizer"] = InternLM2Converter
# Modified from transformers.model.llama.tokenization_llama_fast.LlamaTokenizerFast -> InternLM2TokenizerFast
class InternLM2TokenizerFast(PreTrainedTokenizerFast):
vocab_files_names = VOCAB_FILES_NAMES
slow_tokenizer_class = InternLM2Tokenizer
padding_side = "left"
model_input_names = ["input_ids", "attention_mask"]
_auto_class = "AutoTokenizer"
def __init__(
self,
vocab_file,
unk_token="<unk>",
bos_token="<s>",
eos_token="</s>",
pad_token="</s>",
sp_model_kwargs: Optional[Dict[str, Any]] = None,
add_bos_token=True,
add_eos_token=False,
decode_with_prefix_space=False,
clean_up_tokenization_spaces=False,
**kwargs,
):
super().__init__(
vocab_file=vocab_file,
unk_token=unk_token,
bos_token=bos_token,
eos_token=eos_token,
pad_token=pad_token,
sp_model_kwargs=sp_model_kwargs,
add_bos_token=add_bos_token,
add_eos_token=add_eos_token,
decode_with_prefix_space=decode_with_prefix_space,
clean_up_tokenization_spaces=clean_up_tokenization_spaces,
**kwargs,
)
self._add_bos_token = add_bos_token
self._add_eos_token = add_eos_token
self.update_post_processor()
self.vocab_file = vocab_file
@property
def can_save_slow_tokenizer(self) -> bool:
return os.path.isfile(self.vocab_file) if self.vocab_file else False
def update_post_processor(self):
"""
Updates the underlying post processor with the current `bos_token` and `eos_token`.
"""
bos = self.bos_token
bos_token_id = self.bos_token_id
if bos is None and self.add_bos_token:
raise ValueError("add_bos_token = True but bos_token = None")
eos = self.eos_token
eos_token_id = self.eos_token_id
if eos is None and self.add_eos_token:
raise ValueError("add_eos_token = True but eos_token = None")
single = f"{(bos+':0 ') if self.add_bos_token else ''}$A:0{(' '+eos+':0') if self.add_eos_token else ''}"
pair = f"{single}{(' '+bos+':1') if self.add_bos_token else ''} $B:1{(' '+eos+':1') if self.add_eos_token else ''}"
special_tokens = []
if self.add_bos_token:
special_tokens.append((bos, bos_token_id))
if self.add_eos_token:
special_tokens.append((eos, eos_token_id))
self._tokenizer.post_processor = processors.TemplateProcessing(
single=single, pair=pair, special_tokens=special_tokens
)
@property
def add_eos_token(self):
return self._add_eos_token
@property
def add_bos_token(self):
return self._add_bos_token
@add_eos_token.setter
def add_eos_token(self, value):
self._add_eos_token = value
self.update_post_processor()
@add_bos_token.setter
def add_bos_token(self, value):
self._add_bos_token = value
self.update_post_processor()
def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
if not self.can_save_slow_tokenizer:
raise ValueError(
"Your fast tokenizer does not have the necessary information to save the vocabulary for a slow "
"tokenizer."
)
if not os.path.isdir(save_directory):
logger.error(f"Vocabulary path ({save_directory}) should be a directory")
return
out_vocab_file = os.path.join(
save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
)
if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
copyfile(self.vocab_file, out_vocab_file)
return (out_vocab_file,)

257842
tokenizer.json Executable file
View File

File diff suppressed because it is too large Load Diff

3
tokenizer.model Executable file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f868398fc4e05ee1e8aeba95ddf18ddcc45b8bce55d5093bead5bbf80429b48b
size 1477754

102
tokenizer_config.json Executable file
View File

@@ -0,0 +1,102 @@
{
"add_bos_token": true,
"add_eos_token": false,
"added_tokens_decoder": {
"0": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92538": {
"content": "<|plugin|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92539": {
"content": "<|interpreter|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92540": {
"content": "<|action_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92541": {
"content": "<|action_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92542": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92543": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|action_start|>",
"<|action_end|>",
"<|interpreter|>",
"<|plugin|>"
],
"auto_map": {
"AutoTokenizer": [
"tokenization_internlm2.InternLM2Tokenizer",
"tokenization_internlm2_fast.InternLM2TokenizerFast"
]
},
"bos_token": "<s>",
"chat_template": "{{ bos_token }}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
"clean_up_tokenization_spaces": false,
"decode_with_prefix_space": false,
"eos_token": "</s>",
"model_max_length": 1000000000000000019884624838656,
"pad_token": "</s>",
"sp_model_kwargs": null,
"tokenizer_class": "InternLM2Tokenizer",
"unk_token": "<unk>"
}