AI4Chem/ChemLLM-7B-Chat-1_5-SFT
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: AI4Chem/ChemLLM-7B-Chat-1_5-SFT
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-20 21:15:11 +08:00
commit 2b2bd8d586
19 changed files with 2105 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

56
README.md Normal file
View File

@@ -0,0 +1,56 @@
---
license: other
pipeline_tag: text-generation
tags:
- chemistry
language:
- en
- zh
---
# ChemLLM-7B-Chat-1.5-SFT: LLM for Chemistry and Molecule Science
ChemLLM-7B-Chat-1.5-SFT, The First Open-source Large Language Model for Chemistry and Molecule Science, Build based on InternLM-2 with ❤
[![Paper page](https://huggingface.co/datasets/huggingface/badges/resolve/main/paper-page-sm.svg)](https://huggingface.co/papers/2402.06852)
<center><img src='https://cdn-uploads.huggingface.co/production/uploads/64bce15bafd1e46c5504ad38/wdFV6p3rTBCtskbeuVwNJ.png'></center>
## News
- ChemLLM-1.5 updated! Have a try on [Demo Site](https://chemllm.org/#/chat) or [API Reference](https://api.chemllm.org/docs).[2024-3-23]
- ChemLLM has been featured by HuggingFace on [“Daily Papers” page](https://huggingface.co/papers/2402.06852).[2024-2-13]
- ChemLLM arXiv preprint released.[ChemLLM: A Chemical Large Language Model](https://arxiv.org/abs/2402.06852)[2024-2-10]
- News report from [Shanghai AI Lab](https://mp.weixin.qq.com/s/u-i7lQxJzrytipek4a87fw)[2024-1-26]
- ChemLLM-7B-Chat ver 1.0 released. https://chemllm.org/ [2024-1-18]
- ChemLLM-7B-Chat ver 1.0 open-sourced.[2024-1-17]
- Chepybara ver 0.2 online Demo released. https://chemllm.org/ [2023-12-9]
## Usage
Try [online demo](https://chemllm.org/) instantly, or Deploy it locally.
## Cite this work
```
@misc{zhang2024chemllm,
title={ChemLLM: A Chemical Large Language Model},
author={Di Zhang and Wei Liu and Qian Tan and Jingdan Chen and Hang Yan and Yuliang Yan and Jiatong Li and Weiran Huang and Xiangyu Yue and Dongzhan Zhou and Shufei Zhang and Mao Su and Hansen Zhong and Yuqiang Li and Wanli Ouyang},
year={2024},
eprint={2402.06852},
archivePrefix={arXiv},
primaryClass={cs.AI}
}
```
## Disclaimer
LLM may generate incorrect answers, Please pay attention to proofreading at your own risk.
## Open Source License
The code is licensed under Apache-2.0, while model weights are fully open for academic research and also allow **free** commercial usage. To apply for a commercial license, or other questions and collaborations, please contact <support@chemllm.org>.
## Demo
[Agent Chepybara](https://chemllm.org/)
![image/png](https://cdn-uploads.huggingface.co/production/uploads/64bce15bafd1e46c5504ad38/Vg0ht3KTlFTJBSvnuSptD.png)
## Contact
(AI4Physics Sciecne, Shanghai AI Lab)[support@chemllm.org]

39
config.json Normal file
View File

@@ -0,0 +1,39 @@
{
"_name_or_path": "/mnt/cache/Chemllm/LLaMA-Factory-dev/CHEMLLM_1_5",
"architectures": [
"InternLM2ForCausalLM"
],
"auto_map": {
"AutoConfig": "configuration_internlm.InternLMConfig",
"AutoModel": "modeling_internlm2.InternLM2ForCausalLM",
"AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM"
},
"bias": false,
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"max_position_embeddings": 32768,
"model_type": "internlm",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"pad_token_id": 2,
"rms_norm_eps": 1e-05,
"rope_scaling": {
"base": 1000000,
"factor": 1.0,
"type": "dynamic"
},
"rotary": {
"base": 1000000,
"type": "dynamic"
},
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.37.2",
"use_cache": true,
"vocab_size": 92544
}

135
configuration_internlm.py Normal file
View File

@@ -0,0 +1,135 @@
# coding=utf-8
# Copyright (c) InternLM. All rights reserved.
#
# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
# and OPT implementations in this library. It has been modified from its
# original forms to accommodate minor architectural differences compared
# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" InternLM model configuration"""
from transformers.configuration_utils import PretrainedConfig
from transformers.utils import logging
logger = logging.get_logger(__name__)
INTERNLM_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
class InternLMConfig(PretrainedConfig):
r"""
This is the configuration class to store the configuration of a [`InternLMModel`]. It is used to instantiate
an InternLM model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the InternLM-7B.
Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
documentation from [`PretrainedConfig`] for more information.
Args:
vocab_size (`int`, *optional*, defaults to 32000):
Vocabulary size of the InternLM model. Defines the number of different tokens that can be represented by the
`inputs_ids` passed when calling [`InternLMModel`]
hidden_size (`int`, *optional*, defaults to 4096):
Dimension of the hidden representations.
intermediate_size (`int`, *optional*, defaults to 11008):
Dimension of the MLP representations.
num_hidden_layers (`int`, *optional*, defaults to 32):
Number of hidden layers in the Transformer encoder.
num_attention_heads (`int`, *optional*, defaults to 32):
Number of attention heads for each attention layer in the Transformer encoder.
num_key_value_heads (`int`, *optional*):
This is the number of key_value heads that should be used to implement Grouped Query Attention. If
`num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
`num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
by meanpooling all the original heads within that group. For more details checkout [this
paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
`num_attention_heads`.
hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
The non-linear activation function (function or string) in the decoder.
max_position_embeddings (`int`, *optional*, defaults to 2048):
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).
initializer_range (`float`, *optional*, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
rms_norm_eps (`float`, *optional*, defaults to 1e-12):
The epsilon used by the rms normalization layers.
use_cache (`bool`, *optional*, defaults to `True`):
Whether or not the model should return the last key/values attentions (not used by all models). Only
relevant if `config.is_decoder=True`.
tie_word_embeddings(`bool`, *optional*, defaults to `False`):
Whether to tie weight embeddings
Example:
```python
>>> from transformers import InternLMModel, InternLMConfig
>>> # Initializing a InternLM internlm-7b style configuration
>>> configuration = InternLMConfig()
>>> # Initializing a model from the internlm-7b style configuration
>>> model = InternLMModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
```"""
model_type = "internlm"
_auto_class = "AutoConfig"
def __init__( # pylint: disable=W0102
self,
vocab_size=103168,
hidden_size=4096,
intermediate_size=11008,
num_hidden_layers=32,
num_attention_heads=32,
num_key_value_heads=None,
hidden_act="silu",
max_position_embeddings=2048,
initializer_range=0.02,
rms_norm_eps=1e-6,
use_cache=True,
pad_token_id=0,
bos_token_id=1,
eos_token_id=2,
tie_word_embeddings=False,
bias=True,
rotary={"base": 10000, "type": "dynamic"}, # pylint: disable=W0102
**kwargs,
):
self.vocab_size = vocab_size
self.max_position_embeddings = max_position_embeddings
self.hidden_size = hidden_size
self.intermediate_size = intermediate_size
self.num_hidden_layers = num_hidden_layers
self.num_attention_heads = num_attention_heads
if num_key_value_heads is None:
num_key_value_heads = num_attention_heads
self.num_key_value_heads = num_key_value_heads
self.hidden_act = hidden_act
self.initializer_range = initializer_range
self.rms_norm_eps = rms_norm_eps
self.use_cache = use_cache
self.bias = bias
self.rotary = self.rope_scaling = rotary
super().__init__(
pad_token_id=pad_token_id,
bos_token_id=bos_token_id,
eos_token_id=eos_token_id,
tie_word_embeddings=tie_word_embeddings,
**kwargs,
)

7
generation_config.json Normal file
View File

@@ -0,0 +1,7 @@
{
"_from_model_config": true,
"bos_token_id": 1,
"eos_token_id": 2,
"pad_token_id": 2,
"transformers_version": "4.37.2"
}

3
model-00001-of-00008.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:69893816e3b6a5c5265d5cdde840eb25050da6f72771959a2a3b0309bdb2f19d
size 1949337704

3
model-00002-of-00008.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:afc73d498d460a51e7c313091cd76b5e5b424776cad00f59fe2028aa79f593d2
size 1946242696

3
model-00003-of-00008.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d2092463866ee390b55ae27a1f613ba718617bccd1a91b586ca5d69f3f701f9d
size 1979780440

3
model-00004-of-00008.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8204aa0413b587ed9a4eb05b8cad391e5345ea2ced8e621cbf8c8d8594d3cb99
size 1946242728

3
model-00005-of-00008.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f4c456e481b4a6262752e275cbfe79e55427cb6136dfeee842758dff86724447
size 1979780456

3
model-00006-of-00008.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ed2f94da2ab6e6658e27dc2cd1f8208341666ebc02f52008c896ed2b68c1c3a7
size 1946242728

3
model-00007-of-00008.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2a8945c6da9326a005faa3fff6fd93876801024382882f380972bcc1aa6d58a9
size 1979780456

3
model-00008-of-00008.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a5c32400f5f6335cc974cce6c3e431eef231f4544645619db90724111d8c8c4a
size 1748035640

234
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,234 @@
{
"metadata": {
"total_size": 15475417088
},
"weight_map": {
"model.layers.0.attention.wo.weight": "model-00001-of-00008.safetensors",
"model.layers.0.attention.wqkv.weight": "model-00001-of-00008.safetensors",
"model.layers.0.attention_norm.weight": "model-00001-of-00008.safetensors",
"model.layers.0.feed_forward.w1.weight": "model-00001-of-00008.safetensors",
"model.layers.0.feed_forward.w2.weight": "model-00001-of-00008.safetensors",
"model.layers.0.feed_forward.w3.weight": "model-00001-of-00008.safetensors",
"model.layers.0.ffn_norm.weight": "model-00001-of-00008.safetensors",
"model.layers.1.attention.wo.weight": "model-00001-of-00008.safetensors",
"model.layers.1.attention.wqkv.weight": "model-00001-of-00008.safetensors",
"model.layers.1.attention_norm.weight": "model-00001-of-00008.safetensors",
"model.layers.1.feed_forward.w1.weight": "model-00001-of-00008.safetensors",
"model.layers.1.feed_forward.w2.weight": "model-00001-of-00008.safetensors",
"model.layers.1.feed_forward.w3.weight": "model-00001-of-00008.safetensors",
"model.layers.1.ffn_norm.weight": "model-00001-of-00008.safetensors",
"model.layers.10.attention.wo.weight": "model-00003-of-00008.safetensors",
"model.layers.10.attention.wqkv.weight": "model-00003-of-00008.safetensors",
"model.layers.10.attention_norm.weight": "model-00003-of-00008.safetensors",
"model.layers.10.feed_forward.w1.weight": "model-00003-of-00008.safetensors",
"model.layers.10.feed_forward.w2.weight": "model-00003-of-00008.safetensors",
"model.layers.10.feed_forward.w3.weight": "model-00003-of-00008.safetensors",
"model.layers.10.ffn_norm.weight": "model-00003-of-00008.safetensors",
"model.layers.11.attention.wo.weight": "model-00003-of-00008.safetensors",
"model.layers.11.attention.wqkv.weight": "model-00003-of-00008.safetensors",
"model.layers.11.attention_norm.weight": "model-00004-of-00008.safetensors",
"model.layers.11.feed_forward.w1.weight": "model-00003-of-00008.safetensors",
"model.layers.11.feed_forward.w2.weight": "model-00004-of-00008.safetensors",
"model.layers.11.feed_forward.w3.weight": "model-00003-of-00008.safetensors",
"model.layers.11.ffn_norm.weight": "model-00004-of-00008.safetensors",
"model.layers.12.attention.wo.weight": "model-00004-of-00008.safetensors",
"model.layers.12.attention.wqkv.weight": "model-00004-of-00008.safetensors",
"model.layers.12.attention_norm.weight": "model-00004-of-00008.safetensors",
"model.layers.12.feed_forward.w1.weight": "model-00004-of-00008.safetensors",
"model.layers.12.feed_forward.w2.weight": "model-00004-of-00008.safetensors",
"model.layers.12.feed_forward.w3.weight": "model-00004-of-00008.safetensors",
"model.layers.12.ffn_norm.weight": "model-00004-of-00008.safetensors",
"model.layers.13.attention.wo.weight": "model-00004-of-00008.safetensors",
"model.layers.13.attention.wqkv.weight": "model-00004-of-00008.safetensors",
"model.layers.13.attention_norm.weight": "model-00004-of-00008.safetensors",
"model.layers.13.feed_forward.w1.weight": "model-00004-of-00008.safetensors",
"model.layers.13.feed_forward.w2.weight": "model-00004-of-00008.safetensors",
"model.layers.13.feed_forward.w3.weight": "model-00004-of-00008.safetensors",
"model.layers.13.ffn_norm.weight": "model-00004-of-00008.safetensors",
"model.layers.14.attention.wo.weight": "model-00004-of-00008.safetensors",
"model.layers.14.attention.wqkv.weight": "model-00004-of-00008.safetensors",
"model.layers.14.attention_norm.weight": "model-00004-of-00008.safetensors",
"model.layers.14.feed_forward.w1.weight": "model-00004-of-00008.safetensors",
"model.layers.14.feed_forward.w2.weight": "model-00004-of-00008.safetensors",
"model.layers.14.feed_forward.w3.weight": "model-00004-of-00008.safetensors",
"model.layers.14.ffn_norm.weight": "model-00004-of-00008.safetensors",
"model.layers.15.attention.wo.weight": "model-00004-of-00008.safetensors",
"model.layers.15.attention.wqkv.weight": "model-00004-of-00008.safetensors",
"model.layers.15.attention_norm.weight": "model-00004-of-00008.safetensors",
"model.layers.15.feed_forward.w1.weight": "model-00004-of-00008.safetensors",
"model.layers.15.feed_forward.w2.weight": "model-00004-of-00008.safetensors",
"model.layers.15.feed_forward.w3.weight": "model-00004-of-00008.safetensors",
"model.layers.15.ffn_norm.weight": "model-00004-of-00008.safetensors",
"model.layers.16.attention.wo.weight": "model-00004-of-00008.safetensors",
"model.layers.16.attention.wqkv.weight": "model-00004-of-00008.safetensors",
"model.layers.16.attention_norm.weight": "model-00005-of-00008.safetensors",
"model.layers.16.feed_forward.w1.weight": "model-00005-of-00008.safetensors",
"model.layers.16.feed_forward.w2.weight": "model-00005-of-00008.safetensors",
"model.layers.16.feed_forward.w3.weight": "model-00005-of-00008.safetensors",
"model.layers.16.ffn_norm.weight": "model-00005-of-00008.safetensors",
"model.layers.17.attention.wo.weight": "model-00005-of-00008.safetensors",
"model.layers.17.attention.wqkv.weight": "model-00005-of-00008.safetensors",
"model.layers.17.attention_norm.weight": "model-00005-of-00008.safetensors",
"model.layers.17.feed_forward.w1.weight": "model-00005-of-00008.safetensors",
"model.layers.17.feed_forward.w2.weight": "model-00005-of-00008.safetensors",
"model.layers.17.feed_forward.w3.weight": "model-00005-of-00008.safetensors",
"model.layers.17.ffn_norm.weight": "model-00005-of-00008.safetensors",
"model.layers.18.attention.wo.weight": "model-00005-of-00008.safetensors",
"model.layers.18.attention.wqkv.weight": "model-00005-of-00008.safetensors",
"model.layers.18.attention_norm.weight": "model-00005-of-00008.safetensors",
"model.layers.18.feed_forward.w1.weight": "model-00005-of-00008.safetensors",
"model.layers.18.feed_forward.w2.weight": "model-00005-of-00008.safetensors",
"model.layers.18.feed_forward.w3.weight": "model-00005-of-00008.safetensors",
"model.layers.18.ffn_norm.weight": "model-00005-of-00008.safetensors",
"model.layers.19.attention.wo.weight": "model-00005-of-00008.safetensors",
"model.layers.19.attention.wqkv.weight": "model-00005-of-00008.safetensors",
"model.layers.19.attention_norm.weight": "model-00005-of-00008.safetensors",
"model.layers.19.feed_forward.w1.weight": "model-00005-of-00008.safetensors",
"model.layers.19.feed_forward.w2.weight": "model-00005-of-00008.safetensors",
"model.layers.19.feed_forward.w3.weight": "model-00005-of-00008.safetensors",
"model.layers.19.ffn_norm.weight": "model-00005-of-00008.safetensors",
"model.layers.2.attention.wo.weight": "model-00001-of-00008.safetensors",
"model.layers.2.attention.wqkv.weight": "model-00001-of-00008.safetensors",
"model.layers.2.attention_norm.weight": "model-00002-of-00008.safetensors",
"model.layers.2.feed_forward.w1.weight": "model-00001-of-00008.safetensors",
"model.layers.2.feed_forward.w2.weight": "model-00002-of-00008.safetensors",
"model.layers.2.feed_forward.w3.weight": "model-00001-of-00008.safetensors",
"model.layers.2.ffn_norm.weight": "model-00002-of-00008.safetensors",
"model.layers.20.attention.wo.weight": "model-00005-of-00008.safetensors",
"model.layers.20.attention.wqkv.weight": "model-00005-of-00008.safetensors",
"model.layers.20.attention_norm.weight": "model-00006-of-00008.safetensors",
"model.layers.20.feed_forward.w1.weight": "model-00005-of-00008.safetensors",
"model.layers.20.feed_forward.w2.weight": "model-00006-of-00008.safetensors",
"model.layers.20.feed_forward.w3.weight": "model-00005-of-00008.safetensors",
"model.layers.20.ffn_norm.weight": "model-00006-of-00008.safetensors",
"model.layers.21.attention.wo.weight": "model-00006-of-00008.safetensors",
"model.layers.21.attention.wqkv.weight": "model-00006-of-00008.safetensors",
"model.layers.21.attention_norm.weight": "model-00006-of-00008.safetensors",
"model.layers.21.feed_forward.w1.weight": "model-00006-of-00008.safetensors",
"model.layers.21.feed_forward.w2.weight": "model-00006-of-00008.safetensors",
"model.layers.21.feed_forward.w3.weight": "model-00006-of-00008.safetensors",
"model.layers.21.ffn_norm.weight": "model-00006-of-00008.safetensors",
"model.layers.22.attention.wo.weight": "model-00006-of-00008.safetensors",
"model.layers.22.attention.wqkv.weight": "model-00006-of-00008.safetensors",
"model.layers.22.attention_norm.weight": "model-00006-of-00008.safetensors",
"model.layers.22.feed_forward.w1.weight": "model-00006-of-00008.safetensors",
"model.layers.22.feed_forward.w2.weight": "model-00006-of-00008.safetensors",
"model.layers.22.feed_forward.w3.weight": "model-00006-of-00008.safetensors",
"model.layers.22.ffn_norm.weight": "model-00006-of-00008.safetensors",
"model.layers.23.attention.wo.weight": "model-00006-of-00008.safetensors",
"model.layers.23.attention.wqkv.weight": "model-00006-of-00008.safetensors",
"model.layers.23.attention_norm.weight": "model-00006-of-00008.safetensors",
"model.layers.23.feed_forward.w1.weight": "model-00006-of-00008.safetensors",
"model.layers.23.feed_forward.w2.weight": "model-00006-of-00008.safetensors",
"model.layers.23.feed_forward.w3.weight": "model-00006-of-00008.safetensors",
"model.layers.23.ffn_norm.weight": "model-00006-of-00008.safetensors",
"model.layers.24.attention.wo.weight": "model-00006-of-00008.safetensors",
"model.layers.24.attention.wqkv.weight": "model-00006-of-00008.safetensors",
"model.layers.24.attention_norm.weight": "model-00006-of-00008.safetensors",
"model.layers.24.feed_forward.w1.weight": "model-00006-of-00008.safetensors",
"model.layers.24.feed_forward.w2.weight": "model-00006-of-00008.safetensors",
"model.layers.24.feed_forward.w3.weight": "model-00006-of-00008.safetensors",
"model.layers.24.ffn_norm.weight": "model-00006-of-00008.safetensors",
"model.layers.25.attention.wo.weight": "model-00006-of-00008.safetensors",
"model.layers.25.attention.wqkv.weight": "model-00006-of-00008.safetensors",
"model.layers.25.attention_norm.weight": "model-00007-of-00008.safetensors",
"model.layers.25.feed_forward.w1.weight": "model-00007-of-00008.safetensors",
"model.layers.25.feed_forward.w2.weight": "model-00007-of-00008.safetensors",
"model.layers.25.feed_forward.w3.weight": "model-00007-of-00008.safetensors",
"model.layers.25.ffn_norm.weight": "model-00007-of-00008.safetensors",
"model.layers.26.attention.wo.weight": "model-00007-of-00008.safetensors",
"model.layers.26.attention.wqkv.weight": "model-00007-of-00008.safetensors",
"model.layers.26.attention_norm.weight": "model-00007-of-00008.safetensors",
"model.layers.26.feed_forward.w1.weight": "model-00007-of-00008.safetensors",
"model.layers.26.feed_forward.w2.weight": "model-00007-of-00008.safetensors",
"model.layers.26.feed_forward.w3.weight": "model-00007-of-00008.safetensors",
"model.layers.26.ffn_norm.weight": "model-00007-of-00008.safetensors",
"model.layers.27.attention.wo.weight": "model-00007-of-00008.safetensors",
"model.layers.27.attention.wqkv.weight": "model-00007-of-00008.safetensors",
"model.layers.27.attention_norm.weight": "model-00007-of-00008.safetensors",
"model.layers.27.feed_forward.w1.weight": "model-00007-of-00008.safetensors",
"model.layers.27.feed_forward.w2.weight": "model-00007-of-00008.safetensors",
"model.layers.27.feed_forward.w3.weight": "model-00007-of-00008.safetensors",
"model.layers.27.ffn_norm.weight": "model-00007-of-00008.safetensors",
"model.layers.28.attention.wo.weight": "model-00007-of-00008.safetensors",
"model.layers.28.attention.wqkv.weight": "model-00007-of-00008.safetensors",
"model.layers.28.attention_norm.weight": "model-00007-of-00008.safetensors",
"model.layers.28.feed_forward.w1.weight": "model-00007-of-00008.safetensors",
"model.layers.28.feed_forward.w2.weight": "model-00007-of-00008.safetensors",
"model.layers.28.feed_forward.w3.weight": "model-00007-of-00008.safetensors",
"model.layers.28.ffn_norm.weight": "model-00007-of-00008.safetensors",
"model.layers.29.attention.wo.weight": "model-00007-of-00008.safetensors",
"model.layers.29.attention.wqkv.weight": "model-00007-of-00008.safetensors",
"model.layers.29.attention_norm.weight": "model-00008-of-00008.safetensors",
"model.layers.29.feed_forward.w1.weight": "model-00007-of-00008.safetensors",
"model.layers.29.feed_forward.w2.weight": "model-00008-of-00008.safetensors",
"model.layers.29.feed_forward.w3.weight": "model-00007-of-00008.safetensors",
"model.layers.29.ffn_norm.weight": "model-00008-of-00008.safetensors",
"model.layers.3.attention.wo.weight": "model-00002-of-00008.safetensors",
"model.layers.3.attention.wqkv.weight": "model-00002-of-00008.safetensors",
"model.layers.3.attention_norm.weight": "model-00002-of-00008.safetensors",
"model.layers.3.feed_forward.w1.weight": "model-00002-of-00008.safetensors",
"model.layers.3.feed_forward.w2.weight": "model-00002-of-00008.safetensors",
"model.layers.3.feed_forward.w3.weight": "model-00002-of-00008.safetensors",
"model.layers.3.ffn_norm.weight": "model-00002-of-00008.safetensors",
"model.layers.30.attention.wo.weight": "model-00008-of-00008.safetensors",
"model.layers.30.attention.wqkv.weight": "model-00008-of-00008.safetensors",
"model.layers.30.attention_norm.weight": "model-00008-of-00008.safetensors",
"model.layers.30.feed_forward.w1.weight": "model-00008-of-00008.safetensors",
"model.layers.30.feed_forward.w2.weight": "model-00008-of-00008.safetensors",
"model.layers.30.feed_forward.w3.weight": "model-00008-of-00008.safetensors",
"model.layers.30.ffn_norm.weight": "model-00008-of-00008.safetensors",
"model.layers.31.attention.wo.weight": "model-00008-of-00008.safetensors",
"model.layers.31.attention.wqkv.weight": "model-00008-of-00008.safetensors",
"model.layers.31.attention_norm.weight": "model-00008-of-00008.safetensors",
"model.layers.31.feed_forward.w1.weight": "model-00008-of-00008.safetensors",
"model.layers.31.feed_forward.w2.weight": "model-00008-of-00008.safetensors",
"model.layers.31.feed_forward.w3.weight": "model-00008-of-00008.safetensors",
"model.layers.31.ffn_norm.weight": "model-00008-of-00008.safetensors",
"model.layers.4.attention.wo.weight": "model-00002-of-00008.safetensors",
"model.layers.4.attention.wqkv.weight": "model-00002-of-00008.safetensors",
"model.layers.4.attention_norm.weight": "model-00002-of-00008.safetensors",
"model.layers.4.feed_forward.w1.weight": "model-00002-of-00008.safetensors",
"model.layers.4.feed_forward.w2.weight": "model-00002-of-00008.safetensors",
"model.layers.4.feed_forward.w3.weight": "model-00002-of-00008.safetensors",
"model.layers.4.ffn_norm.weight": "model-00002-of-00008.safetensors",
"model.layers.5.attention.wo.weight": "model-00002-of-00008.safetensors",
"model.layers.5.attention.wqkv.weight": "model-00002-of-00008.safetensors",
"model.layers.5.attention_norm.weight": "model-00002-of-00008.safetensors",
"model.layers.5.feed_forward.w1.weight": "model-00002-of-00008.safetensors",
"model.layers.5.feed_forward.w2.weight": "model-00002-of-00008.safetensors",
"model.layers.5.feed_forward.w3.weight": "model-00002-of-00008.safetensors",
"model.layers.5.ffn_norm.weight": "model-00002-of-00008.safetensors",
"model.layers.6.attention.wo.weight": "model-00002-of-00008.safetensors",
"model.layers.6.attention.wqkv.weight": "model-00002-of-00008.safetensors",
"model.layers.6.attention_norm.weight": "model-00002-of-00008.safetensors",
"model.layers.6.feed_forward.w1.weight": "model-00002-of-00008.safetensors",
"model.layers.6.feed_forward.w2.weight": "model-00002-of-00008.safetensors",
"model.layers.6.feed_forward.w3.weight": "model-00002-of-00008.safetensors",
"model.layers.6.ffn_norm.weight": "model-00002-of-00008.safetensors",
"model.layers.7.attention.wo.weight": "model-00002-of-00008.safetensors",
"model.layers.7.attention.wqkv.weight": "model-00002-of-00008.safetensors",
"model.layers.7.attention_norm.weight": "model-00003-of-00008.safetensors",
"model.layers.7.feed_forward.w1.weight": "model-00003-of-00008.safetensors",
"model.layers.7.feed_forward.w2.weight": "model-00003-of-00008.safetensors",
"model.layers.7.feed_forward.w3.weight": "model-00003-of-00008.safetensors",
"model.layers.7.ffn_norm.weight": "model-00003-of-00008.safetensors",
"model.layers.8.attention.wo.weight": "model-00003-of-00008.safetensors",
"model.layers.8.attention.wqkv.weight": "model-00003-of-00008.safetensors",
"model.layers.8.attention_norm.weight": "model-00003-of-00008.safetensors",
"model.layers.8.feed_forward.w1.weight": "model-00003-of-00008.safetensors",
"model.layers.8.feed_forward.w2.weight": "model-00003-of-00008.safetensors",
"model.layers.8.feed_forward.w3.weight": "model-00003-of-00008.safetensors",
"model.layers.8.ffn_norm.weight": "model-00003-of-00008.safetensors",
"model.layers.9.attention.wo.weight": "model-00003-of-00008.safetensors",
"model.layers.9.attention.wqkv.weight": "model-00003-of-00008.safetensors",
"model.layers.9.attention_norm.weight": "model-00003-of-00008.safetensors",
"model.layers.9.feed_forward.w1.weight": "model-00003-of-00008.safetensors",
"model.layers.9.feed_forward.w2.weight": "model-00003-of-00008.safetensors",
"model.layers.9.feed_forward.w3.weight": "model-00003-of-00008.safetensors",
"model.layers.9.ffn_norm.weight": "model-00003-of-00008.safetensors",
"model.norm.weight": "model-00008-of-00008.safetensors",
"model.tok_embeddings.weight": "model-00001-of-00008.safetensors",
"output.weight": "model-00008-of-00008.safetensors"
}
}

1246
modeling_internlm2.py Normal file
View File

File diff suppressed because it is too large Load Diff

39
special_tokens_map.json Normal file
View File

@@ -0,0 +1,39 @@
{
"additional_special_tokens": [
{
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
],
"bos_token": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

240
tokenization_internlm.py Normal file
View File

@@ -0,0 +1,240 @@
# coding=utf-8
# Copyright (c) InternLM. All rights reserved.
#
# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
# and OPT implementations in this library. It has been modified from its
# original forms to accommodate minor architectural differences compared
# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tokenization classes for IntermLM."""
import os
from shutil import copyfile
from typing import Any, Dict, List, Optional, Tuple
import sentencepiece as spm
from transformers.tokenization_utils import PreTrainedTokenizer
from transformers.utils import logging
logger = logging.get_logger(__name__)
VOCAB_FILES_NAMES = {"vocab_file": "./tokenizer.model"}
PRETRAINED_VOCAB_FILES_MAP = {}
class InternLMTokenizer(PreTrainedTokenizer):
"""
Construct a InternLM tokenizer. Based on byte-level Byte-Pair-Encoding.
Args:
vocab_file (`str`):
Path to the vocabulary file.
"""
vocab_files_names = VOCAB_FILES_NAMES
pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
model_input_names = ["input_ids", "attention_mask"]
_auto_class = "AutoTokenizer"
def __init__(
self,
vocab_file,
unk_token="<unk>",
bos_token="<s>",
eos_token="</s>",
pad_token="</s>",
sp_model_kwargs: Optional[Dict[str, Any]] = None,
add_bos_token=True,
add_eos_token=False,
decode_with_prefix_space=False,
clean_up_tokenization_spaces=False,
**kwargs,
):
self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs
self.vocab_file = vocab_file
self.add_bos_token = add_bos_token
self.add_eos_token = add_eos_token
self.decode_with_prefix_space = decode_with_prefix_space
self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
self.sp_model.Load(vocab_file)
self._no_prefix_space_tokens = None
super().__init__(
bos_token=bos_token,
eos_token=eos_token,
unk_token=unk_token,
pad_token=pad_token,
clean_up_tokenization_spaces=clean_up_tokenization_spaces,
**kwargs,
)
""" Initialization"""
@property
def no_prefix_space_tokens(self):
if self._no_prefix_space_tokens is None:
vocab = self.convert_ids_to_tokens(list(range(self.vocab_size)))
self._no_prefix_space_tokens = {i for i, tok in enumerate(vocab) if not tok.startswith("▁")}
return self._no_prefix_space_tokens
@property
def vocab_size(self):
"""Returns vocab size"""
return self.sp_model.get_piece_size()
@property
def bos_token_id(self) -> Optional[int]:
return self.sp_model.bos_id()
@property
def eos_token_id(self) -> Optional[int]:
return self.sp_model.eos_id()
def get_vocab(self):
"""Returns vocab as a dict"""
vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
vocab.update(self.added_tokens_encoder)
return vocab
def _tokenize(self, text):
"""Returns a tokenized string."""
return self.sp_model.encode(text, out_type=str)
def _convert_token_to_id(self, token):
"""Converts a token (str) in an id using the vocab."""
return self.sp_model.piece_to_id(token)
def _convert_id_to_token(self, index):
"""Converts an index (integer) in a token (str) using the vocab."""
token = self.sp_model.IdToPiece(index)
return token
def _maybe_add_prefix_space(self, tokens, decoded):
if tokens and tokens[0] not in self.no_prefix_space_tokens:
return " " + decoded
else:
return decoded
def convert_tokens_to_string(self, tokens):
"""Converts a sequence of tokens (string) in a single string."""
current_sub_tokens = []
out_string = ""
prev_is_special = False
for token in tokens:
# make sure that special tokens are not decoded using sentencepiece model
if token in self.all_special_tokens:
if not prev_is_special:
out_string += " "
out_string += self.sp_model.decode(current_sub_tokens) + token
prev_is_special = True
current_sub_tokens = []
else:
current_sub_tokens.append(token)
prev_is_special = False
out_string += self.sp_model.decode(current_sub_tokens)
out_string = self.clean_up_tokenization(out_string)
out_string = self._maybe_add_prefix_space(tokens=tokens, decoded=out_string)
return out_string[1:]
def save_vocabulary(self, save_directory, filename_prefix: Optional[str] = None) -> Tuple[str]:
"""
Save the vocabulary and special tokens file to a directory.
Args:
save_directory (`str`):
The directory in which to save the vocabulary.
Returns:
`Tuple(str)`: Paths to the files saved.
"""
if not os.path.isdir(save_directory):
logger.error(f"Vocabulary path ({save_directory}) should be a directory")
return
out_vocab_file = os.path.join(
save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
)
if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
copyfile(self.vocab_file, out_vocab_file)
elif not os.path.isfile(self.vocab_file):
with open(out_vocab_file, "wb") as fi:
content_spiece_model = self.sp_model.serialized_model_proto()
fi.write(content_spiece_model)
return (out_vocab_file,)
def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
if self.add_bos_token:
bos_token_ids = [self.bos_token_id]
else:
bos_token_ids = []
output = bos_token_ids + token_ids_0
if token_ids_1 is not None:
output = output + token_ids_1
if self.add_eos_token:
output = output + [self.eos_token_id]
return output
def get_special_tokens_mask(
self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
) -> List[int]:
"""
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
"""
if already_has_special_tokens:
return super().get_special_tokens_mask(
token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
)
if token_ids_1 is None:
return [1] + ([0] * len(token_ids_0)) + [1]
return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]
def create_token_type_ids_from_sequences(
self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
"""
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
"""
eos = [self.eos_token_id]
if token_ids_1 is None:
return len(token_ids_0 + eos) * [0]
return len(token_ids_0 + eos + token_ids_1 + eos) * [0]

3
tokenizer.model Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f868398fc4e05ee1e8aeba95ddf18ddcc45b8bce55d5093bead5bbf80429b48b
size 1477754

47
tokenizer_config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"added_tokens_decoder": {
"0": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"</s>"
],
"auto_map": {
"AutoTokenizer": [
"tokenization_internlm.InternLMTokenizer",
null
]
},
"bos_token": "<s>",
"chat_template": "{% set system_message = 'You are an AI Chemist assistant whose name is ChemLLM (浦科·浦语).\\n- ChemLLM (浦科·浦语) is a conversational language model that is developed by Shanghai AI Laboratory (上海人工智能实验室). It is designed to be helpful, honest, and harmless.\\n- ChemLLM (浦科·浦语) can understand and communicate fluently in the language chosen by the user such as English and 中文.' %}{% if messages[0]['role'] == 'system' %}{% set system_message = messages[0]['content'] %}{% endif %}{% if system_message is defined %}{{ '<s>' + '<|im_start|>system\\n' + system_message + '<|im_end|>\\n' }}{% endif %}{% for message in messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '<|im_start|>user\\n' + content + '<|im_end|>\\n<|im_start|>assistant\\n' }}{% elif message['role'] == 'assistant' %}{{ content + '\\n' }}{% endif %}{% endfor %}",
"clean_up_tokenization_spaces": false,
"eos_token": "</s>",
"model_max_length": 1000000000000000019884624838656,
"pad_token": "</s>",
"padding_side": "left",
"split_special_tokens": false,
"tokenizer_class": "InternLMTokenizer",
"unk_token": "<unk>"
}