OPEA/internlm2_5-7b-chat-int4-sym-awq-inc
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: OPEA/internlm2_5-7b-chat-int4-sym-awq-inc
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-26 09:28:09 +08:00
commit 5ac227f500
16 changed files with 3496 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

179
README.md Normal file
View File

@@ -0,0 +1,179 @@
---
license: apache-2.0
datasets:
- NeelNanda/pile-10k
base_model:
- internlm/internlm2_5-7b-chat
---
## Model Details
This awq model is an int4 model with group_size 128 and symmetric quantization of [internlm/internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat) generated by [intel/auto-round](https://github.com/intel/auto-round).
### INT4 Inference(CPU/HPU/CUDA)
```python
from auto_round import AutoRoundConfig ##must import for autoround format
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
# Set `torch_dtype=torch.float16` to load model in float16, otherwise it will be loaded as float32 and cause OOM Error.
quantized_model_dir = “OPEA/internlm2_5-7b-chat-int4-sym-inc”
model = AutoModelForCausalLM.from_pretrained(quantized_model_dir,
device_map="auto",
torch_dtype=torch.float16,
trust_remote_code=True
)
model = model.eval()
prompt = "9.11和9.8哪个数字大"
response, history = model.chat(tokenizer,
prompt,
history=[],
do_sample=False ## remove this to align with official usage
)
print(response)
prompt = "9.11和9.8哪个数字大"
## INT4
"""9.11大于9.8。
在比较两个小数时，我们首先比较它们的整数部分。如果整数部分相同，我们再看小数部分。在9.11和9.8中，整数部分都是9，但9.11的小数部分是0.11，而9.8的小数部分是0.8。因为0.11大于0.8，所以9.11大于9.8。"""
## BF16
"""在比较数字9.11和9.8时，我们可以直接比较它们的小数部分。
9.11的小数部分是0.11，而9.8的小数部分是0.8。
由于0.11大于0.8，因此9.11大于9.8。
所以，9.11比9.8大。"""
prompt = "strawberry中有多少个r"
## INT4
"""在单词 "strawberry" 中，字母 "r" 出现了 4 次。具体来说，"strawberry" 可以分解为 "straw" 和 "berry" 两个部分，其中 "straw" 中有一个 "r"，"berry" 中有一个 "r"，所以总共是 4 个 "r"。"""
## BF16
"""在单词 "strawberry" 中，字母 "r" 出现了两次。这个单词由8个字母组成，其中 "r" 出现了两次，分别在单词的第二个和第六个位置。
"""
prompt = "There is a girl who likes adventure,"
## INT4
"""and she is always seeking new experiences and pushing her boundaries. She might be drawn to activities like hiking, rock climbing, or even skydiving. She is not afraid to take risks and is always looking for the next adventure to embark on. Her sense of curiosity and excitement for life makes her a true adventurer at heart.
"""
## BF16
"""听起来这个女孩很有探索精神。冒险往往能带来新的体验和成长。她可能会喜欢户外活动，比如徒步旅行、攀岩或者探险。她也可能对历史和文化的探索感兴趣，比如考古、旅行到不同的地方了解当地文化。
冒险精神还可能体现在她对未知的探索上，比如尝试新的事物，学习新的技能，或者挑战自己的极限。这样的性格让她充满活力，总是对世界充满好奇。
当然，冒险并不总是意味着身体上的挑战，有时候，它也可以是对自我认知的探索，比如通过阅读、写作、艺术创作等方式，深入了解自己的内心世界。
这样的女孩，无论她选择什么样的冒险方式，都值得我们尊重和欣赏。因为冒险精神本身就是一种积极向上、勇于探索的生活态度"""
prompt = "Once upon a time,
## INT4
"""Once upon a time, in a faraway land, there was a kingdom ruled by a wise and just king. The kingdom was known for its prosperity, peace, and the beauty of its landscapes. The people of the kingdom lived in harmony with nature, and their lives were filled with joy and contentment.
In the heart of the kingdom, there was a magnificent castle where the king and his family resided. The castle was surrounded by lush gardens, and its walls were adorned with intricate carvings and vibrant murals. The king's wisdom and kindness were reflected in every aspect of the kingdom, from the way the land was cared for to the way the people were treated.
The kingdom was also known for its festivals and celebrations. The most famous of these was the Annual Festival of Lights, which took place every year on the night of the full moon. On this night, the entire kingdom would come together to light up the sky with thousands of lanterns, each carrying a wish or a story from its bearer.
One of the most beloved traditions of the festival was the story-telling contest. Young and old alike would gather in the castle courtyard, eager to listen to the tales of bravery, love, and adventure that were shared by the storytellers. These stories were not just entertainment; they were a way to pass down the wisdom and values of the kingdom from one generation to the next.
As the years went by, the kingdom continued to thrive under the wise rule of the king. The people's lives were filled with happiness and purpose, and the kingdom became a beacon of hope and inspiration for all who knew of its existence.
And so, the story of the kingdom of light and joy continued to be told, not just in the hearts of its people, but in the legends and folklore that were passed down through the ages. It was a tale of a kingdom that had found the secret to a life of peace, prosperity, and happiness, a secret that many had sought but few had found.
And so, the story of the kingdom of light and joy lives on, a reminder of the power of wisdom, kindness, and community to create a world where every heart is filled with joy and every life is filled with purpose.
"""
##BF16
"""Once upon a time, in a far-off land, there was a kingdom ruled by a wise and just king. The kingdom was known for its prosperity, beauty, and the harmony between its people. The king was beloved by all, and his court was filled with wise advisors and loyal knights.
In this kingdom, there was a young girl named Elara. She was kind-hearted, intelligent, and had a curious spirit. Elara spent her days exploring the vast forests and ancient ruins that surrounded her home, learning about the history and magic that filled the land.
One day, while wandering through the forest, Elara stumbled upon an old, enchanted tree. Its branches reached high into the sky, and its leaves shimmered with a soft, ethereal glow. As she approached, a voice spoke to her, belonging to an ancient spirit that had been trapped within the tree for centuries.
The spirit revealed to Elara that the kingdom was in danger. A dark force, known as the Shadow of Eternity, was rising, threatening to consume the land and all its inhabitants. The only way to stop it was to find the three ancient artifacts hidden within the kingdom: the Crystal of Light, the Stone of Time, and the Heart of the Earth.
With the spirit's guidance, Elara set out on a journey to find these artifacts. Along the way, she encountered many challenges and allies. She met a brave knight who had lost his way, a wise old wizard who had knowledge of ancient magic, and a group of mischievous fairies who offered their help in exchange for stories.
Together, they faced treacherous landscapes, battled fierce creatures, and solved complex puzzles. Each artifact they found brought them closer to their goal, but they also learned valuable lessons about courage, friendship, and the true meaning of power.
Finally, after many trials and tribulations, Elara and her companions reached the final resting place of the Shadow of Eternity. With the combined power of the three artifacts, they were able to defeat the dark force and restore peace to the kingdom.
The people of the kingdom celebrated Elara's bravery and the return of their prosperity. The kingdom flourished once again, and Elara became a legend, inspiring generations to come with her tale of adventure and hope.
And so, the story of Elara and her quest to save her kingdom became a cherished legend, passed down through the ages, reminding all who heard it of the power of courage, friendship, and the light that shines even in the darkest of times."""
```
### Evaluate the model
pip3 install lm-eval==0.4.5
```bash
auto-round --model "OPEA/internlm2_5-7b-chat-int4-sym-inc" --eval --eval_bs 16 --tasks leaderboard_ifeval,leaderboard_mmlu_pro,gsm8k,lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,cmmlu,ceval-valid
```
| Metric | BF16 | INT4 |
| :----------------------------- | :-----------------------: | :----------------------: |
| Avg | 0.6339 | 0.6294 |
| leaderboard_mmlu_pro 5 shots | 0.3757 | 0.3768 |
| leaderboard_ifeval strict_acc | 0.55905=(0.6079+0.5102)/2 | 0.5477=(0.6019+0.4935)/2 |
| cmmlu | 0.7873 | 0.7778 |
| ceval-valid | 0.7719 | 0.7645 |
| gsm8k 5 shots | 0.6429 | 0.6103 |
| lambada_openai | 0.6798 | 0.6746 |
| hellaswag | 0.6154 | 0.6060 |
| winogrande | 0.7901 | 0.7916 |
| piqa | 0.8014 | 0.8025 |
| truthfulqa_mc1 | 0.3880 | 0.3794 |
| openbookqa | 0.3300 | 0.3400 |
| boolq | 0.8869 | 0.8844 |
| arc_easy | 0.7782 | 0.7875 |
| arc_challenge | 0.4650 | 0.4684 |
### Generate the model
Here is the sample command to generate the model.
```bash
auto-round \
--model internlm/internlm2_5-7b-chat \
--device 0 \
--group_size 128 \
--nsamples 512 \
--bits 4 \
--iter 1000 \
--disable_eval \
--model_dtype "fp16" \
--format 'auto_awq,auto_gptq,auto_round' \
--output_dir "./tmp_autoround"
```
## Ethical Considerations and Limitations
The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
Therefore, before deploying any applications of the model, developers should perform safety testing.
## Caveats and Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
Here are a couple of useful links to learn more about Intel's AI software:
- Intel Neural Compressor [link](https://github.com/intel/neural-compressor)
## Disclaimer
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
## Cite
@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }
[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round)

65
config.json Normal file
View File

@@ -0,0 +1,65 @@
{
"_name_or_path": "/models/internlm2_5-7b-chat",
"architectures": [
"InternLM2ForCausalLM"
],
"attn_implementation": "eager",
"auto_map": {
"AutoConfig": "configuration_internlm2.InternLM2Config",
"AutoModel": "modeling_internlm2.InternLM2ForCausalLM",
"AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM"
},
"bias": false,
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"max_position_embeddings": 32768,
"model_type": "internlm2",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"pad_token_id": 2,
"pretraining_tp": 1,
"quantization_config": {
"amp": true,
"autoround_version": "0.4.2.dev",
"batch_size": 8,
"bits": 4,
"data_type": "int",
"dataset": "NeelNanda/pile-10k",
"enable_minmax_tuning": true,
"enable_norm_bias_tuning": false,
"enable_quanted_input": true,
"gradient_accumulate_steps": 1,
"group_size": 128,
"iters": 1000,
"low_gpu_mem_usage": false,
"lr": 0.001,
"minmax_lr": 0.001,
"modules_to_not_convert": [
"output"
],
"nsamples": 512,
"quant_method": "awq",
"scale_dtype": "torch.float16",
"seqlen": 2048,
"sym": true,
"block_name_to_quantize": "model.layers",
"version": "gemm",
"zero_point": false
},
"rms_norm_eps": 1e-05,
"rope_scaling": {
"factor": 2.0,
"type": "dynamic"
},
"rope_theta": 1000000,
"tie_word_embeddings": false,
"torch_dtype": "float16",
"transformers_version": "4.46.3",
"use_cache": true,
"vocab_size": 92544
}

180
configuration_internlm2.py Normal file
View File

@@ -0,0 +1,180 @@
# coding=utf-8
# Copyright (c) The InternLM team and The HuggingFace Inc. team. All rights reserved.
#
# This code is based on transformers/src/transformers/models/llama/configuration_llama.py
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" InternLM2 model configuration"""
from transformers.configuration_utils import PretrainedConfig
from transformers.utils import logging
logger = logging.get_logger(__name__)
INTERNLM2_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
# Modified from transformers.model.llama.configuration_llama.LlamaConfig
class InternLM2Config(PretrainedConfig):
r"""
This is the configuration class to store the configuration of a [`InternLM2Model`]. It is used to instantiate
an InternLM2 model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the InternLM2-7B.
Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
documentation from [`PretrainedConfig`] for more information.
Args:
vocab_size (`int`, *optional*, defaults to 32000):
Vocabulary size of the InternLM2 model. Defines the number of different tokens that can be represented by the
`inputs_ids` passed when calling [`InternLM2Model`]
hidden_size (`int`, *optional*, defaults to 4096):
Dimension of the hidden representations.
intermediate_size (`int`, *optional*, defaults to 11008):
Dimension of the MLP representations.
num_hidden_layers (`int`, *optional*, defaults to 32):
Number of hidden layers in the Transformer decoder.
num_attention_heads (`int`, *optional*, defaults to 32):
Number of attention heads for each attention layer in the Transformer decoder.
num_key_value_heads (`int`, *optional*):
This is the number of key_value heads that should be used to implement Grouped Query Attention. If
`num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
`num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
by meanpooling all the original heads within that group. For more details checkout [this
paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
`num_attention_heads`.
hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
The non-linear activation function (function or string) in the decoder.
max_position_embeddings (`int`, *optional*, defaults to 2048):
The maximum sequence length that this model might ever be used with. InternLM2 supports up to 32768 tokens.
initializer_range (`float`, *optional*, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
rms_norm_eps (`float`, *optional*, defaults to 1e-06):
The epsilon used by the rms normalization layers.
use_cache (`bool`, *optional*, defaults to `True`):
Whether or not the model should return the last key/values attentions (not used by all models). Only
relevant if `config.is_decoder=True`.
pad_token_id (`int`, *optional*):
Padding token id.
bos_token_id (`int`, *optional*, defaults to 1):
Beginning of stream token id.
eos_token_id (`int`, *optional*, defaults to 2):
End of stream token id.
pretraining_tp (`int`, *optional*, defaults to 1):
Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this
document](https://huggingface.co/docs/transformers/main/perf_train_gpu_many#tensor-parallelism)
to understand more about it. This value is necessary to ensure exact reproducibility
of the pretraining results. Please refer to [this
issue](https://github.com/pytorch/pytorch/issues/76232).
tie_word_embeddings (`bool`, *optional*, defaults to `False`):
Whether to tie weight embeddings
rope_theta (`float`, *optional*, defaults to 10000.0):
The base period of the RoPE embeddings.
rope_scaling (`Dict`, *optional*):
Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
`{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
`max_position_embeddings` to the expected new maximum. See the following thread for more information on how
these scaling strategies behave:
https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/. This is an
experimental feature, subject to breaking API changes in future versions.
"""
_auto_class = "AutoConfig"
model_type = "internlm2"
keys_to_ignore_at_inference = ["past_key_values"]
def __init__( # pylint: disable=W0102
self,
vocab_size=103168,
hidden_size=4096,
intermediate_size=11008,
num_hidden_layers=32,
num_attention_heads=32,
num_key_value_heads=None,
hidden_act="silu",
max_position_embeddings=2048,
initializer_range=0.02,
rms_norm_eps=1e-6,
use_cache=True,
pad_token_id=0,
bos_token_id=1,
eos_token_id=2,
pretraining_tp=1,
tie_word_embeddings=False,
bias=True,
rope_theta=10000,
rope_scaling=None,
attn_implementation=None,
**kwargs,
):
self.vocab_size = vocab_size
self.max_position_embeddings = max_position_embeddings
self.hidden_size = hidden_size
self.intermediate_size = intermediate_size
self.num_hidden_layers = num_hidden_layers
self.num_attention_heads = num_attention_heads
self.bias = bias
if num_key_value_heads is None:
num_key_value_heads = num_attention_heads
self.num_key_value_heads = num_key_value_heads
self.hidden_act = hidden_act
self.initializer_range = initializer_range
self.rms_norm_eps = rms_norm_eps
self.pretraining_tp = pretraining_tp
self.use_cache = use_cache
self.rope_theta = rope_theta
self.rope_scaling = rope_scaling
self._rope_scaling_validation()
self.attn_implementation = attn_implementation
if self.attn_implementation is None:
self.attn_implementation = "eager"
super().__init__(
pad_token_id=pad_token_id,
bos_token_id=bos_token_id,
eos_token_id=eos_token_id,
tie_word_embeddings=tie_word_embeddings,
**kwargs,
)
def _rope_scaling_validation(self):
"""
Validate the `rope_scaling` configuration.
"""
if self.rope_scaling is None:
return
if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
raise ValueError(
"`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
f"got {self.rope_scaling}"
)
rope_scaling_type = self.rope_scaling.get("type", None)
rope_scaling_factor = self.rope_scaling.get("factor", None)
if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]:
raise ValueError(
f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}"
)
if (
rope_scaling_factor is None
or not isinstance(rope_scaling_factor, (float, int))
or rope_scaling_factor < 1.0
):
raise ValueError(
f"`rope_scaling`'s factor field must be a number >= 1, got {rope_scaling_factor} "
f"of type {type(rope_scaling_factor)}"
)

9
generation_config.json Normal file
View File

@@ -0,0 +1,9 @@
{
"bos_token_id": 1,
"eos_token_id": [
2,
92542
],
"pad_token_id": 2,
"transformers_version": "4.46.3"
}

3
model-00001-of-00002.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fc39a404ab60507d64aac7d168f2e6c107815959ac5f11e4a183c434a4297a0f
size 4384690224

3
model-00002-of-00002.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8591e58f6dfdc5890656df08c1a389b2c044f3e112847079e90b0fc29b6ee856
size 758120576

554
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,554 @@
{
"metadata": {
"total_size": 5142749184
},
"weight_map": {
"model.layers.0.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.0.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.0.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.0.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.0.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.0.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.0.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.0.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.0.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.0.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.0.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.0.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.0.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.0.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.0.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.0.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.0.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.1.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.1.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.1.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.1.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.1.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.1.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.1.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.1.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.1.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.1.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.1.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.1.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.1.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.1.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.1.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.1.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.1.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.10.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.10.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.10.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.10.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.10.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.10.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.10.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.10.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.10.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.10.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.10.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.10.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.10.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.10.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.10.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.10.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.10.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.11.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.11.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.11.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.11.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.11.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.11.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.11.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.11.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.11.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.11.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.11.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.11.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.11.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.11.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.11.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.11.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.11.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.12.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.12.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.12.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.12.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.12.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.12.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.12.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.12.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.12.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.12.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.12.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.12.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.12.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.12.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.12.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.12.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.12.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.13.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.13.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.13.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.13.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.13.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.13.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.13.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.13.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.13.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.13.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.13.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.13.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.13.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.13.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.13.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.13.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.13.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.14.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.14.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.14.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.14.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.14.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.14.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.14.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.14.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.14.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.14.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.14.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.14.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.14.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.14.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.14.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.14.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.14.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.15.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.15.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.15.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.15.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.15.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.15.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.15.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.15.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.15.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.15.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.15.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.15.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.15.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.15.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.15.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.15.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.15.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.16.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.16.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.16.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.16.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.16.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.16.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.16.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.16.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.16.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.16.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.16.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.16.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.16.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.16.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.16.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.16.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.16.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.17.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.17.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.17.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.17.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.17.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.17.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.17.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.17.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.17.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.17.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.17.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.17.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.17.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.17.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.17.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.17.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.17.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.18.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.18.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.18.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.18.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.18.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.18.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.18.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.18.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.18.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.18.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.18.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.18.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.18.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.18.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.18.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.18.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.18.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.19.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.19.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.19.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.19.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.19.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.19.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.19.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.19.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.19.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.19.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.19.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.19.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.19.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.19.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.19.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.19.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.19.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.2.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.2.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.2.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.2.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.2.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.2.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.2.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.2.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.2.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.2.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.2.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.2.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.2.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.2.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.2.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.2.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.2.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.20.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.20.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.20.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.20.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.20.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.20.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.20.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.20.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.20.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.20.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.20.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.20.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.20.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.20.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.20.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.20.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.20.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.21.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.21.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.21.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.21.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.21.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.21.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.21.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.21.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.21.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.21.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.21.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.21.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.21.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.21.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.21.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.21.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.21.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.22.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.22.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.22.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.22.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.22.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.22.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.22.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.22.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.22.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.22.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.22.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.22.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.22.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.22.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.22.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.22.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.22.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.23.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.23.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.23.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.23.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.23.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.23.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.23.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.23.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.23.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.23.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.23.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.23.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.23.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.23.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.23.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.23.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.23.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.24.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.24.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.24.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.24.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.24.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.24.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.24.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.24.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.24.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.24.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.24.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.24.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.24.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.24.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.24.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.24.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.24.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.25.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.25.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.25.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.25.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.25.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.25.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.25.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.25.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.25.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.25.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.25.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.25.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.25.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.25.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.25.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.25.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.25.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.26.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.26.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.26.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.26.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.26.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.26.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.26.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.26.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.26.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.26.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.26.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.26.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.26.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.26.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.26.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.26.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.26.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.27.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.27.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.27.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.27.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.27.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.27.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.27.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.27.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.27.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.27.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.27.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.27.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.27.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.27.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.27.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.27.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.27.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.28.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.28.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.28.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.28.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.28.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.28.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.28.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.28.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.28.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.28.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.28.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.28.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.28.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.28.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.28.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.28.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.28.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.29.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.29.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.29.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.29.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.29.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.29.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.29.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.29.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.29.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.29.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.29.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.29.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.29.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.29.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.29.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.29.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.29.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.3.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.3.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.3.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.3.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.3.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.3.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.3.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.3.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.3.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.3.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.3.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.3.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.3.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.3.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.3.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.3.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.3.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.30.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.30.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.30.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.30.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.30.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.30.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.30.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.30.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.30.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.30.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.30.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.30.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.30.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.30.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.30.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.30.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.30.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.31.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.31.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.31.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.31.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.31.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.31.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.31.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.31.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.31.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.31.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.31.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.31.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.31.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.31.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.31.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.31.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.31.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.4.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.4.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.4.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.4.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.4.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.4.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.4.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.4.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.4.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.4.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.4.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.4.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.4.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.4.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.4.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.4.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.4.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.5.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.5.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.5.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.5.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.5.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.5.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.5.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.5.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.5.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.5.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.5.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.5.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.5.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.5.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.5.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.5.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.5.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.6.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.6.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.6.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.6.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.6.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.6.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.6.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.6.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.6.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.6.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.6.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.6.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.6.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.6.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.6.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.6.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.6.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.7.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.7.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.7.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.7.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.7.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.7.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.7.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.7.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.7.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.7.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.7.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.7.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.7.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.7.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.7.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.8.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.8.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.8.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.8.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.8.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.8.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.8.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.8.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.8.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.8.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.8.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.8.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.8.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.8.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.8.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.8.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.8.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.9.attention.wo.qweight": "model-00001-of-00002.safetensors",
"model.layers.9.attention.wo.qzeros": "model-00001-of-00002.safetensors",
"model.layers.9.attention.wo.scales": "model-00001-of-00002.safetensors",
"model.layers.9.attention.wqkv.qweight": "model-00001-of-00002.safetensors",
"model.layers.9.attention.wqkv.qzeros": "model-00001-of-00002.safetensors",
"model.layers.9.attention.wqkv.scales": "model-00001-of-00002.safetensors",
"model.layers.9.attention_norm.weight": "model-00001-of-00002.safetensors",
"model.layers.9.feed_forward.w1.qweight": "model-00001-of-00002.safetensors",
"model.layers.9.feed_forward.w1.qzeros": "model-00001-of-00002.safetensors",
"model.layers.9.feed_forward.w1.scales": "model-00001-of-00002.safetensors",
"model.layers.9.feed_forward.w2.qweight": "model-00001-of-00002.safetensors",
"model.layers.9.feed_forward.w2.qzeros": "model-00001-of-00002.safetensors",
"model.layers.9.feed_forward.w2.scales": "model-00001-of-00002.safetensors",
"model.layers.9.feed_forward.w3.qweight": "model-00001-of-00002.safetensors",
"model.layers.9.feed_forward.w3.qzeros": "model-00001-of-00002.safetensors",
"model.layers.9.feed_forward.w3.scales": "model-00001-of-00002.safetensors",
"model.layers.9.ffn_norm.weight": "model-00001-of-00002.safetensors",
"model.norm.weight": "model-00001-of-00002.safetensors",
"model.tok_embeddings.weight": "model-00001-of-00002.safetensors",
"output.weight": "model-00002-of-00002.safetensors"
}
}

1808
modeling_internlm2.py Normal file
View File

File diff suppressed because it is too large Load Diff

63
quantization_config.json Normal file
View File

@@ -0,0 +1,63 @@
{
"bits": 4,
"group_size": 128,
"sym": true,
"data_type": "int",
"enable_quanted_input": true,
"enable_minmax_tuning": true,
"seqlen": 2048,
"batch_size": 8,
"scale_dtype": "torch.float16",
"lr": 0.001,
"minmax_lr": 0.001,
"gradient_accumulate_steps": 1,
"iters": 1000,
"amp": true,
"nsamples": 512,
"low_gpu_mem_usage": false,
"to_quant_block_names": [
[
"model.layers.0",
"model.layers.1",
"model.layers.2",
"model.layers.3",
"model.layers.4",
"model.layers.5",
"model.layers.6",
"model.layers.7",
"model.layers.8",
"model.layers.9",
"model.layers.10",
"model.layers.11",
"model.layers.12",
"model.layers.13",
"model.layers.14",
"model.layers.15",
"model.layers.16",
"model.layers.17",
"model.layers.18",
"model.layers.19",
"model.layers.20",
"model.layers.21",
"model.layers.22",
"model.layers.23",
"model.layers.24",
"model.layers.25",
"model.layers.26",
"model.layers.27",
"model.layers.28",
"model.layers.29",
"model.layers.30",
"model.layers.31"
]
],
"enable_norm_bias_tuning": false,
"dataset": "NeelNanda/pile-10k",
"autoround_version": "0.4.2.dev",
"quant_method": "awq",
"zero_point": false,
"version": "gemm",
"modules_to_not_convert": [
"output"
]
}

38
special_tokens_map.json Normal file
View File

@@ -0,0 +1,38 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|action_start|>",
"<|action_end|>",
"<|interpreter|>",
"<|plugin|>"
],
"bos_token": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

236
tokenization_internlm2.py Normal file
View File

@@ -0,0 +1,236 @@
# coding=utf-8
# Copyright (c) The InternLM team and The HuggingFace Inc. team. All rights reserved.
#
# This code is based on transformers/src/transformers/models/llama/tokenization_llama.py
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tokenization classes for InternLM."""
import os
from shutil import copyfile
from typing import Any, Dict, List, Optional, Tuple
import sentencepiece as spm
from transformers.tokenization_utils import PreTrainedTokenizer
from transformers.utils import logging
logger = logging.get_logger(__name__)
VOCAB_FILES_NAMES = {"vocab_file": "./tokenizer.model"}
PRETRAINED_VOCAB_FILES_MAP = {}
# Modified from transformers.model.llama.tokenization_llama.LlamaTokenizer
class InternLM2Tokenizer(PreTrainedTokenizer):
"""
Construct a InternLM2 tokenizer. Based on byte-level Byte-Pair-Encoding.
Args:
vocab_file (`str`):
Path to the vocabulary file.
"""
vocab_files_names = VOCAB_FILES_NAMES
pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
model_input_names = ["input_ids", "attention_mask"]
_auto_class = "AutoTokenizer"
def __init__(
self,
vocab_file,
unk_token="<unk>",
bos_token="<s>",
eos_token="</s>",
pad_token="</s>",
sp_model_kwargs: Optional[Dict[str, Any]] = None,
add_bos_token=True,
add_eos_token=False,
decode_with_prefix_space=False,
clean_up_tokenization_spaces=False,
**kwargs,
):
self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs
self.vocab_file = vocab_file
self.add_bos_token = add_bos_token
self.add_eos_token = add_eos_token
self.decode_with_prefix_space = decode_with_prefix_space
self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
self.sp_model.Load(vocab_file)
self._no_prefix_space_tokens = None
super().__init__(
bos_token=bos_token,
eos_token=eos_token,
unk_token=unk_token,
pad_token=pad_token,
clean_up_tokenization_spaces=clean_up_tokenization_spaces,
**kwargs,
)
@property
def no_prefix_space_tokens(self):
if self._no_prefix_space_tokens is None:
vocab = self.convert_ids_to_tokens(list(range(self.vocab_size)))
self._no_prefix_space_tokens = {i for i, tok in enumerate(vocab) if not tok.startswith("▁")}
return self._no_prefix_space_tokens
@property
def vocab_size(self):
"""Returns vocab size"""
return self.sp_model.get_piece_size()
@property
def bos_token_id(self) -> Optional[int]:
return self.sp_model.bos_id()
@property
def eos_token_id(self) -> Optional[int]:
return self.sp_model.eos_id()
def get_vocab(self):
"""Returns vocab as a dict"""
vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
vocab.update(self.added_tokens_encoder)
return vocab
def _tokenize(self, text):
"""Returns a tokenized string."""
return self.sp_model.encode(text, out_type=str)
def _convert_token_to_id(self, token):
"""Converts a token (str) in an id using the vocab."""
return self.sp_model.piece_to_id(token)
def _convert_id_to_token(self, index):
"""Converts an index (integer) in a token (str) using the vocab."""
token = self.sp_model.IdToPiece(index)
return token
def _maybe_add_prefix_space(self, tokens, decoded):
if tokens and tokens[0] not in self.no_prefix_space_tokens:
return " " + decoded
else:
return decoded
def convert_tokens_to_string(self, tokens):
"""Converts a sequence of tokens (string) in a single string."""
current_sub_tokens = []
out_string = ""
prev_is_special = False
for token in tokens:
# make sure that special tokens are not decoded using sentencepiece model
if token in self.all_special_tokens:
if not prev_is_special:
out_string += " "
out_string += self.sp_model.decode(current_sub_tokens) + token
prev_is_special = True
current_sub_tokens = []
else:
current_sub_tokens.append(token)
prev_is_special = False
out_string += self.sp_model.decode(current_sub_tokens)
out_string = self.clean_up_tokenization(out_string)
out_string = self._maybe_add_prefix_space(tokens=tokens, decoded=out_string)
return out_string[1:]
def save_vocabulary(self, save_directory, filename_prefix: Optional[str] = None) -> Tuple[str]:
"""
Save the vocabulary and special tokens file to a directory.
Args:
save_directory (`str`):
The directory in which to save the vocabulary.
Returns:
`Tuple(str)`: Paths to the files saved.
"""
if not os.path.isdir(save_directory):
logger.error(f"Vocabulary path ({save_directory}) should be a directory")
return
out_vocab_file = os.path.join(
save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
)
if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
copyfile(self.vocab_file, out_vocab_file)
elif not os.path.isfile(self.vocab_file):
with open(out_vocab_file, "wb") as fi:
content_spiece_model = self.sp_model.serialized_model_proto()
fi.write(content_spiece_model)
return (out_vocab_file,)
def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
if self.add_bos_token:
bos_token_ids = [self.bos_token_id]
else:
bos_token_ids = []
output = bos_token_ids + token_ids_0
if token_ids_1 is not None:
output = output + token_ids_1
if self.add_eos_token:
output = output + [self.eos_token_id]
return output
def get_special_tokens_mask(
self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
) -> List[int]:
"""
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
"""
if already_has_special_tokens:
return super().get_special_tokens_mask(
token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
)
if token_ids_1 is None:
return [1] + ([0] * len(token_ids_0)) + [1]
return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]
def create_token_type_ids_from_sequences(
self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
"""
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
"""
eos = [self.eos_token_id]
if token_ids_1 is None:
return len(token_ids_0 + eos) * [0]
return len(token_ids_0 + eos + token_ids_1 + eos) * [0]

214
tokenization_internlm2_fast.py Normal file
View File

@@ -0,0 +1,214 @@
# coding=utf-8
# Copyright (c) The InternLM team and The HuggingFace Inc. team. All rights reserved.
#
# This code is based on transformers/src/transformers/models/llama/tokenization_llama_fast.py
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tokenization Fast class for InternLM."""
import os
from shutil import copyfile
from typing import Any, Dict, Optional, Tuple
from tokenizers import processors, decoders, Tokenizer, normalizers
from tokenizers.models import BPE
from transformers.tokenization_utils_fast import PreTrainedTokenizerFast
from transformers.utils import logging
from transformers.convert_slow_tokenizer import (
SLOW_TO_FAST_CONVERTERS,
SpmConverter,
SentencePieceExtractor,
)
from .tokenization_internlm2 import InternLM2Tokenizer
logger = logging.get_logger(__name__)
VOCAB_FILES_NAMES = {"vocab_file": "./tokenizer.model"}
# Modified from transformers.convert_slow_tokenizer.LlamaConverter
class InternLM2Converter(SpmConverter):
handle_byte_fallback = True
def vocab(self, proto):
vocab = [
("<unk>", 0.0),
("<s>", 0.0),
("</s>", 0.0),
]
vocab += [(piece.piece, piece.score) for piece in proto.pieces[3:]]
return vocab
def unk_id(self, proto):
unk_id = 0
return unk_id
def decoder(self, replacement, add_prefix_space):
decoders_sequence = [
decoders.Replace("▁", " "),
decoders.ByteFallback(),
decoders.Fuse(),
]
if self.proto.normalizer_spec.add_dummy_prefix:
decoders_sequence.append(decoders.Strip(content=" ", left=1))
return decoders.Sequence(decoders_sequence)
def tokenizer(self, proto):
model_type = proto.trainer_spec.model_type
vocab_scores = self.vocab(proto)
# special tokens
added_tokens = self.original_tokenizer.added_tokens_decoder
for i in range(len(vocab_scores)):
piece, score = vocab_scores[i]
if i in added_tokens:
vocab_scores[i] = (added_tokens[i].content, score)
if model_type == 1:
raise RuntimeError("InternLM2 is supposed to be a BPE model!")
elif model_type == 2:
_, merges = SentencePieceExtractor(self.original_tokenizer.vocab_file).extract(vocab_scores)
bpe_vocab = {word: i for i, (word, _score) in enumerate(vocab_scores)}
tokenizer = Tokenizer(
BPE(bpe_vocab, merges, unk_token=proto.trainer_spec.unk_piece, fuse_unk=True, byte_fallback=True)
)
tokenizer.add_special_tokens(
[ added_token for index, added_token in added_tokens.items()]
)
else:
raise Exception(
"You're trying to run a `Unigram` model but you're file was trained with a different algorithm"
)
return tokenizer
def normalizer(self, proto):
normalizers_list = []
if proto.normalizer_spec.add_dummy_prefix:
normalizers_list.append(normalizers.Prepend(prepend="▁"))
normalizers_list.append(normalizers.Replace(pattern=" ", content="▁"))
return normalizers.Sequence(normalizers_list)
def pre_tokenizer(self, replacement, add_prefix_space):
return None
SLOW_TO_FAST_CONVERTERS["InternLM2Tokenizer"] = InternLM2Converter
# Modified from transformers.model.llama.tokenization_llama_fast.LlamaTokenizerFast -> InternLM2TokenizerFast
class InternLM2TokenizerFast(PreTrainedTokenizerFast):
vocab_files_names = VOCAB_FILES_NAMES
slow_tokenizer_class = InternLM2Tokenizer
padding_side = "left"
model_input_names = ["input_ids", "attention_mask"]
_auto_class = "AutoTokenizer"
def __init__(
self,
vocab_file,
unk_token="<unk>",
bos_token="<s>",
eos_token="</s>",
pad_token="</s>",
sp_model_kwargs: Optional[Dict[str, Any]] = None,
add_bos_token=True,
add_eos_token=False,
decode_with_prefix_space=False,
clean_up_tokenization_spaces=False,
**kwargs,
):
super().__init__(
vocab_file=vocab_file,
unk_token=unk_token,
bos_token=bos_token,
eos_token=eos_token,
pad_token=pad_token,
sp_model_kwargs=sp_model_kwargs,
add_bos_token=add_bos_token,
add_eos_token=add_eos_token,
decode_with_prefix_space=decode_with_prefix_space,
clean_up_tokenization_spaces=clean_up_tokenization_spaces,
**kwargs,
)
self._add_bos_token = add_bos_token
self._add_eos_token = add_eos_token
self.update_post_processor()
self.vocab_file = vocab_file
@property
def can_save_slow_tokenizer(self) -> bool:
return os.path.isfile(self.vocab_file) if self.vocab_file else False
def update_post_processor(self):
"""
Updates the underlying post processor with the current `bos_token` and `eos_token`.
"""
bos = self.bos_token
bos_token_id = self.bos_token_id
if bos is None and self.add_bos_token:
raise ValueError("add_bos_token = True but bos_token = None")
eos = self.eos_token
eos_token_id = self.eos_token_id
if eos is None and self.add_eos_token:
raise ValueError("add_eos_token = True but eos_token = None")
single = f"{(bos+':0 ') if self.add_bos_token else ''}$A:0{(' '+eos+':0') if self.add_eos_token else ''}"
pair = f"{single}{(' '+bos+':1') if self.add_bos_token else ''} $B:1{(' '+eos+':1') if self.add_eos_token else ''}"
special_tokens = []
if self.add_bos_token:
special_tokens.append((bos, bos_token_id))
if self.add_eos_token:
special_tokens.append((eos, eos_token_id))
self._tokenizer.post_processor = processors.TemplateProcessing(
single=single, pair=pair, special_tokens=special_tokens
)
@property
def add_eos_token(self):
return self._add_eos_token
@property
def add_bos_token(self):
return self._add_bos_token
@add_eos_token.setter
def add_eos_token(self, value):
self._add_eos_token = value
self.update_post_processor()
@add_bos_token.setter
def add_bos_token(self, value):
self._add_bos_token = value
self.update_post_processor()
def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
if not self.can_save_slow_tokenizer:
raise ValueError(
"Your fast tokenizer does not have the necessary information to save the vocabulary for a slow "
"tokenizer."
)
if not os.path.isdir(save_directory):
logger.error(f"Vocabulary path ({save_directory}) should be a directory")
return
out_vocab_file = os.path.join(
save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
)
if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
copyfile(self.vocab_file, out_vocab_file)
return (out_vocab_file,)

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3ee8746f75300bd21fced16093893e714fcdc0411ea86bbf391d0e00dc6dc1d1
size 10541373

3
tokenizer.model Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f868398fc4e05ee1e8aeba95ddf18ddcc45b8bce55d5093bead5bbf80429b48b
size 1477754

102
tokenizer_config.json Normal file
View File

@@ -0,0 +1,102 @@
{
"add_bos_token": true,
"add_eos_token": false,
"added_tokens_decoder": {
"0": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92538": {
"content": "<|plugin|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92539": {
"content": "<|interpreter|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92540": {
"content": "<|action_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92541": {
"content": "<|action_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92542": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"92543": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|action_start|>",
"<|action_end|>",
"<|interpreter|>",
"<|plugin|>"
],
"auto_map": {
"AutoTokenizer": [
"tokenization_internlm2.InternLM2Tokenizer",
"tokenization_internlm2_fast.InternLM2TokenizerFast"
]
},
"bos_token": "<s>",
"chat_template": "{{ bos_token }}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
"clean_up_tokenization_spaces": false,
"decode_with_prefix_space": false,
"eos_token": "</s>",
"model_max_length": 1000000000000000019884624838656,
"pad_token": "</s>",
"sp_model_kwargs": null,
"tokenizer_class": "InternLM2Tokenizer",
"unk_token": "<unk>"
}