初始化项目，由ModelHub XC社区提供模型
Model: Salesforce/xgen-small-4B-base-r Source: Original Platform
This commit is contained in:
ModelHub XC
50
.gitattributes
vendored
Normal file
50
.gitattributes
vendored
Normal file
@@ -0,0 +1,50 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zstandard filter=lfs diff=lfs merge=lfs -text
|
||||
*.tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
*.db* filter=lfs diff=lfs merge=lfs -text
|
||||
*.ark* filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.gguf* filter=lfs diff=lfs merge=lfs -text
|
||||
*.ggml filter=lfs diff=lfs merge=lfs -text
|
||||
*.llamafile* filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
vocab.json filter=lfs diff=lfs merge=lfs -text
|
||||
94
README.md
Normal file
94
README.md
Normal file
@@ -0,0 +1,94 @@
|
||||
---
|
||||
license: cc-by-nc-4.0
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
---
|
||||
# Welcome to the xGen-small family!
|
||||
|
||||
**xGen-small** ([blog](https://www.salesforce.com/blog/xgen-small-enterprise-ready-small-language-models/), [arXiv](https://arxiv.org/abs/2505.06496)) is an enterprise-ready compact LM that combines domain-focused data curation, scalable pre-training, length-extension, and RL fine-tuning to deliver long-context performance at predictable, low cost.
|
||||
**This model release is for research purposes only.**
|
||||
|
||||
<p align="center">
|
||||
<img width="60%" src="https://huggingface.co/Salesforce/xgen-small/resolve/main/xgen-small.png?download=true">
|
||||
</p>
|
||||
|
||||
## Model Series
|
||||
|
||||
[xGen-small](https://www.salesforce.com/blog/xgen-small-enterprise-ready-small-language-models/) comes in two sizes (4B and 9B) with two variants (pre-trained and post-trained):
|
||||
|
||||
| Model | # Total Params | Context Length | Variant | Download |
|
||||
|---------------------------------------|----------------|----------------|--------------|----------------|
|
||||
| salesforce/xgen-small-4B-base-r | 4B | 128k | Pre-trained | [🤗 Link](https://huggingface.co/Salesforce/xgen-small-4b-base-r) |
|
||||
| salesforce/xgen-small-4B-instruct-r | 4B | 128k | Post-trained | [🤗 Link](https://huggingface.co/Salesforce/xgen-small-4b-instruct-r) |
|
||||
| salesforce/xgen-small-9B-base-r | 9B | 128k | Pre-trained | [🤗 Link](https://huggingface.co/Salesforce/xgen-small-9b-base-r) |
|
||||
| salesforce/xgen-small-9B-instruct-r | 9B | 128k | Post-trained | [🤗 Link](https://huggingface.co/Salesforce/xgen-small-9b-instruct-r) |
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_name = "Salesforce/xgen-small-4B-base-r"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_name,
|
||||
torch_dtype="auto"
|
||||
).to(device)
|
||||
|
||||
prompt = "What is Salesforce?"
|
||||
inputs = tokenizer(
|
||||
prompt,
|
||||
return_tensors="pt",
|
||||
padding=False,
|
||||
truncation=True
|
||||
).to(device)
|
||||
|
||||
generated = model.generate(**inputs, max_new_tokens=32)
|
||||
output = tokenizer.decode(
|
||||
generated[0],
|
||||
skip_special_tokens=True,
|
||||
)
|
||||
print(output)
|
||||
```
|
||||
|
||||
## Evaluation
|
||||
|
||||
| Category | Task | Llama 3.2-3B | Gemma 3-4B | Qwen2.5-3B | xGen-small 4B Base |
|
||||
| :------------------------------- | :------------- | :----------- | :--------- | :--------- | :------------------|
|
||||
| General Knowledge & Reasoning | ARC-Challenge | 50.7 | 57.9 | 55.5 | 57.3 |
|
||||
| General Knowledge & Reasoning | Big-Bench Hard | 39.1 | 40.6 | 46.1 | 50.9 |
|
||||
| General Knowledge & Reasoning | HellaSwag | 76.3 | 77.6 | 74.6 | 78.5 |
|
||||
| General Knowledge & Reasoning | MMLU | 56.1 | 59.5 | 65.6 | 62.6 |
|
||||
| General Knowledge & Reasoning | MMLU-Pro | 25.1 | 28.1 | 32.0 | 31.8 |
|
||||
| General Knowledge & Reasoning | TruthfulQA | 39.3 | 39.8 | 48.9 | 42.8 |
|
||||
| General Knowledge & Reasoning | WinoGrande | 71.6 | 72.3 | 70.0 | 72.7 |
|
||||
| Math & Science | GPQA | 28.3 | 29.3 | 29.3 | 29.4 |
|
||||
| Math & Science | GSM8K | 28.0 | 40.6 | 59.1 | 71.9 |
|
||||
| Math & Science | MATH | 9.0 | 25.2 | 40.7 | 43.1 |
|
||||
| Coding | HumanEval | 28.1 | 34.6 | 38.3 | 42.5 |
|
||||
| Coding | HumanEval+ | 25.6 | 28.3 | 33.9 | 37.3 |
|
||||
| Coding | MBPP | 33.2 | 42.2 | 42.0 | 45.0 |
|
||||
| Coding | MBPP+ | 38.5 | 52.4 | 52.0 | 54.0 |
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{xgensmall,
|
||||
title={xGen-small Technical Report},
|
||||
author={Erik Nijkamp and Bo Pang and Egor Pakhomov and Akash Gokul and Jin Qu and Silvio Savarese and Yingbo Zhou and Caiming Xiong},
|
||||
year={2025},
|
||||
eprint={2505.06496},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CL},
|
||||
url={https://arxiv.org/abs/2505.06496},
|
||||
}
|
||||
```
|
||||
|
||||
## Ethical Considerations
|
||||
This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly impact people's lives, rights, or safety. For further guidance on use cases, refer to our AUP and AI AUP.
|
||||
|
||||
## Model Licenses
|
||||
The models are being released under CC-BY-NC-4.0, Copyright © Salesforce, Inc. All Rights Reserved.
|
||||
5
added_tokens.json
Normal file
5
added_tokens.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<|endofprompt|>": 100276,
|
||||
"<|im_end|>": 100265,
|
||||
"<|im_start|>": 100264
|
||||
}
|
||||
29
config.json
Normal file
29
config.json
Normal file
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 8192,
|
||||
"max_position_embeddings": 262144,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 28,
|
||||
"num_key_value_heads": 8,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 128000000,
|
||||
"tie_word_embeddings": true,
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": "4.51.3",
|
||||
"use_cache": true,
|
||||
"vocab_size": 102400
|
||||
}
|
||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
||||
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}
|
||||
6
generation_config.json
Normal file
6
generation_config.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"transformers_version": "4.51.3"
|
||||
}
|
||||
100001
merges.txt
Normal file
100001
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model-00001-of-00004.safetensors
Normal file
3
model-00001-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:802255d52d871d34f9600fc39320522546174b245e2bc85d5ea780c930b8290f
|
||||
size 4966225752
|
||||
3
model-00002-of-00004.safetensors
Normal file
3
model-00002-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:44394b4ba83d1c08abb2665fd3b93bfc8e40884669a37e03ad4b6ef3f725e2d3
|
||||
size 4999914480
|
||||
3
model-00003-of-00004.safetensors
Normal file
3
model-00003-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3b159009660cd6432cb260dd06ad49d9561428026cf0ffea3ece9485b1716728
|
||||
size 4999914512
|
||||
3
model-00004-of-00004.safetensors
Normal file
3
model-00004-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2e3dc0ffe9b548b4327e37dc652e4634b47b33e18688c4f0606a298a09823536
|
||||
size 2684539600
|
||||
261
model.safetensors.index.json
Normal file
261
model.safetensors.index.json
Normal file
@@ -0,0 +1,261 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_size": 17650565120
|
||||
},
|
||||
"weight_map": {
|
||||
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.14.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.15.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.15.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.15.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.16.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.16.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.16.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.17.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.17.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.17.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.23.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.23.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.23.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.24.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.24.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.24.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.24.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.25.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.25.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.25.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.25.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.25.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.26.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.26.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.26.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.26.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.26.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.27.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.27.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.27.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.27.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.27.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.5.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.6.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.6.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.6.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.7.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.7.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.7.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.norm.weight": "model-00004-of-00004.safetensors"
|
||||
}
|
||||
}
|
||||
23
special_tokens_map.json
Normal file
23
special_tokens_map.json
Normal file
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2bdbab692e14b23f1f9daee01196cd0a29c7b31dc22e47f5d5a77dda1b73632b
|
||||
size 7133808
|
||||
69
tokenizer_config.json
Normal file
69
tokenizer_config.json
Normal file
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"100257": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100258": {
|
||||
"content": "<|fim_prefix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100259": {
|
||||
"content": "<|fim_middle|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100260": {
|
||||
"content": "<|fim_suffix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100264": {
|
||||
"content": "<|im_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100265": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"100276": {
|
||||
"content": "<|endofprompt|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"bos_token": "<|endoftext|>",
|
||||
"chat_template": "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '\n<|im_end|>' + '\n'}}{% endfor %}{{ '<|im_start|>assistant\n' }}",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|endoftext|>",
|
||||
"extra_special_tokens": {},
|
||||
"model_max_length": 16384,
|
||||
"tokenizer_class": "GPT2Tokenizer",
|
||||
"unk_token": "<|endoftext|>"
|
||||
}
|
||||
3
vocab.json
Normal file
3
vocab.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:240b7e18e79826df8c0dbeff7133a3ab4708d7958fede6e27e9409e09159375f
|
||||
size 1610691
|
||||
Reference in New Issue
Block a user