初始化项目，由ModelHub XC社区提供模型
Model: stabilityai/stablelm-base-alpha-7b Source: Original Platform
This commit is contained in:
ModelHub XC
38
.gitattributes
vendored
Normal file
38
.gitattributes
vendored
Normal file
@@ -0,0 +1,38 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
pytorch_model-00001-of-00004.bin filter=lfs diff=lfs merge=lfs -text
|
||||
pytorch_model-00002-of-00004.bin filter=lfs diff=lfs merge=lfs -text
|
||||
pytorch_model-00003-of-00004.bin filter=lfs diff=lfs merge=lfs -text
|
||||
pytorch_model-00004-of-00004.bin filter=lfs diff=lfs merge=lfs -text
|
||||
85
README.md
Normal file
85
README.md
Normal file
@@ -0,0 +1,85 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
tags:
|
||||
- causal-lm
|
||||
license: cc-by-sa-4.0
|
||||
---
|
||||
|
||||
# StableLM-Base-Alpha
|
||||
|
||||
📢 **DISCLAIMER**: The StableLM-Base-Alpha models have been superseded. Find the latest versions in the Stable LM Collection [here](https://huggingface.co/collections/stabilityai/stable-lm-650852cfd55dd4e15cdcb30a).
|
||||
|
||||
## Model Description
|
||||
|
||||
`StableLM-Base-Alpha` is a suite of 3B and 7B parameter decoder-only language models pre-trained on a diverse collection of English datasets with a sequence length of 4096 to push beyond the context window limitations of existing open-source language models.
|
||||
|
||||
## Usage
|
||||
|
||||
Get started generating text with `StableLM-Base-Alpha` by using the following code snippet:
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("StabilityAI/stablelm-base-alpha-7b")
|
||||
model = AutoModelForCausalLM.from_pretrained("StabilityAI/stablelm-base-alpha-7b")
|
||||
model.half().cuda()
|
||||
|
||||
inputs = tokenizer("What's your mood today?", return_tensors="pt").to("cuda")
|
||||
tokens = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=64,
|
||||
temperature=0.7,
|
||||
do_sample=True,
|
||||
)
|
||||
print(tokenizer.decode(tokens[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## Model Details
|
||||
|
||||
* **Developed by**: [Stability AI](https://stability.ai/)
|
||||
* **Model type**: StableLM-Base-Alpha models are auto-regressive language models based on the NeoX transformer architecture.
|
||||
* **Language(s)**: English
|
||||
* **Library**: [GPT-NeoX](https://github.com/EleutherAI/gpt-neox)
|
||||
* **License**: Base model checkpoints (`StableLM-Base-Alpha`) are licensed under the Creative Commons license ([CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/)). Under the license, you must give [credit](https://creativecommons.org/licenses/by/4.0/#) to Stability AI, provide a link to the license, and [indicate if changes were made](https://creativecommons.org/licenses/by/4.0/#). You may do so in any reasonable manner, but not in any way that suggests the Stability AI endorses you or your use.
|
||||
|
||||
* **Contact**: For questions and comments about the model, please email `lm@stability.ai`
|
||||
|
||||
## Training
|
||||
|
||||
| Parameters | Hidden Size | Layers | Heads | Sequence Length |
|
||||
|------------|-------------|--------|-------|-----------------|
|
||||
| 3B | 4096 | 16 | 32 | 4096 |
|
||||
| 7B | 6144 | 16 | 48 | 4096 |
|
||||
|
||||
### Training Dataset
|
||||
|
||||
`StableLM-Base-Alpha` is pre-trained on a new experimental dataset built atop [The Pile](https://huggingface.co/datasets/EleutherAI/the_pile) and is threes times larger at approximately 1.5T tokens.
|
||||
|
||||
### Training Procedure
|
||||
|
||||
Models are pre-trained on the aforementioned dataset in mixed-precision (FP16), optimized with Adam, and trained using the NeoX tokenizer with a vocabulary size of 50,257. We outline the complete hyperparameters choices in the project's [GitHub repository](https://github.com/Stability-AI/StableLM/blob/main/configs/stablelm-base-alpha-7b.yaml).
|
||||
|
||||
## Use and Limitations
|
||||
|
||||
### Intended Use
|
||||
|
||||
These models are intended to be used by all individuals as foundational models for application-specific fine-tuning without strict limitations on commercial use.
|
||||
|
||||
### Limitations and bias
|
||||
|
||||
The pre-training dataset may have contained offensive or inappropriate content even after applying data cleansing filters which can be reflected in the model generated text. We recommend users exercise reasonable caution when using these models in production systems. Do not use the models for any applications that may cause harm or distress to individuals or groups.
|
||||
|
||||
## Citations
|
||||
|
||||
```bibtext
|
||||
@software{gpt-neox-library,
|
||||
title = {{GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch}},
|
||||
author = {Andonian, Alex and Anthony, Quentin and Biderman, Stella and Black, Sid and Gali, Preetham and Gao, Leo and Hallahan, Eric and Levy-Kramer, Josh and Leahy, Connor and Nestler, Lucas and Parker, Kip and Pieler, Michael and Purohit, Shivanshu and Songz, Tri and Phil, Wang and Weinbach, Samuel},
|
||||
url = {https://www.github.com/eleutherai/gpt-neox},
|
||||
doi = {10.5281/zenodo.5879544},
|
||||
month = {8},
|
||||
year = {2021},
|
||||
version = {0.0.1},
|
||||
}
|
||||
```
|
||||
25
config.json
Normal file
25
config.json
Normal file
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"_name_or_path": "/fsx/ckpts/7b_tok=neox_data=pilev2-recontam_lower-code_bs=8m_tp=4_pp=1_init=wang-small-init/global_step69000_hf",
|
||||
"architectures": [
|
||||
"GPTNeoXForCausalLM"
|
||||
],
|
||||
"bos_token_id": 0,
|
||||
"eos_token_id": 0,
|
||||
"hidden_act": "gelu",
|
||||
"hidden_size": 6144,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 24576,
|
||||
"layer_norm_eps": 1e-05,
|
||||
"max_position_embeddings": 4096,
|
||||
"model_type": "gpt_neox",
|
||||
"num_attention_heads": 48,
|
||||
"num_hidden_layers": 16,
|
||||
"rotary_emb_base": 10000,
|
||||
"rotary_pct": 0.25,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": "4.27.4",
|
||||
"use_cache": true,
|
||||
"use_parallel_residual": true,
|
||||
"vocab_size": 50432
|
||||
}
|
||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
||||
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}
|
||||
6
generation_config.json
Normal file
6
generation_config.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 0,
|
||||
"eos_token_id": 0,
|
||||
"transformers_version": "4.27.4"
|
||||
}
|
||||
3
pytorch_model-00001-of-00004.bin
Normal file
3
pytorch_model-00001-of-00004.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:6d92cc51dd745fe03be2e5509c1b719676204c857bdd37cec5bf046d41225342
|
||||
size 9780618427
|
||||
3
pytorch_model-00002-of-00004.bin
Normal file
3
pytorch_model-00002-of-00004.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e5a52081181cc79abae6d9ea1204f8cc36172e86fef69d28aaf5e0f81497d7cb
|
||||
size 9766089215
|
||||
3
pytorch_model-00003-of-00004.bin
Normal file
3
pytorch_model-00003-of-00004.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ebcf7f3aacf746dd5a85995acbc50a17e3a43f9a0d5702635ad40a7a1d99ed6c
|
||||
size 9749285761
|
||||
3
pytorch_model-00004-of-00004.bin
Normal file
3
pytorch_model-00004-of-00004.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:6f7dc1e04951c32a121a40cffa764f892eca8744a151712737c3fdea37094770
|
||||
size 2447551184
|
||||
251
pytorch_model.bin.index.json
Normal file
251
pytorch_model.bin.index.json
Normal file
@@ -0,0 +1,251 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_size": 31508579392.0
|
||||
},
|
||||
"weight_map": {
|
||||
"embed_out.weight": "pytorch_model-00004-of-00004.bin",
|
||||
"gpt_neox.embed_in.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.final_layer_norm.bias": "pytorch_model-00004-of-00004.bin",
|
||||
"gpt_neox.final_layer_norm.weight": "pytorch_model-00004-of-00004.bin",
|
||||
"gpt_neox.layers.0.attention.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.attention.dense.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.attention.dense.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.attention.masked_bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.attention.query_key_value.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.attention.query_key_value.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.attention.rotary_emb.inv_freq": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.input_layernorm.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.post_attention_layernorm.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.0.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.attention.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.attention.dense.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.attention.dense.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.attention.masked_bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.attention.query_key_value.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.attention.query_key_value.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.attention.rotary_emb.inv_freq": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.input_layernorm.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.post_attention_layernorm.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.1.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.10.attention.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.10.attention.dense.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.10.attention.dense.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.10.attention.masked_bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.10.attention.query_key_value.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.10.attention.query_key_value.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.10.attention.rotary_emb.inv_freq": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.10.input_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.10.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.10.mlp.dense_4h_to_h.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.10.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.10.mlp.dense_h_to_4h.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.10.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.10.post_attention_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.10.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.11.attention.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.attention.dense.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.attention.dense.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.attention.masked_bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.attention.query_key_value.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.attention.query_key_value.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.attention.rotary_emb.inv_freq": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.input_layernorm.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.mlp.dense_4h_to_h.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.mlp.dense_h_to_4h.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.post_attention_layernorm.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.11.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.attention.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.attention.dense.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.attention.dense.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.attention.masked_bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.attention.query_key_value.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.attention.query_key_value.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.attention.rotary_emb.inv_freq": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.input_layernorm.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.mlp.dense_4h_to_h.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.mlp.dense_h_to_4h.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.post_attention_layernorm.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.12.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.attention.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.attention.dense.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.attention.dense.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.attention.masked_bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.attention.query_key_value.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.attention.query_key_value.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.attention.rotary_emb.inv_freq": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.input_layernorm.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.mlp.dense_4h_to_h.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.mlp.dense_h_to_4h.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.post_attention_layernorm.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.13.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.attention.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.attention.dense.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.attention.dense.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.attention.masked_bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.attention.query_key_value.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.attention.query_key_value.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.attention.rotary_emb.inv_freq": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.input_layernorm.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.mlp.dense_4h_to_h.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.mlp.dense_h_to_4h.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.post_attention_layernorm.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.14.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.15.attention.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.15.attention.dense.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.15.attention.dense.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.15.attention.masked_bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.15.attention.query_key_value.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.15.attention.query_key_value.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.15.attention.rotary_emb.inv_freq": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.15.input_layernorm.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.15.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.15.mlp.dense_4h_to_h.bias": "pytorch_model-00004-of-00004.bin",
|
||||
"gpt_neox.layers.15.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00004.bin",
|
||||
"gpt_neox.layers.15.mlp.dense_h_to_4h.bias": "pytorch_model-00004-of-00004.bin",
|
||||
"gpt_neox.layers.15.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00004.bin",
|
||||
"gpt_neox.layers.15.post_attention_layernorm.bias": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.15.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
|
||||
"gpt_neox.layers.2.attention.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.attention.dense.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.attention.dense.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.attention.masked_bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.attention.query_key_value.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.attention.query_key_value.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.attention.rotary_emb.inv_freq": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.input_layernorm.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.post_attention_layernorm.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.2.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.attention.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.attention.dense.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.attention.dense.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.attention.masked_bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.attention.query_key_value.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.attention.query_key_value.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.attention.rotary_emb.inv_freq": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.input_layernorm.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.post_attention_layernorm.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.3.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.attention.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.attention.dense.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.attention.dense.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.attention.masked_bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.attention.query_key_value.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.attention.query_key_value.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.attention.rotary_emb.inv_freq": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.input_layernorm.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.4.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.4.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.post_attention_layernorm.bias": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.4.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
|
||||
"gpt_neox.layers.5.attention.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.attention.dense.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.attention.dense.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.attention.masked_bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.attention.query_key_value.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.attention.query_key_value.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.attention.rotary_emb.inv_freq": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.input_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.post_attention_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.5.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.attention.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.attention.dense.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.attention.dense.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.attention.masked_bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.attention.query_key_value.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.attention.query_key_value.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.attention.rotary_emb.inv_freq": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.input_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.post_attention_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.6.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.attention.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.attention.dense.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.attention.dense.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.attention.masked_bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.attention.query_key_value.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.attention.query_key_value.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.attention.rotary_emb.inv_freq": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.input_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.post_attention_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.7.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.attention.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.attention.dense.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.attention.dense.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.attention.masked_bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.attention.query_key_value.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.attention.query_key_value.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.attention.rotary_emb.inv_freq": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.input_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.post_attention_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.8.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.attention.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.attention.dense.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.attention.dense.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.attention.masked_bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.attention.query_key_value.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.attention.query_key_value.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.attention.rotary_emb.inv_freq": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.input_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.post_attention_layernorm.bias": "pytorch_model-00002-of-00004.bin",
|
||||
"gpt_neox.layers.9.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin"
|
||||
}
|
||||
}
|
||||
5
special_tokens_map.json
Normal file
5
special_tokens_map.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"bos_token": "<|endoftext|>",
|
||||
"eos_token": "<|endoftext|>",
|
||||
"unk_token": "<|endoftext|>"
|
||||
}
|
||||
100529
tokenizer.json
Normal file
100529
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
9
tokenizer_config.json
Normal file
9
tokenizer_config.json
Normal file
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"add_prefix_space": false,
|
||||
"bos_token": "<|endoftext|>",
|
||||
"clean_up_tokenization_spaces": true,
|
||||
"eos_token": "<|endoftext|>",
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"tokenizer_class": "GPTNeoXTokenizer",
|
||||
"unk_token": "<|endoftext|>"
|
||||
}
|
||||
Reference in New Issue
Block a user