初始化项目，由ModelHub XC社区提供模型
Model: speakleash/Bielik-11B-v2 Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
204
README.md
Normal file
204
README.md
Normal file
@@ -0,0 +1,204 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- pl
|
||||
library_name: transformers
|
||||
inference:
|
||||
parameters:
|
||||
temperature: 0.9
|
||||
extra_gated_description: If you want to learn more about how you can use the model, please refer to our <a href="https://bielik.ai/terms/">Terms of Use</a>.
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
<img src="https://huggingface.co/speakleash/Bielik-11B-v2/raw/main/speakleash_cyfronet.png">
|
||||
</p>
|
||||
|
||||
# Bielik-11B-v2
|
||||
|
||||
Bielik-11B-v2 is a generative text model featuring 11 billion parameters. It is initialized from its predecessor, Mistral-7B-v0.2, and trained on 400 billion tokens.
|
||||
The aforementioned model stands as a testament to the unique collaboration between the open-science/open-source project SpeakLeash and the High Performance Computing (HPC) center: ACK Cyfronet AGH.
|
||||
Developed and trained on Polish text corpora, which have been cherry-picked and processed by the SpeakLeash team, this endeavor leverages Polish large-scale computing infrastructure, specifically within the PLGrid environment,
|
||||
and more precisely, the HPC center: ACK Cyfronet AGH. The creation and training of the Bielik-11B-v2 was propelled by the support of computational grant number PLG/2024/016951, conducted on the Athena and Helios supercomputer,
|
||||
enabling the use of cutting-edge technology and computational resources essential for large-scale machine learning processes. As a result, the model exhibits an exceptional ability to understand and process the Polish language,
|
||||
providing accurate responses and performing a variety of linguistic tasks with high precision.
|
||||
|
||||
⚠️ This is a base model intended for further fine-tuning across most use cases. If you're looking for a model ready for chatting or following instructions out-of-the-box, please use [Bielik-11B-v.2.2-Instruct](https://huggingface.co/speakleash/Bielik-11B-v2.2-Instruct).
|
||||
|
||||
🎥 Demo: https://chat.bielik.ai
|
||||
|
||||
🗣️ Chat Arena<span style="color:red;">*</span>: https://arena.speakleash.org.pl/
|
||||
|
||||
<span style="color:red;">*</span>Chat Arena is a platform for testing and comparing different AI language models, allowing users to evaluate their performance and quality.
|
||||
|
||||
|
||||
## Model
|
||||
|
||||
Bielik-11B-v2 has been trained with [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) using different parallelization techniques.
|
||||
|
||||
The model training was conducted on the Helios Supercomputer at the ACK Cyfronet AGH, utilizing 256 NVidia GH200 cards.
|
||||
|
||||
The training dataset was composed of Polish texts collected and made available through the [SpeakLeash](https://speakleash.org/) project, as well as a subset of CommonCrawl data. We used 200 billion tokens (over 700 GB of plain text) for two epochs of training.
|
||||
|
||||
### Model description:
|
||||
|
||||
* **Developed by:** [SpeakLeash](https://speakleash.org/) & [ACK Cyfronet AGH](https://www.cyfronet.pl/)
|
||||
* **Language:** Polish
|
||||
* **Model type:** causal decoder-only
|
||||
* **Initialized from:** [Mistral-7B-v0.2](https://models.mistralcdn.com/mistral-7b-v0-2/mistral-7B-v0.2.tar)
|
||||
* **License:** Apache 2.0 and [Terms of Use](https://bielik.ai/terms/)
|
||||
* **Model ref:** speakleash:45b6efdb701991181a05968fc53d2a8e
|
||||
|
||||
### Quality evaluation
|
||||
|
||||
An XGBoost classification model was prepared and created to evaluate the quality of texts in native Polish language. It is based on 93 features, such as the ratio of out-of-vocabulary words to all words (OOVs), the number of nouns, verbs, average sentence length etc. The model outputs the category of a given document (either HIGH, MEDIUM or LOW) along with the probability. This approach allows implementation of a dedicated pipeline to choose documents, from which we've used entries with HIGH quality index and probability exceeding 90%.
|
||||
|
||||
This filtration and appropriate selection of texts enable the provision of a condensed and high-quality database of texts in Polish for training purposes.
|
||||
|
||||
### Quickstart
|
||||
|
||||
This model can be easily loaded using the AutoModelForCausalLM functionality.
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
model_name = "speakleash/Bielik-11B-v2"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_name)
|
||||
```
|
||||
|
||||
In order to reduce the memory usage, you can use smaller precision (`bfloat16`).
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)
|
||||
```
|
||||
|
||||
And then you can use HuggingFace Pipelines to generate text:
|
||||
|
||||
```python
|
||||
import transformers
|
||||
|
||||
text = "Najważniejszym celem człowieka na ziemi jest"
|
||||
|
||||
pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer)
|
||||
sequences = pipeline(max_new_tokens=100, do_sample=True, top_k=50, eos_token_id=tokenizer.eos_token_id, text_inputs=text)
|
||||
for seq in sequences:
|
||||
print(f"Result: {seq['generated_text']}")
|
||||
```
|
||||
Generated output:
|
||||
> Najważniejszym celem człowieka na ziemi jest życie w pokoju, harmonii i miłości. Dla każdego z nas bardzo ważne jest, aby otaczać się kochanymi osobami.
|
||||
|
||||
|
||||
## Evaluation
|
||||
|
||||
Models have been evaluated on two leaderboards: [Open PL LLM Leaderboard](https://huggingface.co/spaces/speakleash/open_pl_llm_leaderboard) and [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). The Open PL LLM Leaderboard uses a 5-shot evaluation and focuses on NLP tasks in Polish, while the Open LLM Leaderboard evaluates models on various English language tasks.
|
||||
|
||||
### Open PL LLM Leaderboard
|
||||
|
||||
The benchmark evaluates models in NLP tasks like sentiment analysis, categorization, text classification but does not test chatting skills. Average column is an average score among all tasks normalized by baseline scores.
|
||||
|
||||
| Model | Parameters (B) | Average |
|
||||
|------------------------|------------|---------|
|
||||
| Meta-Llama-3-70B | 70 | 62.07 |
|
||||
| Qwen1.5-72B | 72 | 61.11 |
|
||||
| Meta-Llama-3.1-70B | 70 | 60.87 |
|
||||
| Mixtral-8x22B-v0.1 | 141 | 60.75 |
|
||||
| Qwen1.5-32B | 32 | 58.71 |
|
||||
| **Bielik-11B-v2** | **11** | **58.14** |
|
||||
| Qwen2-7B | 7 | 49.39 |
|
||||
| SOLAR-10.7B-v1.0 | 10.7 | 47.54 |
|
||||
| Mistral-Nemo-Base-2407 | 12 | 47.28 |
|
||||
| internlm2-20b | 20 | 47.15 |
|
||||
| Meta-Llama-3.1-8B | 8 | 43.77 |
|
||||
| Meta-Llama-3-8B | 8 | 43.30 |
|
||||
| Mistral-7B-v0.2 | 7 | 38.81 |
|
||||
| Bielik-7B-v0.1 | 7 | 34.34 |
|
||||
| Qra-13b | 13 | 33.90 |
|
||||
| Qra-7b | 7 | 16.60 |
|
||||
|
||||
The results from the Open PL LLM Leaderboard show that the Bielik-11B-v2 model, with 11 billion parameters, achieved an average score of 58.14. This makes it the best performing model among those under 20B parameters, outperforming the second-best model in this category by an impressive 8.75 percentage points. This significant lead not only places it ahead of its predecessor, the Bielik-7B-v0.1 (which scored 34.34), but also demonstrates its superiority over other larger models. The substantial improvement highlights the remarkable advancements and optimizations made in this newer version.
|
||||
|
||||
Other Polish models listed include Qra-13b and Qra-7b, scoring 33.90 and 16.60 respectively, indicating that Bielik-11B-v2 outperforms these models by a considerable margin.
|
||||
|
||||
Additionally, the Bielik-11B-v2 was initialized from the weights of Mistral-7B-v0.2, which itself scored 38.81, further demonstrating the effective enhancements incorporated into the Bielik-11B-v2 model.
|
||||
|
||||
### Open LLM Leaderboard
|
||||
|
||||
The Open LLM Leaderboard evaluates models on various English language tasks, providing insights into the model's performance across different linguistic challenges.
|
||||
|
||||
| Model | AVG | arc_challenge | hellaswag | truthfulqa_mc2 | mmlu | winogrande | gsm8k |
|
||||
|-------------------------|-------|---------------|-----------|----------------|-------|------------|-------|
|
||||
| **Bielik-11B-v2** | **65.87** | 60.58 | 79.84 | 46.13 | 63.06 | 77.82 | 67.78 |
|
||||
| Mistral-7B-v0.2 | 60.37 | 60.84 | 83.08 | 41.76 | 63.62 | 78.22 | 34.72 |
|
||||
| Bielik-7B-v0.1 | 49.98 | 45.22 | 67.92 | 47.16 | 43.20 | 66.85 | 29.49 |
|
||||
|
||||
The results from the Open LLM Leaderboard demonstrate the impressive performance of Bielik-11B-v2 across various NLP tasks. With an average score of 65.87, it significantly outperforms its predecessor, Bielik-7B-v0.1, and even surpasses Mistral-7B-v0.2, which served as its initial weight basis.
|
||||
|
||||
Key observations:
|
||||
1. Bielik-11B-v2 shows substantial improvements in most categories compared to Bielik-7B-v0.1, highlighting the effectiveness of the model's enhancements.
|
||||
2. It performs exceptionally well in tasks like hellaswag (common sense reasoning), winogrande (commonsense reasoning), and gsm8k (mathematical problem-solving), indicating its versatility across different types of language understanding and generation tasks.
|
||||
3. While Mistral-7B-v0.2 outperforms in truthfulqa_mc2, Bielik-11B-v2 maintains competitive performance in this truth-discernment task.
|
||||
|
||||
Although Bielik-11B-v2 was primarily trained on Polish data, it has retained and even improved its ability to understand and operate in English, as evidenced by its strong performance across these English-language benchmarks. This suggests that the model has effectively leveraged cross-lingual transfer learning, maintaining its Polish language expertise while enhancing its English language capabilities.
|
||||
|
||||
|
||||
## Limitations and Biases
|
||||
|
||||
Bielik-11B-v2 is not intended for deployment without fine-tuning. It should not be used for human-facing interactions without further guardrails and user consent.
|
||||
|
||||
Bielik-11B-v2 can produce factually incorrect output, and should not be relied on to produce factually accurate data. Bielik-11B-v2 was trained on various public datasets. While great efforts have been taken to clear the training data, it is possible that this model can generate lewd, false, biased or otherwise offensive outputs.
|
||||
|
||||
|
||||
## Citation
|
||||
Please cite this model using the following format:
|
||||
|
||||
```
|
||||
@misc{Bielik11Bv2b,
|
||||
title = {Bielik-11B-v2 model card},
|
||||
author = {Ociepa, Krzysztof and Flis, Łukasz and Wróbel, Krzysztof and Gwoździej, Adrian and {SpeakLeash Team} and {Cyfronet Team}},
|
||||
year = {2024},
|
||||
url = {https://huggingface.co/speakleash/Bielik-11B-v2},
|
||||
note = {Accessed: 2024-08-28},
|
||||
urldate = {2024-08-28}
|
||||
}
|
||||
@unpublished{Bielik11Bv2a,
|
||||
author = {Ociepa, Krzysztof and Flis, Łukasz and Kinas, Remigiusz and Gwoździej, Adrian and Wróbel, Krzysztof},
|
||||
title = {Bielik: A Family of Large Language Models for the Polish Language - Development, Insights, and Evaluation},
|
||||
year = {2024},
|
||||
}
|
||||
@misc{ociepa2024bielik7bv01polish,
|
||||
title={Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation},
|
||||
author={Krzysztof Ociepa and Łukasz Flis and Krzysztof Wróbel and Adrian Gwoździej and Remigiusz Kinas},
|
||||
year={2024},
|
||||
eprint={2410.18565},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CL},
|
||||
url={https://arxiv.org/abs/2410.18565},
|
||||
}
|
||||
```
|
||||
|
||||
## Responsible for training the model
|
||||
|
||||
* [Krzysztof Ociepa](https://www.linkedin.com/in/krzysztof-ociepa-44886550/)<sup>SpeakLeash</sup> - team leadership, conceptualizing, data preparation, process optimization and oversight of training
|
||||
* [Łukasz Flis](https://www.linkedin.com/in/lukasz-flis-0a39631/)<sup>Cyfronet AGH</sup> - coordinating and supervising the training
|
||||
* [Adrian Gwoździej](https://www.linkedin.com/in/adrgwo/)<sup>SpeakLeash</sup> - data cleaning and quality
|
||||
* [Krzysztof Wróbel](https://www.linkedin.com/in/wrobelkrzysztof/)<sup>SpeakLeash</sup> - benchmarks
|
||||
|
||||
The model could not have been created without the commitment and work of the entire SpeakLeash team, whose contribution is invaluable. Thanks to the hard work of many individuals, it was possible to gather a large amount of content in Polish and establish collaboration between the open-science SpeakLeash project and the HPC center: ACK Cyfronet AGH. Individuals who contributed to the creation of the model:
|
||||
[Grzegorz Urbanowicz](https://www.linkedin.com/in/grzegorz-urbanowicz-05823469/),
|
||||
[Igor Ciuciura](https://www.linkedin.com/in/igor-ciuciura-1763b52a6/),
|
||||
[Jacek Chwiła](https://www.linkedin.com/in/jacek-chwila/),
|
||||
[Szymon Baczyński](https://www.linkedin.com/in/szymon-baczynski/),
|
||||
[Paweł Kiszczak](https://www.linkedin.com/in/paveu-kiszczak/),
|
||||
[Aleksander Smywiński-Pohl](https://www.linkedin.com/in/apohllo/).
|
||||
|
||||
Members of the ACK Cyfronet AGH team providing valuable support and expertise:
|
||||
[Szymon Mazurek](https://www.linkedin.com/in/sz-mazurek-ai/),
|
||||
[Marek Magryś](https://www.linkedin.com/in/magrys/).
|
||||
|
||||
|
||||
## Contact Us
|
||||
|
||||
If you have any questions or suggestions, please use the discussion tab. If you want to contact us directly, join our [Discord SpeakLeash](https://discord.gg/pv4brQMDTy).
|
||||
130
added_tokens.json
Normal file
130
added_tokens.json
Normal file
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"<|control_100|>": 32099,
|
||||
"<|control_101|>": 32100,
|
||||
"<|control_102|>": 32101,
|
||||
"<|control_103|>": 32102,
|
||||
"<|control_104|>": 32103,
|
||||
"<|control_105|>": 32104,
|
||||
"<|control_106|>": 32105,
|
||||
"<|control_107|>": 32106,
|
||||
"<|control_108|>": 32107,
|
||||
"<|control_109|>": 32108,
|
||||
"<|control_10|>": 32009,
|
||||
"<|control_110|>": 32109,
|
||||
"<|control_111|>": 32110,
|
||||
"<|control_112|>": 32111,
|
||||
"<|control_113|>": 32112,
|
||||
"<|control_114|>": 32113,
|
||||
"<|control_115|>": 32114,
|
||||
"<|control_116|>": 32115,
|
||||
"<|control_117|>": 32116,
|
||||
"<|control_118|>": 32117,
|
||||
"<|control_119|>": 32118,
|
||||
"<|control_11|>": 32010,
|
||||
"<|control_120|>": 32119,
|
||||
"<|control_121|>": 32120,
|
||||
"<|control_122|>": 32121,
|
||||
"<|control_123|>": 32122,
|
||||
"<|control_124|>": 32123,
|
||||
"<|control_125|>": 32124,
|
||||
"<|control_126|>": 32125,
|
||||
"<|control_127|>": 32126,
|
||||
"<|control_128|>": 32127,
|
||||
"<|control_12|>": 32011,
|
||||
"<|control_13|>": 32012,
|
||||
"<|control_14|>": 32013,
|
||||
"<|control_15|>": 32014,
|
||||
"<|control_16|>": 32015,
|
||||
"<|control_17|>": 32016,
|
||||
"<|control_18|>": 32017,
|
||||
"<|control_19|>": 32018,
|
||||
"<|control_20|>": 32019,
|
||||
"<|control_21|>": 32020,
|
||||
"<|control_22|>": 32021,
|
||||
"<|control_23|>": 32022,
|
||||
"<|control_24|>": 32023,
|
||||
"<|control_25|>": 32024,
|
||||
"<|control_26|>": 32025,
|
||||
"<|control_27|>": 32026,
|
||||
"<|control_28|>": 32027,
|
||||
"<|control_29|>": 32028,
|
||||
"<|control_30|>": 32029,
|
||||
"<|control_31|>": 32030,
|
||||
"<|control_32|>": 32031,
|
||||
"<|control_33|>": 32032,
|
||||
"<|control_34|>": 32033,
|
||||
"<|control_35|>": 32034,
|
||||
"<|control_36|>": 32035,
|
||||
"<|control_37|>": 32036,
|
||||
"<|control_38|>": 32037,
|
||||
"<|control_39|>": 32038,
|
||||
"<|control_40|>": 32039,
|
||||
"<|control_41|>": 32040,
|
||||
"<|control_42|>": 32041,
|
||||
"<|control_43|>": 32042,
|
||||
"<|control_44|>": 32043,
|
||||
"<|control_45|>": 32044,
|
||||
"<|control_46|>": 32045,
|
||||
"<|control_47|>": 32046,
|
||||
"<|control_48|>": 32047,
|
||||
"<|control_49|>": 32048,
|
||||
"<|control_50|>": 32049,
|
||||
"<|control_51|>": 32050,
|
||||
"<|control_52|>": 32051,
|
||||
"<|control_53|>": 32052,
|
||||
"<|control_54|>": 32053,
|
||||
"<|control_55|>": 32054,
|
||||
"<|control_56|>": 32055,
|
||||
"<|control_57|>": 32056,
|
||||
"<|control_58|>": 32057,
|
||||
"<|control_59|>": 32058,
|
||||
"<|control_60|>": 32059,
|
||||
"<|control_61|>": 32060,
|
||||
"<|control_62|>": 32061,
|
||||
"<|control_63|>": 32062,
|
||||
"<|control_64|>": 32063,
|
||||
"<|control_65|>": 32064,
|
||||
"<|control_66|>": 32065,
|
||||
"<|control_67|>": 32066,
|
||||
"<|control_68|>": 32067,
|
||||
"<|control_69|>": 32068,
|
||||
"<|control_6|>": 32005,
|
||||
"<|control_70|>": 32069,
|
||||
"<|control_71|>": 32070,
|
||||
"<|control_72|>": 32071,
|
||||
"<|control_73|>": 32072,
|
||||
"<|control_74|>": 32073,
|
||||
"<|control_75|>": 32074,
|
||||
"<|control_76|>": 32075,
|
||||
"<|control_77|>": 32076,
|
||||
"<|control_78|>": 32077,
|
||||
"<|control_79|>": 32078,
|
||||
"<|control_7|>": 32006,
|
||||
"<|control_80|>": 32079,
|
||||
"<|control_81|>": 32080,
|
||||
"<|control_82|>": 32081,
|
||||
"<|control_83|>": 32082,
|
||||
"<|control_84|>": 32083,
|
||||
"<|control_85|>": 32084,
|
||||
"<|control_86|>": 32085,
|
||||
"<|control_87|>": 32086,
|
||||
"<|control_88|>": 32087,
|
||||
"<|control_89|>": 32088,
|
||||
"<|control_8|>": 32007,
|
||||
"<|control_90|>": 32089,
|
||||
"<|control_91|>": 32090,
|
||||
"<|control_92|>": 32091,
|
||||
"<|control_93|>": 32092,
|
||||
"<|control_94|>": 32093,
|
||||
"<|control_95|>": 32094,
|
||||
"<|control_96|>": 32095,
|
||||
"<|control_97|>": 32096,
|
||||
"<|control_98|>": 32097,
|
||||
"<|control_99|>": 32098,
|
||||
"<|control_9|>": 32008,
|
||||
"<|function_call|>": 32004,
|
||||
"<|function_list|>": 32002,
|
||||
"<|function_output|>": 32003,
|
||||
"<|im_end|>": 32001,
|
||||
"<|im_start|>": 32000
|
||||
}
|
||||
26
config.json
Normal file
26
config.json
Normal file
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"architectures": [
|
||||
"MistralForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 14336,
|
||||
"max_position_embeddings": 32768,
|
||||
"model_type": "mistral",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 50,
|
||||
"num_key_value_heads": 8,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_theta": 1000000,
|
||||
"sliding_window": null,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.40.2",
|
||||
"use_cache": true,
|
||||
"vocab_size": 32128
|
||||
}
|
||||
6
generation_config.json
Normal file
6
generation_config.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"transformers_version": "4.40.2"
|
||||
}
|
||||
3
model-00001-of-00005.safetensors
Normal file
3
model-00001-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1cfdf340a7a54bbcd06a18be3ff66c129f11d508332a66bef184fba82a7780ce
|
||||
size 4944210912
|
||||
3
model-00002-of-00005.safetensors
Normal file
3
model-00002-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:46ccb684e6e9811320e97d10f17494fd8f8920662ee0230d5ddd56c02cf7c1f3
|
||||
size 4999819336
|
||||
3
model-00003-of-00005.safetensors
Normal file
3
model-00003-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:a3408af06742b438f3dddac3efc02aa8c0c7a56114655bdacec5f4e4e2708d3f
|
||||
size 4915916184
|
||||
3
model-00004-of-00005.safetensors
Normal file
3
model-00004-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:f91968f4d9b390cfea0d7dde6a3771a0c536eaf67ffc5cbe08991f3bddde9142
|
||||
size 4915916176
|
||||
3
model-00005-of-00005.safetensors
Normal file
3
model-00005-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8c56dff2b60ae00175e89be91e25e00b67c8b77e616e4d170a61c112b124992e
|
||||
size 2561783456
|
||||
460
model.safetensors.index.json
Normal file
460
model.safetensors.index.json
Normal file
@@ -0,0 +1,460 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_size": 22337593344
|
||||
},
|
||||
"weight_map": {
|
||||
"lm_head.weight": "model-00005-of-00005.safetensors",
|
||||
"model.embed_tokens.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.10.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.10.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.24.input_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.25.input_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.26.input_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.27.input_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.28.input_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.28.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.28.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.28.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.28.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.29.input_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.29.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.29.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.29.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.29.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.29.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.29.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.30.input_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.30.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.30.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.30.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.30.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.30.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.30.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.31.input_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.31.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.31.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.31.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.31.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.31.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.31.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.31.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.32.input_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.32.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.32.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.32.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.32.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.32.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.32.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.32.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.32.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.33.input_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.33.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.33.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.33.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.33.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.33.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.33.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.33.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.33.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
|
||||
"model.layers.34.input_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.34.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.34.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.34.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.34.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.34.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.34.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.34.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.34.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.35.input_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.35.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.35.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.35.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.35.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.35.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.35.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.35.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.35.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.36.input_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.36.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.36.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.36.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.36.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.36.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.36.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.36.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.36.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.37.input_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.37.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.37.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.37.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.37.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.37.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.37.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.37.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.37.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.38.input_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.38.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.38.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.38.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.38.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.38.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.38.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.38.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.38.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.39.input_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.39.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.39.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.39.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.39.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.39.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.39.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.39.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.39.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.40.input_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.40.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.40.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.40.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.40.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.40.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.40.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.40.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.40.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.41.input_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.41.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.41.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.41.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.41.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.41.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.41.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.41.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.41.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.42.input_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.42.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.42.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.42.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.42.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.42.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.42.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.42.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.42.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.43.input_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.43.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.43.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.43.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.43.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.43.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.43.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.43.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.43.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.44.input_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.44.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.44.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.44.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.44.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.44.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.44.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.44.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.44.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
|
||||
"model.layers.45.input_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.45.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.45.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.45.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.45.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.45.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.45.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.45.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.45.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.46.input_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.46.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.46.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.46.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.46.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.46.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.46.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.46.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.46.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.47.input_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.47.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.47.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.47.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.47.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.47.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.47.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.47.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.47.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.48.input_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.48.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.48.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.48.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.48.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.48.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.48.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.48.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.48.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.49.input_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.49.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.49.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.49.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.49.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.49.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.49.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.49.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.49.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.9.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.9.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
|
||||
"model.norm.weight": "model-00005-of-00005.safetensors"
|
||||
}
|
||||
}
|
||||
12
params.json
Normal file
12
params.json
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"dim": 4096,
|
||||
"n_layers": 50,
|
||||
"head_dim": 128,
|
||||
"hidden_dim": 14336,
|
||||
"n_heads": 32,
|
||||
"n_kv_heads": 8,
|
||||
"norm_eps": 1e-05,
|
||||
"vocab_size": 32128,
|
||||
"rope_theta": 1000000.0
|
||||
}
|
||||
|
||||
BIN
speakleash_cyfronet.png
Normal file
BIN
speakleash_cyfronet.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
160
special_tokens_map.json
Normal file
160
special_tokens_map.json
Normal file
@@ -0,0 +1,160 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<|function_list|>",
|
||||
"<|function_output|>",
|
||||
"<|function_call|>",
|
||||
"<|control_6|>",
|
||||
"<|control_7|>",
|
||||
"<|control_8|>",
|
||||
"<|control_9|>",
|
||||
"<|control_10|>",
|
||||
"<|control_11|>",
|
||||
"<|control_12|>",
|
||||
"<|control_13|>",
|
||||
"<|control_14|>",
|
||||
"<|control_15|>",
|
||||
"<|control_16|>",
|
||||
"<|control_17|>",
|
||||
"<|control_18|>",
|
||||
"<|control_19|>",
|
||||
"<|control_20|>",
|
||||
"<|control_21|>",
|
||||
"<|control_22|>",
|
||||
"<|control_23|>",
|
||||
"<|control_24|>",
|
||||
"<|control_25|>",
|
||||
"<|control_26|>",
|
||||
"<|control_27|>",
|
||||
"<|control_28|>",
|
||||
"<|control_29|>",
|
||||
"<|control_30|>",
|
||||
"<|control_31|>",
|
||||
"<|control_32|>",
|
||||
"<|control_33|>",
|
||||
"<|control_34|>",
|
||||
"<|control_35|>",
|
||||
"<|control_36|>",
|
||||
"<|control_37|>",
|
||||
"<|control_38|>",
|
||||
"<|control_39|>",
|
||||
"<|control_40|>",
|
||||
"<|control_41|>",
|
||||
"<|control_42|>",
|
||||
"<|control_43|>",
|
||||
"<|control_44|>",
|
||||
"<|control_45|>",
|
||||
"<|control_46|>",
|
||||
"<|control_47|>",
|
||||
"<|control_48|>",
|
||||
"<|control_49|>",
|
||||
"<|control_50|>",
|
||||
"<|control_51|>",
|
||||
"<|control_52|>",
|
||||
"<|control_53|>",
|
||||
"<|control_54|>",
|
||||
"<|control_55|>",
|
||||
"<|control_56|>",
|
||||
"<|control_57|>",
|
||||
"<|control_58|>",
|
||||
"<|control_59|>",
|
||||
"<|control_60|>",
|
||||
"<|control_61|>",
|
||||
"<|control_62|>",
|
||||
"<|control_63|>",
|
||||
"<|control_64|>",
|
||||
"<|control_65|>",
|
||||
"<|control_66|>",
|
||||
"<|control_67|>",
|
||||
"<|control_68|>",
|
||||
"<|control_69|>",
|
||||
"<|control_70|>",
|
||||
"<|control_71|>",
|
||||
"<|control_72|>",
|
||||
"<|control_73|>",
|
||||
"<|control_74|>",
|
||||
"<|control_75|>",
|
||||
"<|control_76|>",
|
||||
"<|control_77|>",
|
||||
"<|control_78|>",
|
||||
"<|control_79|>",
|
||||
"<|control_80|>",
|
||||
"<|control_81|>",
|
||||
"<|control_82|>",
|
||||
"<|control_83|>",
|
||||
"<|control_84|>",
|
||||
"<|control_85|>",
|
||||
"<|control_86|>",
|
||||
"<|control_87|>",
|
||||
"<|control_88|>",
|
||||
"<|control_89|>",
|
||||
"<|control_90|>",
|
||||
"<|control_91|>",
|
||||
"<|control_92|>",
|
||||
"<|control_93|>",
|
||||
"<|control_94|>",
|
||||
"<|control_95|>",
|
||||
"<|control_96|>",
|
||||
"<|control_97|>",
|
||||
"<|control_98|>",
|
||||
"<|control_99|>",
|
||||
"<|control_100|>",
|
||||
"<|control_101|>",
|
||||
"<|control_102|>",
|
||||
"<|control_103|>",
|
||||
"<|control_104|>",
|
||||
"<|control_105|>",
|
||||
"<|control_106|>",
|
||||
"<|control_107|>",
|
||||
"<|control_108|>",
|
||||
"<|control_109|>",
|
||||
"<|control_110|>",
|
||||
"<|control_111|>",
|
||||
"<|control_112|>",
|
||||
"<|control_113|>",
|
||||
"<|control_114|>",
|
||||
"<|control_115|>",
|
||||
"<|control_116|>",
|
||||
"<|control_117|>",
|
||||
"<|control_118|>",
|
||||
"<|control_119|>",
|
||||
"<|control_120|>",
|
||||
"<|control_121|>",
|
||||
"<|control_122|>",
|
||||
"<|control_123|>",
|
||||
"<|control_124|>",
|
||||
"<|control_125|>",
|
||||
"<|control_126|>",
|
||||
"<|control_127|>",
|
||||
"<|control_128|>"
|
||||
],
|
||||
"bos_token": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
92275
tokenizer.json
Normal file
92275
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
Binary file not shown.
1196
tokenizer_config.json
Normal file
1196
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user