occiglot/occiglot-7b-eu5
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: occiglot/occiglot-7b-eu5
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-04-13 04:58:58 +08:00
commit ed60fe45fd
19 changed files with 92149 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

187
README.md Normal file
View File

@@ -0,0 +1,187 @@
---
license: apache-2.0
language:
- en
- es
- de
- fr
- it
pipeline_tag: text-generation
---
![image/png](https://huggingface.co/datasets/malteos/images/resolve/main/occiglot.medium.png)
# Occiglot-7B-EU5
> A [polyglot](https://en.wikipedia.org/wiki/Multilingualism#In_individuals) language model for the [Occident](https://en.wikipedia.org/wiki/Occident).
>
**Occiglot-7B-EU5** is a generative language model with 7B parameters supporting the top-5 EU languages (English, Spanish, French, German, and Italian) and trained by the [Occiglot Research Collective](https://occiglot.github.io/occiglot/).
It is based on [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) and trained on 293B tokens of additional multilingual and code data with a block size of 8,192 tokens per sample.
Note that the model is a general-purpose base model and was not instruction-fine-tuned nor optimized for chat or other applications. We make an instruction tuned variant available as [occiglot-7b-eu5-instruct](https://huggingface.co/occiglot/occiglot-7b-eu5-instruct)
This is the first release of an ongoing open research project for multilingual language models.
If you want to train a model for your own language or are working on evaluations, please contact us or join our [Discord server](https://discord.gg/wUpvYs4XvM). **We are open for collaborations!**
### Model details
- **Continued-pretraining from:** [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
- **Model type:** Causal decoder-only transformer language model
- **Languages:** English, Spanish, French, German, Italian, and code.
- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.html)
- **Compute resources:** [HessianAI's 42](https://hessian.ai/)
- **Contributors:** Manuel Brack, Patrick Schramowski, Pedro Ortiz, Malte Ostendorff, Fabio Barth, Georg Rehm, Kristian Kersting
- **Research labs:** [Occiglot](https://occiglot.github.io/occiglot/) with support from [SAINT](https://www.dfki.de/en/web/research/research-departments/foundations-of-systems-ai) and [SLT](https://www.dfki.de/en/web/research/research-departments/speech-and-language-technology)
- **Contact:** [Discord](https://discord.gg/wUpvYs4XvM)
### How to use
You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
set a seed for reproducibility:
```python
>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='occiglot/occiglot-7b-eu5')
>>> set_seed(42)
>>> generator("Hallo, Ich bin ein Sprachmodell,", max_length=40, num_return_sequences=1)
[{'generated_text': 'Hallo, Ich bin ein Sprachmodell, das dir bei der Übersetzung von Texten zwischen Deutsch und Englisch helfen kann. Wenn du mir einen Text in Deutsch'}]
```
## Dataset
The training data was split amongst the 4 target languages (de, es, fr, it) and the continuous training in English and code.
The data distribution by language (estimated) is as follows:
- English: ~13%
- Code: ~5%
- German: ~20%
- Spanish: ~20%
- French: ~20%
- Italian: ~20%
The training data was prepared using [lm-datasets](https://github.com/malteos/lm-datasets).
The exact data configuration is [here](https://huggingface.co/occiglot/occiglot-7b-eu5/blob/main/lm-datasets-config.yml).
## Training settings
- Continual pre-training on 128 x A100-80GB on [HessianAI's 42](https://hessian.ai/).
- Framework: [Determined](https://www.determined.ai/)
- Precision: bf16
- Optimizer: AdamW (lr: 0.00001, warmup_steps: 420)
- Global batch size: 512 (with 8192 blocksize) split over 128 GPUs
- Cosine Annealing with Warmup
## Tokenizer
Tokenizer is unchanged from [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).
## Evaluation
Preliminary evaluation results can be found below.
Please note that the non-English results are based on partially machine-translated datasets and English prompts ([Belebele](https://huggingface.co/datasets/facebook/belebele) and [Okapi framework](https://github.com/nlp-uoregon/Okapi)) and thus should be interpreted with caution, e.g., biased towards English model performance.
Currently, we are working on more suitable benchmarks for Spanish, French, German, and Italian.
<details>
<summary>Evaluation results</summary>
### All 5 Languages
| | avg | arc_challenge | belebele | hellaswag | mmlu | truthfulqa |
|:---------------------------|---------:|----------------:|-----------:|------------:|---------:|-------------:|
| Occiglot-7b-eu5 | 0.516895 | 0.508109 | 0.675556 | 0.718963 | 0.402064 | 0.279782 |
| Occiglot-7b-eu5-instruct | 0.537799 | 0.53632 | 0.691111 | 0.731918 | 0.405198 | 0.32445 |
| Occiglot-7b-de-en | 0.518337 | 0.496297 | 0.715111 | 0.669034 | 0.412545 | 0.298697 |
| Occiglot-7b-de-en-instruct | 0.543173 | 0.530826 | 0.745778 | 0.67676 | 0.411326 | 0.351176 |
| Occiglot-7b-it-en | 0.513221 | 0.500564 | 0.694444 | 0.668099 | 0.413528 | 0.289469 |
| Occiglot-7b-it-en-instruct | 0.53721 | 0.523128 | 0.726667 | 0.683414 | 0.414918 | 0.337927 |
| Occiglot-7b-fr-en | 0.509209 | 0.496806 | 0.691333 | 0.667475 | 0.409129 | 0.281303 |
| Occiglot-7b-fr-en-instruct | 0.52884 | 0.515613 | 0.723333 | 0.67371 | 0.413024 | 0.318521 |
| Occiglot-7b-es-en | 0.483388 | 0.482949 | 0.606889 | 0.653902 | 0.398922 | 0.274277 |
| Occiglot-7b-es-en-instruct | 0.504023 | 0.494576 | 0.65 | 0.670847 | 0.406176 | 0.298513 |
| Leo-mistral-hessianai-7b | 0.484806 | 0.462103 | 0.653556 | 0.642242 | 0.379208 | 0.28692 |
| Claire-mistral-7b-0.1 | 0.514226 | 0.502773 | 0.705111 | 0.666871 | 0.412128 | 0.284245 |
| Lince-mistral-7b-it-es | 0.543427 | 0.540222 | 0.745111 | 0.692931 | 0.426241 | 0.312629 |
| Cerbero-7b | 0.532385 | 0.513714 | 0.743111 | 0.654061 | 0.427566 | 0.323475 |
| Mistral-7b-v0.1 | 0.547111 | 0.528937 | 0.768444 | 0.682516 | 0.448253 | 0.307403 |
| Mistral-7b-instruct-v0.2 | 0.56713 | 0.547228 | 0.741111 | 0.69455 | 0.422501 | 0.430262 |
### English
| | avg | arc_challenge | belebele | hellaswag | mmlu | truthfulqa |
|:---------------------------|---------:|----------------:|-----------:|------------:|---------:|-------------:|
| Occiglot-7b-eu5 | 0.59657 | 0.530717 | 0.726667 | 0.789882 | 0.531904 | 0.403678 |
| Occiglot-7b-eu5-instruct | 0.617905 | 0.558874 | 0.746667 | 0.799841 | 0.535109 | 0.449 |
| Leo-mistral-hessianai-7b | 0.600949 | 0.522184 | 0.736667 | 0.777833 | 0.538812 | 0.429248 |
| Mistral-7b-v0.1 | 0.668385 | 0.612628 | 0.844444 | 0.834097 | 0.624555 | 0.426201 |
| Mistral-7b-instruct-v0.2 | 0.713657 | 0.637372 | 0.824444 | 0.846345 | 0.59201 | 0.668116 |
### German
| | avg | arc_challenge_de | belebele_de | hellaswag_de | mmlu_de | truthfulqa_de |
|:---------------------------|---------:|-------------------:|--------------:|---------------:|----------:|----------------:|
| Occiglot-7b-eu5 | 0.508311 | 0.493584 | 0.646667 | 0.666631 | 0.483406 | 0.251269 |
| Occiglot-7b-eu5-instruct | 0.531506 | 0.529512 | 0.667778 | 0.685205 | 0.488234 | 0.286802 |
| Occiglot-7b-de-en | 0.540085 | 0.50556 | 0.743333 | 0.67421 | 0.514633 | 0.26269 |
| Occiglot-7b-de-en-instruct | 0.566474 | 0.54491 | 0.772222 | 0.688407 | 0.515915 | 0.310914 |
| Leo-mistral-hessianai-7b | 0.517766 | 0.474765 | 0.691111 | 0.682109 | 0.488309 | 0.252538 |
| Mistral-7b-v0.1 | 0.527957 | 0.476476 | 0.738889 | 0.610589 | 0.529567 | 0.284264 |
| Mistral-7b-instruct-v0.2 | 0.535215 | 0.485885 | 0.688889 | 0.622438 | 0.501961 | 0.376904 |
### Spanish
| | avg | arc_challenge_es | belebele_es | hellaswag_es | mmlu_es | truthfulqa_es |
|:---------------------------|---------:|-------------------:|--------------:|---------------:|----------:|----------------:|
| Occiglot-7b-eu5 | 0.533194 | 0.508547 | 0.676667 | 0.725411 | 0.499325 | 0.25602 |
| Occiglot-7b-eu5-instruct | 0.548155 | 0.535043 | 0.68 | 0.737039 | 0.503525 | 0.285171 |
| Occiglot-7b-es-en | 0.527264 | 0.529915 | 0.627778 | 0.72253 | 0.512749 | 0.243346 |
| Occiglot-7b-es-en-instruct | 0.5396 | 0.545299 | 0.636667 | 0.734372 | 0.524374 | 0.257288 |
| Lince-mistral-7b-it-es | 0.547212 | 0.52906 | 0.721111 | 0.687967 | 0.512749 | 0.285171 |
| Mistral-7b-v0.1 | 0.554817 | 0.528205 | 0.747778 | 0.672712 | 0.544023 | 0.281369 |
| Mistral-7b-instruct-v0.2 | 0.568575 | 0.54188 | 0.73 | 0.685406 | 0.511699 | 0.373891 |
### French
| | avg | arc_challenge_fr | belebele_fr | hellaswag_fr | mmlu_fr | truthfulqa_fr |
|:---------------------------|---------:|-------------------:|--------------:|---------------:|----------:|----------------:|
| Occiglot-7b-eu5 | 0.525017 | 0.506416 | 0.675556 | 0.712358 | 0.495684 | 0.23507 |
| Occiglot-7b-eu5-instruct | 0.554216 | 0.541488 | 0.7 | 0.724245 | 0.499122 | 0.306226 |
| Occiglot-7b-fr-en | 0.542903 | 0.532934 | 0.706667 | 0.718891 | 0.51333 | 0.242694 |
| Occiglot-7b-fr-en-instruct | 0.567079 | 0.542344 | 0.752222 | 0.72553 | 0.52051 | 0.29479 |
| Claire-mistral-7b-0.1 | 0.515127 | 0.486741 | 0.694444 | 0.642964 | 0.479566 | 0.271919 |
| Cerbero-7b | 0.526044 | 0.462789 | 0.735556 | 0.624438 | 0.516462 | 0.290978 |
| Mistral-7b-v0.1 | 0.558129 | 0.525235 | 0.776667 | 0.66481 | 0.543121 | 0.280813 |
| Mistral-7b-instruct-v0.2 | 0.575821 | 0.551754 | 0.758889 | 0.67916 | 0.506837 | 0.382465 |
### Italian
| | avg | arc_challenge_it | belebele_it | hellaswag_it | mmlu_it | truthfulqa_it |
|:---------------------------|---------:|-------------------:|--------------:|---------------:|----------:|----------------:|
| Occiglot-7b-eu5 | 0.421382 | 0.501283 | 0.652222 | 0.700533 | 0 | 0.252874 |
| Occiglot-7b-eu5-instruct | 0.437214 | 0.516681 | 0.661111 | 0.71326 | 0 | 0.295019 |
| Occiglot-7b-it-en | 0.432667 | 0.536356 | 0.684444 | 0.694768 | 0 | 0.247765 |
| Occiglot-7b-it-en-instruct | 0.456261 | 0.545766 | 0.717778 | 0.713804 | 0 | 0.303959 |
| Cerbero-7b | 0.434939 | 0.522669 | 0.717778 | 0.631567 | 0 | 0.302682 |
| Mistral-7b-v0.1 | 0.426264 | 0.502139 | 0.734444 | 0.630371 | 0 | 0.264368 |
| Mistral-7b-instruct-v0.2 | 0.442383 | 0.519247 | 0.703333 | 0.6394 | 0 | 0.349936 |
</details>
## Acknowledgements
The model training was supported by a compute grant at the [42 supercomputer](https://hessian.ai/) which is a central component in the development of [hessian AI](https://hessian.ai/), the [AI Innovation Lab](https://hessian.ai/infrastructure/ai-innovationlab/) (funded by the [Hessian Ministry of Higher Education, Research and the Art (HMWK)](https://wissenschaft.hessen.de) & the [Hessian Ministry of the Interior, for Security and Homeland Security (HMinD)](https://innen.hessen.de)) and the [AI Service Centers](https://hessian.ai/infrastructure/ai-service-centre/) (funded by the [German Federal Ministry for Economic Affairs and Climate Action (BMWK)](https://www.bmwk.de/Navigation/EN/Home/home.html)).
The curation of the training data is partially funded by the [German Federal Ministry for Economic Affairs and Climate Action (BMWK)](https://www.bmwk.de/Navigation/EN/Home/home.html)
through the project [OpenGPT-X](https://opengpt-x.de/en/) (project no. 68GX21007D).
## License
[Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.html)
## See also
- https://huggingface.co/NikolayKozloff/occiglot-7b-eu5-GGUF
- https://huggingface.co/collections/occiglot/occiglot-eu5-7b-v01-65dbed502a6348b052695e01

26
config.json Normal file
View File

@@ -0,0 +1,26 @@
{
"_name_or_path": "occiglot/occiglot-7b-eu5",
"architectures": [
"MistralForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"max_position_embeddings": 32768,
"model_type": "mistral",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"rms_norm_eps": 1e-05,
"rope_theta": 10000.0,
"sliding_window": 4096,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.35.2",
"use_cache": true,
"vocab_size": 32000
}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 1,
"eos_token_id": 2,
"transformers_version": "4.35.2"
}

398
lm-datasets-config.yml Normal file
View File

@@ -0,0 +1,398 @@
# Config file for https://github.com/malteos/lm-datasets
#
# EU top-5 (en,fr,es,de,it) + code
# target size: 300B tokens (train first for 200B tokens)
# a fixed random seed for shuffling etc.
seed: 0
# data split settings
validation_ratio: 0.005 # number of documents in the split: len(dataset) * ratio
validation_min_total_docs: 1_000 # to be used as validation set, the dataset must have at least n docs
validation_max_split_docs: 1_000 # number of documents in validation split are capped at this numbers
validation_min_split_docs: 10 # split must have at least this number of documents, otherwise it will be discarded
tokenizer_train_ratio: 0.1
selected_source_ids:
- starcoder
selected_dataset_ids:
# english
- pes2o
- math_amps
- eurlex_en
- wikipedia_20231101_en
- wikibooks_en
- wikiquote_en
- wikinews_en
- wikisource_en
- wikivoyage_en
- colossal_oscar_2015-14_en
- colossal_oscar_2016-40_en
- colossal_oscar_2017-43_en
- colossal_oscar_2018-47_en
- colossal_oscar_2019-22_en
- colossal_oscar_2020-24_en
- colossal_oscar_2020-45_en
- colossal_oscar_2021-49_en
- colossal_oscar_2022-27_en
- colossal_oscar_2022-49_en
- colossal_oscar_2023-14_en
- colossal_oscar_2023-23_en
- pile_of_law_r_legaladvice
- pile_of_law_atticus_contracts
- pile_of_law_un_debates
- proof_pile2_open_web_math
- parlamint_gb
- redpajama_stackexchange
# french
- cabernet
- eurlex_fr
- legal_mc4_fr
- wikipedia_20231101_fr
- wikibooks_fr
- wikiquote_fr
- wikinews_fr
- wikisource_fr
- wikivoyage_fr
- colossal_oscar_2015-14_fr
- colossal_oscar_2016-40_fr
- colossal_oscar_2017-43_fr
- colossal_oscar_2018-47_fr
- colossal_oscar_2019-22_fr
- colossal_oscar_2020-24_fr
- colossal_oscar_2020-45_fr
- colossal_oscar_2021-49_fr
- colossal_oscar_2022-27_fr
- colossal_oscar_2022-49_fr
- colossal_oscar_2023-14_fr
- colossal_oscar_2023-23_fr
- opensubtitles_fr
- parlamint_fr
# spanish
- spanish_legal
- eurlex_es
- legal_mc4_es
- wikipedia_20231101_es
- wikibooks_es
- wikiquote_es
- wikinews_es
- wikisource_es
- wikivoyage_es
- colossal_oscar_2015-14_es
- colossal_oscar_2016-40_es
- colossal_oscar_2017-43_es
- colossal_oscar_2018-47_es
- colossal_oscar_2019-22_es
- colossal_oscar_2020-24_es
- colossal_oscar_2020-45_es
- colossal_oscar_2021-49_es
- colossal_oscar_2022-27_es
- colossal_oscar_2022-49_es
- colossal_oscar_2023-14_es
- colossal_oscar_2023-23_es
- opensubtitles_es
- parlamint_es
# german
- openlegaldata
- dewac
- eurlex_de
- legal_mc4_de
- wikipedia_20231101_de
- wikibooks_de
- wikiquote_de
- wikinews_de
- wikisource_de
- wikivoyage_de
- colossal_oscar_2015-14_de
- colossal_oscar_2016-40_de
- colossal_oscar_2017-43_de
- colossal_oscar_2018-47_de
- colossal_oscar_2019-22_de
- colossal_oscar_2020-24_de
- colossal_oscar_2020-45_de
- colossal_oscar_2021-49_de
- colossal_oscar_2022-27_de
- colossal_oscar_2022-49_de
- colossal_oscar_2023-14_de
- colossal_oscar_2023-23_de
- open_discourse_bundestag
- tagesschau_2018_2023
- opensubtitles_de
- parlamint_at
# italian
- itwac
- eurlex_it
- legal_mc4_it
- wikipedia_20231101_it
- wikibooks_it
- wikiquote_it
- wikinews_it
- wikisource_it
- wikivoyage_it
- colossal_oscar_2015-14_it
- colossal_oscar_2016-40_it
- colossal_oscar_2017-43_it
- colossal_oscar_2018-47_it
- colossal_oscar_2019-22_it
- colossal_oscar_2020-24_it
- colossal_oscar_2020-45_it
- colossal_oscar_2021-49_it
- colossal_oscar_2022-27_it
- colossal_oscar_2022-49_it
- colossal_oscar_2023-14_it
- colossal_oscar_2023-23_it
- opensubtitles_it
- parlamint_it
- tatoeba_translation_en_fr
- tatoeba_translation_en_es
- tatoeba_translation_en_it
- tatoeba_translation_fr_it
- tatoeba_translation_es_fr
- tatoeba_translation_es_it
- tatoeba_translation_de_en
- tatoeba_translation_de_fr
- tatoeba_translation_de_es
- tatoeba_translation_de_it
- opus100_translation_de_en
- opus100_translation_en_es
- opus100_translation_en_fr
- opus100_translation_en_it
- wmt19_translation_de_en
- wmt19_translation_fr_de
sampling_factor_by_dataset_id:
redpajama_stackexchange: 0.1
pes2o: 0.1
math_amps: 0.1
openlegaldata: 0.75
dewac: 0.05
itwac: 1
cabernet: 1
spanish_legal: 0.1
eurlex_de: 0.5
eurlex_en: 0.5
eurlex_es: 1
eurlex_fr: 1
eurlex_it: 1
legal_mc4_de: 0.1
legal_mc4_es: 0.25
legal_mc4_fr: 0.25
legal_mc4_it: 1
wikipedia_20231101_de: 2
wikibooks_de: 1
wikiquote_de: 1
wikinews_de: 2
wikisource_de: 1
wikivoyage_de: 1
wikipedia_20231101_en: 1
wikibooks_en: 1
wikiquote_en: 0.25
wikinews_en: 1
wikisource_en: 1
wikivoyage_en: 1
wikipedia_20231101_es: 2
wikibooks_es: 1
wikiquote_es: 1
wikinews_es: 2
wikisource_es: 1
wikivoyage_es: 1
wikipedia_20231101_fr: 2
wikibooks_fr: 1
wikiquote_fr: 1
wikinews_fr: 2
wikisource_fr: 1
wikivoyage_fr: 1
wikipedia_20231101_it: 2
wikibooks_it: 1
wikiquote_it: 1
wikinews_it: 2
wikisource_it: 1
wikivoyage_it: 1
colossal_oscar_2015-14_de: 1
colossal_oscar_2016-40_de: 0.95
colossal_oscar_2017-43_de: 0.1
colossal_oscar_2018-47_de: 0.1
colossal_oscar_2019-22_de: 0.1
colossal_oscar_2020-24_de: 0.1
colossal_oscar_2020-45_de: 0.1
colossal_oscar_2021-49_de: 0.1
colossal_oscar_2022-27_de: 0.1
colossal_oscar_2022-49_de: 0.1
colossal_oscar_2023-14_de: 0.95
colossal_oscar_2023-23_de: 1
colossal_oscar_2015-14_en: 0.05
colossal_oscar_2016-40_en: 0.05
colossal_oscar_2017-43_en: 0.001
colossal_oscar_2018-47_en: 0.001
colossal_oscar_2019-22_en: 0.001
colossal_oscar_2020-24_en: 0.001
colossal_oscar_2020-45_en: 0.001
colossal_oscar_2021-49_en: 0.001
colossal_oscar_2022-27_en: 0.001
colossal_oscar_2022-49_en: 0.001
colossal_oscar_2023-14_en: 0.05
colossal_oscar_2023-23_en: 0.05
colossal_oscar_2015-14_es: 1
colossal_oscar_2016-40_es: 1
colossal_oscar_2017-43_es: 0.25
colossal_oscar_2018-47_es: 0.1
colossal_oscar_2019-22_es: 0.1
colossal_oscar_2020-24_es: 0.1
colossal_oscar_2020-45_es: 0.1
colossal_oscar_2021-49_es: 0.1
colossal_oscar_2022-27_es: 0.1
colossal_oscar_2022-49_es: 0.3
colossal_oscar_2023-14_es: 1
colossal_oscar_2023-23_es: 1
colossal_oscar_2015-14_fr: 1
colossal_oscar_2016-40_fr: 1
colossal_oscar_2017-43_fr: 0.25
colossal_oscar_2018-47_fr: 0.25
colossal_oscar_2019-22_fr: 0.1
colossal_oscar_2020-24_fr: 0.1
colossal_oscar_2020-45_fr: 0.1
colossal_oscar_2021-49_fr: 0.1
colossal_oscar_2022-27_fr: 0.1
colossal_oscar_2022-49_fr: 0.75
colossal_oscar_2023-14_fr: 1
colossal_oscar_2023-23_fr: 1
starcoder_emacs-lisp: 0.1
starcoder_literate-haskell: 0.1
starcoder_shell: 0.1
starcoder_ada: 0.1
starcoder_erlang: 0.1
starcoder_lua: 0.1
starcoder_smalltalk: 0.1
starcoder_agda: 0.1
starcoder_f-sharp: 0.1
starcoder_makefile: 0.1
starcoder_solidity: 0.1
starcoder_alloy: 0.1
starcoder_fortran: 0.1
starcoder_maple: 0.1
starcoder_sparql: 0.1
starcoder_antlr: 0.1
starcoder_git-commits-cleaned: 0.05
starcoder_markdown: 0.05
starcoder_sql: 0.1
starcoder_applescript: 0.1
starcoder_github-issues-filtered-structured: 0.075
starcoder_mathematica: 0.1
starcoder_stan: 0.1
starcoder_assembly: 0.1
starcoder_glsl: 0.1
starcoder_matlab: 0.1
starcoder_standard-ml: 0.1
starcoder_augeas: 0.1
starcoder_go: 0.05
starcoder_ocaml: 0.1
starcoder_stata: 0.1
starcoder_awk: 0.1
starcoder_groovy: 0.1
starcoder_pascal: 0.1
starcoder_systemverilog: 0.1
starcoder_batchfile: 0.1
starcoder_haskell: 0.1
starcoder_perl: 0.1
starcoder_tcl: 0.1
starcoder_bluespec: 0.1
starcoder_html: 0.05
starcoder_php: 0.05
starcoder_tcsh: 0.1
starcoder_c: 0.05
starcoder_idris: 0.1
starcoder_powershell: 0.1
starcoder_tex: 0.1
starcoder_c-sharp: 0.05
starcoder_isabelle: 0.1
starcoder_prolog: 0.1
starcoder_thrift: 0.1
starcoder_clojure: 0.1
starcoder_java: 0.05
starcoder_protocol-buffer: 0.1
starcoder_typescript: 0.05
starcoder_cmake: 0.1
starcoder_java-server-pages: 0.1
starcoder_python: 0.05
starcoder_verilog: 0.1
starcoder_coffeescript: 0.1
starcoder_javascript: 0.05
starcoder_r: 0.1
starcoder_vhdl: 0.1
starcoder_common-lisp: 0.1
starcoder_json: 0.1
starcoder_racket: 0.1
starcoder_visual-basic: 0.1
starcoder_cpp: 0.05
starcoder_julia: 0.1
starcoder_restructuredtext: 0.1
starcoder_xslt: 0.1
starcoder_css: 0.1
starcoder_jupyter-scripts-dedup-filtered: 0.1
starcoder_rmarkdown: 0.1
starcoder_yacc: 0.1
starcoder_cuda: 0.1
starcoder_jupyter-structured-clean-dedup: 0.1
starcoder_ruby: 0.1
starcoder_yaml: 0.1
starcoder_dart: 0.1
starcoder_kotlin: 0.1
starcoder_rust: 0.1
starcoder_zig: 0.1
starcoder_dockerfile: 0.1
starcoder_lean: 0.1
starcoder_sas: 0.1
starcoder_elixir: 0.1
starcoder_literate-agda: 0.1
starcoder_scala: 0.1
starcoder_elm: 0.1
starcoder_literate-coffeescript: 0.1
starcoder_scheme: 0.1
pile_of_law_r_legaladvice: 1
pile_of_law_atticus_contracts: 0.25
pile_of_law_un_debates: 1
open_discourse_bundestag: 0.5
tagesschau_2018_2023: 1
proof_pile2_open_web_math: 0.25
tatoeba_translation_en_fr: 1
tatoeba_translation_en_es: 1
tatoeba_translation_en_it: 1
tatoeba_translation_fr_it: 1
tatoeba_translation_es_fr: 1
tatoeba_translation_es_it: 1
tatoeba_translation_de_en: 1
tatoeba_translation_de_fr: 1
tatoeba_translation_de_es: 1
tatoeba_translation_de_it: 1
opus100_translation_de_en: 1
opus100_translation_en_es: 1
opus100_translation_en_fr: 1
opus100_translation_en_it: 1
wmt19_translation_de_en: 1
wmt19_translation_fr_de: 1
opensubtitles_es: 1
opensubtitles_fr: 1
opensubtitles_de: 1
opensubtitles_it: 1
parlamint_es: 1
parlamint_it: 1
parlamint_at: 1
parlamint_fr: 1
parlamint_gb: 1
colossal_oscar_2015-14_it: 1
colossal_oscar_2016-40_it: 1
colossal_oscar_2017-43_it: 0.75
colossal_oscar_2018-47_it: 0.75
colossal_oscar_2019-22_it: 0.75
colossal_oscar_2020-24_it: 0.75
colossal_oscar_2020-45_it: 0.75
colossal_oscar_2021-49_it: 0.75
colossal_oscar_2022-27_it: 0.75
colossal_oscar_2022-49_it: 0.75
colossal_oscar_2023-14_it: 0.9
colossal_oscar_2023-23_it: 1

3
model-00001-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b821be98562201ed577b896d8c85bb23ca3d07e24e61d12ce168f60986a09ff8
size 4943162336

3
model-00001-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3572c45423614e2b874ae7d421fd4f850e7cabc7e966980547b2cdce71e515f0
size 4987196936

3
model-00002-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2c8afdd179e5dd99634d7d72e5d10a7af2bd729d80b16bee9d0e73dfefbbb13c
size 4999819336

3
model-00002-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fdb2cd75b9f7dbb8ba0256ba6771bc7fbc54310ea9bb3f57d57eed297ea2d3f4
size 4899116440

3
model-00003-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1cf8944af4feb1069e779072fbebd8d90a8c1e0ea9f2fa72d60c8ae00ccfafff
size 4540516344

3
model-00003-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1c560c912b037e5af4c78bab14eacc2c3fc5f7724371fe68e6f0b0fe4ffafc19
size 4999813120

3
model-00004-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:729463205aaed76447ca86177110de7a825fe8e6d8b7db5dfd35caef42cd2531
size 4999813128

3
model-00005-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:897a386e7f7a04a78c0c130538f715c7d4c699ba471919b2dc3d381a67f358bb
size 4832007496

3
model-00006-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e8f37c13b68f2eda900c982f0dd6cae586d3df65dfa7bcf87dbb6717a8e434f6
size 4249014896

298
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,298 @@
{
"metadata": {
"total_size": 14483464192
},
"weight_map": {
"lm_head.weight": "model-00003-of-00003.safetensors",
"model.embed_tokens.weight": "model-00001-of-00003.safetensors",
"model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.10.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.10.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.11.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.17.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.18.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.19.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.20.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.21.input_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.22.input_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"model.layers.23.input_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.24.input_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.25.input_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.26.input_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.27.input_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.28.input_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.28.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.28.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.28.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.28.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.29.input_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.29.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.29.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.29.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.29.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.29.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.29.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.30.input_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.30.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.30.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.30.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.30.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.30.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.30.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.31.input_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.31.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.31.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.31.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.31.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"model.layers.31.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.31.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.31.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"model.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.8.input_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.9.input_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.9.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.9.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"model.norm.weight": "model-00003-of-00003.safetensors"
}
}

5
special_tokens_map.json Normal file
View File

@@ -0,0 +1,5 @@
{
"bos_token": "<s>",
"eos_token": "</s>",
"unk_token": "<unk>"
}

91122
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
tokenizer.model (Stored with Git LFS) Normal file
View File

Binary file not shown.

42
tokenizer_config.json Normal file
View File

@@ -0,0 +1,42 @@
{
"add_bos_token": true,
"add_eos_token": false,
"added_tokens_decoder": {
"0": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [],
"bos_token": "<s>",
"clean_up_tokenization_spaces": false,
"eos_token": "</s>",
"legacy": true,
"model_max_length": 1000000000000000019884624838656,
"pad_token": null,
"sp_model_kwargs": {},
"spaces_between_special_tokens": false,
"tokenizer_class": "LlamaTokenizer",
"unk_token": "<unk>",
"use_default_system_prompt": false
}