初始化项目，由ModelHub XC社区提供模型
Model: projecte-aina/aguila-7b Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
251
README.md
Normal file
251
README.md
Normal file
@@ -0,0 +1,251 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
- es
|
||||
- ca
|
||||
licence:
|
||||
- apache-2.0
|
||||
tags:
|
||||
- aguila
|
||||
- falcon
|
||||
- spanish
|
||||
- catalan
|
||||
metrics:
|
||||
- ppl
|
||||
model-index:
|
||||
- name: aguila_7b
|
||||
results:
|
||||
- task:
|
||||
name: Causal Language Modeling
|
||||
type: text-generation
|
||||
metrics:
|
||||
- name: Perplexity
|
||||
type: ppl
|
||||
value: 8.59
|
||||
pipeline_tag: text-generation
|
||||
widget:
|
||||
- text: |-
|
||||
Respon a la pregunta següent.
|
||||
Pregunta: "Quina és la capital de Suècia?"
|
||||
Resposta: "La capital de Suècia és Estocolm."
|
||||
----
|
||||
Respon a la pregunta següent.
|
||||
Pregunta: "Quina beguda es consumeix als matins per despertar-se?"
|
||||
Resposta: "La majoria de gent consumeix cafè per despertar-se."
|
||||
----
|
||||
Respon a la pregunta següent.
|
||||
Pregunta: "Explica com funciona un motor de combustió"
|
||||
Resposta:
|
||||
example_title: Pregunta-Resposta
|
||||
- text: |-
|
||||
Extrae las entidades nombradas del siguiente texto:
|
||||
Texto: "Me llamo Wolfgang y vivo en Berlin"
|
||||
Entidades: Wolfgang:PER, Berlin:LOC
|
||||
----
|
||||
Extrae las entidades nombradas del siguiente texto:
|
||||
Texto: "Hoy voy a visitar el parc güell tras salir del barcelona supercomputing center"
|
||||
Entidades: parc güell:LOC, barcelona supercomputing center:LOC
|
||||
----
|
||||
Extrae las entidades nombradas del siguiente texto:
|
||||
Texto: "Maria y Miguel no tienen ningún problema contigo"
|
||||
Entidades: Maria:PER, Miguel:PER
|
||||
----
|
||||
Extrae las entidades nombradas del siguiente texto:
|
||||
Texto: "Damián se cortó el pelo"
|
||||
Entidades: Damián:PER
|
||||
----
|
||||
Extrae las entidades nombradas del siguiente texto:
|
||||
Texto: "Lo mejor de Barcelona és el bar de mi amigo Pablo"
|
||||
Entidades: Pablo:PER, Barcelona:LOC
|
||||
----
|
||||
Extrae las entidades nombradas del siguiente texto:
|
||||
Texto: "Carlos comparte piso con Marc"
|
||||
Entidades:
|
||||
example_title: Entidades-Nombradas
|
||||
---
|
||||
|
||||
# Ǎguila-7B
|
||||
|
||||
<strong><span style="color:darkred">⚠️NOTICE⚠️: This model has been deprecated and is no longer actively maintained or supported. To access the latest models with enhanced features, better performance,
|
||||
and ongoing support, please visit <a style="color:darkred" href="https://huggingface.co/BSC-LT">https://huggingface.co/BSC-LT</a></span></strong>
|
||||
|
||||
|
||||
## Table of Contents
|
||||
<details>
|
||||
<summary>Click to expand</summary>
|
||||
|
||||
- [Model description](#model-description)
|
||||
- [Intended uses and limitations](#intended-uses-and-limitations)
|
||||
- [How to use](#how-to-use)
|
||||
- [Limitations and bias](#limitations-and-bias)
|
||||
- [Language adaptation](#language-adaptation)
|
||||
- [Training](#training)
|
||||
- [Training data](#training-data)
|
||||
- [Training procedure](#training-procedure)
|
||||
- [Additional information](#additional-information)
|
||||
- [Author](#author)
|
||||
- [Contact](#contact)
|
||||
- [Copyright](#copyright)
|
||||
- [License](#license)
|
||||
- [Funding](#funding)
|
||||
- [Disclaimer](#disclaimer)
|
||||
|
||||
</details>
|
||||
|
||||
## Model description
|
||||
|
||||
**Ǎguila-7B** is a transformer-based causal language model for Catalan, Spanish, and English.
|
||||
It is based on the [Falcon-7B](https://huggingface.co/tiiuae/falcon-7b) model and has been trained on a 26B token
|
||||
trilingual corpus collected from publicly available corpora and crawlers.
|
||||
|
||||
More information available in the following post from Medium.com: Introducing Ǎguila, a new open-source LLM for Spanish and Catalan (https://medium.com/@mpamies247/introducing-a%CC%8Cguila-a-new-open-source-llm-for-spanish-and-catalan-ee1ebc70bc79)
|
||||
|
||||
|
||||
## Intended uses and limitations
|
||||
|
||||
The **Ǎguila-7B** model is ready-to-use only for causal language modeling to perform text-generation tasks.
|
||||
However, it is intended to be fine-tuned for downstream tasks.
|
||||
|
||||
## How to use
|
||||
|
||||
Here is how to use this model:
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
input_text = "El mercat del barri és fantàstic, hi pots trobar"
|
||||
|
||||
model_id = "projecte-aina/aguila-7b"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
generator = pipeline(
|
||||
"text-generation",
|
||||
model=model_id,
|
||||
tokenizer=tokenizer,
|
||||
torch_dtype=torch.bfloat16,
|
||||
trust_remote_code=True,
|
||||
device_map="auto",
|
||||
)
|
||||
generation = generator(
|
||||
input_text,
|
||||
do_sample=True,
|
||||
top_k=10,
|
||||
eos_token_id=tokenizer.eos_token_id,
|
||||
)
|
||||
|
||||
print(f"Result: {generation[0]['generated_text']}")
|
||||
```
|
||||
|
||||
## Limitations and bias
|
||||
At the time of submission, no measures have been taken to estimate the bias and toxicity embedded in the model.
|
||||
However, we are well aware that our models may be biased since the corpora have been collected using crawling techniques
|
||||
on multiple web sources. We intend to conduct research in these areas in the future, and if completed, this model card will be updated.
|
||||
|
||||
|
||||
## Language adaptation
|
||||
|
||||
We adapted the original [Falcon-7B](https://huggingface.co/tiiuae/falcon-7b) model to Spanish and Catalan by swapping the tokenizer and adjusting the embedding layer.
|
||||
|
||||
The adaptation procedure is explained in [this blog post](https://medium.com/@mpamies247/ee1ebc70bc79).
|
||||
|
||||
## Training
|
||||
|
||||
### Training data
|
||||
|
||||
The training corpus consists of 26B tokens of several corpora gathered from web crawlings and public domain data.
|
||||
|
||||
| Dataset | Language | Words (per-epoch) | Epochs |
|
||||
|---------------------|----------|--------------------|--------------|
|
||||
| Wikipedia | en | 2169.97M | 1.428144485 |
|
||||
| C4_es | es | 53709.80M | 0.1049686196 |
|
||||
| Biomedical | es | 455.03M | 0.7140722425 |
|
||||
| Legal | es | 995.70M | 0.7140722425 |
|
||||
| Wikipedia | es | 693.60M | 1.428144485 |
|
||||
| Gutenberg | es | 53.18M | 0.7140722425 |
|
||||
| C4_ca | ca | 2826.00M | 2.142216727 |
|
||||
| Biomedical | ca | 11.80M | 1.428144485 |
|
||||
| RacoCatalà Noticias | ca | 17.16M | 2.142216727 |
|
||||
| RacoCatalà Forums | ca | 333.73M | 2.142216727 |
|
||||
| CaWaC | ca | 57.79M | 2.142216727 |
|
||||
| Wikipedia | ca | 228.01M | 3.570361212 |
|
||||
| Vilaweb | ca | 50.34M | 2.142216727 |
|
||||
|
||||
The dataset has the following language distribution:
|
||||
|
||||
|Language|Percentage|
|
||||
|--------|----------|
|
||||
| En | 16.84% |
|
||||
| Es | 41.38% |
|
||||
| Ca | 41.79% |
|
||||
|
||||
Note: A small amount of English data was kept to avoid catastrophic forgetting.
|
||||
|
||||
## Training procedure
|
||||
|
||||
The training corpus has been tokenized using a byte version of [Byte-Pair Encoding (BPE)](https://github.com/openai/gpt-2) with a vocabulary size of 50,257 tokens.
|
||||
After training a new tokenizer and adapting [falcon-7b](https://huggingface.co/tiiuae/falcon-7b)'s embedding layer, the model was
|
||||
further pre-trained in three target languages: Catalan, Spanish and English.
|
||||
|
||||
The training lasted a total of 320 hours on 8 NVIDIA H100 GPUs with 80GB RAM.
|
||||
|
||||
|
||||
### Training hyperparameters
|
||||
|
||||
- seed: 42
|
||||
- distributed_type: multi-GPU
|
||||
- num_devices: 8
|
||||
- train_batch_size: 1
|
||||
- eval_batch_size: 1
|
||||
- total_train_batch_size: 8
|
||||
- total_eval_batch_size: 8
|
||||
- optimizer: Adam
|
||||
- betas: (0.9,0.999)
|
||||
- epsilon: 1e-08
|
||||
- learning_rate: 5e-05
|
||||
- lr_scheduler_type: linear
|
||||
- num_epochs: 1.0
|
||||
|
||||
|
||||
### Framework versions
|
||||
|
||||
- Pytorch 2.0.0
|
||||
- Transformers 4.30.2
|
||||
- Datasets 2.13.1
|
||||
- Tokenizers 0.13.3
|
||||
|
||||
## Additional information
|
||||
|
||||
### Author
|
||||
The Language Technologies Unit from Barcelona Supercomputing Center.
|
||||
|
||||
### Contact
|
||||
For further information, please send an email to <langtech@bsc.es>.
|
||||
|
||||
### Copyright
|
||||
Copyright(c) 2023 by Language Technologies Unit, Barcelona Supercomputing Center.
|
||||
|
||||
### License
|
||||
[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
|
||||
|
||||
### Funding
|
||||
This work was funded by:
|
||||
- The [Departament de la Vicepresidència i de Polítiques Digitals i Territori de la Generalitat de Catalunya](https://politiquesdigitals.gencat.cat/ca/inici/index.html#googtrans(ca|en) within the framework of [Projecte AINA](https://politiquesdigitals.gencat.cat/ca/economia/catalonia-ai/aina).
|
||||
- The [Spanish State Secretariat for Digitalization and Artificial Intelligence](https://portal.mineco.gob.es/en-us/digitalizacionIA/Pages/sedia.aspx) within the framework of the [Plan de Impulso de las Tecnologías del Lenguaje](https://plantl.mineco.gob.es/Paginas/index.aspx).
|
||||
|
||||
### Disclaimer
|
||||
|
||||
<details>
|
||||
<summary>Click to expand</summary>
|
||||
|
||||
The model published in this repository is intended for a generalist purpose and is available to third parties under a permissive Apache License, Version 2.0.
|
||||
|
||||
Be aware that the model may have biases and/or any other undesirable distortions.
|
||||
|
||||
When third parties deploy or provide systems and/or services to other parties using this model (or any system based on it)
|
||||
or become users of the model, they should note that it is their responsibility to mitigate the risks arising from its use and,
|
||||
in any event, to comply with applicable regulations, including regulations regarding the use of Artificial Intelligence.
|
||||
|
||||
In no event shall the owner and creator of the model (Barcelona Supercomputing Center)
|
||||
be liable for any results arising from the use made by third parties.
|
||||
|
||||
</details>
|
||||
15
all_results.json
Normal file
15
all_results.json
Normal file
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"epoch": 1.0,
|
||||
"eval_accuracy": 0.5258444783433934,
|
||||
"eval_loss": 2.150390625,
|
||||
"eval_runtime": 41.4599,
|
||||
"eval_samples": 3568,
|
||||
"eval_samples_per_second": 86.059,
|
||||
"eval_steps_per_second": 10.757,
|
||||
"perplexity": 8.5882125125473,
|
||||
"train_loss": 0.2576859601399347,
|
||||
"train_runtime": 150388.7494,
|
||||
"train_samples": 26705019,
|
||||
"train_samples_per_second": 177.573,
|
||||
"train_steps_per_second": 22.197
|
||||
}
|
||||
34
config.json
Executable file
34
config.json
Executable file
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"alibi": false,
|
||||
"apply_residual_connection_post_layernorm": false,
|
||||
"architectures": [
|
||||
"RWForCausalLM"
|
||||
],
|
||||
"attention_dropout": 0.0,
|
||||
"auto_map": {
|
||||
"AutoConfig": "configuration_RW.RWConfig",
|
||||
"AutoModel": "modelling_RW.RWModel",
|
||||
"AutoModelForCausalLM": "modelling_RW.RWForCausalLM",
|
||||
"AutoModelForQuestionAnswering": "modelling_RW.RWForQuestionAnswering",
|
||||
"AutoModelForSequenceClassification": "modelling_RW.RWForSequenceClassification",
|
||||
"AutoModelForTokenClassification": "modelling_RW.RWForTokenClassification"
|
||||
},
|
||||
"bias": false,
|
||||
"bos_token_id": 50256,
|
||||
"eos_token_id": 50256,
|
||||
"hidden_dropout": 0.0,
|
||||
"hidden_size": 4544,
|
||||
"initializer_range": 0.02,
|
||||
"layer_norm_epsilon": 1e-05,
|
||||
"max_position_embeddings": 2048,
|
||||
"model_type": "RefinedWebModel",
|
||||
"multi_query": true,
|
||||
"n_head": 71,
|
||||
"n_layer": 32,
|
||||
"pad_token_id": 50256,
|
||||
"parallel_attn": true,
|
||||
"torch_dtype": "float16",
|
||||
"transformers_version": "4.35.2",
|
||||
"use_cache": true,
|
||||
"vocab_size": 50257
|
||||
}
|
||||
79
configuration_RW.py
Executable file
79
configuration_RW.py
Executable file
@@ -0,0 +1,79 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2022 the Big Science Workshop and HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
""" Bloom configuration"""
|
||||
from transformers.configuration_utils import PretrainedConfig
|
||||
from transformers.utils import logging
|
||||
|
||||
|
||||
logger = logging.get_logger(__name__)
|
||||
|
||||
|
||||
class RWConfig(PretrainedConfig):
|
||||
model_type = "RefinedWebModel"
|
||||
keys_to_ignore_at_inference = ["past_key_values"]
|
||||
attribute_map = {
|
||||
"num_hidden_layers": "n_layer",
|
||||
"num_attention_heads": "n_head",
|
||||
}
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vocab_size=250880,
|
||||
hidden_size=64,
|
||||
n_layer=2,
|
||||
n_head=8,
|
||||
layer_norm_epsilon=1e-5,
|
||||
initializer_range=0.02,
|
||||
use_cache=True,
|
||||
bos_token_id=1,
|
||||
eos_token_id=2,
|
||||
apply_residual_connection_post_layernorm=False,
|
||||
hidden_dropout=0.0,
|
||||
attention_dropout=0.0,
|
||||
multi_query=False,
|
||||
alibi=False,
|
||||
bias=False,
|
||||
parallel_attn=False,
|
||||
**kwargs,
|
||||
):
|
||||
self.vocab_size = vocab_size
|
||||
# Backward compatibility with n_embed kwarg
|
||||
n_embed = kwargs.pop("n_embed", None)
|
||||
self.hidden_size = hidden_size if n_embed is None else n_embed
|
||||
self.n_layer = n_layer
|
||||
self.n_head = n_head
|
||||
self.layer_norm_epsilon = layer_norm_epsilon
|
||||
self.initializer_range = initializer_range
|
||||
self.use_cache = use_cache
|
||||
self.apply_residual_connection_post_layernorm = apply_residual_connection_post_layernorm
|
||||
self.hidden_dropout = hidden_dropout
|
||||
self.attention_dropout = attention_dropout
|
||||
|
||||
self.bos_token_id = bos_token_id
|
||||
self.eos_token_id = eos_token_id
|
||||
self.multi_query = multi_query
|
||||
self.alibi = alibi
|
||||
self.bias = bias
|
||||
self.parallel_attn = parallel_attn
|
||||
|
||||
super().__init__(bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs)
|
||||
|
||||
@property
|
||||
def head_dim(self):
|
||||
return self.hidden_size // self.n_head
|
||||
|
||||
@property
|
||||
def rotary(self):
|
||||
return not self.alibi
|
||||
10
eval_results.json
Normal file
10
eval_results.json
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"epoch": 1.0,
|
||||
"eval_accuracy": 0.5258444783433934,
|
||||
"eval_loss": 2.150390625,
|
||||
"eval_runtime": 41.4599,
|
||||
"eval_samples": 3568,
|
||||
"eval_samples_per_second": 86.059,
|
||||
"eval_steps_per_second": 10.757,
|
||||
"perplexity": 8.5882125125473
|
||||
}
|
||||
6
generation_config.json
Normal file
6
generation_config.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"transformers_version": "4.30.2"
|
||||
}
|
||||
BIN
images/accuracy_aguila_7b.png
Normal file
BIN
images/accuracy_aguila_7b.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 21 KiB |
BIN
images/training_loss_aguila_7b.png
Normal file
BIN
images/training_loss_aguila_7b.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
BIN
images/validation_loss_aguila_7b.png
Normal file
BIN
images/validation_loss_aguila_7b.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 22 KiB |
50001
merges.txt
Normal file
50001
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model-00001-of-00008.safetensors
Executable file
3
model-00001-of-00008.safetensors
Executable file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:aada0644189869b0d1c8833e83a755e5af0a0d442b765c0fe351b15f0e31d3c7
|
||||
size 1948115592
|
||||
3
model-00002-of-00008.safetensors
Executable file
3
model-00002-of-00008.safetensors
Executable file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:fa84f6e2b1648310efefe55ca17647cbc4a978ccf50e37817ff897b6aad95233
|
||||
size 1905520744
|
||||
3
model-00003-of-00008.safetensors
Executable file
3
model-00003-of-00008.safetensors
Executable file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:85c0ce9c7b06b355c40779e7254d41915d3f4de62e487093470f81d3f5d80f0d
|
||||
size 1986949096
|
||||
3
model-00004-of-00008.safetensors
Executable file
3
model-00004-of-00008.safetensors
Executable file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:f08675c673027f2071de8b79c4ba914de726bd4d530c809d66bb67066af6d966
|
||||
size 1905502384
|
||||
3
model-00005-of-00008.safetensors
Executable file
3
model-00005-of-00008.safetensors
Executable file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2c0ada5ca5ed5017178812d339e18767d916ff064a3975384dff82cb02ec7ba3
|
||||
size 1905520776
|
||||
3
model-00006-of-00008.safetensors
Executable file
3
model-00006-of-00008.safetensors
Executable file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:4761b9a726c55687205294bbb94c0cc8aa0e53d7ff2047c8331f0ff006a5e243
|
||||
size 1986949128
|
||||
3
model-00007-of-00008.safetensors
Executable file
3
model-00007-of-00008.safetensors
Executable file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:c538621d0cd8f8624414580551d337f980714ceb427f39b84ca3fcdd74cb7a2f
|
||||
size 1905502384
|
||||
3
model-00008-of-00008.safetensors
Executable file
3
model-00008-of-00008.safetensors
Executable file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:60b794f0ba8146ca3b915a2f39da15b566b84aac04c528a4e25161f3df5d0dd8
|
||||
size 165202000
|
||||
202
model.safetensors.index.json
Executable file
202
model.safetensors.index.json
Executable file
@@ -0,0 +1,202 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_size": 13709238912
|
||||
},
|
||||
"weight_map": {
|
||||
"transformer.h.0.input_layernorm.bias": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.0.input_layernorm.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.0.mlp.dense_4h_to_h.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.0.mlp.dense_h_to_4h.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.0.self_attention.dense.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.0.self_attention.query_key_value.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.1.input_layernorm.bias": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.1.input_layernorm.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.1.mlp.dense_4h_to_h.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.1.mlp.dense_h_to_4h.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.1.self_attention.dense.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.1.self_attention.query_key_value.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.10.input_layernorm.bias": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.10.input_layernorm.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.10.mlp.dense_4h_to_h.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.10.mlp.dense_h_to_4h.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.10.self_attention.dense.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.10.self_attention.query_key_value.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.11.input_layernorm.bias": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.11.input_layernorm.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.11.mlp.dense_4h_to_h.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.11.mlp.dense_h_to_4h.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.11.self_attention.dense.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.11.self_attention.query_key_value.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.12.input_layernorm.bias": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.12.input_layernorm.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.12.mlp.dense_4h_to_h.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.12.mlp.dense_h_to_4h.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.12.self_attention.dense.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.12.self_attention.query_key_value.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.13.input_layernorm.bias": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.13.input_layernorm.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.13.mlp.dense_4h_to_h.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.13.mlp.dense_h_to_4h.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.13.self_attention.dense.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.13.self_attention.query_key_value.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.14.input_layernorm.bias": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.14.input_layernorm.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.14.mlp.dense_4h_to_h.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.14.mlp.dense_h_to_4h.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.14.self_attention.dense.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.14.self_attention.query_key_value.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.15.input_layernorm.bias": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.15.input_layernorm.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.15.mlp.dense_4h_to_h.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.15.mlp.dense_h_to_4h.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.15.self_attention.dense.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.15.self_attention.query_key_value.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.16.input_layernorm.bias": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.16.input_layernorm.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.16.mlp.dense_4h_to_h.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.16.mlp.dense_h_to_4h.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.16.self_attention.dense.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.16.self_attention.query_key_value.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.17.input_layernorm.bias": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.17.input_layernorm.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.17.mlp.dense_4h_to_h.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.17.mlp.dense_h_to_4h.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.17.self_attention.dense.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.17.self_attention.query_key_value.weight": "model-00004-of-00008.safetensors",
|
||||
"transformer.h.18.input_layernorm.bias": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.18.input_layernorm.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.18.mlp.dense_4h_to_h.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.18.mlp.dense_h_to_4h.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.18.self_attention.dense.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.18.self_attention.query_key_value.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.19.input_layernorm.bias": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.19.input_layernorm.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.19.mlp.dense_4h_to_h.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.19.mlp.dense_h_to_4h.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.19.self_attention.dense.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.19.self_attention.query_key_value.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.2.input_layernorm.bias": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.2.input_layernorm.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.2.mlp.dense_4h_to_h.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.2.mlp.dense_h_to_4h.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.2.self_attention.dense.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.2.self_attention.query_key_value.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.20.input_layernorm.bias": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.20.input_layernorm.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.20.mlp.dense_4h_to_h.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.20.mlp.dense_h_to_4h.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.20.self_attention.dense.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.20.self_attention.query_key_value.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.21.input_layernorm.bias": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.21.input_layernorm.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.21.mlp.dense_4h_to_h.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.21.mlp.dense_h_to_4h.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.21.self_attention.dense.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.21.self_attention.query_key_value.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.22.input_layernorm.bias": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.22.input_layernorm.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.22.mlp.dense_4h_to_h.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.22.mlp.dense_h_to_4h.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.22.self_attention.dense.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.22.self_attention.query_key_value.weight": "model-00005-of-00008.safetensors",
|
||||
"transformer.h.23.input_layernorm.bias": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.23.input_layernorm.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.23.mlp.dense_4h_to_h.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.23.mlp.dense_h_to_4h.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.23.self_attention.dense.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.23.self_attention.query_key_value.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.24.input_layernorm.bias": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.24.input_layernorm.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.24.mlp.dense_4h_to_h.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.24.mlp.dense_h_to_4h.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.24.self_attention.dense.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.24.self_attention.query_key_value.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.25.input_layernorm.bias": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.25.input_layernorm.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.25.mlp.dense_4h_to_h.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.25.mlp.dense_h_to_4h.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.25.self_attention.dense.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.25.self_attention.query_key_value.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.26.input_layernorm.bias": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.26.input_layernorm.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.26.mlp.dense_4h_to_h.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.26.mlp.dense_h_to_4h.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.26.self_attention.dense.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.26.self_attention.query_key_value.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.27.input_layernorm.bias": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.27.input_layernorm.weight": "model-00006-of-00008.safetensors",
|
||||
"transformer.h.27.mlp.dense_4h_to_h.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.27.mlp.dense_h_to_4h.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.27.self_attention.dense.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.27.self_attention.query_key_value.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.28.input_layernorm.bias": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.28.input_layernorm.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.28.mlp.dense_4h_to_h.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.28.mlp.dense_h_to_4h.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.28.self_attention.dense.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.28.self_attention.query_key_value.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.29.input_layernorm.bias": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.29.input_layernorm.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.29.mlp.dense_4h_to_h.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.29.mlp.dense_h_to_4h.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.29.self_attention.dense.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.29.self_attention.query_key_value.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.3.input_layernorm.bias": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.3.input_layernorm.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.3.mlp.dense_4h_to_h.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.3.mlp.dense_h_to_4h.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.3.self_attention.dense.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.3.self_attention.query_key_value.weight": "model-00001-of-00008.safetensors",
|
||||
"transformer.h.30.input_layernorm.bias": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.30.input_layernorm.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.30.mlp.dense_4h_to_h.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.30.mlp.dense_h_to_4h.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.30.self_attention.dense.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.30.self_attention.query_key_value.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.31.input_layernorm.bias": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.31.input_layernorm.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.31.mlp.dense_4h_to_h.weight": "model-00008-of-00008.safetensors",
|
||||
"transformer.h.31.mlp.dense_h_to_4h.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.31.self_attention.dense.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.31.self_attention.query_key_value.weight": "model-00007-of-00008.safetensors",
|
||||
"transformer.h.4.input_layernorm.bias": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.4.input_layernorm.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.4.mlp.dense_4h_to_h.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.4.mlp.dense_h_to_4h.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.4.self_attention.dense.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.4.self_attention.query_key_value.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.5.input_layernorm.bias": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.5.input_layernorm.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.5.mlp.dense_4h_to_h.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.5.mlp.dense_h_to_4h.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.5.self_attention.dense.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.5.self_attention.query_key_value.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.6.input_layernorm.bias": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.6.input_layernorm.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.6.mlp.dense_4h_to_h.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.6.mlp.dense_h_to_4h.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.6.self_attention.dense.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.6.self_attention.query_key_value.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.7.input_layernorm.bias": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.7.input_layernorm.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.7.mlp.dense_4h_to_h.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.7.mlp.dense_h_to_4h.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.7.self_attention.dense.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.7.self_attention.query_key_value.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.8.input_layernorm.bias": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.8.input_layernorm.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.8.mlp.dense_4h_to_h.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.8.mlp.dense_h_to_4h.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.8.self_attention.dense.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.8.self_attention.query_key_value.weight": "model-00002-of-00008.safetensors",
|
||||
"transformer.h.9.input_layernorm.bias": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.9.input_layernorm.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.9.mlp.dense_4h_to_h.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.9.mlp.dense_h_to_4h.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.9.self_attention.dense.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.h.9.self_attention.query_key_value.weight": "model-00003-of-00008.safetensors",
|
||||
"transformer.ln_f.bias": "model-00008-of-00008.safetensors",
|
||||
"transformer.ln_f.weight": "model-00008-of-00008.safetensors",
|
||||
"transformer.word_embeddings.weight": "model-00001-of-00008.safetensors"
|
||||
}
|
||||
}
|
||||
1100
modelling_RW.py
Executable file
1100
modelling_RW.py
Executable file
File diff suppressed because it is too large
Load Diff
203
pytorch_model.bin.index.json
Normal file
203
pytorch_model.bin.index.json
Normal file
@@ -0,0 +1,203 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_size": 13709238912
|
||||
},
|
||||
"weight_map": {
|
||||
"lm_head.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.0.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.0.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.0.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.0.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.0.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.0.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.1.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.1.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.1.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.1.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.1.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.1.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.10.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.10.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.10.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.10.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.10.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.10.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.11.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.11.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.11.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.11.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.11.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.11.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.12.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.12.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.12.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.12.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.12.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.12.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.13.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.13.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.13.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.13.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.13.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.13.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.14.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.14.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.14.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.14.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.14.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.14.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.15.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.15.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.15.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.15.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.15.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.15.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.16.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.16.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.16.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.16.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.16.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.16.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.17.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.17.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.17.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.17.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.17.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.17.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.18.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.18.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.18.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.18.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.18.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.18.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.19.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.19.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.19.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.19.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.19.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.19.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.2.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.2.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.2.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.2.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.2.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.2.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.20.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.20.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.20.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.20.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.20.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.20.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.21.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.21.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.21.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.21.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.21.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.21.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.22.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.22.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.22.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.22.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.22.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.22.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.23.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.23.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.23.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.23.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.23.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.23.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.24.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.24.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.24.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.24.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.24.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.24.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.25.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.25.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.25.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.25.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.25.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.25.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.26.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.26.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.26.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.26.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.26.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.26.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.27.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.27.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.27.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.27.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.27.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.27.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.28.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.28.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.28.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.28.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.28.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.28.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.29.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.29.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.29.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.29.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.29.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.29.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.3.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.3.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.3.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.3.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.3.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.3.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.30.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.30.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.30.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.30.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.30.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.30.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.31.input_layernorm.bias": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.31.input_layernorm.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.31.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.31.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.31.self_attention.dense.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.31.self_attention.query_key_value.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.h.4.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.4.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.4.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.4.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.4.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.4.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.5.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.5.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.5.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.5.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.5.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.5.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.6.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.6.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.6.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.6.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.6.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.6.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.7.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.7.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.7.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.7.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.7.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.7.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.8.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.8.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.8.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.8.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.8.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.8.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.9.input_layernorm.bias": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.9.input_layernorm.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.9.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.9.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.9.self_attention.dense.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.h.9.self_attention.query_key_value.weight": "pytorch_model-00001-of-00002.bin",
|
||||
"transformer.ln_f.bias": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.ln_f.weight": "pytorch_model-00002-of-00002.bin",
|
||||
"transformer.word_embeddings.weight": "pytorch_model-00001-of-00002.bin"
|
||||
}
|
||||
}
|
||||
24
special_tokens_map.json
Normal file
24
special_tokens_map.json
Normal file
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": "<|endoftext|>",
|
||||
"unk_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
100305
tokenizer.json
Normal file
100305
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
33
tokenizer_config.json
Normal file
33
tokenizer_config.json
Normal file
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_prefix_space": false,
|
||||
"bos_token": {
|
||||
"__type": "AddedToken",
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"clean_up_tokenization_spaces": true,
|
||||
"eos_token": {
|
||||
"__type": "AddedToken",
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"errors": "replace",
|
||||
"model_max_length": 2048,
|
||||
"pad_token": null,
|
||||
"tokenizer_class": "GPT2Tokenizer",
|
||||
"unk_token": {
|
||||
"__type": "AddedToken",
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
8
train_results.json
Normal file
8
train_results.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"epoch": 1.0,
|
||||
"train_loss": 0.2576859601399347,
|
||||
"train_runtime": 150388.7494,
|
||||
"train_samples": 26705019,
|
||||
"train_samples_per_second": 177.573,
|
||||
"train_steps_per_second": 22.197
|
||||
}
|
||||
10030
trainer_state.json
Normal file
10030
trainer_state.json
Normal file
File diff suppressed because it is too large
Load Diff
3
training_args.bin
Normal file
3
training_args.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3e3730b770a7a69a97a94adeeead1c692defea9bd7b7bf89372f99be11a0b762
|
||||
size 4987
|
||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user