251 lines
8.9 KiB
Markdown
251 lines
8.9 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
- es
|
||
- ca
|
||
licence:
|
||
- apache-2.0
|
||
tags:
|
||
- aguila
|
||
- falcon
|
||
- spanish
|
||
- catalan
|
||
metrics:
|
||
- ppl
|
||
model-index:
|
||
- name: aguila_7b
|
||
results:
|
||
- task:
|
||
name: Causal Language Modeling
|
||
type: text-generation
|
||
metrics:
|
||
- name: Perplexity
|
||
type: ppl
|
||
value: 8.59
|
||
pipeline_tag: text-generation
|
||
widget:
|
||
- text: |-
|
||
Respon a la pregunta següent.
|
||
Pregunta: "Quina és la capital de Suècia?"
|
||
Resposta: "La capital de Suècia és Estocolm."
|
||
----
|
||
Respon a la pregunta següent.
|
||
Pregunta: "Quina beguda es consumeix als matins per despertar-se?"
|
||
Resposta: "La majoria de gent consumeix cafè per despertar-se."
|
||
----
|
||
Respon a la pregunta següent.
|
||
Pregunta: "Explica com funciona un motor de combustió"
|
||
Resposta:
|
||
example_title: Pregunta-Resposta
|
||
- text: |-
|
||
Extrae las entidades nombradas del siguiente texto:
|
||
Texto: "Me llamo Wolfgang y vivo en Berlin"
|
||
Entidades: Wolfgang:PER, Berlin:LOC
|
||
----
|
||
Extrae las entidades nombradas del siguiente texto:
|
||
Texto: "Hoy voy a visitar el parc güell tras salir del barcelona supercomputing center"
|
||
Entidades: parc güell:LOC, barcelona supercomputing center:LOC
|
||
----
|
||
Extrae las entidades nombradas del siguiente texto:
|
||
Texto: "Maria y Miguel no tienen ningún problema contigo"
|
||
Entidades: Maria:PER, Miguel:PER
|
||
----
|
||
Extrae las entidades nombradas del siguiente texto:
|
||
Texto: "Damián se cortó el pelo"
|
||
Entidades: Damián:PER
|
||
----
|
||
Extrae las entidades nombradas del siguiente texto:
|
||
Texto: "Lo mejor de Barcelona és el bar de mi amigo Pablo"
|
||
Entidades: Pablo:PER, Barcelona:LOC
|
||
----
|
||
Extrae las entidades nombradas del siguiente texto:
|
||
Texto: "Carlos comparte piso con Marc"
|
||
Entidades:
|
||
example_title: Entidades-Nombradas
|
||
---
|
||
|
||
# Ǎguila-7B
|
||
|
||
<strong><span style="color:darkred">⚠️NOTICE⚠️: This model has been deprecated and is no longer actively maintained or supported. To access the latest models with enhanced features, better performance,
|
||
and ongoing support, please visit <a style="color:darkred" href="https://huggingface.co/BSC-LT">https://huggingface.co/BSC-LT</a></span></strong>
|
||
|
||
|
||
## Table of Contents
|
||
<details>
|
||
<summary>Click to expand</summary>
|
||
|
||
- [Model description](#model-description)
|
||
- [Intended uses and limitations](#intended-uses-and-limitations)
|
||
- [How to use](#how-to-use)
|
||
- [Limitations and bias](#limitations-and-bias)
|
||
- [Language adaptation](#language-adaptation)
|
||
- [Training](#training)
|
||
- [Training data](#training-data)
|
||
- [Training procedure](#training-procedure)
|
||
- [Additional information](#additional-information)
|
||
- [Author](#author)
|
||
- [Contact](#contact)
|
||
- [Copyright](#copyright)
|
||
- [License](#license)
|
||
- [Funding](#funding)
|
||
- [Disclaimer](#disclaimer)
|
||
|
||
</details>
|
||
|
||
## Model description
|
||
|
||
**Ǎguila-7B** is a transformer-based causal language model for Catalan, Spanish, and English.
|
||
It is based on the [Falcon-7B](https://huggingface.co/tiiuae/falcon-7b) model and has been trained on a 26B token
|
||
trilingual corpus collected from publicly available corpora and crawlers.
|
||
|
||
More information available in the following post from Medium.com: Introducing Ǎguila, a new open-source LLM for Spanish and Catalan (https://medium.com/@mpamies247/introducing-a%CC%8Cguila-a-new-open-source-llm-for-spanish-and-catalan-ee1ebc70bc79)
|
||
|
||
|
||
## Intended uses and limitations
|
||
|
||
The **Ǎguila-7B** model is ready-to-use only for causal language modeling to perform text-generation tasks.
|
||
However, it is intended to be fine-tuned for downstream tasks.
|
||
|
||
## How to use
|
||
|
||
Here is how to use this model:
|
||
|
||
```python
|
||
import torch
|
||
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
|
||
|
||
input_text = "El mercat del barri és fantàstic, hi pots trobar"
|
||
|
||
model_id = "projecte-aina/aguila-7b"
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
generator = pipeline(
|
||
"text-generation",
|
||
model=model_id,
|
||
tokenizer=tokenizer,
|
||
torch_dtype=torch.bfloat16,
|
||
trust_remote_code=True,
|
||
device_map="auto",
|
||
)
|
||
generation = generator(
|
||
input_text,
|
||
do_sample=True,
|
||
top_k=10,
|
||
eos_token_id=tokenizer.eos_token_id,
|
||
)
|
||
|
||
print(f"Result: {generation[0]['generated_text']}")
|
||
```
|
||
|
||
## Limitations and bias
|
||
At the time of submission, no measures have been taken to estimate the bias and toxicity embedded in the model.
|
||
However, we are well aware that our models may be biased since the corpora have been collected using crawling techniques
|
||
on multiple web sources. We intend to conduct research in these areas in the future, and if completed, this model card will be updated.
|
||
|
||
|
||
## Language adaptation
|
||
|
||
We adapted the original [Falcon-7B](https://huggingface.co/tiiuae/falcon-7b) model to Spanish and Catalan by swapping the tokenizer and adjusting the embedding layer.
|
||
|
||
The adaptation procedure is explained in [this blog post](https://medium.com/@mpamies247/ee1ebc70bc79).
|
||
|
||
## Training
|
||
|
||
### Training data
|
||
|
||
The training corpus consists of 26B tokens of several corpora gathered from web crawlings and public domain data.
|
||
|
||
| Dataset | Language | Words (per-epoch) | Epochs |
|
||
|---------------------|----------|--------------------|--------------|
|
||
| Wikipedia | en | 2169.97M | 1.428144485 |
|
||
| C4_es | es | 53709.80M | 0.1049686196 |
|
||
| Biomedical | es | 455.03M | 0.7140722425 |
|
||
| Legal | es | 995.70M | 0.7140722425 |
|
||
| Wikipedia | es | 693.60M | 1.428144485 |
|
||
| Gutenberg | es | 53.18M | 0.7140722425 |
|
||
| C4_ca | ca | 2826.00M | 2.142216727 |
|
||
| Biomedical | ca | 11.80M | 1.428144485 |
|
||
| RacoCatalà Noticias | ca | 17.16M | 2.142216727 |
|
||
| RacoCatalà Forums | ca | 333.73M | 2.142216727 |
|
||
| CaWaC | ca | 57.79M | 2.142216727 |
|
||
| Wikipedia | ca | 228.01M | 3.570361212 |
|
||
| Vilaweb | ca | 50.34M | 2.142216727 |
|
||
|
||
The dataset has the following language distribution:
|
||
|
||
|Language|Percentage|
|
||
|--------|----------|
|
||
| En | 16.84% |
|
||
| Es | 41.38% |
|
||
| Ca | 41.79% |
|
||
|
||
Note: A small amount of English data was kept to avoid catastrophic forgetting.
|
||
|
||
## Training procedure
|
||
|
||
The training corpus has been tokenized using a byte version of [Byte-Pair Encoding (BPE)](https://github.com/openai/gpt-2) with a vocabulary size of 50,257 tokens.
|
||
After training a new tokenizer and adapting [falcon-7b](https://huggingface.co/tiiuae/falcon-7b)'s embedding layer, the model was
|
||
further pre-trained in three target languages: Catalan, Spanish and English.
|
||
|
||
The training lasted a total of 320 hours on 8 NVIDIA H100 GPUs with 80GB RAM.
|
||
|
||
|
||
### Training hyperparameters
|
||
|
||
- seed: 42
|
||
- distributed_type: multi-GPU
|
||
- num_devices: 8
|
||
- train_batch_size: 1
|
||
- eval_batch_size: 1
|
||
- total_train_batch_size: 8
|
||
- total_eval_batch_size: 8
|
||
- optimizer: Adam
|
||
- betas: (0.9,0.999)
|
||
- epsilon: 1e-08
|
||
- learning_rate: 5e-05
|
||
- lr_scheduler_type: linear
|
||
- num_epochs: 1.0
|
||
|
||
|
||
### Framework versions
|
||
|
||
- Pytorch 2.0.0
|
||
- Transformers 4.30.2
|
||
- Datasets 2.13.1
|
||
- Tokenizers 0.13.3
|
||
|
||
## Additional information
|
||
|
||
### Author
|
||
The Language Technologies Unit from Barcelona Supercomputing Center.
|
||
|
||
### Contact
|
||
For further information, please send an email to <langtech@bsc.es>.
|
||
|
||
### Copyright
|
||
Copyright(c) 2023 by Language Technologies Unit, Barcelona Supercomputing Center.
|
||
|
||
### License
|
||
[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
|
||
|
||
### Funding
|
||
This work was funded by:
|
||
- The [Departament de la Vicepresidència i de Polítiques Digitals i Territori de la Generalitat de Catalunya](https://politiquesdigitals.gencat.cat/ca/inici/index.html#googtrans(ca|en) within the framework of [Projecte AINA](https://politiquesdigitals.gencat.cat/ca/economia/catalonia-ai/aina).
|
||
- The [Spanish State Secretariat for Digitalization and Artificial Intelligence](https://portal.mineco.gob.es/en-us/digitalizacionIA/Pages/sedia.aspx) within the framework of the [Plan de Impulso de las Tecnologías del Lenguaje](https://plantl.mineco.gob.es/Paginas/index.aspx).
|
||
|
||
### Disclaimer
|
||
|
||
<details>
|
||
<summary>Click to expand</summary>
|
||
|
||
The model published in this repository is intended for a generalist purpose and is available to third parties under a permissive Apache License, Version 2.0.
|
||
|
||
Be aware that the model may have biases and/or any other undesirable distortions.
|
||
|
||
When third parties deploy or provide systems and/or services to other parties using this model (or any system based on it)
|
||
or become users of the model, they should note that it is their responsibility to mitigate the risks arising from its use and,
|
||
in any event, to comply with applicable regulations, including regulations regarding the use of Artificial Intelligence.
|
||
|
||
In no event shall the owner and creator of the model (Barcelona Supercomputing Center)
|
||
be liable for any results arising from the use made by third parties.
|
||
|
||
</details> |