306 lines
10 KiB
Markdown
306 lines
10 KiB
Markdown
---
|
|
language:
|
|
- pt
|
|
license: apache-2.0
|
|
library_name: transformers
|
|
tags:
|
|
- Misral
|
|
- Portuguese
|
|
- 7b
|
|
- chat
|
|
- portugues
|
|
base_model: mistralai/Mistral-7B-Instruct-v0.2
|
|
datasets:
|
|
- rhaymison/ultrachat-easy-use
|
|
pipeline_tag: text-generation
|
|
model-index:
|
|
- name: Mistral-portuguese-luana-7b-chat
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: ENEM Challenge (No Images)
|
|
type: eduagarcia/enem_challenge
|
|
split: train
|
|
args:
|
|
num_few_shot: 3
|
|
metrics:
|
|
- type: acc
|
|
value: 59.13
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=rhaymison/Mistral-portuguese-luana-7b-chat
|
|
name: Open Portuguese LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: BLUEX (No Images)
|
|
type: eduagarcia-temp/BLUEX_without_images
|
|
split: train
|
|
args:
|
|
num_few_shot: 3
|
|
metrics:
|
|
- type: acc
|
|
value: 49.24
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=rhaymison/Mistral-portuguese-luana-7b-chat
|
|
name: Open Portuguese LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: OAB Exams
|
|
type: eduagarcia/oab_exams
|
|
split: train
|
|
args:
|
|
num_few_shot: 3
|
|
metrics:
|
|
- type: acc
|
|
value: 36.58
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=rhaymison/Mistral-portuguese-luana-7b-chat
|
|
name: Open Portuguese LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: Assin2 RTE
|
|
type: assin2
|
|
split: test
|
|
args:
|
|
num_few_shot: 15
|
|
metrics:
|
|
- type: f1_macro
|
|
value: 90.47
|
|
name: f1-macro
|
|
source:
|
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=rhaymison/Mistral-portuguese-luana-7b-chat
|
|
name: Open Portuguese LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: Assin2 STS
|
|
type: eduagarcia/portuguese_benchmark
|
|
split: test
|
|
args:
|
|
num_few_shot: 15
|
|
metrics:
|
|
- type: pearson
|
|
value: 76.55
|
|
name: pearson
|
|
source:
|
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=rhaymison/Mistral-portuguese-luana-7b-chat
|
|
name: Open Portuguese LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: FaQuAD NLI
|
|
type: ruanchaves/faquad-nli
|
|
split: test
|
|
args:
|
|
num_few_shot: 15
|
|
metrics:
|
|
- type: f1_macro
|
|
value: 66.75
|
|
name: f1-macro
|
|
source:
|
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=rhaymison/Mistral-portuguese-luana-7b-chat
|
|
name: Open Portuguese LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: HateBR Binary
|
|
type: ruanchaves/hatebr
|
|
split: test
|
|
args:
|
|
num_few_shot: 25
|
|
metrics:
|
|
- type: f1_macro
|
|
value: 77.46
|
|
name: f1-macro
|
|
source:
|
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=rhaymison/Mistral-portuguese-luana-7b-chat
|
|
name: Open Portuguese LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: PT Hate Speech Binary
|
|
type: hate_speech_portuguese
|
|
split: test
|
|
args:
|
|
num_few_shot: 25
|
|
metrics:
|
|
- type: f1_macro
|
|
value: 69.45
|
|
name: f1-macro
|
|
source:
|
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=rhaymison/Mistral-portuguese-luana-7b-chat
|
|
name: Open Portuguese LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: tweetSentBR
|
|
type: eduagarcia-temp/tweetsentbr
|
|
split: test
|
|
args:
|
|
num_few_shot: 25
|
|
metrics:
|
|
- type: f1_macro
|
|
value: 59.63
|
|
name: f1-macro
|
|
source:
|
|
url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=rhaymison/Mistral-portuguese-luana-7b-chat
|
|
name: Open Portuguese LLM Leaderboard
|
|
---
|
|
|
|
# Mistral-portuguese-luana-7b-chat
|
|
|
|
<p align="center">
|
|
<img src="https://raw.githubusercontent.com/rhaymisonbetini/huggphotos/main/luana-chat.webp" width="50%" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
|
|
</p>
|
|
|
|
|
|
This model was trained with a superset of 250,000 chat in Portuguese.
|
|
The model comes to help fill the gap in models in Portuguese. Tuned from the Mistral 7b, the model was adjusted mainly for chat.
|
|
|
|
# How to use
|
|
|
|
### FULL MODEL : A100
|
|
### HALF MODEL: L4
|
|
### 8bit or 4bit : T4 or V100
|
|
|
|
You can use the model in its normal form up to 4-bit quantization. Below we will use both approaches.
|
|
Remember that verbs are important in your prompt. Tell your model how to act or behave so that you can guide them along the path of their response.
|
|
Important points like these help models (even smaller models like 7b) to perform much better.
|
|
|
|
```python
|
|
!pip install -q -U transformers
|
|
!pip install -q -U accelerate
|
|
!pip install -q -U bitsandbytes
|
|
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
|
|
model = AutoModelForCausalLM.from_pretrained("rhaymison/Mistral-portuguese-luana-7b-chat", device_map= {"": 0})
|
|
tokenizer = AutoTokenizer.from_pretrained("rhaymison/Mistral-portuguese-luana-7b-chat")
|
|
model.eval()
|
|
|
|
```
|
|
|
|
You can use with Pipeline but in this example i will use such as Streaming
|
|
```python
|
|
|
|
inputs = tokenizer([f"""<s>[INST] Estou pensando em levar minha esposa e meu gatinha para fazer algo diferente. Você teria alguma coisa em mente ?.
|
|
[/INST]"""], return_tensors="pt")
|
|
|
|
inputs.to(model.device)
|
|
|
|
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
|
|
_ = model.generate(**inputs, streamer=streamer, max_new_tokens=128)
|
|
|
|
```
|
|
|
|
If you are having a memory problem such as "CUDA Out of memory", you should use 4-bit or 8-bit quantization.
|
|
For the complete model in colab you will need the A100.
|
|
If you want to use 4bits or 8bits, T4 or L4 will already solve the problem.
|
|
|
|
# 4bits example
|
|
|
|
```python
|
|
from transformers import BitsAndBytesConfig
|
|
import torch
|
|
nb_4bit_config = BitsAndBytesConfig(
|
|
load_in_4bit=True,
|
|
bnb_4bit_quant_type="nf4",
|
|
bnb_4bit_compute_dtype=torch.bfloat16,
|
|
bnb_4bit_use_double_quant=True
|
|
)
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
base_model,
|
|
quantization_config=bnb_config,
|
|
device_map={"": 0}
|
|
)
|
|
|
|
```
|
|
|
|
# LangChain
|
|
|
|
<div style="background-color:#f0f0f0; padding:10px; border-radius:10px; font-family: 'Arial';">
|
|
<p><strong><span style="color:#3366cc;">Human:</span></strong> Ola Luana, estou precisando de umas dicas para o final de semana.
|
|
Poderia me ajudar ?</p>
|
|
|
|
<p><strong><span style="color:#ff6600;">Luana:</span></strong> Olá! Claro, ficarei feliz em ajudá-lo. O que você tem em mente para o final de semana?</p>
|
|
|
|
<p><strong><span style="color:#3366cc;">Human:</span></strong> Estou pensando em levar minha esposa e meu gatinha para fazer algo diferente.
|
|
Você teria alguma coisa em mente ?</p>
|
|
|
|
<p><strong><span style="color:#ff6600;">Luana:</span></strong> Claro! Você pode considerar fazer um piquenique na praia, visitar um museu
|
|
ou fazer uma caminhada na natureza. O que você e sua esposa gostam de fazer juntos?</p>
|
|
|
|
<p><strong><span style="color:#3366cc;">Human:</span></strong> Eu e minha esposa gostamos muito de caminhadas.
|
|
Temos uma mochila que carregamos nosso gatinha para todo lado.</p>
|
|
|
|
<p><strong><span style="color:#ff6600;">Luana:</span></strong> Parece ótimo! Você pode considerar fazer uma caminhada em um parque local ou reserva natural. Você
|
|
também pode pesquisar trilhas mais longas para fazer em um fim de semana.
|
|
Além disso, você pode trazer um piquenique com você para saborear durante a caminhada.</p>
|
|
|
|
<p><strong><span style="color:#3366cc;">Human:</span></strong> Otimas ideias voce tem me dado. O que será que é gostoso levar em um piquenique ?</p>
|
|
|
|
<p><strong><span style="color:#ff6600;">Luana:</span></strong> Você pode trazer sanduíches, frutas, vegetais, queijo e biscoitos.
|
|
Você também pode trazer uma garrafa de água ou um refrigerante para beber.
|
|
Além disso, você pode trazer um saco de lixo para descartar resíduos de maneira responsável.</p>
|
|
|
|
<p><strong><span style="color:#3366cc;">Human:</span></strong> Olha só que dica bacana. Realmente devo levar uma sacola para nao descartar
|
|
nada na Natureza e assim me divertir com minha esposa.</p>
|
|
|
|
<p><strong><span style="color:#ff6600;">Luana:</span></strong> Sim, é importante descartar resíduos de maneira responsável e proteger o meio ambiente.
|
|
Você pode considerar fazer um piquenique em um parque local ou reserva natural onde
|
|
possa descartar resíduos de maneira responsável.</p>
|
|
|
|
<p><strong><span style="color:#3366cc;">Human:</span></strong> Perfeito. Muito Obrigado Luana. Otimas dicas. Até a proxima !</p>
|
|
|
|
<p><strong><span style="color:#ff6600;">Luana:</span></strong> De nada! Fique em contato se precisar de mais ajuda. Tenha um ótimo fim de semana!</p>
|
|
</div>
|
|
|
|
|
|
# [Open Portuguese LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard)
|
|
Detailed results can be found [here](https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/tree/main/rhaymison/Mistral-portuguese-luana-7b-chat)
|
|
|
|
| Metric | Value |
|
|
|--------------------------|---------|
|
|
|Average |**65.03**|
|
|
|ENEM Challenge (No Images)| 59.13|
|
|
|BLUEX (No Images) | 49.24|
|
|
|OAB Exams | 36.58|
|
|
|Assin2 RTE | 90.47|
|
|
|Assin2 STS | 76.55|
|
|
|FaQuAD NLI | 66.75|
|
|
|HateBR Binary | 77.46|
|
|
|PT Hate Speech Binary | 69.45|
|
|
|tweetSentBR | 59.63|
|
|
|
|
|
|
|
|
### Comments
|
|
|
|
Any idea, help or report will always be welcome.
|
|
|
|
email: rhaymisoncristian@gmail.com
|
|
|
|
<div style="display:flex; flex-direction:row; justify-content:left">
|
|
<a href="https://www.linkedin.com/in/rhaymison-cristian-betini-2b3016175/" target="_blank">
|
|
<img src="https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white">
|
|
</a>
|
|
<a href="https://github.com/rhaymisonbetini" target="_blank">
|
|
<img src="https://img.shields.io/badge/GitHub-100000?style=for-the-badge&logo=github&logoColor=white">
|
|
</a>
|
|
</div>
|