Qwen2.5-0.5B-Portuguese-v1/README.md

license, language, base_model, pipeline_tag, datasets, model-index

license	language	base_model	pipeline_tag	datasets	model-index
mit	zho eng fra spa por deu ita rus jpn kor vie tha ara	Qwen/Qwen2.5-0.5B-Instruct	text-generation	adalbertojunior/openHermes_portuguese cnmoro/smoltalk-555k-ptbr cnmoro/RagMixPTBR-Legal-Alpaca-2M	name results Qwen2.5-0.5B-Portuguese-v1 task dataset metrics source type name text-generation Text Generation name type split args ENEM Challenge (No Images) eduagarcia/enem_challenge train num_few_shot 3 type value name acc 37.86 accuracy url name https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Qwen2.5-0.5B-Portuguese-v1 Open Portuguese LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args BLUEX (No Images) eduagarcia-temp/BLUEX_without_images train num_few_shot 3 type value name acc 34.63 accuracy url name https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Qwen2.5-0.5B-Portuguese-v1 Open Portuguese LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args OAB Exams eduagarcia/oab_exams train num_few_shot 3 type value name acc 33.12 accuracy url name https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Qwen2.5-0.5B-Portuguese-v1 Open Portuguese LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args Assin2 RTE assin2 test num_few_shot 15 type value name f1_macro 86.3 f1-macro url name https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Qwen2.5-0.5B-Portuguese-v1 Open Portuguese LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args Assin2 STS eduagarcia/portuguese_benchmark test num_few_shot 15 type value name pearson 54.3 pearson url name https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Qwen2.5-0.5B-Portuguese-v1 Open Portuguese LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args FaQuAD NLI ruanchaves/faquad-nli test num_few_shot 15 type value name f1_macro 65.33 f1-macro url name https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Qwen2.5-0.5B-Portuguese-v1 Open Portuguese LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args HateBR Binary ruanchaves/hatebr test num_few_shot 25 type value name f1_macro 44.06 f1-macro url name https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Qwen2.5-0.5B-Portuguese-v1 Open Portuguese LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args PT Hate Speech Binary hate_speech_portuguese test num_few_shot 25 type value name f1_macro 55.1 f1-macro url name https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Qwen2.5-0.5B-Portuguese-v1 Open Portuguese LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args tweetSentBR eduagarcia/tweetsentbr_fewshot test num_few_shot 25 type value name f1_macro 45.96 f1-macro url name https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Qwen2.5-0.5B-Portuguese-v1 Open Portuguese LLM Leaderboard

Qwen2.5-0.5B finetuned for proficiency in Portuguese language and increased intelligence.

https://ollama.com/cnmoro/Qwen2.5-0.5B-Portuguese-v1

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "cnmoro/Qwen2.5-0.5B-Portuguese-v1"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Escreva uma breve introdução sobre LLMs (Large Language Models) e suas aplicações."

# System prompt is always injected and hardcoded automatically
# for ideal performance in portuguese language.
# No need to write it again.
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
response
# LLM significa Large Language Models, que são modelos de linguagem computacional
# projetados para simular a inteligência humana no processamento e geração de texto.
# Esses modelos usam técnicas avançadas de aprendizado de máquina e redes neurais para
# compreender e gerar texto com base em dados de entrada. As aplicações de LLM incluem
# tradução automática, análise de sentimento, modelagem de tópicos e resposta a perguntas
# automatizadas. Eles estão sendo cada vez mais utilizados em diversas áreas, como
# saúde, educação e finanças, para melhorar a comunicação, as experiências dos clientes
# e os resultados da pesquisa.

Overall Results

Task	Metric	Value	Stdev
assin2_rte	f1_macro	0.391	0.006
assin2_rte	acc	0.527	0.007
assin2_sts	pearson	0.115	0.014
assin2_sts	mse	1.011	N/A
bluex	acc	0.349	0.010
enem_challenge	acc	0.363	0.007
faquad_nli	f1_macro	0.595	0.017
faquad_nli	acc	0.791	0.011
hatebr_offensive	f1_macro	0.338	0.005
hatebr_offensive	acc	0.502	0.009
oab_exams	acc	0.326	0.006
portuguese_hate_speech	f1_macro	0.412	0.004
portuguese_hate_speech	acc	0.702	0.011
tweetsentbr	f1_macro	0.455	0.005
tweetsentbr	acc	0.594	0.008

Detailed Results

assin2_rte

Metric	Value	Stdev
f1_macro	0.391	0.006
acc	0.527	0.007

assin2_sts

Metric	Value	Stdev
pearson	0.115	0.014
mse	1.011	N/A

bluex

Exam ID	Metric	Value	Stdev
all	acc	0.349	0.010
USP_2019	acc	0.225	0.038
USP_2024	acc	0.293	0.041
USP_2021	acc	0.423	0.040
UNICAMP_2018	acc	0.241	0.034
UNICAMP_2024	acc	0.444	0.043
USP_2020	acc	0.393	0.038
UNICAMP_2020	acc	0.291	0.035
UNICAMP_2021_1	acc	0.326	0.040
UNICAMP_2022	acc	0.487	0.046
USP_2022	acc	0.388	0.040
UNICAMP_2019	acc	0.280	0.037
UNICAMP_2021_2	acc	0.294	0.037
UNICAMP_2023	acc	0.558	0.044
USP_2023	acc	0.364	0.042
USP_2018	acc	0.278	0.035

enem_challenge

Exam ID	Metric	Value	Stdev
all	acc	0.363	0.007
2016_2	acc	0.390	0.025
2015	acc	0.319	0.025
2011	acc	0.410	0.026
2013	acc	0.398	0.027
2017	acc	0.319	0.025
2022	acc	0.376	0.024
2009	acc	0.226	0.023
2010	acc	0.444	0.026
2012	acc	0.345	0.025
2014	acc	0.339	0.026
2016	acc	0.397	0.026
2023	acc	0.385	0.024

faquad_nli

Metric	Value	Stdev
f1_macro	0.595	0.017
acc	0.791	0.011

hatebr_offensive

Metric	Value	Stdev
f1_macro	0.338	0.005
acc	0.502	0.009

oab_exams

Exam ID	Metric	Value	Stdev
all	acc	0.326	0.006
2018-25	acc	0.400	0.032
2016-20a	acc	0.238	0.027
2011-05	acc	0.400	0.032
2012-08	acc	0.325	0.030
2012-09	acc	0.260	0.029
2014-13	acc	0.325	0.030
2011-03	acc	0.313	0.027
2016-20	acc	0.275	0.029
2012-06a	acc	0.325	0.030
2017-22	acc	0.338	0.031
2015-16	acc	0.325	0.030
2013-12	acc	0.300	0.030
2017-24	acc	0.250	0.028
2012-06	acc	0.238	0.027
2014-14	acc	0.325	0.030
2013-11	acc	0.325	0.030
2013-10	acc	0.413	0.032
2010-02	acc	0.390	0.028
2016-21	acc	0.375	0.031
2015-18	acc	0.300	0.030
2015-17	acc	0.282	0.029
2016-19	acc	0.333	0.031
2012-07	acc	0.388	0.031
2017-23	acc	0.325	0.030
2011-04	acc	0.350	0.031
2010-01	acc	0.282	0.028
2014-15	acc	0.385	0.032

portuguese_hate_speech

Metric	Value	Stdev
f1_macro	0.412	0.004
acc	0.702	0.011

tweetsentbr

Metric	Value	Stdev
f1_macro	0.455	0.005
acc	0.594	0.008

Model Meta Information

• Truncated Samples: 3863
• Non-Truncated Samples: 10287
• Padded Samples: 0
• Non-Padded Samples: 14150
• Fewshots Truncated: 3863
• Has Chat Template: True
• Chat Type: system_user_assistant
• Number of GPUs: 1
• Accelerate Number of Processes: N/A
• Model SHA: None
• Model Data Type: torch.bfloat16
• Model Memory Footprint: 988065664 bytes
• Model Number of Parameters: 494032768
• Model is Loaded in 4bit: N/A
• Model is Loaded in 8bit: N/A
• Model is Quantized: N/A
• Model Device: cuda:0
• Batch Size: 1
• Max Length: 512
• Max Context Length 480
• Max Generation Tokens: 32
• Effective Batch Size: 1.0

Open Portuguese LLM Leaderboard Evaluation Results

Detailed results can be found here and on the 🚀 Open Portuguese LLM Leaderboard

Metric	Value
Average	50.74
ENEM Challenge (No Images)	37.86
BLUEX (No Images)	34.63
OAB Exams	33.12
Assin2 RTE	86.30
Assin2 STS	54.30
FaQuAD NLI	65.33
HateBR Binary	44.06
PT Hate Speech Binary	55.10
tweetSentBR	45.96