calme-3.2-baguette-3b/README.md

language, license, library_name, tags, base_model, datasets, model_name, license_name, license_link, pipeline_tag, inference, model_creator, quantized_by, model-index

language	license	library_name	tags	base_model	datasets	model_name	license_name	license_link	pipeline_tag	inference	model_creator	quantized_by	model-index
fr en	other	transformers	chat qwen qwen2.5 finetune french english	Qwen/Qwen2.5-3B	MaziyarPanahi/french_instruct_sharegpt MaziyarPanahi/calme-legalkit-v0.2	calme-3.2-baguette-3b	qwen-research	https://huggingface.co/Qwen/Qwen2.5-3B/blob/main/LICENSE	text-generation	false	MaziyarPanahi	MaziyarPanahi	name results calme-3.2-baguette-3b task dataset metrics source type name text-generation Text Generation name type args IFEval (0-Shot) HuggingFaceH4/ifeval num_few_shot 0 type value name inst_level_strict_acc and prompt_level_strict_acc 63.38 strict accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MaziyarPanahi/calme-3.2-baguette-3b Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args BBH (3-Shot) BBH num_few_shot 3 type value name acc_norm 25.87 normalized accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MaziyarPanahi/calme-3.2-baguette-3b Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MATH Lvl 5 (4-Shot) hendrycks/competition_math num_few_shot 4 type value name exact_match 3.1 exact match url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MaziyarPanahi/calme-3.2-baguette-3b Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args GPQA (0-shot) Idavidrein/gpqa num_few_shot 0 type value name acc_norm 5.93 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MaziyarPanahi/calme-3.2-baguette-3b Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MuSR (0-shot) TAUR-Lab/MuSR num_few_shot 0 type value name acc_norm 8.6 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MaziyarPanahi/calme-3.2-baguette-3b Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args MMLU-PRO (5-shot) TIGER-Lab/MMLU-Pro main test num_few_shot 5 type value name acc 25.98 accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MaziyarPanahi/calme-3.2-baguette-3b Open LLM Leaderboard

Tip

This is avery small model, so it might not perform well for some prompts and may be sensitive to hyper parameters. I would appreciate any feedback to see if I can fix any issues in the next iteration. ❤️

MaziyarPanahi/calme-3.2-baguette-3b

This model is an advanced iteration of the powerful Qwen/Qwen2.5-3B, fine-tuned specifically to enhance its capabilities across general domains in both French and English.

⚡ Quantized GGUF

All GGUF models are available here: MaziyarPanahi/calme-3.2-baguette-3b-GGUF

🏆 Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	22.14
IFEval (0-Shot)	63.38
BBH (3-Shot)	25.87
MATH Lvl 5 (4-Shot)	3.10
GPQA (0-shot)	5.93
MuSR (0-shot)	8.60
MMLU-PRO (5-shot)	25.98

Prompt Template

This model uses ChatML prompt template:

<|im_start|>system
{System}
<|im_end|>
<|im_start|>user
{User}
<|im_end|>
<|im_start|>assistant
{Assistant}

How to use

# Use a pipeline as a high-level helper

from transformers import pipeline

messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe = pipeline("text-generation", model="MaziyarPanahi/calme-3.2-baguette-3b")
pipe(messages)


# Load model directly

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MaziyarPanahi/calme-3.2-baguette-3b")
model = AutoModelForCausalLM.from_pretrained("MaziyarPanahi/calme-3.2-baguette-3b")

Ethical Considerations

As with any large language model, users should be aware of potential biases and limitations. We recommend implementing appropriate safeguards and human oversight when deploying this model in production environments.