--- library_name: transformers pipeline_tag: text-generation language: - it license: llama3 base_model: - mii-llm/zagreus-0.4B-ita datasets: - efederici/pinocchio - FinancialSupport/italic_sft - FinancialSupport/italic_sft_ext - FinancialSupport/quiz_militare tags: - llama - italian - small-language-model - multiple-choice - supervised-fine-tuning - post-training - italic --- # Zagreus 0.4B xmoons improved Zagreus 0.4B xmoons improved is an Italian multiple-choice model fine-tuned from [`mii-llm/zagreus-0.4B-ita`](https://huggingface.co/mii-llm/zagreus-0.4B-ita). It is the selected checkpoint from the reproducible H100 run that reached **44.35% accuracy** on the official 10,000-question ITALIC benchmark in fast, five-shot mode through vLLM. The model was developed for the [`mii-llm/Post-Training-Challenge`](https://huggingface.co/spaces/mii-llm/Post-Training-Challenge). The model is a compact Llama-style causal language model with 437,760,960 parameters. Its tokenizer and Llama 3 chat template come from [`swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA`](https://huggingface.co/swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA). The repository is a self-contained Transformers checkpoint and uses `<|eot_id|>` as both the end-of-turn and generation stop token. ## Intended use The model is intended for: - research on compact Italian language models; - Italian multiple-choice question answering; - participation in the MII Post-Training Challenge; - reproduction and analysis of the ITALIC post-training experiment. It is not designed as a general-purpose assistant, a factual authority, or a component for medical, legal, financial, or other high-stakes decisions. ## Usage Install PyTorch and Transformers, then load the checkpoint from the repository root: ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "fgoose180/zagreus-0.4B-xmoons-improved" device = "cuda" if torch.cuda.is_available() else "cpu" dtype = torch.bfloat16 if device == "cuda" else torch.float32 tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, dtype=dtype, ).to(device) messages = [ {"role": "system", "content": "Sei un assistente utile."}, { "role": "user", "content": """Rispondi alla seguente domanda a scelta multipla sull'argomento 'data_geography'. La tua risposta deve essere nel seguente formato: 'LETTERA' (senza virgolette) dove LETTERA è una tra ABCD. Scrivi solo la lettera corrispondente alla tua risposta senza spiegazioni. Qual è la capitale d'Italia? A) Roma B) Milano C) Torino D) Napoli Risposta:""", }, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt", return_dict=True, ).to(device) with torch.inference_mode(): output = model.generate( **inputs, max_new_tokens=8, do_sample=False, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id, ) answer = tokenizer.decode( output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True, ).strip() print(answer) ``` The reported benchmark result uses the five fixed official ITALIC demonstrations. Zero-shot use, different prompts, sampling, or another chat template should not be expected to reproduce the reported score. ## Model architecture | Property | Value | |---|---| | Architecture | `LlamaForCausalLM` | | Model type | Llama-style decoder-only transformer | | Parameters | 437,760,960 | | Hidden size | 960 | | Layers | 32 | | Attention heads / KV heads | 15 / 5 | | Vocabulary size | 128,256 | | Maximum trained context | 2,048 tokens during SFT | | Checkpoint serialization | SafeTensors | ## Training ### Data The final training stream was built online from pinned revisions of four public datasets: | Dataset | Main contribution | |---|---| | `efederici/pinocchio` | Italian language, culture, and general-knowledge MCQs | | `FinancialSupport/italic_sft` | ITALIC-style Italian MCQs | | `FinancialSupport/italic_sft_ext` | Extended ITALIC-style MCQs | | `FinancialSupport/quiz_militare` | Italian civic and general-knowledge MCQs | Rows were normalized, exactly deduplicated, split before upsampling, and decontaminated against the official ITALIC test and five demonstrations using character TF-IDF cosine and word-shingle MinHash/LSH. The final artifacts were: | Artifact | Rows | Unique questions | |---|---:|---:| | Clean training stream | 108,288 | 67,654 | | Clean validation holdout | 6,745 | 6,745 | The training stream contains 53,591 language draws and 54,697 culture draws. The benchmark labels were not used for checkpoint selection. ### Objective and hyperparameters The model was trained for one epoch with completion-only supervised loss. Each target was rendered with five distinct training-pool demonstrations and independently permuted answer options. The official five ITALIC demonstrations were reserved for validation and final evaluation. | Parameter | Value | |---|---:| | Hardware | 1x NVIDIA H100 | | Optimizer steps | 6,768 | | Batch size | 16 | | Initial learning rate | `3e-4` | | Schedule | 50-step warmup, cosine decay | | Weight decay | `0.0` | | Maximum sequence length | 2,048 | | Forward precision | BF16 autocast | | Parameters and optimizer updates | FP32 | | Gradient clipping | `1.0` | | Random seed | 0 | | Few-shot probability / count | `1.0 / 5` | | Option permutation probability | `1.0` | The detached Modal pipeline took approximately 84 minutes end to end. Carbon emissions were not measured, so no emissions estimate is reported. ## Evaluation The selected checkpoint was chosen at step 6,768 using only a clean validation holdout and the criterion `0.5 * language_accuracy + 0.5 * culture_accuracy`. | Evaluation | Questions | Accuracy | Balanced accuracy | Unparsed | |---|---:|---:|---:|---:| | Validation holdout | 6,745 | 55.97% | 61.38% | 0 | | Official ITALIC fast five-shot, vLLM 0.26 | 10,000 | **44.35%** | not reported | 0 | The official evaluation used greedy decoding, the official fast answer extractor, a pinned ITALIC harness, and the official five demonstrations. The result contained 4,435 correct answers out of 10,000. ## Reproducibility The training and evaluation implementation is available in [`mattiacurri/zagreus-italic-challenge-xmoons`](https://github.com/mattiacurri/zagreus-italic-challenge-xmoons). The exact local source snapshot used for this release is identified by Git commit `afc6b7f09a1e107f38ae04358ece7ed85f6be7a3`. Selected artifact identities: ```text d679f3e79221aea95c64ef0be177530b02dce25d4745b0247cbae7b677f1ceab training pool 749dbcdea8b244c3609acecddf618d9780559f501da18b09f650dfe10f2fb881 validation pool 0d534fd8eceec72b4fc3179d77afe7d6291766f5831bc4c8b69b9c45463c1656 selection.json 2ac2f6e412ea1e8f67ce6d8395c1c9845270101590d84de1be748e5919833c93 official result JSON ``` The source repository contains the full experiment and reproducibility reports. ## Limitations and risks - The model is optimized for Italian multiple-choice prompts and often emits only an answer letter. It is not a broadly instruction-tuned chat model. - The 44.35% figure is one benchmark result under one exact prompt and backend configuration; it does not measure general Italian language competence. - Exact and fuzzy decontamination reduce known overlap but cannot prove the absence of semantic contamination or benchmark-distribution overfitting. - The public training sources may contain factual errors, social biases, stereotypes, or outdated information that can be inherited by the model. - Outputs outside the trained answer format may be unreliable. ## License and attribution The base model weights are published under Apache-2.0. The included tokenizer and chat template come from a Llama 3 derivative and remain subject to the published Llama 3 terms; the metadata therefore uses the more restrictive `llama3` license identifier. Training datasets and benchmark assets retain their own terms. Some `FinancialSupport` dataset cards do not declare a license, so users should verify those terms before redistribution or commercial use. ## Acknowledgements - [`firekern/zagreus-italic-challenge-xmoons`](https://github.com/firekern/zagreus-italic-challenge-xmoons), the upstream repository - [`mii-llm/zagreus-0.4B-ita`](https://huggingface.co/mii-llm/zagreus-0.4B-ita) - [`swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA`](https://huggingface.co/swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA) - [`mii-llm/Post-Training-Challenge`](https://huggingface.co/spaces/mii-llm/Post-Training-Challenge) - [ITALIC](https://italicbench.it/)