---
language:
- it
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: mii-llm/zagreus-0.4B-ita
datasets:
- efederici/pinocchio
- mik3ml/italian-dictionary
- fraug-library/synonyms_dictionnaries
tags:
- italian
- multiple-choice
- knowledge-distillation
- italic-benchmark
model-index:
- name: Zagreus 0.4B ITALIC KL
  results:
  - task:
      type: text-generation
      name: ITALIC multiple-choice generation
    dataset:
      name: ITALIC
      type: Crisp-Unimib/ITALIC
    metrics:
    - type: accuracy
      value: 0.3381
      name: Official zero-shot fast accuracy
---

# Zagreus 0.4B ITALIC KL

This model is a full BF16 fine-tune of
[`mii-llm/zagreus-0.4B-ita`](https://huggingface.co/mii-llm/zagreus-0.4B-ita) for the
[Italian Post-Training Challenge 2026](https://huggingface.co/spaces/mii-llm/Post-Training-Challenge).
It is trained to express the base model's answer-text probabilities through the first generated
multiple-choice letter.

- Training code: https://github.com/GitMarco27/zagreus-italic-kl
- Training data and KL targets: https://huggingface.co/datasets/GitMarco27/zagreus-0.4b-italic-kl-data
- Complete evaluation: [`evaluation/official_results.json`](evaluation/official_results.json)

## Results

| Evaluation | Setting | Questions | Accuracy |
|---|---|---:|---:|
| ITALIC official `run_eval.py` | zero-shot, `fast=true`, temperature 0 | 10,000 | **0.3381** |
| Public domain development set | zero-shot, greedy | 480 | 0.5813 |
| KL held-out set | first-token letter accuracy | 924 records after filtering | 0.6190 |

The official evaluator is pinned to commit `92df420ff686babeea54e217b9f90f8471374916`. Zero-shot evaluation is
configured by setting `few_shot_file: null` in the official ITALIC configuration. This differs from the
repository's five-shot default and was accepted for this challenge submission.

## Method

For each answer option, the base model produces a domain-conditional cloze-PMI score:

```text
s_i = log P(option_i | question) - log P(option_i | domain prompt)
q   = softmax(s / tau)
```

Two balanced option permutations are created for each question. At the first generation position, the model is
trained with the following loss:

```text
L = KL(q || softmax(z_ABCD))
  + lambda * CE(z_ABCD, gold)
  + mu * (-log P(first token is A, B, C, or D))
```

The final hyperparameters are `tau=1`, `lambda=1`, `mu=0.1`, learning rate `1e-4`, batch size 16, five epochs,
weight decay 0.01, and seed 42. The teacher and student both start from the mandatory base model. No external
teacher checkpoint or generation API is used.

## Training data

The 21,847-question source pool contains:

- 9,847 culture and civics questions from `efederici/pinocchio`;
- 6,000 definition questions derived from `mik3ml/italian-dictionary`;
- 6,000 synonym questions derived from `fraug-library/synonyms_dictionnaries`.

The KL build selects 5,000 training questions and 500 question-disjoint held-out questions, then creates two
permutations per question. The 256-token prompt filter leaves 9,294 training records and 924 held-out records.

The source pool is deduplicated and checked against all ITALIC questions with exact normalized matching,
MinHash LSH, and RapidFuzz confirmation. The final audit found no question overlap. ITALIC labels are not used
for training.

The dataset repository records the original terms for each source. The Hugging Face card for the synonym mirror
does not declare a license, so users should review its upstream terms before redistributing derived data.

## Usage

This checkpoint is specialized for four-option multiple-choice questions. The following example uses the same
prompt style as the official fast evaluation:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "GitMarco27/zagreus-0.4b-italic-kl"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "system", "content": "Sei un assistente utile."},
    {
        "role": "user",
        "content": (
            "Rispondi alla seguente domanda a scelta multipla. "
            "Scrivi solo la lettera corretta.\n\n"
            "Qual è la capitale d'Italia?\n\n"
            "A) Milano\nB) Roma\nC) Torino\nD) Napoli\n\nRisposta:"
        ),
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)
output = model.generate(**inputs, do_sample=False, max_new_tokens=1)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Final run

The final run used one NVIDIA A100 80GB PCIe GPU. The allocation lasted 0.7531 hours, including setup, target
generation, training, evaluation, transfer, and verification. Based on the recorded provider price ratio, this
equals 0.3622 H100-equivalent hours.

The immutable base revision is `f463a5beabf054809ee3cc702c711215a5fde690`. The model weights have SHA-256
`5e0284467f727f22000b897a3c40f3e5c24b519de463f71a4212149fe1913c6e`.

## Limitations

This is a benchmark-specific model, not a general instruction model. Training directly supervises the first
generated token, so later tokens may repeat. For normal multiple-choice inference, use greedy decoding with
`max_new_tokens=1`.

The model has about 438 million parameters. Its training data covers lexical, cultural, and civic questions
better than syntax, orthography, and morphology.

## Development tools

Development of this project relied on AI assistants: Claude Fable 5, GPT-5.6-sol, Claude Opus, and
Qwen3.6 27B. They supported code writing, experiment analysis, and documentation. The assistants were not
used to generate training data: all KL targets were produced with the mandatory base model itself, as
described above, and no external API was used for synthetic generation.

## References

- Seveso et al. (2025), [ITALIC: An Italian Culture-Aware Natural Language Benchmark](https://aclanthology.org/2025.naacl-long.68/).
- Holtzman et al. (2021), [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://aclanthology.org/2021.emnlp-main.564/).
- Hinton, Vinyals, and Dean (2015), [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531).