zagreus-0.4B-xmoons/README.md

---
license: apache-2.0
language:
- it
base_model: mii-llm/zagreus-0.4B-ita
pipeline_tag: text-generation
tags:
- italian
- multiple-choice
- italic
---

# zagreus-0.4B-xmoons

Supervised fine-tune of [`mii-llm/zagreus-0.4B-ita`](https://huggingface.co/mii-llm/zagreus-0.4B-ita)
for the [mii-llm Post-Training Challenge](https://huggingface.co/spaces/mii-llm/Post-Training-Challenge),
evaluated on [ITALIC](https://italicbench.it/) (10,000 Italian multiple-choice questions).

## Result

| | Accuracy |
|---|---|
| baseline `mii-llm/zagreus-0.4B-ita` | 28.88 |
| **this model** | **41.02** (4102/10000) |
| random uniform (options vary 2-5) | 27.02 |

Culture and commonsense **43.40** · Language capability **37.51** · unparsed answers **0**.

The gain is statistically significant on the paired test set: McNemar chi2 = 326.4,
p = 5.9e-73; paired bootstrap 95% CI of the delta [+10.84, +13.46]; Wilson 95% CI
of the accuracy [40.06, 41.99].

## Method

One supervised fine-tuning pass on gold answers — **no distillation**.

- **Benchmark**: the official `italic.jsonl` (10,000) and `5_shots.jsonl` are fetched
  byte-for-byte from the ITALIC repository at a pinned commit and verified by SHA-256.
  Evaluation data is never rebuilt or modified.
- **Data**: Italian MCQ from four public HF datasets (revisions pinned), balanced by
  **inverse frequency** so language and culture are equally represented. The weighting
  derives from the data counts alone — no decision is taken by looking at the test set.
- **Holdout**: a dev split of 7,568 **unique** questions is carved before any
  upsampling and used for recipe selection; the ITALIC test is never used for tuning.
- **Decontamination**: two independent signals (TF-IDF char-ngram cosine >= 0.80 and
  MinHash/LSH word-shingle Jaccard >= 0.70) against both the test set and the official
  few-shot exemplars. Measured overlap of the final training pool: **0** with the
  10,000 test questions, **0** with the 5-shot exemplars.
- **Training**: 1 epoch, loss on the completion only, `max_len` 2048 so the supervised
  target is never truncated, option permutation to remove the positional answer prior,
  few-shot demonstrations drawn from the training pool.
- **Evaluation**: the official `fast` protocol — templates, system message and
  `extract_answer_fast` reproduced verbatim from ITALIC's `run_eval.py`.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "icekern/zagreus-0.4B-xmoons"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [
    {{"role": "system", "content": "Sei un assistente utile."}},
    {{"role": "user", "content": "Rispondi alla seguente domanda a scelta multipla ..."}},
]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
```

The model answers multiple-choice questions with a single letter, following the
ITALIC `fast` prompt format.

## Limitations

- The four weakest categories are all linguistic (morphology 30.71, synonyms 35.43,
  orthography 35.63, syntax 35.77). Public Italian MCQ corpora hold only ~2.5k unique
  grammar questions, so the balanced pool repeats them: the model sees little *unique*
  linguistic information.
- `efederici/pinocchio` and ITALIC draw on overlapping source material (Italian
  public-competition questions). Exact and near-duplicate contamination is removed and
  audited, but source-level overlap cannot be fully excluded.
- The reported score comes from a local HF-generate harness reproducing the official
  `fast` protocol.

## Code

Full pipeline, raw evaluation output and per-example predictions:
https://github.com/firekern/italic-post-training-challenge