162 lines
6.2 KiB
Markdown
162 lines
6.2 KiB
Markdown
---
|
|
language:
|
|
- it
|
|
- en
|
|
license: apache-2.0
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
base_model: mii-llm/zagreus-0.4B-ita
|
|
datasets:
|
|
- efederici/pinocchio
|
|
- mik3ml/italian-dictionary
|
|
- fraug-library/synonyms_dictionnaries
|
|
tags:
|
|
- italian
|
|
- multiple-choice
|
|
- knowledge-distillation
|
|
- italic-benchmark
|
|
model-index:
|
|
- name: Zagreus 0.4B ITALIC KL
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
name: ITALIC multiple-choice generation
|
|
dataset:
|
|
name: ITALIC
|
|
type: Crisp-Unimib/ITALIC
|
|
metrics:
|
|
- type: accuracy
|
|
value: 0.3381
|
|
name: Official zero-shot fast accuracy
|
|
---
|
|
|
|
# Zagreus 0.4B ITALIC KL
|
|
|
|
This model is a full BF16 fine-tune of
|
|
[`mii-llm/zagreus-0.4B-ita`](https://huggingface.co/mii-llm/zagreus-0.4B-ita) for the
|
|
[Italian Post-Training Challenge 2026](https://huggingface.co/spaces/mii-llm/Post-Training-Challenge).
|
|
It is trained to express the base model's answer-text probabilities through the first generated
|
|
multiple-choice letter.
|
|
|
|
- Training code: https://github.com/GitMarco27/zagreus-italic-kl
|
|
- Training data and KL targets: https://huggingface.co/datasets/GitMarco27/zagreus-0.4b-italic-kl-data
|
|
- Complete evaluation: [`evaluation/official_results.json`](evaluation/official_results.json)
|
|
|
|
## Results
|
|
|
|
| Evaluation | Setting | Questions | Accuracy |
|
|
|---|---|---:|---:|
|
|
| ITALIC official `run_eval.py` | zero-shot, `fast=true`, temperature 0 | 10,000 | **0.3381** |
|
|
| Public domain development set | zero-shot, greedy | 480 | 0.5813 |
|
|
| KL held-out set | first-token letter accuracy | 924 records after filtering | 0.6190 |
|
|
|
|
The official evaluator is pinned to commit `92df420ff686babeea54e217b9f90f8471374916`. Zero-shot evaluation is
|
|
configured by setting `few_shot_file: null` in the official ITALIC configuration. This differs from the
|
|
repository's five-shot default and was accepted for this challenge submission.
|
|
|
|
## Method
|
|
|
|
For each answer option, the base model produces a domain-conditional cloze-PMI score:
|
|
|
|
```text
|
|
s_i = log P(option_i | question) - log P(option_i | domain prompt)
|
|
q = softmax(s / tau)
|
|
```
|
|
|
|
Two balanced option permutations are created for each question. At the first generation position, the model is
|
|
trained with the following loss:
|
|
|
|
```text
|
|
L = KL(q || softmax(z_ABCD))
|
|
+ lambda * CE(z_ABCD, gold)
|
|
+ mu * (-log P(first token is A, B, C, or D))
|
|
```
|
|
|
|
The final hyperparameters are `tau=1`, `lambda=1`, `mu=0.1`, learning rate `1e-4`, batch size 16, five epochs,
|
|
weight decay 0.01, and seed 42. The teacher and student both start from the mandatory base model. No external
|
|
teacher checkpoint or generation API is used.
|
|
|
|
## Training data
|
|
|
|
The 21,847-question source pool contains:
|
|
|
|
- 9,847 culture and civics questions from `efederici/pinocchio`;
|
|
- 6,000 definition questions derived from `mik3ml/italian-dictionary`;
|
|
- 6,000 synonym questions derived from `fraug-library/synonyms_dictionnaries`.
|
|
|
|
The KL build selects 5,000 training questions and 500 question-disjoint held-out questions, then creates two
|
|
permutations per question. The 256-token prompt filter leaves 9,294 training records and 924 held-out records.
|
|
|
|
The source pool is deduplicated and checked against all ITALIC questions with exact normalized matching,
|
|
MinHash LSH, and RapidFuzz confirmation. The final audit found no question overlap. ITALIC labels are not used
|
|
for training.
|
|
|
|
The dataset repository records the original terms for each source. The Hugging Face card for the synonym mirror
|
|
does not declare a license, so users should review its upstream terms before redistributing derived data.
|
|
|
|
## Usage
|
|
|
|
This checkpoint is specialized for four-option multiple-choice questions. The following example uses the same
|
|
prompt style as the official fast evaluation:
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "GitMarco27/zagreus-0.4b-italic-kl"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
|
|
|
|
messages = [
|
|
{"role": "system", "content": "Sei un assistente utile."},
|
|
{
|
|
"role": "user",
|
|
"content": (
|
|
"Rispondi alla seguente domanda a scelta multipla. "
|
|
"Scrivi solo la lettera corretta.\n\n"
|
|
"Qual è la capitale d'Italia?\n\n"
|
|
"A) Milano\nB) Roma\nC) Torino\nD) Napoli\n\nRisposta:"
|
|
),
|
|
},
|
|
]
|
|
|
|
inputs = tokenizer.apply_chat_template(
|
|
messages,
|
|
add_generation_prompt=True,
|
|
return_tensors="pt",
|
|
return_dict=True,
|
|
).to(model.device)
|
|
output = model.generate(**inputs, do_sample=False, max_new_tokens=1)
|
|
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
|
```
|
|
|
|
## Final run
|
|
|
|
The final run used one NVIDIA A100 80GB PCIe GPU. The allocation lasted 0.7531 hours, including setup, target
|
|
generation, training, evaluation, transfer, and verification. Based on the recorded provider price ratio, this
|
|
equals 0.3622 H100-equivalent hours.
|
|
|
|
The immutable base revision is `f463a5beabf054809ee3cc702c711215a5fde690`. The model weights have SHA-256
|
|
`5e0284467f727f22000b897a3c40f3e5c24b519de463f71a4212149fe1913c6e`.
|
|
|
|
## Limitations
|
|
|
|
This is a benchmark-specific model, not a general instruction model. Training directly supervises the first
|
|
generated token, so later tokens may repeat. For normal multiple-choice inference, use greedy decoding with
|
|
`max_new_tokens=1`.
|
|
|
|
The model has about 438 million parameters. Its training data covers lexical, cultural, and civic questions
|
|
better than syntax, orthography, and morphology.
|
|
|
|
## Development tools
|
|
|
|
Development of this project relied on AI assistants: Claude Fable 5, GPT-5.6-sol, Claude Opus, and
|
|
Qwen3.6 27B. They supported code writing, experiment analysis, and documentation. The assistants were not
|
|
used to generate training data: all KL targets were produced with the mandatory base model itself, as
|
|
described above, and no external API was used for synthetic generation.
|
|
|
|
## References
|
|
|
|
- Seveso et al. (2025), [ITALIC: An Italian Culture-Aware Natural Language Benchmark](https://aclanthology.org/2025.naacl-long.68/).
|
|
- Holtzman et al. (2021), [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://aclanthology.org/2021.emnlp-main.564/).
|
|
- Hinton, Vinyals, and Dean (2015), [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531).
|