53 lines
3.4 KiB
Markdown
53 lines
3.4 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- it
|
|
base_model: mii-llm/zagreus-0.4B-ita
|
|
pipeline_tag: text-generation
|
|
datasets:
|
|
- idealab-cs2/italic-softkd-pool
|
|
tags:
|
|
- italian
|
|
- small-language-model
|
|
- knowledge-distillation
|
|
- italic
|
|
---
|
|
|
|
# zagreus-0.4B-italic-softkd
|
|
|
|
Submission for the [mii-llm Italian Post-Training Challenge](https://huggingface.co/spaces/mii-llm/Post-Training-Challenge). This is [mii-llm/zagreus-0.4B-ita](https://huggingface.co/mii-llm/zagreus-0.4B-ita) (437M, Llama architecture) post-trained for the [ITALIC benchmark](https://github.com/Crisp-Unimib/ITALIC), 10,000 multiple-choice questions on Italian culture and language.
|
|
|
|
| Model | ITALIC full 10K (5-shot fast, temp 0, official harness) |
|
|
|---|---|
|
|
| zagreus-0.4B-ita (base) | 0.2802 |
|
|
| public leaderboard best at submission time | 0.372 |
|
|
| **this model** | **0.4787** (confirmed 0.4784 on an independent rerun) |
|
|
|
|
**Update (2026-08-05):** superseded by [idealab-cs2/zagreus-0.4B-italic-extkd](https://huggingface.co/idealab-cs2/zagreus-0.4B-italic-extkd) — same recipe extended with two more softKD stages (culture pool, then external in-distribution pool), 0.4921/0.4929/0.4932 full-10K.
|
|
|
|
## Training
|
|
|
|
One soft knowledge-distillation run from the base model, within the challenge budget of 10 H100 GPU hours (measured replay: ~4.9 H100h including data generation, teacher labeling and evaluation).
|
|
|
|
For every training question a 3-teacher committee (Mistral-Small-3.2-24B, Gemma-3-27B, Qwen3-32B, each fitting a single H100 in bf16) provides its probability distribution over the answer letters at the answer token. The student is trained in fp32 with a KL term on that distribution (temperature 2.0) plus cross-entropy on the gold letter (weight 0.5). The training pool has 21,606 items: pinocchio-derived knowledge MCQs, committee-labeled grammar MCQs, and generated orthography/morphology MCQs for the categories where public data is scarce. Every pool is decontaminated against the ITALIC test set (exact match plus semantic similarity at cosine 0.80; 0 exact leaks, max similarity 0.7999).
|
|
|
|
Training data is published on the Hub: [italic-softkd-pool](https://huggingface.co/datasets/idealab-cs2/italic-softkd-pool) (the exact training file) (the exact training file; provenance splits knowledge/grammar/orthomorph included as named splits). Training code, the decontamination manifest and the complete report (including what did not work) are in the submission repository: https://github.com/ruggsea/italic-challenge-submission
|
|
|
|
## Evaluation
|
|
|
|
Official [ITALIC harness](https://github.com/Crisp-Unimib/ITALIC), unmodified: vLLM server, fast mode (letter-only answers), 5-shot, temperature 0, accuracy over all 10,000 questions. The chat template is embedded in the tokenizer config; serve the model with its own template.
|
|
|
|
```bash
|
|
vllm serve idealab-cs2/zagreus-0.4B-italic-softkd --chat-template-content-format auto
|
|
```
|
|
|
|
Per-category accuracy: civic education .56, lexicon .52, tourism .50, geography .47, art history .45, history .45, synonyms .47, syntax .42, orthography .39, morphology .36.
|
|
|
|
## Acknowledgments
|
|
|
|
Training, data curation and evaluation by [ruggsea](https://huggingface.co/ruggsea), using compute resources of the Complex Social & Computational Systems (CS²) group, IDea_Lab, University of Graz.
|
|
|
|
## Intended use
|
|
|
|
This model is specialized for Italian multiple-choice question answering in the ITALIC format. It is a 0.4B model: outside this format expect base-model-level capabilities.
|