99 lines
4.0 KiB
Markdown
99 lines
4.0 KiB
Markdown
|
---
|
|||
|
|
language:
|
|||
|
|
- it
|
|||
|
|
base_model:
|
|||
|
|
- mii-llm/zagreus-0.4B-ita
|
|||
|
|
datasets:
|
|||
|
|
- antoniogr7/italic-lexical-elicitation
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
library_name: transformers
|
|||
|
|
tags:
|
|||
|
|
- italian
|
|||
|
|
- italic
|
|||
|
|
- elicitation
|
|||
|
|
- multiple-choice
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# zagreus-0.4B-ita-elicit
|
|||
|
|
|
|||
|
|
An **elicitation** fine-tune of [`mii-llm/zagreus-0.4B-ita`](https://huggingface.co/mii-llm/zagreus-0.4B-ita)
|
|||
|
|
for the [ITALIC](https://github.com/Crisp-Unimib/ITALIC) benchmark. It does **not** inject new
|
|||
|
|
knowledge (the model is only 0.4B); it surfaces the lexical/semantic knowledge the base model
|
|||
|
|
*already* holds and routes it onto the answer-letter channel, while removing the base model's
|
|||
|
|
heavy answer-letter bias. **No ITALIC data is used for training** — the benchmark is only read
|
|||
|
|
to guard against leakage.
|
|||
|
|
|
|||
|
|
Net result: **ITALIC 0.280 → 0.3126** (fast, zero-shot, greedy, n=10,000).
|
|||
|
|
|
|||
|
|
## Results
|
|||
|
|
|
|||
|
|
ITALIC, **fast / zero-shot**, greedy, n=10,000 (0 unparsed). Base = `mii-llm/zagreus-0.4B-ita`.
|
|||
|
|
|
|||
|
|
| | base | **this model** | Δ |
|
|||
|
|
|---|---|---|---|
|
|||
|
|
| **overall** | 0.280 | **0.3126** | **+3.2** |
|
|||
|
|
| language capability (macro) | 0.2705 | 0.3133 | +4.3 |
|
|||
|
|
| culture and commonsense (macro) | 0.2868 | 0.3121 | +2.5 |
|
|||
|
|
| `lexicon` | 0.2584 | **0.3841** | **+12.6** |
|
|||
|
|
| `synonyms_and_antonyms` | 0.2915 | 0.3275 | +3.6 |
|
|||
|
|
|
|||
|
|
The intended target (`lexicon`) moves the most. The culture categories rise modestly too — not
|
|||
|
|
from injected facts but from repairing the shared letter-emission head; per-fact culture remains
|
|||
|
|
a wall for a 0.4B, so those gains are small.
|
|||
|
|
|
|||
|
|
## Method (brief)
|
|||
|
|
|
|||
|
|
Full fine-tune with an elicitation objective, per rendered option ordering:
|
|||
|
|
|
|||
|
|
1. **`CE(gold)`** — commit probability mass to the correct answer letter (peaking anchor, the
|
|||
|
|
primary driver).
|
|||
|
|
2. **`λ·KL(q ‖ p)`** — distill a **PMI-cloze teacher** computed over the option *text* (`q`) onto
|
|||
|
|
the answer-letter logits (`p`), down-weighted (`λ=0.3`) to avoid flattening.
|
|||
|
|
3. **format term** — keep probability mass on the option-letter tokens.
|
|||
|
|
|
|||
|
|
Options are re-permuted every epoch so the model must bind meaning to the letter it lands on, not
|
|||
|
|
to a position. Training data is a synthetic Italian lexical/semantic MCQ pool built from **Italian
|
|||
|
|
Wiktionary** (kaikki.org): [`antoniogr7/italic-lexical-elicitation`](https://huggingface.co/datasets/antoniogr7/italic-lexical-elicitation).
|
|||
|
|
Trained and evaluated in **bfloat16**.
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
**Serve/evaluate zero-shot.** The model was trained and measured 0-shot; few-shot demos shift the
|
|||
|
|
emission distribution it was tuned for and regress the score. The ITALIC chat template is baked
|
|||
|
|
into the tokenizer.
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import torch
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
name = "antoniogr7/zagreus-0.4B-ita-elicit"
|
|||
|
|
tok = AutoTokenizer.from_pretrained(name)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16).cuda().eval()
|
|||
|
|
|
|||
|
|
question = "Qual è il sinonimo di «celere»?"
|
|||
|
|
options = {"A": "lento", "B": "rapido", "C": "grande", "D": "scuro"}
|
|||
|
|
body = question + "\n\n" + "\n".join(f"{k}) {v}" for k, v in options.items()) + "\n\nRisposta:\n"
|
|||
|
|
msgs = [{"role": "system", "content": "Sei un assistente utile."},
|
|||
|
|
{"role": "user", "content": body}]
|
|||
|
|
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").cuda()
|
|||
|
|
out = model.generate(ids, max_new_tokens=2, do_sample=False)
|
|||
|
|
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).strip()) # -> "B"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
For faithful benchmark scoring use the official ITALIC harness (vLLM + `run_eval.py`, `fast: true`,
|
|||
|
|
`few_shot_file: null`).
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
- **0.4B model.** Only the lexical/semantic slice of ITALIC (~19.5%) is addressable by elicitation;
|
|||
|
|
culture is per-fact knowledge (a wall) and grammar is a capacity wall. The realistic ceiling of
|
|||
|
|
this approach is ~0.31–0.33 overall.
|
|||
|
|
- **Zero-shot only** (see Usage).
|
|||
|
|
- Italian only.
|
|||
|
|
|
|||
|
|
## License & attribution
|
|||
|
|
|
|||
|
|
The model inherits the license of the base model `mii-llm/zagreus-0.4B-ita`. The training dataset
|
|||
|
|
is derived from Italian Wiktionary via kaikki.org and released under **CC BY-SA 4.0** (attribution:
|
|||
|
|
Wiktionary contributors; extraction by wiktextract / kaikki.org).
|