初始化项目，由ModelHub XC社区提供模型

Model: GitMarco27/zagreus-0.4b-italic-kl Source: Original Platform
2026-07-25 11:26:09 +08:00
commit 30e7dd7c9b
14 changed files with 514 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+evaluation/official_results.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,161 @@
+---
+language:
+- it
+- en
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+base_model: mii-llm/zagreus-0.4B-ita
+datasets:
+- efederici/pinocchio
+- mik3ml/italian-dictionary
+- fraug-library/synonyms_dictionnaries
+tags:
+- italian
+- multiple-choice
+- knowledge-distillation
+- italic-benchmark
+model-index:
+- name: Zagreus 0.4B ITALIC KL
+  results:
+  - task:
+      type: text-generation
+      name: ITALIC multiple-choice generation
+    dataset:
+      name: ITALIC
+      type: Crisp-Unimib/ITALIC
+    metrics:
+    - type: accuracy
+      value: 0.3381
+      name: Official zero-shot fast accuracy
+---
+
+# Zagreus 0.4B ITALIC KL
+
+This model is a full BF16 fine-tune of
+[`mii-llm/zagreus-0.4B-ita`](https://huggingface.co/mii-llm/zagreus-0.4B-ita) for the
+[Italian Post-Training Challenge 2026](https://huggingface.co/spaces/mii-llm/Post-Training-Challenge).
+It is trained to express the base model's answer-text probabilities through the first generated
+multiple-choice letter.
+
+- Training code: https://github.com/GitMarco27/zagreus-italic-kl
+- Training data and KL targets: https://huggingface.co/datasets/GitMarco27/zagreus-0.4b-italic-kl-data
+- Complete evaluation: [`evaluation/official_results.json`](evaluation/official_results.json)
+
+## Results
+
+| Evaluation | Setting | Questions | Accuracy |
+|---|---|---:|---:|
+| ITALIC official `run_eval.py` | zero-shot, `fast=true`, temperature 0 | 10,000 | **0.3381** |
+| Public domain development set | zero-shot, greedy | 480 | 0.5813 |
+| KL held-out set | first-token letter accuracy | 924 records after filtering | 0.6190 |
+
+The official evaluator is pinned to commit `92df420ff686babeea54e217b9f90f8471374916`. Zero-shot evaluation is
+configured by setting `few_shot_file: null` in the official ITALIC configuration. This differs from the
+repository's five-shot default and was accepted for this challenge submission.
+
+## Method
+
+For each answer option, the base model produces a domain-conditional cloze-PMI score:
+
+```text
+s_i = log P(option_i | question) - log P(option_i | domain prompt)
+q   = softmax(s / tau)
+```
+
+Two balanced option permutations are created for each question. At the first generation position, the model is
+trained with the following loss:
+
+```text
+L = KL(q || softmax(z_ABCD))
+  + lambda * CE(z_ABCD, gold)
+  + mu * (-log P(first token is A, B, C, or D))
+```
+
+The final hyperparameters are `tau=1`, `lambda=1`, `mu=0.1`, learning rate `1e-4`, batch size 16, five epochs,
+weight decay 0.01, and seed 42. The teacher and student both start from the mandatory base model. No external
+teacher checkpoint or generation API is used.
+
+## Training data
+
+The 21,847-question source pool contains:
+
+- 9,847 culture and civics questions from `efederici/pinocchio`;
+- 6,000 definition questions derived from `mik3ml/italian-dictionary`;
+- 6,000 synonym questions derived from `fraug-library/synonyms_dictionnaries`.
+
+The KL build selects 5,000 training questions and 500 question-disjoint held-out questions, then creates two
+permutations per question. The 256-token prompt filter leaves 9,294 training records and 924 held-out records.
+
+The source pool is deduplicated and checked against all ITALIC questions with exact normalized matching,
+MinHash LSH, and RapidFuzz confirmation. The final audit found no question overlap. ITALIC labels are not used
+for training.
+
+The dataset repository records the original terms for each source. The Hugging Face card for the synonym mirror
+does not declare a license, so users should review its upstream terms before redistributing derived data.
+
+## Usage
+
+This checkpoint is specialized for four-option multiple-choice questions. The following example uses the same
+prompt style as the official fast evaluation:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "GitMarco27/zagreus-0.4b-italic-kl"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
+
+messages = [
+    {"role": "system", "content": "Sei un assistente utile."},
+    {
+        "role": "user",
+        "content": (
+            "Rispondi alla seguente domanda a scelta multipla. "
+            "Scrivi solo la lettera corretta.\n\n"
+            "Qual è la capitale d'Italia?\n\n"
+            "A) Milano\nB) Roma\nC) Torino\nD) Napoli\n\nRisposta:"
+        ),
+    },
+]
+
+inputs = tokenizer.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    return_tensors="pt",
+    return_dict=True,
+).to(model.device)
+output = model.generate(**inputs, do_sample=False, max_new_tokens=1)
+print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+## Final run
+
+The final run used one NVIDIA A100 80GB PCIe GPU. The allocation lasted 0.7531 hours, including setup, target
+generation, training, evaluation, transfer, and verification. Based on the recorded provider price ratio, this
+equals 0.3622 H100-equivalent hours.
+
+The immutable base revision is `f463a5beabf054809ee3cc702c711215a5fde690`. The model weights have SHA-256
+`5e0284467f727f22000b897a3c40f3e5c24b519de463f71a4212149fe1913c6e`.
+
+## Limitations
+
+This is a benchmark-specific model, not a general instruction model. Training directly supervises the first
+generated token, so later tokens may repeat. For normal multiple-choice inference, use greedy decoding with
+`max_new_tokens=1`.
+
+The model has about 438 million parameters. Its training data covers lexical, cultural, and civic questions
+better than syntax, orthography, and morphology.
+
+## Development tools
+
+Development of this project relied on AI assistants: Claude Fable 5, GPT-5.6-sol, Claude Opus, and
+Qwen3.6 27B. They supported code writing, experiment analysis, and documentation. The assistants were not
+used to generate training data: all KL targets were produced with the mandatory base model itself, as
+described above, and no external API was used for synthetic generation.
+
+## References
+
+- Seveso et al. (2025), [ITALIC: An Italian Culture-Aware Natural Language Benchmark](https://aclanthology.org/2025.naacl-long.68/).
+- Holtzman et al. (2021), [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://aclanthology.org/2021.emnlp-main.564/).
+- Hinton, Vinyals, and Dean (2015), [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531).
--- a/UPLOAD_MANIFEST.json
+++ b/UPLOAD_MANIFEST.json
@@ -0,0 +1,65 @@
+{
+  "schema": 1,
+  "files": [
+    {
+      "path": "README.md",
+      "bytes": 6329,
+      "sha256": "fb116cd24701c6d0c08a5bd55c0fc613b5f8ad1d462591097cc5da2feebd5f70"
+    },
+    {
+      "path": "chat_template.jinja",
+      "bytes": 414,
+      "sha256": "7ceb896ae93b545e4b52c21534de63aae63588b7129120ec3df3a8af0b2f21f6"
+    },
+    {
+      "path": "config.json",
+      "bytes": 735,
+      "sha256": "8b3d600450368daa7023bd374ebe1305f011c5875bb81a5a8bf8c8a64af0b2d8"
+    },
+    {
+      "path": "evaluation/config.yaml",
+      "bytes": 427,
+      "sha256": "1b283ba62494fd53a264c20511e9457f92e8cec86c6a1f37757a1cadd8589c15"
+    },
+    {
+      "path": "evaluation/domain_dev.json",
+      "bytes": 747,
+      "sha256": "21c8919e71d79400ae022602b9787211674985fe46bc2af0032e8e09a094b5d1"
+    },
+    {
+      "path": "evaluation/official_results.json",
+      "bytes": 18695104,
+      "sha256": "a8cecd85d6577725611a6ce7cef873fabe41028a66a12276e99238646892f383"
+    },
+    {
+      "path": "evaluation/provenance.json",
+      "bytes": 1561,
+      "sha256": "e3cba34436c3f36454258b3b5c497377458358c7df319cdbbfe31f74a7c91d8b"
+    },
+    {
+      "path": "generation_config.json",
+      "bytes": 143,
+      "sha256": "57a77228b0aef0671bb5bfa6a72060465dd87e6c0d64c4828e374314e2ed2fe7"
+    },
+    {
+      "path": "model.safetensors",
+      "bytes": 875554600,
+      "sha256": "5e0284467f727f22000b897a3c40f3e5c24b519de463f71a4212149fe1913c6e"
+    },
+    {
+      "path": "tokenizer.json",
+      "bytes": 17210099,
+      "sha256": "a65c6c5f9764771aa485e6a1f5e63d7d9af8477fe0777148c17476ecb2e09a05"
+    },
+    {
+      "path": "tokenizer_config.json",
+      "bytes": 374,
+      "sha256": "7e447094613755fff6f95c963bfd91286a356d4d78866915543039ccab0f062d"
+    },
+    {
+      "path": "training_run.json",
+      "bytes": 2014,
+      "sha256": "ddad603e6bd1cbe63210ba2f575dc2bd5d00bd32a525a8f246b7f5bd52b60b3c"
+    }
+  ]
+}
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,5 @@
+{{- bos_token }}{%- for message in messages %}{{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>
+
+' }}{%- if message['role'] == 'assistant' %}{% generation %}{{- message['content'] }}{{- '<|eot_id|>' }}{% endgeneration %}{%- else %}{{- message['content'] }}{{- '<|eot_id|>' }}{%- endif %}{%- endfor %}{%- if add_generation_prompt %}{{- '<|start_header_id|>assistant<|end_header_id|>
+
+' }}{%- endif %}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 128000,
+  "dtype": "bfloat16",
+  "eos_token_id": 128001,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 960,
+  "initializer_range": 0.02,
+  "intermediate_size": 2560,
+  "max_position_embeddings": 4096,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 15,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 5,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.13.0",
+  "use_cache": true,
+  "vocab_size": 128256
+}
--- a/evaluation/config.yaml
+++ b/evaluation/config.yaml
@@ -0,0 +1,22 @@
+provider: custom_openai
+model: zagreus-0.4b-italic-kl
+api_key: EMPTY
+temperature: 0.0
+max_tokens: 350
+system_message: null
+provider_kwargs:
+  base_url: http://127.0.0.1:8000/v1
+rate_limiting:
+  enabled: false
+  requests_per_minute: 15
+limit: null
+fast: true
+data:
+  data_file: ./italic.jsonl
+  few_shot_file: null
+  output_dir: results
+num_threads: 30
+auto_resume: true
+checkpointing:
+  enabled: true
+  checkpoint_interval: 50
--- a/evaluation/domain_dev.json
+++ b/evaluation/domain_dev.json
@@ -0,0 +1,42 @@
+{
+  "schema": 1,
+  "model": "zagreus-0.4b-italic-kl",
+  "data": "domain_pool_dev.jsonl",
+  "n": 480,
+  "shots": 0,
+  "seed": 42,
+  "accuracy": 0.58125,
+  "prediction_distribution": {
+    "B": 139,
+    "C": 92,
+    "A": 147,
+    "D": 102
+  },
+  "gold_distribution": {
+    "A": 120,
+    "C": 132,
+    "B": 120,
+    "D": 108
+  },
+  "per_category": {
+    "lexicon": {
+      "accuracy": 0.85,
+      "n": 120
+    },
+    "synonyms_and_antonyms": {
+      "accuracy": 0.8083333333333333,
+      "n": 120
+    },
+    "culture": {
+      "accuracy": 0.375,
+      "n": 120
+    },
+    "civic_education": {
+      "accuracy": 0.2916666666666667,
+      "n": 120
+    }
+  },
+  "empty_predictions": 0,
+  "invalid_predictions": 0,
+  "wall_seconds": 18.380833387374878
+}
--- a/evaluation/official_results.json
+++ b/evaluation/official_results.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8cecd85d6577725611a6ce7cef873fabe41028a66a12276e99238646892f383
+size 18695104
--- a/evaluation/provenance.json
+++ b/evaluation/provenance.json
@@ -0,0 +1,43 @@
+{
+  "schema": 1,
+  "model": "zagreus-0.4b-italic-kl",
+  "model_weights_sha256": "5e0284467f727f22000b897a3c40f3e5c24b519de463f71a4212149fe1913c6e",
+  "base_model": "mii-llm/zagreus-0.4B-ita",
+  "base_revision": "f463a5beabf054809ee3cc702c711215a5fde690",
+  "evaluator": "Crisp-Unimib/ITALIC",
+  "evaluator_revision": "92df420ff686babeea54e217b9f90f8471374916",
+  "evaluator_sha256": "ed4e4c63109245146040b234018ffb720ed088e6d55aef074cd9755dc3ed94f1",
+  "italic_data_sha256": "ca877846f19a6d781ba151382e4f43b10efad1a7e375b5e0c50b047a3917e0af",
+  "original_official_results_sha256": "eb5d61796c868d583c8b37794589ce1ac24984a08ccae896ee96d3996889c0b0",
+  "public_official_results_sha256": "a8cecd85d6577725611a6ce7cef873fabe41028a66a12276e99238646892f383",
+  "public_result_transformations": [
+    "Changed only the local served-model alias to zagreus-0.4b-italic-kl."
+  ],
+  "evaluation": {
+    "questions": 10000,
+    "zero_shot": true,
+    "fast": true,
+    "temperature": 0.0,
+    "metrics": {
+      "accuracy": 0.3381,
+      "accuracy:std": 0.4730627759610768
+    },
+    "prediction_counts": {
+      "A": 3780,
+      "B": 2611,
+      "C": 2400,
+      "D": 1209
+    },
+    "empty_parses": 0,
+    "started_at": "2026-07-15T07:38:58Z",
+    "ended_at": "2026-07-15T07:46:32Z",
+    "wall_seconds": 454
+  },
+  "final_run": {
+    "hardware": "NVIDIA A100 80GB PCIe",
+    "resource_wall_hours": 0.7531477777777777,
+    "h100_equivalent_hours": 0.3622406266820453,
+    "estimated_provider_cost_usd": 1.046875411111111,
+    "within_10_h100_equivalent_hours": true
+  }
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 128000,
+  "eos_token_id": [
+    128009,
+    128001
+  ],
+  "transformers_version": "5.13.0"
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e0284467f727f22000b897a3c40f3e5c24b519de463f71a4212149fe1913c6e
+size 875554600
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a65c6c5f9764771aa485e6a1f5e63d7d9af8477fe0777148c17476ecb2e09a05
+size 17210099
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|end_of_text|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 131072,
+  "pad_token": "<|finetune_right_pad_id|>",
+  "tokenizer_class": "TokenizersBackend"
+}
--- a/training_run.json
+++ b/training_run.json
@@ -0,0 +1,74 @@
+{
+  "schema": 1,
+  "method": "cloze_pmi_to_first_token_kl",
+  "model": "mii-llm/zagreus-0.4B-ita",
+  "requested_revision": "f463a5beabf054809ee3cc702c711215a5fde690",
+  "resolved_revision": "f463a5beabf054809ee3cc702c711215a5fde690",
+  "data": "kl",
+  "data_sha256": {
+    "train": "ad9f071607f8f17e0bd624dfa1fdd30158f6a1eae8f243aeeed47f5212094b21",
+    "heldout": "ca81ef92405c7ab895f00cac27a4a5d69b2c550db782f90780882a0f7a84cd99"
+  },
+  "hyperparameters": {
+    "tau": 1.0,
+    "lambda_gold_ce": 1.0,
+    "mu_letter_mass": 0.1,
+    "learning_rate": 0.0001,
+    "epochs": 5,
+    "batch_size": 16,
+    "zero_shot": true,
+    "max_question_tokens": 256,
+    "seed": 42,
+    "weight_decay": 0.01
+  },
+  "counts_after_length_filter": {
+    "train_records": 9294,
+    "heldout_records": 924,
+    "optimizer_steps": 2905
+  },
+  "heldout_history": [
+    {
+      "epoch": 0,
+      "p_eq_q_cloze": 0.2564935064935065,
+      "letter_accuracy_vs_gold": 0.2510822510822511,
+      "letter_mass": 0.12870299325876938
+    },
+    {
+      "epoch": 1,
+      "p_eq_q_cloze": 0.2694805194805195,
+      "letter_accuracy_vs_gold": 0.26731601731601734,
+      "letter_mass": 0.9791770796755176
+    },
+    {
+      "epoch": 2,
+      "p_eq_q_cloze": 0.2911255411255411,
+      "letter_accuracy_vs_gold": 0.27164502164502163,
+      "letter_mass": 0.9890340902072526
+    },
+    {
+      "epoch": 3,
+      "p_eq_q_cloze": 0.43398268398268397,
+      "letter_accuracy_vs_gold": 0.5227272727272727,
+      "letter_mass": 0.9932343846275693
+    },
+    {
+      "epoch": 4,
+      "p_eq_q_cloze": 0.525974025974026,
+      "letter_accuracy_vs_gold": 0.6071428571428571,
+      "letter_mass": 0.9928273122548025
+    },
+    {
+      "epoch": 5,
+      "p_eq_q_cloze": 0.5292207792207793,
+      "letter_accuracy_vs_gold": 0.6190476190476191,
+      "letter_mass": 0.9943214587835006
+    }
+  ],
+  "wall_seconds": 393.98932337760925,
+  "environment": {
+    "python": "3.12.3",
+    "torch": "2.13.0+cu129",
+    "transformers": "5.13.0",
+    "device": "cuda"
+  }
+}