初始化项目，由ModelHub XC社区提供模型

Model: mii-llm/zagreus-0.4B-ita
Source: Original Platform

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,412 @@
+---
+language:
+- it
+- en
+license: apache-2.0
+tags:
+- small-language-model
+- slm
+- edge-ai
+- italian
+- bilingual
+- base-model
+- pretrained
+- llama
+- nanotron
+- from-scratch
+model_type: llama
+pipeline_tag: text-generation
+library_name: transformers
+datasets:
+- HuggingFaceFW/fineweb
+- HuggingFaceFW/fineweb-2
+- HuggingFaceFW/finepdfs
+- bigcode/starcoderdata
+---
+
+# Zagreus-0.4B-ita
+
+**Zagreus-0.4B-ita** is a bilingual English/Italian foundational Small Language Model (SLM) trained **from scratch** by the [mii-llm](https://mii-llm.ai) community (*Made in Italy – Large Language Model*) on the [Seeweb](https://www.seeweb.it) HPC infrastructure.
+
+This is a **base (pre-trained) model** — it is not instruction-tuned and is intended for researchers, developers, and practitioners who want to fine-tune or build upon a high-quality bilingual English/Italian foundation. It serves as the base for the entire [Nesso model family](https://huggingface.co/mii-llm).
+
+The Zagreus family represents one of the few openly released, high-performing small language models dedicated to European Romance languages, trained entirely from first principles with a fully transparent pipeline.
+
+---
+
+## Model Details
+
+| Property | Value |
+|---|---|
+| **Architecture** | Modified Llama-3.2 (fully dense) |
+| **Parameters** | ~400M |
+| **Hidden size** | 960 |
+| **Intermediate size** | 2560 |
+| **Layers** | 32 |
+| **Attention heads** | 15 (KV heads: 5) |
+| **Activation** | SiLU |
+| **Context length** | 4096 tokens |
+| **Tokenizer** | Llama-3.2 (`vocab_size`: 128,256) |
+| **Positional encoding** | RoPE (`theta`: 10000.0) |
+| **Tied embeddings** | Yes |
+| **Precision** | BF16 |
+| **Languages** | English (~400B tokens), Italian (~400B tokens) |
+| **Training tokens** | ~1 trillion |
+| **Training framework** | [Nanotron (mii-llm fork)](https://github.com/mii-llm/nanotron) |
+| **Infrastructure** | 64× NVIDIA A100 GPUs (8 nodes × 8 GPUs), Seeweb HPC |
+
+---
+
+## Training Data
+
+All datasets used are fully open source and released by Hugging Face:
+
+| Dataset | Tokens | Description |
+|---|---|---|
+| [FineWeb (350BT sample)](https://huggingface.co/datasets/HuggingFaceFW/fineweb/viewer/sample-350BT) | ~350B | High-quality English web text |
+| [FineWeb-2 (ita_Latn)](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2/viewer/ita_Latn) | — | Italian web text |
+| [FinePDFs (ita_Latn)](https://huggingface.co/datasets/HuggingFaceFW/finepdfs/viewer/ita_Latn) | — | Italian PDF documents |
+| [StarCoder Data](https://huggingface.co/datasets/bigcode/starcoderdata) | ~250B | Multilingual code |
+
+**Token distribution**: ~400B English + ~400B Italian + ~200B Code ≈ **1 trillion tokens total**
+
+### Tokenization
+
+Raw datasets were tokenized using the **Llama-3.2 tokenizer** (`meta-llama/Llama-3.2-1B`) via the [datatrove](https://github.com/huggingface/datatrove) library. The process ran for over **three weeks of continuous computation** on CPU nodes via Slurm, generating approximately 3–5 TB of tokenized data shards.
+
+---
+
+## Architecture Choice
+
+We adopted a **modified Llama-3.2 fully dense architecture**. The choice of a dense model over Mixture-of-Experts (MoE) in the small-parameter regime (~500M) was deliberate: in tightly constrained capacity settings, routing overhead and expert under-utilization typical of MoE architectures may offset their theoretical efficiency advantages. Dense models provide better compute utilization and more stable training dynamics at this scale.
+
+---
+
+## Pre-training Configuration
+
+Full Nanotron YAML configuration used for training:
+
+```yaml
+checkpoints:
+  checkpoint_interval: 5000
+  checkpoints_path: checkpoints_zagreus_ita_v2
+  checkpoints_path_is_shared_file_system: false
+  resume_checkpoint_path: /training/pretraining/nanotron/checkpoints_zagreus_ita_v2/630000
+  save_final_state: false
+  save_initial_state: false
+data_stages:
+- data:
+    dataset:
+      dataset_folder:
+      - /training/pretraining/fineweb-ita/tokenized
+      - /training/pretraining/fineweb-edu-350BT/000_tokenized_output
+      - /training/pretraining/fineweb-edu-350BT/011_tokenized_output
+      - /training/pretraining/fineweb-edu-350BT/012_tokenized_output
+      - /training/pretraining/fineweb-edu-350BT/013_tokenized_output
+      - /training/pretraining/fineweb-edu-350BT/014_tokenized_output
+      - /training/pretraining/fineweb-edu-350BT/015_tokenized_output
+      - /training/pretraining/fineweb-edu-350BT/016_tokenized_output
+      - /training/pretraining/finepdf-ita/000_tokenized_output
+      - /training/pretraining/starcoder_tokenized/000_tokenized_output
+    num_loading_workers: 0
+    seed: 8
+  name: stable phase
+  start_training_step: 1
+general:
+  benchmark_csv_path: null
+  consumed_train_samples: null
+  ignore_sanity_checks: true
+  project: zagreus
+  run: zagreus-350M
+  seed: 8
+  step: null
+logging:
+  iteration_step_info_interval: 1
+  log_level: info
+  log_level_replica: info
+model:
+  ddp_bucket_cap_mb: 100
+  dtype: bfloat16
+  init_method:
+    std: 0.03227
+  make_vocab_size_divisible_by: 1
+  model_config:
+    bos_token_id: 128000
+    eos_token_id: 128001
+    hidden_act: silu
+    hidden_size: 960
+    initializer_range: 0.02
+    intermediate_size: 2560
+    is_llama_config: true
+    max_position_embeddings: 4096
+    num_attention_heads: 15
+    num_hidden_layers: 32
+    num_key_value_heads: 5
+    pad_token_id: null
+    pretraining_tp: 1
+    rms_norm_eps: 1.0e-05
+    rope_interleaved: false
+    rope_scaling: null
+    rope_theta: 10000.0
+    tie_word_embeddings: true
+    use_cache: true
+    vocab_size: 128256
+optimizer:
+  accumulate_grad_in_fp32: true
+  clip_grad: 1.0
+  learning_rate_scheduler:
+    learning_rate: 0.003
+    lr_decay_starting_step: 750000
+    lr_decay_steps: 50000
+    lr_decay_style: linear
+    lr_warmup_steps: 4000
+    lr_warmup_style: linear
+    min_decay_lr: 1.0e-7
+  optimizer_factory:
+    adam_beta1: 0.9
+    adam_beta2: 0.95
+    adam_eps: 1.0e-08
+    name: adamW
+    torch_adam_is_fused: true
+  weight_decay: 0.01
+  zero_stage: 0
+parallelism:
+  dp: 64
+  expert_parallel_size: 1
+  pp: 1
+  pp_engine: 1f1b
+  recompute_layer: false
+  tp: 1
+  tp_linear_async_communication: true
+  tp_mode: REDUCE_SCATTER
+  tp_recompute_allgather: true
+profiler: null
+tokenizer:
+  tokenizer_max_length: null
+  tokenizer_name_or_path: meta-llama/Llama-3.2-1B
+  tokenizer_revision: null
+tokens:
+  batch_accumulation_per_replica: 1
+  limit_test_batches: 0
+  limit_val_batches: 0
+  micro_batch_size: 4
+  sequence_length: 4096
+  train_steps: 2000000
+  val_check_interval: 5000
+```
+
+### Slurm Launch Script
+
+```bash
+#SBATCH --job-name=350_it
+#SBATCH --account=YOUR_ACCOUNT
+#SBATCH --partition=PARTITION
+#SBATCH --nodes=8
+#SBATCH --gres=gpu:8            # 8 A100 per node = 64 total
+#SBATCH --cpus-per-task=32
+#SBATCH --time=4-00:00:00
+#SBATCH --output=slurm-%j.out
+
+################ 0. Environment ################
+module purge
+module load profile/global
+module load python/3.11 cuda/12.2 cudnn nccl gcc
+
+source /path/to/venv/nanotron/bin/activate
+
+export HF_HOME=/path/to/hf_home
+export TRANSFORMERS_OFFLINE=1
+export HF_HUB_OFFLINE=1
+export HF_DATASETS_OFFLINE=1
+export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
+export NCCL_IB_DISABLE=0
+export NCCL_SOCKET_IFNAME="ib0,eno,eth"
+export WANDB_MODE=disabled
+
+################ 1. Distributed vars ############
+GPUS_PER_NODE=4
+NNODES=$SLURM_JOB_NUM_NODES
+NODE_RANK=$SLURM_NODEID
+MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n1)
+MASTER_PORT=29400
+RDZV_ID=$SLURM_JOB_ID
+
+################ 2. Launch ######################
+srun torchrun \
+      --nnodes $NNODES \
+      --nproc_per_node $GPUS_PER_NODE \
+      --rdzv_id $RDZV_ID \
+      --rdzv_backend c10d \
+      --rdzv_endpoint $MASTER_ADDR:$MASTER_PORT \
+      /path/to/nanotron/run_train.py \
+      --config-file smollm2/zagreus_350M_ita.yaml
+```
+
+### Checkpoint Conversion to Hugging Face Format
+
+```bash
+torchrun --nproc_per_node=1 -m examples.llama.convert_nanotron_to_hf \
+  --checkpoint_path=checkpoints/544000 \
+  --save_path=hf_checkpoints/544000 \
+  --tokenizer_name meta-llama/Llama-3.2-1B
+```
+
+---
+
+## Evaluation
+
+### Evaluation Commands
+
+```bash
+lm-eval --model hf --model_args pretrained=<checkpoint> \
+  --tasks m_mmlu_it --num_fewshot 5 --device cuda:0 --batch_size 1
+
+lm-eval --model hf --model_args pretrained=<checkpoint> \
+  --tasks hellaswag_it,arc_it --device cuda:0 --batch_size 1
+```
+
+### Checkpoint Progression
+
+The table below tracks benchmark scores across training checkpoints, demonstrating steady model improvement throughout pre-training:
+
+| Checkpoint | MMLU IT ↑ | HellaSwag IT ↑ | ARC IT ↑ | **Average** |
+|---|---|---|---|---|
+| v2-95k | 0.2529 | 0.3366 | 0.2652 | 0.2849 |
+| v2-205k | 0.2628 | — | — | 0.2628 |
+| v2-290k | 0.2428 | 0.3492 | 0.2335 | 0.2752 |
+| v2-305k | 0.2598 | 0.3562 | 0.2652 | 0.2937 |
+| v2-365k | 0.2566 | 0.3664 | 0.2712 | **0.2981** |
+| v2-390k | 0.2556 | 0.3438 | 0.2498 | 0.2831 |
+| v2-460k | 0.2540 | 0.3778 | 0.2549 | 0.2956 |
+| v2-520k | 0.2540 | 0.3778 | 0.2549 | 0.2956 |
+| v2-590k | 0.2547 | 0.3651 | 0.2455 | 0.2884 |
+| v2-630k | 0.2562 | 0.3632 | 0.2643 | 0.2946 |
+| v2-680k | 0.2538 | 0.3740 | 0.2592 | 0.2957 |
+| v2-775k | 0.2535 | 0.3750 | 0.2583 | 0.2956 |
+
+---
+
+### Evalita Benchmark
+
+Evalita is a comprehensive Italian NLP evaluation suite benchmarking models across a wide range of linguistic tasks, from classification and extraction to generation and semantic understanding. Evaluation was conducted using the `evalita-mp` task suite from [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).
+
+#### Evaluation Command
+
+```bash
+lm_eval --model hf \
+  --model_args pretrained=mii-llm/zagreus-0.4B-ita \
+  --tasks evalita-mp \
+  --device cuda:0 \
+  --batch_size 1
+```
+
+#### Results
+
+| Task | Metric | Score |
+|---|---|---|
+| **Evalita-LLM (Overall)** | acc | **0.3226** |
+| Admission Test | acc | 0.2137 |
+| FAQ | acc | 0.2681 |
+| Hate Speech Detection | f1 | 0.6056 |
+| Lexical Substitution | f1 | 0.0000 |
+| NER | f1 | 0.1611 |
+| Relation Extraction | f1 | 0.1244 |
+| Sentiment Analysis | f1 | 0.3660 |
+| Summarization (Fanpage) | rouge1 | 0.1947 |
+| Text Entailment | acc | 0.5133 |
+| Word in Context | f1 | 0.4697 |
+
+> Evalita results serve as a zero-shot baseline for the base model. For the comparison between the base model and the SFT variant, see the [Open-Zagreus-0.4B](https://huggingface.co/mii-llm/open-zagreus-0.4B) model card.
+
+---
+
+## Usage
+
+This is a **base model** — it performs causal language modelling (text completion) and is not instruction-tuned. It is best suited as a starting point for fine-tuning.
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_id = "mii-llm/zagreus-0.4B-ita"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto"
+)
+
+# Base model: text completion, not instruction following
+prompt = "L'intelligenza artificiale è una disciplina che"
+
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+output = model.generate(
+    **inputs,
+    max_new_tokens=200,
+    temperature=0.8,
+    do_sample=True,
+    repetition_penalty=1.1
+)
+
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+For instruction-following, use the post-trained variants:
+- 🗣️ [Nesso-0.4B-instruct](https://huggingface.co/mii-llm/nesso-0.4B-instruct) — conversational and instruction following
+- 🤖 [Nesso-0.4B-agentic](https://huggingface.co/mii-llm/nesso-0.4B-agentic) — function calling and agentic tasks
+- 🔓 [Open-Zagreus-0.4B](https://huggingface.co/mii-llm/open-zagreus-0.4B) — fully open-source SFT variant
+
+---
+
+## Full Model Family
+
+### Base Models (Zagreus)
+
+| Model | Languages | HuggingFace |
+|---|---|---|
+| **Zagreus-0.4B-ita** *(this model)* | English + Italian | [🤗 Link]() |
+| [Zagreus-0.4B-spa](https://huggingface.co/mii-llm/zagreus-0.4B-spa) | English + Spanish | [🤗 Link](https://huggingface.co/mii-llm/zagreus-0.4B-spa) |
+| [Zagreus-0.4B-por](https://huggingface.co/mii-llm/zagreus-0.4B-por) | English + Portuguese | [🤗 Link](https://huggingface.co/mii-llm/zagreus-0.4B-por) |
+| [Zagreus-0.4B-fra](https://huggingface.co/mii-llm/zagreus-0.4B-fra) | English + French | [🤗 Link](https://huggingface.co/mii-llm/zagreus-0.4B-fra) |
+
+### Post-trained Models (Nesso)
+
+| Model | Use Case | HuggingFace |
+|---|---|---|
+| [Nesso-0.4B-instruct](https://huggingface.co/mii-llm/nesso-0.4B-instruct) | Conversational / Instruction following | [🤗 Link](https://huggingface.co/mii-llm/nesso-0.4B-instruct) |
+| [Nesso-0.4B-agentic](https://huggingface.co/mii-llm/nesso-0.4B-agentic) | Function calling / Agentic | [🤗 Link](https://huggingface.co/mii-llm/nesso-0.4B-agentic) |
+| [Open-Zagreus-0.4B](https://huggingface.co/mii-llm/open-zagreus-0.4B) | Fully open source | [🤗 Link](https://huggingface.co/mii-llm/open-zagreus-0.4B) |
+
+---
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```bibtex
+@misc{zagreus2025,
+  title        = {The Joy and Pain of Training an LLM from Scratch:
+                  A Technical Report on the Zagreus and Nesso Model Families},
+  author       = {mii-llm community},
+  year         = {2025},
+  howpublished = {\url{https://github.com/mii-llm/zagreus-nesso-slm}},
+}
+```
+
+---
+
+## Acknowledgements
+
+- **Antonio Baldassarra** (CEO, Seeweb) and **Marco Cristofanilli** (Head of AI, Seeweb) for commissioning and sponsoring the infrastructure
+- The **Hugging Face** team for Nanotron, datatrove, FineWeb, FineWeb-2, and FinePDFs
+- The **mii-llm** open-source community for contributions to multilingual evaluation harnesses and the Nanotron fork
+
+---
+
+## License
+
+Released under the **Apache 2.0** license.
+
+> Made with ❤️ in Italy by [mii-llm](https://mii-llm.ai)

config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 128000,
+  "eos_token_id": 128001,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 960,
+  "initializer_range": 0.02,
+  "intermediate_size": 2560,
+  "max_position_embeddings": 4096,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 15,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 5,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.53.2",
+  "use_cache": true,
+  "vocab_size": 128256
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 128000,
+  "eos_token_id": 128001,
+  "transformers_version": "4.53.2"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3625f74bfd7250a6bf6450a2e03bec4f119b4a2f9df64d9e444947064afaaec6
+size 875554600

special_tokens_map.json

@@ -0,0 +1,16 @@
+{
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|end_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}