mii-llm/nesso-0.4B-instruct
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: mii-llm/nesso-0.4B-instruct
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-05-25 00:15:21 +08:00
commit 288fe79979
9 changed files with 2510 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

251
README.md Normal file
View File

@@ -0,0 +1,251 @@
---
language:
- it
- en
license: apache-2.0
tags:
- small-language-model
- slm
- edge-ai
- italian
- bilingual
- instruction-following
- chat
- llama
- nanotron
- axolotl
base_model: mii-llm/zagreus-0.4B-ita
model_type: llama
pipeline_tag: text-generation
library_name: transformers
datasets:
- HuggingFaceFW/fineweb-2
- HuggingFaceFW/finepdfs
- bigcode/starcoderdata
- HuggingFaceFW/fineweb
---
# Nesso-0.4B-Instruct
**Nesso-0.4B-Instruct** is a bilingual English/Italian Small Language Model (SLM) optimized for **conversational and instruction-following** use cases. It is post-trained on top of [Zagreus-0.4B-ita](https://huggingface.co/mii-llm/zagreus-0.4B-ita), a foundational model trained from scratch by the [mii-llm](https://mii-llm.ai) community (*Made in Italy – Large Language Model*) on the [Seeweb](https://www.seeweb.it) HPC infrastructure.
Designed for **sovereign edge inference**, Nesso-0.4B-Instruct delivers competitive instruction-following performance in both Italian and English at a fraction of the compute cost of larger models.
> ⚠️ This model is currently at the **SFT (Supervised Fine-Tuning)** stage. DPO (Direct Preference Optimization) training is planned and updated results will be published upon completion.
---
## Model Details
| Property | Value |
|---|---|
| **Architecture** | Modified Llama-3.2 (fully dense) |
| **Parameters** | ~400M |
| **Hidden size** | 960 |
| **Layers** | 32 |
| **Attention heads** | 15 (KV heads: 5) |
| **Context length** | 4096 tokens |
| **Tokenizer** | Llama-3.2 (`vocab_size`: 128,256) |
| **Precision** | BF16 |
| **Languages** | English, Italian |
| **Base model** | mii-llm/zagreus-0.4B-ita |
| **Post-training framework** | Axolotl + FSDP |
| **Chat template** | ChatML |
---
## Training Details
### Base Model Pre-training
`Nesso-0.4B-Instruct` is built on `Zagreus-0.4B-ita`, which was pre-trained on approximately **1 trillion tokens** using the following data mix:
| Dataset | Description |
|---|---|
| [FineWeb (350BT sample)](https://huggingface.co/datasets/HuggingFaceFW/fineweb/viewer/sample-350BT) | ~350B tokens of English web text |
| [FineWeb-2 (ita_Latn)](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2/viewer/ita_Latn) | Italian web text |
| [FinePDFs (ita_Latn)](https://huggingface.co/datasets/HuggingFaceFW/finepdfs/viewer/ita_Latn) | Italian PDF documents |
| [StarCoder Data](https://huggingface.co/datasets/bigcode/starcoderdata) | ~250B tokens of code |
**Token distribution**: ~400B English + ~400B Italian + ~200B Code
**Infrastructure**: 64× NVIDIA A100 GPUs (8 nodes × 8 GPUs) on Seeweb HPC
**Framework**: [Nanotron (mii-llm fork)](https://github.com/mii-llm/nanotron)
### Post-training (SFT)
Post-training was performed using **Axolotl** with FSDP across 4 nodes (32× A100 GPUs).
The instruction dataset is a **proprietary bilingual (English/Italian)** corpus curated by the mii-llm team, with long-term iteration across domains including instruction following, conversational AI, and general knowledge. This dataset is considered a strategic research asset and is not released as open source.
**Key hyperparameters:**
| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW (fused) |
| Learning rate | `1e-3` |
| LR scheduler | Cosine (constant ratio: 0.8, min ratio: 0.3) |
| Epochs | 3 |
| Micro batch size | 1 |
| Gradient accumulation steps | 8 |
| Sequence length | 4096 |
| Max grad norm | 1.0 |
| Precision | BF16 + Flash Attention |
| FSDP strategy | FULL_SHARD |
---
## Chat Template
This model uses the **ChatML** format:
```
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Ciao! Come stai?<|im_end|>
<|im_start|>assistant
```
Special tokens:
- `pad_token`: `<|im_end|>`
- `eos_token`: `<|im_end|>`
---
## Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_id = "mii-llm/nesso-0.4B-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto"
)
messages = [
{"role": "system", "content": "Sei un assistente utile e preciso."},
{"role": "user", "content": "Spiegami cos'è un modello linguistico di grandi dimensioni."}
]
input_ids = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
output = model.generate(
input_ids,
max_new_tokens=512,
temperature=0.7,
do_sample=True
)
print(tokenizer.decode(output[0][input_ids.shape[-1]:], skip_special_tokens=True))
```
---
## Evaluation
### Evaluation Commands
```bash
# Italian benchmarks
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
--tasks m_mmlu_it --num_fewshot 5 --device cuda:0 --batch_size 1
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
--tasks hellaswag_it,arc_it --device cuda:0 --batch_size 1
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
--tasks ifeval-ita --device cuda:0 --batch_size 1
# English benchmarks
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
--tasks mmlu --num_fewshot 5 --device cuda:0 --batch_size 1
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
--tasks hellaswag,arc --device cuda:0 --batch_size 1
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
--tasks ifeval --device cuda:0 --batch_size 1
```
### Results
#### English Benchmarks
| Model | IFEval EN ↑ | ARC EN ↑ | HellaSwag EN ↑ | MMLU EN ↑ | **Avg EN** |
|---|---|---|---|---|---|
| Qwen/Qwen3-0.6B | 0.2758 | 0.3430 | 0.4742 | **0.4013** | 0.3736 |
| **Nesso-0.4B-instruct** | **0.3465** | 0.3003 | 0.4629 | 0.2871 | 0.3492 |
| LiquidAI/LFM2-350M | 0.1595 | 0.2457 | 0.3092 | 0.3445 | 0.2647 |
#### Italian Benchmarks
| Model | IFEval IT ↑ | ARC IT ↑ | HellaSwag IT ↑ | MMLU IT ↑ | **Avg IT** |
|---|---|---|---|---|---|
| Qwen/Qwen3-0.6B | **0.3058** | 0.2729 | 0.3598 | **0.4025** | **0.3353** |
| **Nesso-0.4B-instruct** | 0.2962 | **0.2874** | **0.4076** | 0.2875 | 0.3197 |
| LiquidAI/LFM2-350M | 0.1427 | 0.2464 | 0.2994 | 0.3132 | 0.2504 |
#### Overall
| Model | Avg EN | Avg IT | **Overall** |
|---|---|---|---|
| Qwen/Qwen3-0.6B | 0.3736 | 0.3353 | 0.3545 |
| **Nesso-0.4B-instruct** | 0.3492 | 0.3197 | **0.3345** |
| LiquidAI/LFM2-350M | 0.2647 | 0.2504 | 0.2576 |
### Discussion
Nesso-0.4B-Instruct achieves the **highest IFEval English score (0.3465)** among all compared models — including the larger Qwen3-0.6B — demonstrating strong instruction-following capability. On Italian HellaSwag, it also leads with **0.4076**.
Qwen3-0.6B maintains a clear advantage on MMLU in both languages. MMLU is a widely used benchmark that is frequently represented in training corpora; we believe our results nonetheless demonstrate a highly competitive SLM for English/Italian edge deployment scenarios.
---
## Related Models
| Model | Description |
|---|---|
| [Zagreus-0.4B-ita](https://huggingface.co/mii-llm/zagreus-0.4B-ita) | Base pre-trained model (this model's foundation) |
| [Nesso-0.4B-agentic](https://huggingface.co/mii-llm/nesso-0.4B-agentic) | Optimized for function calling and agentic tasks |
| [Open-Zagreus-0.4B](https://huggingface.co/mii-llm/open-zagreus-0.4B) | Fully open-source SFT variant |
---
## Citation
If you use this model in your research, please cite:
```bibtex
@misc{nesso2025,
title = {The Joy and Pain of Training an LLM from Scratch:
A Technical Report on the Zagreus and Nesso Model Families},
author = {mii-llm community},
year = {2025},
howpublished = {\url{https://github.com/mii-llm/zagreus-nesso-slm}},
}
```
---
## Acknowledgements
- **Antonio Baldassarra** (CEO, Seeweb) and **Marco Cristofanilli** (Head of AI, Seeweb) for infrastructure sponsorship
- The **Hugging Face** team for Nanotron, datatrove, FineWeb, and FineWeb-2
- The **mii-llm** open-source community
---
## License
Released under the **Apache 2.0** license.
> Made with ❤️ in Italy by [mii-llm](https://mii-llm.ai)