初始化项目，由ModelHub XC社区提供模型
Model: mii-llm/nesso-0.4B-instruct Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
251
README.md
Normal file
251
README.md
Normal file
@@ -0,0 +1,251 @@
|
||||
---
|
||||
language:
|
||||
- it
|
||||
- en
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- small-language-model
|
||||
- slm
|
||||
- edge-ai
|
||||
- italian
|
||||
- bilingual
|
||||
- instruction-following
|
||||
- chat
|
||||
- llama
|
||||
- nanotron
|
||||
- axolotl
|
||||
base_model: mii-llm/zagreus-0.4B-ita
|
||||
model_type: llama
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
datasets:
|
||||
- HuggingFaceFW/fineweb-2
|
||||
- HuggingFaceFW/finepdfs
|
||||
- bigcode/starcoderdata
|
||||
- HuggingFaceFW/fineweb
|
||||
---
|
||||
|
||||
# Nesso-0.4B-Instruct
|
||||
|
||||
**Nesso-0.4B-Instruct** is a bilingual English/Italian Small Language Model (SLM) optimized for **conversational and instruction-following** use cases. It is post-trained on top of [Zagreus-0.4B-ita](https://huggingface.co/mii-llm/zagreus-0.4B-ita), a foundational model trained from scratch by the [mii-llm](https://mii-llm.ai) community (*Made in Italy – Large Language Model*) on the [Seeweb](https://www.seeweb.it) HPC infrastructure.
|
||||
|
||||
Designed for **sovereign edge inference**, Nesso-0.4B-Instruct delivers competitive instruction-following performance in both Italian and English at a fraction of the compute cost of larger models.
|
||||
|
||||
> ⚠️ This model is currently at the **SFT (Supervised Fine-Tuning)** stage. DPO (Direct Preference Optimization) training is planned and updated results will be published upon completion.
|
||||
|
||||
---
|
||||
|
||||
## Model Details
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| **Architecture** | Modified Llama-3.2 (fully dense) |
|
||||
| **Parameters** | ~400M |
|
||||
| **Hidden size** | 960 |
|
||||
| **Layers** | 32 |
|
||||
| **Attention heads** | 15 (KV heads: 5) |
|
||||
| **Context length** | 4096 tokens |
|
||||
| **Tokenizer** | Llama-3.2 (`vocab_size`: 128,256) |
|
||||
| **Precision** | BF16 |
|
||||
| **Languages** | English, Italian |
|
||||
| **Base model** | mii-llm/zagreus-0.4B-ita |
|
||||
| **Post-training framework** | Axolotl + FSDP |
|
||||
| **Chat template** | ChatML |
|
||||
|
||||
---
|
||||
|
||||
## Training Details
|
||||
|
||||
### Base Model Pre-training
|
||||
|
||||
`Nesso-0.4B-Instruct` is built on `Zagreus-0.4B-ita`, which was pre-trained on approximately **1 trillion tokens** using the following data mix:
|
||||
|
||||
| Dataset | Description |
|
||||
|---|---|
|
||||
| [FineWeb (350BT sample)](https://huggingface.co/datasets/HuggingFaceFW/fineweb/viewer/sample-350BT) | ~350B tokens of English web text |
|
||||
| [FineWeb-2 (ita_Latn)](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2/viewer/ita_Latn) | Italian web text |
|
||||
| [FinePDFs (ita_Latn)](https://huggingface.co/datasets/HuggingFaceFW/finepdfs/viewer/ita_Latn) | Italian PDF documents |
|
||||
| [StarCoder Data](https://huggingface.co/datasets/bigcode/starcoderdata) | ~250B tokens of code |
|
||||
|
||||
**Token distribution**: ~400B English + ~400B Italian + ~200B Code
|
||||
**Infrastructure**: 64× NVIDIA A100 GPUs (8 nodes × 8 GPUs) on Seeweb HPC
|
||||
**Framework**: [Nanotron (mii-llm fork)](https://github.com/mii-llm/nanotron)
|
||||
|
||||
### Post-training (SFT)
|
||||
|
||||
Post-training was performed using **Axolotl** with FSDP across 4 nodes (32× A100 GPUs).
|
||||
|
||||
The instruction dataset is a **proprietary bilingual (English/Italian)** corpus curated by the mii-llm team, with long-term iteration across domains including instruction following, conversational AI, and general knowledge. This dataset is considered a strategic research asset and is not released as open source.
|
||||
|
||||
**Key hyperparameters:**
|
||||
|
||||
| Hyperparameter | Value |
|
||||
|---|---|
|
||||
| Optimizer | AdamW (fused) |
|
||||
| Learning rate | `1e-3` |
|
||||
| LR scheduler | Cosine (constant ratio: 0.8, min ratio: 0.3) |
|
||||
| Epochs | 3 |
|
||||
| Micro batch size | 1 |
|
||||
| Gradient accumulation steps | 8 |
|
||||
| Sequence length | 4096 |
|
||||
| Max grad norm | 1.0 |
|
||||
| Precision | BF16 + Flash Attention |
|
||||
| FSDP strategy | FULL_SHARD |
|
||||
|
||||
---
|
||||
|
||||
## Chat Template
|
||||
|
||||
This model uses the **ChatML** format:
|
||||
|
||||
```
|
||||
<|im_start|>system
|
||||
You are a helpful assistant.<|im_end|>
|
||||
<|im_start|>user
|
||||
Ciao! Come stai?<|im_end|>
|
||||
<|im_start|>assistant
|
||||
```
|
||||
|
||||
Special tokens:
|
||||
- `pad_token`: `<|im_end|>`
|
||||
- `eos_token`: `<|im_end|>`
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
import torch
|
||||
|
||||
model_id = "mii-llm/nesso-0.4B-instruct"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto"
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "Sei un assistente utile e preciso."},
|
||||
{"role": "user", "content": "Spiegami cos'è un modello linguistico di grandi dimensioni."}
|
||||
]
|
||||
|
||||
input_ids = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
add_generation_prompt=True,
|
||||
return_tensors="pt"
|
||||
).to(model.device)
|
||||
|
||||
output = model.generate(
|
||||
input_ids,
|
||||
max_new_tokens=512,
|
||||
temperature=0.7,
|
||||
do_sample=True
|
||||
)
|
||||
|
||||
print(tokenizer.decode(output[0][input_ids.shape[-1]:], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Evaluation
|
||||
|
||||
### Evaluation Commands
|
||||
|
||||
```bash
|
||||
# Italian benchmarks
|
||||
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
|
||||
--tasks m_mmlu_it --num_fewshot 5 --device cuda:0 --batch_size 1
|
||||
|
||||
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
|
||||
--tasks hellaswag_it,arc_it --device cuda:0 --batch_size 1
|
||||
|
||||
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
|
||||
--tasks ifeval-ita --device cuda:0 --batch_size 1
|
||||
|
||||
# English benchmarks
|
||||
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
|
||||
--tasks mmlu --num_fewshot 5 --device cuda:0 --batch_size 1
|
||||
|
||||
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
|
||||
--tasks hellaswag,arc --device cuda:0 --batch_size 1
|
||||
|
||||
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
|
||||
--tasks ifeval --device cuda:0 --batch_size 1
|
||||
```
|
||||
|
||||
### Results
|
||||
|
||||
#### English Benchmarks
|
||||
|
||||
| Model | IFEval EN ↑ | ARC EN ↑ | HellaSwag EN ↑ | MMLU EN ↑ | **Avg EN** |
|
||||
|---|---|---|---|---|---|
|
||||
| Qwen/Qwen3-0.6B | 0.2758 | 0.3430 | 0.4742 | **0.4013** | 0.3736 |
|
||||
| **Nesso-0.4B-instruct** | **0.3465** | 0.3003 | 0.4629 | 0.2871 | 0.3492 |
|
||||
| LiquidAI/LFM2-350M | 0.1595 | 0.2457 | 0.3092 | 0.3445 | 0.2647 |
|
||||
|
||||
#### Italian Benchmarks
|
||||
|
||||
| Model | IFEval IT ↑ | ARC IT ↑ | HellaSwag IT ↑ | MMLU IT ↑ | **Avg IT** |
|
||||
|---|---|---|---|---|---|
|
||||
| Qwen/Qwen3-0.6B | **0.3058** | 0.2729 | 0.3598 | **0.4025** | **0.3353** |
|
||||
| **Nesso-0.4B-instruct** | 0.2962 | **0.2874** | **0.4076** | 0.2875 | 0.3197 |
|
||||
| LiquidAI/LFM2-350M | 0.1427 | 0.2464 | 0.2994 | 0.3132 | 0.2504 |
|
||||
|
||||
#### Overall
|
||||
|
||||
| Model | Avg EN | Avg IT | **Overall** |
|
||||
|---|---|---|---|
|
||||
| Qwen/Qwen3-0.6B | 0.3736 | 0.3353 | 0.3545 |
|
||||
| **Nesso-0.4B-instruct** | 0.3492 | 0.3197 | **0.3345** |
|
||||
| LiquidAI/LFM2-350M | 0.2647 | 0.2504 | 0.2576 |
|
||||
|
||||
### Discussion
|
||||
|
||||
Nesso-0.4B-Instruct achieves the **highest IFEval English score (0.3465)** among all compared models — including the larger Qwen3-0.6B — demonstrating strong instruction-following capability. On Italian HellaSwag, it also leads with **0.4076**.
|
||||
|
||||
Qwen3-0.6B maintains a clear advantage on MMLU in both languages. MMLU is a widely used benchmark that is frequently represented in training corpora; we believe our results nonetheless demonstrate a highly competitive SLM for English/Italian edge deployment scenarios.
|
||||
|
||||
---
|
||||
|
||||
## Related Models
|
||||
|
||||
| Model | Description |
|
||||
|---|---|
|
||||
| [Zagreus-0.4B-ita](https://huggingface.co/mii-llm/zagreus-0.4B-ita) | Base pre-trained model (this model's foundation) |
|
||||
| [Nesso-0.4B-agentic](https://huggingface.co/mii-llm/nesso-0.4B-agentic) | Optimized for function calling and agentic tasks |
|
||||
| [Open-Zagreus-0.4B](https://huggingface.co/mii-llm/open-zagreus-0.4B) | Fully open-source SFT variant |
|
||||
|
||||
---
|
||||
|
||||
## Citation
|
||||
|
||||
If you use this model in your research, please cite:
|
||||
|
||||
```bibtex
|
||||
@misc{nesso2025,
|
||||
title = {The Joy and Pain of Training an LLM from Scratch:
|
||||
A Technical Report on the Zagreus and Nesso Model Families},
|
||||
author = {mii-llm community},
|
||||
year = {2025},
|
||||
howpublished = {\url{https://github.com/mii-llm/zagreus-nesso-slm}},
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
- **Antonio Baldassarra** (CEO, Seeweb) and **Marco Cristofanilli** (Head of AI, Seeweb) for infrastructure sponsorship
|
||||
- The **Hugging Face** team for Nanotron, datatrove, FineWeb, and FineWeb-2
|
||||
- The **mii-llm** open-source community
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
Released under the **Apache 2.0** license.
|
||||
|
||||
> Made with ❤️ in Italy by [mii-llm](https://mii-llm.ai)
|
||||
31
chat_template.jinja
Normal file
31
chat_template.jinja
Normal file
@@ -0,0 +1,31 @@
|
||||
{% if tools %}<|im_start|>system
|
||||
{% if messages[0].role == 'system' %}{{ messages[0].content }}
|
||||
|
||||
{% endif %}# Tools
|
||||
|
||||
You may call one or more functions to assist with the user query.
|
||||
|
||||
You are provided with function signatures within <tools></tools> XML tags:
|
||||
<tools>{% for tool in tools %}
|
||||
{{ tool | tojson }}{% endfor %}
|
||||
</tools>
|
||||
|
||||
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
|
||||
<tool_call>
|
||||
{"name": <function-name>, "arguments": <args-json-object>}
|
||||
</tool_call><|im_end|>
|
||||
{% else %}{% if messages[0].role == 'system' %}<|im_start|>system
|
||||
{{ messages[0].content }}<|im_end|>
|
||||
{% endif %}{% endif %}{% for message in messages %}{% if message.content is string %}{% set content = message.content %}{% else %}{% set content = '' %}{% endif %}{% if (message.role == "user") or (message.role == "system" and not loop.first) %}<|im_start|>{{ message.role }}
|
||||
{{ content }}<|im_end|>
|
||||
{% elif message.role == "assistant" %}<|im_start|>{{ message.role }}
|
||||
{{ content }}{% if message.tool_calls %}{% for tool_call in message.tool_calls %}{% if (loop.first and content) or (not loop.first) %}
|
||||
{% endif %}{% if tool_call.function %}{% set tool_call = tool_call.function %}{% endif %}<tool_call>
|
||||
{"name": "{{ tool_call.name }}", "arguments": {% if tool_call.arguments is string %}{{ tool_call.arguments }}{% else %}{{ tool_call.arguments | tojson }}{% endif %}}
|
||||
</tool_call>{% endfor %}{% endif %}<|im_end|>
|
||||
{% elif message.role == "tool" %}{% if loop.first or (messages[loop.index0 - 1].role != "tool") %}<|im_start|>user{% endif %}
|
||||
<tool_response>
|
||||
{{ content }}
|
||||
</tool_response>{% if loop.last or (messages[loop.index0 + 1].role != "tool") %}<|im_end|>
|
||||
{% endif %}{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
|
||||
{% endif %}
|
||||
29
config.json
Normal file
29
config.json
Normal file
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 128000,
|
||||
"eos_token_id": 128256,
|
||||
"head_dim": 64,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 960,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 2560,
|
||||
"max_position_embeddings": 4096,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 15,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 5,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 10000.0,
|
||||
"tie_word_embeddings": true,
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": "4.55.2",
|
||||
"use_cache": false,
|
||||
"vocab_size": 128262
|
||||
}
|
||||
7
generation_config.json
Normal file
7
generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 128000,
|
||||
"do_sample": true,
|
||||
"eos_token_id": 128001,
|
||||
"transformers_version": "4.55.2"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b12a105a0fba31b21dc0aa4098595823ab4a55c836e63fed041b5c93d9e909cf
|
||||
size 2243626016
|
||||
31
special_tokens_map.json
Normal file
31
special_tokens_map.json
Normal file
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<tool_call>",
|
||||
"</tool_call>",
|
||||
"<tool_response>",
|
||||
"</tool_response>"
|
||||
],
|
||||
"bos_token": {
|
||||
"content": "<|begin_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:4a80502037c38583156839ad1486db1e7e5aa0b3f9d47e072f16e6b30a0eb2dd
|
||||
size 17211058
|
||||
2119
tokenizer_config.json
Normal file
2119
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user