165 lines
5.2 KiB
Markdown
165 lines
5.2 KiB
Markdown
---
|
||
base_model: OdaxAI/DANTE-Mosaic-3.5B
|
||
language:
|
||
- en
|
||
- it
|
||
- es
|
||
- fr
|
||
- de
|
||
- pt
|
||
- ja
|
||
- zh
|
||
- ar
|
||
license: apache-2.0
|
||
tags:
|
||
- llama-cpp
|
||
- gguf
|
||
- quantized
|
||
- text-generation
|
||
- small-language-model
|
||
- medical
|
||
- multilingual
|
||
- 3b
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# DANTE-Mosaic-3.5B — GGUF Quantizations
|
||
|
||
GGUF quantized versions of [OdaxAI/DANTE-Mosaic-3.5B](https://huggingface.co/OdaxAI/DANTE-Mosaic-3.5B) for use with **llama.cpp**, **LM Studio**, **Ollama**, and other GGUF-compatible runtimes.
|
||
|
||
> DANTE-Mosaic-3.5B is a multilingual foundation model by [OdaxAI](https://odaxai.com), trained in **21 A100-GPU-hours** via generative cross-architecture distillation from the trillion-scale Kimi K2 teacher. **#1 on MMLU and MMLU-Pro**, **#1 on GSM8K** (tied Qwen3-4B-Base), **#1 on HellaSwag** — across the standard 3B–4B open-weight basket.
|
||
|
||
---
|
||
|
||
## Available Quantizations
|
||
|
||
| File | Quant | Size | Use case |
|
||
|------|-------|------|----------|
|
||
| `DANTE-Mosaic-3.5B-Q4_K_M.gguf` | Q4_K_M | 1.92 GB | **Recommended** — best quality/size tradeoff |
|
||
| `DANTE-Mosaic-3.5B-Q5_K_M.gguf` | Q5_K_M | 2.21 GB | Higher quality, still runs on 8 GB RAM |
|
||
| `DANTE-Mosaic-3.5B-Q8_0.gguf` | Q8_0 | 3.28 GB | Near-lossless, for GPU inference |
|
||
|
||
---
|
||
|
||
## Quick Start
|
||
|
||
### llama.cpp
|
||
|
||
```bash
|
||
# Install (macOS)
|
||
brew install llama.cpp
|
||
|
||
# Run directly from HuggingFace
|
||
llama-cli -hf OdaxAI/DANTE-Mosaic-3.5B-GGUF:Q4_K_M
|
||
|
||
# Or start a local OpenAI-compatible server
|
||
llama-server -hf OdaxAI/DANTE-Mosaic-3.5B-GGUF:Q4_K_M
|
||
```
|
||
|
||
### Ollama
|
||
|
||
```bash
|
||
ollama run OdaxAI_00/dante-mosaic-3.5b
|
||
```
|
||
|
||
Or via HuggingFace (no account needed):
|
||
|
||
```bash
|
||
ollama run hf.co/OdaxAI/DANTE-Mosaic-3.5B-GGUF:Q4_K_M
|
||
```
|
||
|
||
Or build from the Modelfile included in this repo:
|
||
|
||
```bash
|
||
ollama create dante-mosaic -f Modelfile
|
||
ollama run dante-mosaic
|
||
```
|
||
|
||
### LM Studio
|
||
|
||
1. Open LM Studio → **Discover** tab
|
||
2. Search `OdaxAI/DANTE-Mosaic-3.5B-GGUF`
|
||
3. Download `Q4_K_M` (recommended) and start chatting
|
||
|
||
### Python (llama-cpp-python)
|
||
|
||
```python
|
||
from llama_cpp import Llama
|
||
|
||
llm = Llama.from_pretrained(
|
||
repo_id="OdaxAI/DANTE-Mosaic-3.5B-GGUF",
|
||
filename="DANTE-Mosaic-3.5B-Q4_K_M.gguf",
|
||
n_ctx=4096,
|
||
n_gpu_layers=-1, # offload all layers to GPU if available
|
||
)
|
||
|
||
response = llm(
|
||
"Solve step by step: if a train travels 120 km in 1.5 hours, what is its average speed?",
|
||
max_tokens=256,
|
||
temperature=0.7,
|
||
echo=False,
|
||
)
|
||
print(response["choices"][0]["text"])
|
||
```
|
||
|
||
### Docker
|
||
|
||
```bash
|
||
docker model run hf.co/OdaxAI/DANTE-Mosaic-3.5B-GGUF:Q4_K_M
|
||
```
|
||
|
||
---
|
||
|
||
## Benchmark Results
|
||
|
||
> All scores measured on the released checkpoint with **lm-evaluation-harness v0.4.5** / **bigcode-evaluation-harness** (pinned), full datasets, 1× A100-40GB, BF16, greedy (T=0), seed 42. No subsets, no prompt engineering.
|
||
|
||
### Canonical scores (DANTE-Mosaic-3.5B)
|
||
|
||
| Benchmark | N | Setting | Score |
|
||
|-----------|---|---------|-------|
|
||
| **HellaSwag** | 10 042 | 10-shot, acc_norm | **76.73 %** |
|
||
| **GSM8K** | 1 319 | 8-shot, strict-match | **74.45 %** |
|
||
| **ARC-Challenge** | 1 172 | 25-shot, acc_norm | **62.71 %** |
|
||
| **MMLU** | 14 042 | 5-shot, acc | **59.38 %** |
|
||
| **MBPP** | 374 | pass@1, 0-shot greedy | **42.60 %** |
|
||
| **MMLU-Pro** | 4 500 | 5-shot, exact_match | **39.74 %** |
|
||
| **HumanEval** | 164 | pass@1, greedy | 6.70 % |
|
||
|
||
### Comparison vs. 3B–4B open-weight basket
|
||
|
||
| Benchmark | **DANTE-Mosaic** 3.08B | SmolLM3-3B | Qwen2.5-3B | Llama3.2-3B | Qwen3-1.7B-B | Qwen3-4B-B |
|
||
|-----------|:---:|:---:|:---:|:---:|:---:|:---:|
|
||
| HellaSwag | **76.7** | 76.2 | 74.2 | 75.5 | 60.5 | 74.4 |
|
||
| ARC-Challenge | 62.7 | **65.6** | 59.8 | 58.6 | 55.9 | 62.1 |
|
||
| MMLU★ | **59.4** | 44.1ᶜᶠ | 42.9ᶜᶠ | 41.3ᶜᶠ | 39.1ᶜᶠ | 47.7ᶜᶠ |
|
||
| MMLU-Pro | **39.7** | 32.7 | 31.3 | 25.1 | 30.4 | 41.1 |
|
||
| GSM8K | **74.5** | 67.6 | 70.1 | 25.9 | 65.9 | 74.1 |
|
||
| MBPP⁺ | 42.6 | 52.9 | 52.1 | 38.9 | 59.3 | **63.8** |
|
||
| HumanEval⁺ | 6.7 | 30.5 | 34.1 | 25.0 | 43.3 | **54.9** |
|
||
|
||
★ DANTE evaluated on canonical 5-shot MMLU; peers use harder MMLU-CF (cloze). ᶜᶠ MMLU-CF. ⁺ Harder + variants. Peer numbers from the SmolLM3 technical report (Table 1), single harness run.
|
||
|
||
**Wins vs. the basket:** #1 MMLU (+11.7 pp over next-best), #1 MMLU-Pro, #1 GSM8K (tied Qwen3-4B-Base), #1 HellaSwag. Code (HumanEval/MBPP) is the honest gap — targeted in the next iteration.
|
||
|
||
---
|
||
|
||
## About DANTE-Mosaic-3.5B
|
||
|
||
| | |
|
||
|---|---|
|
||
| **Architecture** | Dense Transformer (SmolLM3-3B base) + DANTE distillation |
|
||
| **Parameters** | ~3.08 B |
|
||
| **Teacher** | Kimi K2 (~1T params MoE, W4A16, vLLM TP=16) |
|
||
| **Training compute** | **~21 A100-GPU-hours** (~27 000× cheaper than SmolLM3-3B pretraining) |
|
||
| **Context length** | 131 072 tokens |
|
||
| **Languages** | EN, IT, ES, FR, DE, PT, JA, ZH, AR |
|
||
| **License** | Apache 2.0 |
|
||
| **Developer** | [OdaxAI](https://odaxai.com) — European foundation model company |
|
||
|
||
Full technical details, training objective (CWCE + TEA + entropy curriculum), and reproducibility scripts on the [original model card](https://huggingface.co/OdaxAI/DANTE-Mosaic-3.5B) and [OdaxAI research page](https://odaxai.com/research/dante-mosaic-3.5b).
|
||
|
||
---
|
||
|
||
*Quantizations produced with [llama.cpp](https://github.com/ggerganov/llama.cpp). Original weights: [OdaxAI/DANTE-Mosaic-3.5B](https://huggingface.co/OdaxAI/DANTE-Mosaic-3.5B).*
|