初始化项目，由ModelHub XC社区提供模型
Model: jc-builds/SmolLM3-3B-Instruct-GGUF Source: Original Platform
This commit is contained in:
ModelHub XC
158
README.md
Normal file
158
README.md
Normal file
@@ -0,0 +1,158 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
license_link: https://www.apache.org/licenses/LICENSE-2.0
|
||||
base_model: HuggingFaceTB/SmolLM3-3B
|
||||
library_name: gguf
|
||||
pipeline_tag: text-generation
|
||||
language:
|
||||
- en
|
||||
- fr
|
||||
- es
|
||||
- de
|
||||
- it
|
||||
- pt
|
||||
tags:
|
||||
- gguf
|
||||
- llama.cpp
|
||||
- quantized
|
||||
- on-device
|
||||
- mobile
|
||||
- apple-silicon
|
||||
- haplo
|
||||
- smollm3
|
||||
inference: false
|
||||
---
|
||||
|
||||
# SmolLM3-3B — GGUF (iPhone-optimized)
|
||||
|
||||
GGUF quantizations of [`HuggingFaceTB/SmolLM3-3B`](https://huggingface.co/HuggingFaceTB/SmolLM3-3B), built and optimized for on-device inference on iPhone, iPad, and Apple Silicon Macs via [llama.cpp](https://github.com/ggerganov/llama.cpp) or apps that wrap it (e.g. [Haplo](https://haploapp.com)).
|
||||
|
||||
> Built and quantized by **jc-builds** for the [Haplo](https://haploapp.com) ecosystem. Original weights © Hugging Face, redistributed under Apache 2.0 per the upstream license.
|
||||
|
||||
## TL;DR
|
||||
|
||||
A 3B-parameter decoder-only transformer with **hybrid reasoning** (toggle "thinking mode" via `/think` or `/no_think` system prompts), **128k context** (with YaRN), and 6 native languages. SmolLM3 is the rare model where **everything is open** — weights, training data mixture, and training configs. At the 3B scale it outperforms Llama-3.2-3B and Qwen2.5-3B across most benchmarks and stays competitive with many 4B-class models.
|
||||
|
||||
## Available quantizations
|
||||
|
||||
| File | Size | Bits/weight | Recommended use |
|
||||
|------|------|-------------|-----------------|
|
||||
| `SmolLM3-3B-Q4_K_M.gguf` | 1.8 GB | 4.8 | **Default — best size/quality tradeoff for phone & laptop** |
|
||||
| `SmolLM3-3B-Q5_K_M.gguf` | 2.1 GB | 5.7 | Slightly better quality, ~17% bigger; good for iPad / Mac |
|
||||
| `SmolLM3-3B-Q8_0.gguf` | 3.0 GB | 8.5 | Near-FP16 quality; only worth it on Apple Silicon Mac |
|
||||
|
||||
**Pick `Q4_K_M`** unless you have a reason not to — it's the sweet spot for on-device on Apple Silicon. Q5_K_M is ~5-10% smarter on hard reasoning prompts but ~20% bigger; Q8_0 is essentially indistinguishable from FP16 but 2× the size of Q4_K_M.
|
||||
|
||||
## Performance on Apple Silicon
|
||||
|
||||
Approximate decode throughput at single-batch greedy decode, 2048-token context. Measured with `llama-cli`.
|
||||
|
||||
| Device | RAM | Q4_K_M tok/s | Notes |
|
||||
|--------|-----|--------------|-------|
|
||||
| iPhone 15 Pro | 8 GB | ~22 tok/s | Smooth chat experience |
|
||||
| iPhone 14 Pro | 6 GB | ~18 tok/s | Comfortable |
|
||||
| iPad Pro M2 | 8 GB | ~45 tok/s | Snappy |
|
||||
| MacBook Pro M3 | 16 GB | ~80 tok/s | Effectively instant |
|
||||
|
||||
> Reference numbers — your throughput will vary with prompt length, KV cache, and what else is running. Q5_K_M and Q8_0 are roughly 15% / 40% slower than Q4_K_M respectively.
|
||||
|
||||
## How to use
|
||||
|
||||
### 1. Haplo (iPhone / iPad / Mac)
|
||||
|
||||
The model appears automatically in Haplo's model browser on Kuzco-1.1.0+ builds. The download URL for Q4_K_M is:
|
||||
|
||||
```
|
||||
https://huggingface.co/jc-builds/SmolLM3-3B-Instruct-GGUF/resolve/main/SmolLM3-3B-Q4_K_M.gguf
|
||||
```
|
||||
|
||||
### 2. llama.cpp (CLI)
|
||||
|
||||
```bash
|
||||
huggingface-cli download jc-builds/SmolLM3-3B-Instruct-GGUF SmolLM3-3B-Q4_K_M.gguf --local-dir .
|
||||
|
||||
./llama-cli \
|
||||
-m SmolLM3-3B-Q4_K_M.gguf \
|
||||
-p "Explain gravity in two sentences." \
|
||||
-n 256 \
|
||||
--temp 0.6 \
|
||||
--top-p 0.95
|
||||
```
|
||||
|
||||
### 3. Ollama
|
||||
|
||||
```bash
|
||||
cat <<'EOF' > Modelfile
|
||||
FROM ./SmolLM3-3B-Q4_K_M.gguf
|
||||
PARAMETER temperature 0.6
|
||||
PARAMETER top_p 0.95
|
||||
EOF
|
||||
ollama create smollm3 -f Modelfile
|
||||
ollama run smollm3
|
||||
```
|
||||
|
||||
## Reasoning modes (think / no_think)
|
||||
|
||||
SmolLM3 ships with hybrid reasoning. You toggle it via system prompt:
|
||||
|
||||
| System prompt | Behavior |
|
||||
|---|---|
|
||||
| `/think` (default) | Emits a `<think>…</think>` reasoning block, then the answer. Better on math / code / multi-step problems. |
|
||||
| `/no_think` | Skips the reasoning block. Use for fast chat / simple Q&A. |
|
||||
|
||||
Example:
|
||||
```
|
||||
<|im_start|>system
|
||||
/no_think<|im_end|>
|
||||
<|im_start|>user
|
||||
Capital of Australia?<|im_end|>
|
||||
<|im_start|>assistant
|
||||
```
|
||||
|
||||
## Sampling defaults
|
||||
|
||||
The upstream team recommends `temperature=0.6` and `top_p=0.95`. The GGUF metadata stores these as the defaults — most clients (llama.cpp, Haplo, Ollama) will use them automatically.
|
||||
|
||||
## Chat template
|
||||
|
||||
The HuggingFaceTB chat template is preserved in the GGUF metadata (so llama.cpp's `--chat-template` flag is *not* required). It uses ChatML-style turns:
|
||||
|
||||
```
|
||||
<|im_start|>system
|
||||
{system}<|im_end|>
|
||||
<|im_start|>user
|
||||
{user}<|im_end|>
|
||||
<|im_start|>assistant
|
||||
{assistant}<|im_end|>
|
||||
```
|
||||
|
||||
## Quantization recipe
|
||||
|
||||
Built with [llama.cpp](https://github.com/ggerganov/llama.cpp) at commit `e43431b` (May 7, 2026).
|
||||
|
||||
1. Downloaded `HuggingFaceTB/SmolLM3-3B` safetensors checkpoint via `huggingface-cli`.
|
||||
2. Converted to GGUF FP16 via `convert_hf_to_gguf.py --outtype f16`.
|
||||
3. Quantized to each target type via `llama-quantize`:
|
||||
```bash
|
||||
llama-quantize SmolLM3-3B-F16.gguf SmolLM3-3B-Q4_K_M.gguf Q4_K_M
|
||||
llama-quantize SmolLM3-3B-F16.gguf SmolLM3-3B-Q5_K_M.gguf Q5_K_M
|
||||
llama-quantize SmolLM3-3B-F16.gguf SmolLM3-3B-Q8_0.gguf Q8_0
|
||||
```
|
||||
|
||||
No imatrix calibration was used — the weights come from the upstream FP16 directly.
|
||||
|
||||
## Original model card
|
||||
|
||||
See the upstream model card for full architecture, training, and benchmark details: [HuggingFaceTB/SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B).
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0, inherited from the original model. Commercial use, modification, and redistribution are permitted. See [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) for the full terms.
|
||||
|
||||
> SmolLM3 by Hugging Face. Licensed under Apache 2.0.
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
- The [Hugging Face SmolLM team](https://huggingface.co/HuggingFaceTB) for the original weights and an unusually generous open-everything release (training data, recipe, configs).
|
||||
- The [llama.cpp](https://github.com/ggerganov/llama.cpp) team for the GGUF format and quantization tooling.
|
||||
- The [Haplo](https://haploapp.com) ecosystem this drop is built for.
|
||||
Reference in New Issue
Block a user