68 lines
1.4 KiB
Markdown
68 lines
1.4 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
pipeline_tag: text-generation
|
|
|
|
base_model: Avtrkrb/SmolLM2-135M-Instruct-bnb-4bit-reasoning
|
|
|
|
tags:
|
|
|
|
- gguf
|
|
- llama.cpp
|
|
- ollama
|
|
- reasoning
|
|
- smollm2
|
|
---
|
|
|
|
# SmolLM2-135M-Instruct-Reasoning-GGUF
|
|
|
|
GGUF conversions of:
|
|
|
|
Avtrkrb/SmolLM2-135M-Instruct-bnb-4bit-reasoning
|
|
|
|
## Available Quantizations
|
|
|
|
| Quantization | Recommended Use |
|
|
| ------------ | --------------------------------- |
|
|
| Q4_0 | Same as Q4_K_M |
|
|
| Q4_K_M | Best balance of quality and size |
|
|
| Q5_K_M | Better quality |
|
|
| Q6_K | Better quaity smaller than Q8_0 |
|
|
| Q8_0 | Highest quality quantized version |
|
|
|
|
## Chat Template
|
|
|
|
This model uses the standard SmolLM2 chat template.
|
|
|
|
<|im_start|>system
|
|
You are a helpful AI assistant named SmolLM, trained by Hugging Face
|
|
<|im_end|>
|
|
|
|
<|im_start|>user
|
|
Hello
|
|
<|im_end|>
|
|
|
|
<|im_start|>assistant
|
|
|
|
## Ollama Example
|
|
|
|
FROM ./SmolLM2-135M-Instruct-bnb-4bit-reasoning-Q4_K_M.gguf
|
|
|
|
TEMPLATE """
|
|
{{ .Prompt }}
|
|
"""
|
|
|
|
## llama.cpp Example
|
|
|
|
./llama-cli
|
|
-m SmolLM2-135M-Instruct-bnb-4bit-reasoning-Q4_K_M.gguf
|
|
|
|
## Notes
|
|
|
|
When using GGUF versions, ensure the same chat template used during training is applied during inference. Incorrect templates can significantly reduce model quality.
|
|
|
|
## Source Model
|
|
|
|
Avtrkrb/SmolLM2-135M-Instruct-bnb-4bit-reasoning
|