107 lines
4.2 KiB
Markdown
107 lines
4.2 KiB
Markdown
---
|
||
base_model: mrcuddle/Mistral-Heretica-12B
|
||
base_model_relation: quantized
|
||
quantized_by: AnkitAI
|
||
library_name: gguf
|
||
pipeline_tag: text-generation
|
||
license: other
|
||
language:
|
||
- en
|
||
tags:
|
||
- gguf
|
||
- llama.cpp
|
||
- quantized
|
||
- mistral
|
||
- merge
|
||
- roleplay
|
||
- creative-writing
|
||
---
|
||
|
||
# Mistral-Heretica-12B-GGUF
|
||
|
||
|
||
|
||
GGUF quantizations of [mrcuddle/Mistral-Heretica-12B](https://huggingface.co/mrcuddle/Mistral-Heretica-12B).
|
||
|
||
Original model: a [Task Arithmetic](https://arxiv.org/abs/2212.04089) merge of Mistral-Nemo-Instruct-2407, [Lumimaid-v0.2-12B](https://huggingface.co/NeverSleep/Lumimaid-v0.2-12B), and [absolute-heresy](https://huggingface.co/MuXodious/Mistral-Nemo-Instruct-2407-absolute-heresy) — tuned for uncensored roleplay and creative writing. 12B params, Mistral architecture.
|
||
|
||
Quantized with [llama.cpp](https://github.com/ggml-org/llama.cpp) (release b9890). Run these in [LM Studio](https://lmstudio.ai/), [Ollama](https://ollama.com/), [KoboldCpp](https://github.com/LostRuins/koboldcpp), [SillyTavern](https://github.com/SillyTavern/SillyTavern), or `llama.cpp` directly.
|
||
|
||
## Download
|
||
|
||
| File | Quant | Size | Description |
|
||
|---|---|---|---|
|
||
| [Q8_0](./Mistral-Heretica-12B-GGUF-Q8_0.gguf) | Q8_0 | 12 GB | Maximum quality, near-lossless. Overkill for most. |
|
||
| [Q6_K](./Mistral-Heretica-12B-GGUF-Q6_K.gguf) | Q6_K | 9.4 GB | Very high quality, effectively lossless. |
|
||
| [Q5_K_M](./Mistral-Heretica-12B-GGUF-Q5_K_M.gguf) | Q5_K_M | 8.1 GB | High quality. Balanced pick. |
|
||
| [Q4_K_M](./Mistral-Heretica-12B-GGUF-Q4_K_M.gguf) | Q4_K_M | 7.0 GB | Good quality, best size/speed tradeoff. **Recommended default.** |
|
||
|
||
F16 (23 GB) is the unquantized conversion — only needed if you want to re-quantize yourself.
|
||
|
||
## Which file should I choose?
|
||
|
||
Pick the largest quant that fits in your RAM/VRAM with room to spare (leave ~1–2 GB for context).
|
||
|
||
- 8 GB RAM/VRAM → **Q4_K_M**
|
||
- 12 GB → **Q5_K_M** or **Q6_K**
|
||
- 16 GB+ → **Q6_K** or **Q8_0**
|
||
|
||
For roleplay/creative use, Q4_K_M is plenty — quality loss over Q8 is negligible in practice.
|
||
|
||
## Prompt format
|
||
|
||
Mistral instruct format (primary):
|
||
|
||
```
|
||
<s>[INST] {prompt} [/INST]
|
||
```
|
||
|
||
Most front-ends (SillyTavern, LM Studio) apply this automatically when you select a **Mistral / Mistral-Nemo** template. The Lumimaid component was also trained with ChatML, so ChatML works too — but Mistral format is the safe default.
|
||
|
||
## Recommended settings
|
||
|
||
Mistral-Nemo (this model's base) is **temperature-sensitive** — high temperature makes it incoherent. Keep it low:
|
||
|
||
| Setting | Value | Note |
|
||
|---|---|---|
|
||
| Temperature | **0.5 – 0.7** | Do not exceed ~1.0. Lower = more coherent, higher = more creative. |
|
||
| Min-P | **0.05** | Primary truncation. Leave Top-P/Top-K off if using Min-P. |
|
||
| Repetition penalty | 1.05 – 1.1 | Or use DRY (0.8 / 1.75 / 2) if your front-end supports it. |
|
||
| Context | up to 16k reliable | Nemo's trained context is 128k but quality degrades well before that. |
|
||
|
||
Start at temp 0.6 / min-p 0.05 and adjust from there.
|
||
|
||
## Run it
|
||
|
||
**llama.cpp:**
|
||
```bash
|
||
llama-cli -m Mistral-Heretica-12B-GGUF-Q4_K_M.gguf --jinja -p "Write a short noir opening."
|
||
```
|
||
|
||
**Download one file with the HF CLI (skip the rest):**
|
||
```bash
|
||
hf download AnkitAI/Mistral-Heretica-12B-GGUF \
|
||
Mistral-Heretica-12B-GGUF-Q4_K_M.gguf --local-dir .
|
||
```
|
||
|
||
**Ollama:**
|
||
```bash
|
||
ollama run hf.co/AnkitAI/Mistral-Heretica-12B-GGUF:Q4_K_M
|
||
```
|
||
|
||
## Verified
|
||
|
||
Q4_K_M smoke-tested: loads correctly, follows instructions (valid JSON output), and produces coherent creative prose. ~5–6 tok/s generation on Apple Silicon (M-series).
|
||
|
||
## License
|
||
|
||
The base model declares no explicit license. This merge inherits terms from its source models — notably **Lumimaid-v0.2-12B, which is CC-BY-NC-4.0 (non-commercial)**. Treat this model as **non-commercial** unless you verify otherwise with the original authors. These are quantizations only; all model rights belong to the original creators.
|
||
|
||
## Credits
|
||
|
||
- Original model: [mrcuddle](https://huggingface.co/mrcuddle)
|
||
- Merge components: [NeverSleep](https://huggingface.co/NeverSleep), [MuXodious](https://huggingface.co/MuXodious), [Mistral AI](https://huggingface.co/mistralai)
|
||
- Quantization tooling: [llama.cpp](https://github.com/ggml-org/llama.cpp)
|
||
|
||
More models: [ankitaglawe.com](https://ankitaglawe.com)
|