bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-23 21:11:05 +08:00
commit c81587a8ed
28 changed files with 261 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
.gitattributes vendored Normal file
View File

@@ -0,0 +1,60 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q4_0_8_8.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q4_0_4_8.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q4_0_4_4.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct-f16.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-NeMo-Minitron-8B-Instruct.imatrix filter=lfs diff=lfs merge=lfs -text

3
Mistral-NeMo-Minitron-8B-Instruct-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d3cd45ccb918d6a98c0875c79b7d633f358a610931d28f8251f4e5222b257cd1
size 3099936448

3
Mistral-NeMo-Minitron-8B-Instruct-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a57b185fbf457b2dc9d9649071dfa826dbb6b8003302621f457fe10dc04ff7f7
size 3976963776

3
Mistral-NeMo-Minitron-8B-Instruct-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:82f988637bf0316c41b27818d9e49ff4c663688fa0c84656b1ca4133e03f512f
size 3684468416

3
Mistral-NeMo-Minitron-8B-Instruct-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5830f077bf9e35d6c1aad6e9630d9ec83dc654de3244c65788693f16acbc9c6b
size 4660430528

3
Mistral-NeMo-Minitron-8B-Instruct-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2225cea310dbea9d5e7a682132fe160b1ad28b4cc417cfeb98aa93082ea9f4c8
size 3333392064

3
Mistral-NeMo-Minitron-8B-Instruct-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dbb1fcce6acc38e107fbe61fad37c1c91d6f1022db3f525aaf884cb0e5759d68
size 3857680064

3
Mistral-NeMo-Minitron-8B-Instruct-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9079e52d21467281d269dca6168053ec65092e30d4a6f81b29cb3c323844b119
size 4537091776

3
Mistral-NeMo-Minitron-8B-Instruct-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4a49b37136027890b0fb5c4215266de1e8092efd0d949c0d896a100d9165b0e0
size 4209149632

3
Mistral-NeMo-Minitron-8B-Instruct-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b1788101fcfe5a0dd5e582d4041392ac282ac8e7f6b888759d3c49bdd9980d14
size 3834218176

3
Mistral-NeMo-Minitron-8B-Instruct-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:687c4b819a2d1ce22e21726d797dfe3037d0d874c4ef6ffb5dab8a99a5ff60a1
size 5006853824

3
Mistral-NeMo-Minitron-8B-Instruct-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7de8460ed23fec81850737356d2fe17f787fdba6a28cfae9d35cf8c7dcd48326
size 4895114944

3
Mistral-NeMo-Minitron-8B-Instruct-Q4_0_4_4.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5d944b23b831b5145a992d214916d1ae2b60d938677d8cc5b5bbaf1bc367dc2e
size 4880369344

3
Mistral-NeMo-Minitron-8B-Instruct-Q4_0_4_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:229ed0e8d6a97d6add91a33c1c17a40d81c055ebd7b696c99fd4533223920aef
size 4880369344

3
Mistral-NeMo-Minitron-8B-Instruct-Q4_0_8_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1a6e5446298e0ede781c3cd07882343610c145f8aa7d9942759dcfed2eee231e
size 4880369344

3
Mistral-NeMo-Minitron-8B-Instruct-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c08dfc5a78d78e8efeae5bbc601a12bc9cce5e5247d1da951c3655003a450daf
size 5543757504

3
Mistral-NeMo-Minitron-8B-Instruct-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4edc37b54cd542af79de298729a0a479c5d273ea6170f0abd610e1133e90d09e
size 5145298624

3
Mistral-NeMo-Minitron-8B-Instruct-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ff58ae862182bcf55c4ea789ae5df50f60ac7826408a76ece37e984e9f62d4c4
size 4911957696

3
Mistral-NeMo-Minitron-8B-Instruct-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4b5790d8e476bfe51713a79328498b004ca3969388c385258bfa94c8f415ce1e
size 6332810944

3
Mistral-NeMo-Minitron-8B-Instruct-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c49571bcfcfe15dec01b54c911c219b0c2eafc876e26fd37f931bb076fb8bee0
size 6001460928

3
Mistral-NeMo-Minitron-8B-Instruct-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0124dd9e3f79844587ecd256fbaccd604308305463f3b0192ff5122b00d41aad
size 5864982208

3
Mistral-NeMo-Minitron-8B-Instruct-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:352163e248ee45e155836fe8203f4e0fb0b95f3bc31342620a7a4fa06340240d
size 6911133376

3
Mistral-NeMo-Minitron-8B-Instruct-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2ae624444e1612e488b68b239fdd5e193fd708b23e2ece3e3993d7002216923f
size 7171180224

3
Mistral-NeMo-Minitron-8B-Instruct-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d1a432771db8414a629bdf1b928b732b50c30ae7d4d05838a9ca0fb3e10c0c9d
size 8948844224

3
Mistral-NeMo-Minitron-8B-Instruct-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a91d231281b8df7bfdc1da9cb5f12d013990ae23ed53c82fe6c9e8b67fcfaa99
size 16836756896

3
Mistral-NeMo-Minitron-8B-Instruct.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1410fcd8a07c4f5fdebc28c0d2d449aab504520856744cf15bfcf401ad1d2f05
size 5784658

125
README.md Normal file
View File

@@ -0,0 +1,125 @@
---
base_model: nvidia/Mistral-NeMo-Minitron-8B-Instruct
pipeline_tag: text-generation
quantized_by: bartowski
---
## Llamacpp imatrix Quantizations of Mistral-NeMo-Minitron-8B-Instruct
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b3901">b3901</a> for quantization.
Original model: https://huggingface.co/nvidia/Mistral-NeMo-Minitron-8B-Instruct
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<extra_id_0>System
{system_prompt}
<extra_id_1>User
{prompt}
<extra_id_1>Assistant
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Mistral-NeMo-Minitron-8B-Instruct-f16.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-f16.gguf) | f16 | 16.84GB | false | Full F16 weights. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q8_0.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q8_0.gguf) | Q8_0 | 8.95GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q6_K_L.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q6_K_L.gguf) | Q6_K_L | 7.17GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q6_K.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q6_K.gguf) | Q6_K | 6.91GB | false | Very high quality, near perfect, *recommended*. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q5_K_L.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q5_K_L.gguf) | Q5_K_L | 6.33GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q5_K_M.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q5_K_M.gguf) | Q5_K_M | 6.00GB | false | High quality, *recommended*. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q5_K_S.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q5_K_S.gguf) | Q5_K_S | 5.86GB | false | High quality, *recommended*. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q4_K_L.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q4_K_L.gguf) | Q4_K_L | 5.54GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q4_K_M.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q4_K_M.gguf) | Q4_K_M | 5.15GB | false | Good quality, default size for must use cases, *recommended*. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q3_K_XL.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q3_K_XL.gguf) | Q3_K_XL | 5.01GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q4_K_S.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q4_K_S.gguf) | Q4_K_S | 4.91GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q4_0.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q4_0.gguf) | Q4_0 | 4.90GB | false | Legacy format, generally not worth using over similarly sized formats |
| [Mistral-NeMo-Minitron-8B-Instruct-Q4_0_8_8.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q4_0_8_8.gguf) | Q4_0_8_8 | 4.88GB | false | Optimized for ARM inference. Requires 'sve' support (see link below). *Don't use on Mac or Windows*. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q4_0_4_8.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q4_0_4_8.gguf) | Q4_0_4_8 | 4.88GB | false | Optimized for ARM inference. Requires 'i8mm' support (see link below). *Don't use on Mac or Windows*. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q4_0_4_4.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q4_0_4_4.gguf) | Q4_0_4_4 | 4.88GB | false | Optimized for ARM inference. Should work well on all ARM chips, pick this if you're unsure. *Don't use on Mac or Windows*. |
| [Mistral-NeMo-Minitron-8B-Instruct-IQ4_XS.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-IQ4_XS.gguf) | IQ4_XS | 4.66GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q3_K_L.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q3_K_L.gguf) | Q3_K_L | 4.54GB | false | Lower quality but usable, good for low RAM availability. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q3_K_M.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q3_K_M.gguf) | Q3_K_M | 4.21GB | false | Low quality. |
| [Mistral-NeMo-Minitron-8B-Instruct-IQ3_M.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-IQ3_M.gguf) | IQ3_M | 3.98GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q2_K_L.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q2_K_L.gguf) | Q2_K_L | 3.86GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q3_K_S.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q3_K_S.gguf) | Q3_K_S | 3.83GB | false | Low quality, not recommended. |
| [Mistral-NeMo-Minitron-8B-Instruct-IQ3_XS.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-IQ3_XS.gguf) | IQ3_XS | 3.68GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Mistral-NeMo-Minitron-8B-Instruct-Q2_K.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-Q2_K.gguf) | Q2_K | 3.33GB | false | Very low quality but surprisingly usable. |
| [Mistral-NeMo-Minitron-8B-Instruct-IQ2_M.gguf](https://huggingface.co/bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF/blob/main/Mistral-NeMo-Minitron-8B-Instruct-IQ2_M.gguf) | IQ2_M | 3.10GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
Some say that this improves the quality, others don't notice any difference. If you use these models PLEASE COMMENT with your findings. I would like feedback that these are actually used and useful so I don't keep uploading quants no one is using.
Thanks!
## Downloading using huggingface-cli
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF --include "Mistral-NeMo-Minitron-8B-Instruct-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Mistral-NeMo-Minitron-8B-Instruct-GGUF --include "Mistral-NeMo-Minitron-8B-Instruct-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Mistral-NeMo-Minitron-8B-Instruct-Q8_0) or download them all in place (./)
## Q4_0_X_X
These are *NOT* for Metal (Apple) offloading, only ARM chips.
If you're using an ARM chip, the Q4_0_X_X quants will have a substantial speedup. Check out Q4_0_4_4 speed comparisons [on the original pull request](https://github.com/ggerganov/llama.cpp/pull/5780#pullrequestreview-21657544660)
To check which one would work best for your ARM chip, you can check [AArch64 SoC features](https://gpages.juszkiewicz.com.pl/arm-socs-table/arm-socs.html) (thanks EloyOn!).
## Which file should I choose?
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset
Thank you ZeroWw for the inspiration to experiment with embed/output
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}