bartowski/Mistral-Nemo-Instruct-2407-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Mistral-Nemo-Instruct-2407-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-02 19:42:12 +08:00
commit e15950740b
29 changed files with 277 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
.gitattributes vendored Normal file
View File

@@ -0,0 +1,61 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-f32.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407.imatrix filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q4_0_8_8.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q4_0_4_8.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q4_0_4_4.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-Nemo-Instruct-2407-f16.gguf filter=lfs diff=lfs merge=lfs -text

3
Mistral-Nemo-Instruct-2407-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:72194914c723b1126f104d62b73c6da6adb4e47686bf2dfafa5def42dcbb41e9
size 4435027072

3
Mistral-Nemo-Instruct-2407-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b695c69b285c17a602bb614b30c22e96100a9046681f4a773e231228ebc67a96
size 5722236032

3
Mistral-Nemo-Instruct-2407-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d68db0c60a66623392422b134eade128b77458581d9dc845f06a5c7b7144193a
size 5306492032

3
Mistral-Nemo-Instruct-2407-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4635276546fba351d9567641e55dc35ab9bce7fdc04a6f513ebb522326d3f41d
size 6742713472

3
Mistral-Nemo-Instruct-2407-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5a1a7b2df7e21d6a3e41a1d99f7c2e69892adf061d99e6b9353e920cf3f5acbb
size 4791051392

3
Mistral-Nemo-Instruct-2407-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fafcdd92d1e5854b4564ab30c348ac9b227e7fad5a9516338f7f971ea2b93de2
size 5446411392

3
Mistral-Nemo-Instruct-2407-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f9844586fc2809a6018bf6086753f655ce3528b38354d4b019eef5121938fd7c
size 6561506432

3
Mistral-Nemo-Instruct-2407-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:42a78c7db9dc76c049c985eca8b3a0a7c218607e226e132168728a53f5104cc9
size 6083093632

3
Mistral-Nemo-Instruct-2407-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:feb33ad62929e746dd1ac857a76cad9ae8093e1a675c5edd17d3e7533840bbd4
size 5534229632

3
Mistral-Nemo-Instruct-2407-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:768a20da7b0f99af2d5e09e0e63f667616a817d905012b78282b299289cd47d6
size 7148708992

3
Mistral-Nemo-Instruct-2407-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc7a72b30db1d9eac3ed7cab1f69d408621a01e12b4dc29a1c54639a74311017
size 7094641792

3
Mistral-Nemo-Instruct-2407-Q4_0_4_4.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:594f996e64ca85803a391e506dde49d0f3cba16aea15880c2851052ebb090223
size 7071704192

3
Mistral-Nemo-Instruct-2407-Q4_0_4_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:035bda8d0fdd52cdf991ff8d34f3515f51849044f76fcfdb4da98f90bfb9092d
size 7071704192

3
Mistral-Nemo-Instruct-2407-Q4_0_8_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7d4ca1fa31e8cdff20eb130565b897430427e8002631aaaeb2d420331ee0a5b2
size 7071704192

3
Mistral-Nemo-Instruct-2407-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:43e1d001f51830b2191d39c401fd266e1f0b875176069f8fd0cd6be4792dda5f
size 7975281792

3
Mistral-Nemo-Instruct-2407-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7c1a10d202d8788dbe5628dc962254d10654c853cae6aaeca0618f05490d4a46
size 7477208192

3
Mistral-Nemo-Instruct-2407-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:458c1775659eeb08647014ae91446c3646db098654e3054790f4ef3f354385c8
size 7120200832

3
Mistral-Nemo-Instruct-2407-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2e0ccbeabb69988d93abc2e421589f6f8b10f56449c5e4fadc348a10eb652bd9
size 9141822592

3
Mistral-Nemo-Instruct-2407-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f2118506b57d31403cba021b476ceb95600cc278fbc2feeaaf884b6e64fa6ee5
size 8727635072

3
Mistral-Nemo-Instruct-2407-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4637aba0d49de640521d5072c639e18238856f2a51ecb0cb3cbba086ffcf1f91
size 8518739072

3
Mistral-Nemo-Instruct-2407-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:da9ac0c204363b96dcfdfdf9fedea0d6dd0a923dee0e22bf589994e118482482
size 10056213632

3
Mistral-Nemo-Instruct-2407-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:54dfc2a56db2cd63f1dca3645a7fe67743c1ccc80e0d21753d2d2854058dc4e6
size 10381272192

3
Mistral-Nemo-Instruct-2407-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7985eab9b84d7d514253a53f372095d51f941f84a62cadaa0ccf6e1b3fbef494
size 13022372992

3
Mistral-Nemo-Instruct-2407-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:816cde814bb83890089d08fbcf8e7268a622cd859e530fcee12442565cc7a964
size 24504279872

3
Mistral-Nemo-Instruct-2407-f32.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:95cd4c88a408cf0b4ce9c6132ee43324f4462056f38ada8da606e7bf824d1e06
size 48999012032

3
Mistral-Nemo-Instruct-2407.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e4a024a3550a134fa76de0b4f55296f79f35719b00c71fbe1416f8c2df7328ef
size 7054418

137
README.md Normal file
View File

@@ -0,0 +1,137 @@
---
base_model: mistralai/Mistral-Nemo-Instruct-2407
language:
- en
- fr
- de
- es
- it
- pt
- ru
- zh
- ja
license: apache-2.0
pipeline_tag: text-generation
quantized_by: bartowski
extra_gated_description: If you want to learn more about how we process your personal
data, please read our <a href="https://mistral.ai/terms/">Privacy Policy</a>.
---
## Llamacpp imatrix Quantizations of Mistral-Nemo-Instruct-2407
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b4014">b4014</a> for quantization.
Original model: https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<s>[INST]{prompt}[/INST]
```
## What's new:
Update chat template
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Mistral-Nemo-Instruct-2407-f16.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-f16.gguf) | f16 | 24.50GB | false | Full F16 weights. |
| [Mistral-Nemo-Instruct-2407-Q8_0.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q8_0.gguf) | Q8_0 | 13.02GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Mistral-Nemo-Instruct-2407-Q6_K_L.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q6_K_L.gguf) | Q6_K_L | 10.38GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Mistral-Nemo-Instruct-2407-Q6_K.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q6_K.gguf) | Q6_K | 10.06GB | false | Very high quality, near perfect, *recommended*. |
| [Mistral-Nemo-Instruct-2407-Q5_K_L.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q5_K_L.gguf) | Q5_K_L | 9.14GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Mistral-Nemo-Instruct-2407-Q5_K_M.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q5_K_M.gguf) | Q5_K_M | 8.73GB | false | High quality, *recommended*. |
| [Mistral-Nemo-Instruct-2407-Q5_K_S.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q5_K_S.gguf) | Q5_K_S | 8.52GB | false | High quality, *recommended*. |
| [Mistral-Nemo-Instruct-2407-Q4_K_L.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q4_K_L.gguf) | Q4_K_L | 7.98GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Mistral-Nemo-Instruct-2407-Q4_K_M.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q4_K_M.gguf) | Q4_K_M | 7.48GB | false | Good quality, default size for must use cases, *recommended*. |
| [Mistral-Nemo-Instruct-2407-Q3_K_XL.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q3_K_XL.gguf) | Q3_K_XL | 7.15GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Mistral-Nemo-Instruct-2407-Q4_K_S.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q4_K_S.gguf) | Q4_K_S | 7.12GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Mistral-Nemo-Instruct-2407-Q4_0.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q4_0.gguf) | Q4_0 | 7.09GB | false | Legacy format, generally not worth using over similarly sized formats |
| [Mistral-Nemo-Instruct-2407-Q4_0_8_8.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q4_0_8_8.gguf) | Q4_0_8_8 | 7.07GB | false | Optimized for ARM inference. Requires 'sve' support (see link below). *Don't use on Mac or Windows*. |
| [Mistral-Nemo-Instruct-2407-Q4_0_4_8.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q4_0_4_8.gguf) | Q4_0_4_8 | 7.07GB | false | Optimized for ARM inference. Requires 'i8mm' support (see link below). *Don't use on Mac or Windows*. |
| [Mistral-Nemo-Instruct-2407-Q4_0_4_4.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q4_0_4_4.gguf) | Q4_0_4_4 | 7.07GB | false | Optimized for ARM inference. Should work well on all ARM chips, pick this if you're unsure. *Don't use on Mac or Windows*. |
| [Mistral-Nemo-Instruct-2407-IQ4_XS.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-IQ4_XS.gguf) | IQ4_XS | 6.74GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Mistral-Nemo-Instruct-2407-Q3_K_L.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q3_K_L.gguf) | Q3_K_L | 6.56GB | false | Lower quality but usable, good for low RAM availability. |
| [Mistral-Nemo-Instruct-2407-Q3_K_M.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q3_K_M.gguf) | Q3_K_M | 6.08GB | false | Low quality. |
| [Mistral-Nemo-Instruct-2407-IQ3_M.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-IQ3_M.gguf) | IQ3_M | 5.72GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Mistral-Nemo-Instruct-2407-Q3_K_S.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q3_K_S.gguf) | Q3_K_S | 5.53GB | false | Low quality, not recommended. |
| [Mistral-Nemo-Instruct-2407-Q2_K_L.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q2_K_L.gguf) | Q2_K_L | 5.45GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Mistral-Nemo-Instruct-2407-IQ3_XS.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-IQ3_XS.gguf) | IQ3_XS | 5.31GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Mistral-Nemo-Instruct-2407-Q2_K.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q2_K.gguf) | Q2_K | 4.79GB | false | Very low quality but surprisingly usable. |
| [Mistral-Nemo-Instruct-2407-IQ2_M.gguf](https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-IQ2_M.gguf) | IQ2_M | 4.44GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
Some say that this improves the quality, others don't notice any difference. If you use these models PLEASE COMMENT with your findings. I would like feedback that these are actually used and useful so I don't keep uploading quants no one is using.
Thanks!
## Downloading using huggingface-cli
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Mistral-Nemo-Instruct-2407-GGUF --include "Mistral-Nemo-Instruct-2407-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Mistral-Nemo-Instruct-2407-GGUF --include "Mistral-Nemo-Instruct-2407-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Mistral-Nemo-Instruct-2407-Q8_0) or download them all in place (./)
## Q4_0_X_X
These are *NOT* for Metal (Apple) offloading, only ARM chips.
If you're using an ARM chip, the Q4_0_X_X quants will have a substantial speedup. Check out Q4_0_4_4 speed comparisons [on the original pull request](https://github.com/ggerganov/llama.cpp/pull/5780#pullrequestreview-21657544660)
To check which one would work best for your ARM chip, you can check [AArch64 SoC features](https://gpages.juszkiewicz.com.pl/arm-socs-table/arm-socs.html) (thanks EloyOn!).
## Which file should I choose?
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset
Thank you ZeroWw for the inspiration to experiment with embed/output
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}