bartowski/Lumimaid-Magnum-12B-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Lumimaid-Magnum-12B-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-08 10:33:05 +08:00
commit d45bf62b87
28 changed files with 260 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
.gitattributes vendored Normal file
View File

@@ -0,0 +1,60 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q4_0_8_8.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q4_0_4_8.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q4_0_4_4.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B-f16.gguf filter=lfs diff=lfs merge=lfs -text
Lumimaid-Magnum-12B.imatrix filter=lfs diff=lfs merge=lfs -text

3
Lumimaid-Magnum-12B-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0c40a5f9fe83f820b95138cbf6df478c3ddbd7da5f1599183fd6799eb92be935
size 4435024832

3
Lumimaid-Magnum-12B-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d89f9b10bef0cd3905d433c115d1ba4de73a986c11e653214f7455dc74d7fa5e
size 5722233792

3
Lumimaid-Magnum-12B-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e6b5dd3b9c6f8ae63de69318b44043aef77a3c94854a69ee1e918624d6ac4578
size 5306489792

3
Lumimaid-Magnum-12B-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1a6296b10ce2aa77c8d8d0bda9ac990796b34ccd7d3cbd3e69e0ba1ed81c00e4
size 6742711232

3
Lumimaid-Magnum-12B-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:21e22ceb99808e04007b161c52b4cb64467c72be312516a742194c0c5282a3ff
size 4791049152

3
Lumimaid-Magnum-12B-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:73a859f1506d5fe3427fda92ae6bd50857fc6ec0af628aa9d9cffb7be1bca83f
size 5446409152

3
Lumimaid-Magnum-12B-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cd6c1252d9a212b700c99634611818e76e29421841196231a48c8f63dc20e990
size 6561504192

3
Lumimaid-Magnum-12B-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a4f7b2f95c1bc4a5f41eee18bf9ed18fd3fb51b084a7dcbef9b16fcd64288d64
size 6083091392

3
Lumimaid-Magnum-12B-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:30a41e81b2152d9d0f17a1e832040c6dd3b004d1066baa0b400ce642fbdc0f68
size 5534227392

3
Lumimaid-Magnum-12B-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d5acc8f9b7b879e5c345e0c7a6fa9b1a27d208647ff519ec82c1a750b169cd49
size 7148706752

3
Lumimaid-Magnum-12B-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1d76dc2c96590f26468ff1972af4cba5c66a1080e94b737e47da3c09f054bf3c
size 7094639552

3
Lumimaid-Magnum-12B-Q4_0_4_4.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9cb76a6f69a03da785ff169262f880402ba81be7b0f34bd7d6d6a15ebd2ab65e
size 7071701952

3
Lumimaid-Magnum-12B-Q4_0_4_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ea88c2328feaf5d20289ee041b4b9c84bc7a88b0755b9d7f3172a13ad94952e6
size 7071701952

3
Lumimaid-Magnum-12B-Q4_0_8_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cb937f74476c0e50887d2117483fa3a03e3ba03812285354d4671dc6fb782632
size 7071701952

3
Lumimaid-Magnum-12B-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:31833f27c824d58946ded5d764552cff40ba26928ee76b1714e262e85855f911
size 7975279552

3
Lumimaid-Magnum-12B-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:758b9c969489bc20097dd0c23dedc3bff1b1b92d5852431724c9b7c0961c1ba2
size 7477205952

3
Lumimaid-Magnum-12B-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d7e96c14cfa206a249af7a0fcdcf0e41407200d75cfdfe4ac0656a9c6e23759c
size 7120198592

3
Lumimaid-Magnum-12B-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1c353785afaa8ce65242cec6ce4711597b0429b2d3b930e5d1c230bdbee117c2
size 9141820352

3
Lumimaid-Magnum-12B-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:364e3573963a16f4a34ec74ccefa820490ad1ed14fc4ccc66e4448c9bd9b8463
size 8727632832

3
Lumimaid-Magnum-12B-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d2da427b1241ab0f3328a8638d575fd49fd8c7df9e032321f2293e5c110064aa
size 8518736832

3
Lumimaid-Magnum-12B-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eccb93aaadbdec879f6a3658901a488992d799200b900cfc6de7d7c884a3bb91
size 10056211392

3
Lumimaid-Magnum-12B-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f62e169100c262a2218080c32f87c8cbfddab0ed52ec74d5682914b0898dabfe
size 10381269952

3
Lumimaid-Magnum-12B-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f6e039991696eab0d6a3e0c0a2acd0052219f5698369ab9adc768d49495d5622
size 13022370752

3
Lumimaid-Magnum-12B-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8f9be9391a281cf0c2352cdf5191acd5f268ede8c8d857a62e520f89553a453c
size 24504277664

3
Lumimaid-Magnum-12B.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:18e608a9f77f4e09bdde94585a66a2d5d9bb25a06947c332c599cd6834fbdaa4
size 7054418

124
README.md Normal file
View File

@@ -0,0 +1,124 @@
---
base_model: Undi95/Lumimaid-Magnum-12B
library_name: transformers
pipeline_tag: text-generation
tags:
- mergekit
- merge
quantized_by: bartowski
---
## Llamacpp imatrix Quantizations of Lumimaid-Magnum-12B
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b3825">b3825</a> for quantization.
Original model: https://huggingface.co/Undi95/Lumimaid-Magnum-12B
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<s>[INST] {prompt}[/INST]
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Lumimaid-Magnum-12B-f16.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-f16.gguf) | f16 | 24.50GB | false | Full F16 weights. |
| [Lumimaid-Magnum-12B-Q8_0.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q8_0.gguf) | Q8_0 | 13.02GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Lumimaid-Magnum-12B-Q6_K_L.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q6_K_L.gguf) | Q6_K_L | 10.38GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Lumimaid-Magnum-12B-Q6_K.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q6_K.gguf) | Q6_K | 10.06GB | false | Very high quality, near perfect, *recommended*. |
| [Lumimaid-Magnum-12B-Q5_K_L.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q5_K_L.gguf) | Q5_K_L | 9.14GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Lumimaid-Magnum-12B-Q5_K_M.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q5_K_M.gguf) | Q5_K_M | 8.73GB | false | High quality, *recommended*. |
| [Lumimaid-Magnum-12B-Q5_K_S.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q5_K_S.gguf) | Q5_K_S | 8.52GB | false | High quality, *recommended*. |
| [Lumimaid-Magnum-12B-Q4_K_L.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q4_K_L.gguf) | Q4_K_L | 7.98GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Lumimaid-Magnum-12B-Q4_K_M.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q4_K_M.gguf) | Q4_K_M | 7.48GB | false | Good quality, default size for must use cases, *recommended*. |
| [Lumimaid-Magnum-12B-Q3_K_XL.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q3_K_XL.gguf) | Q3_K_XL | 7.15GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Lumimaid-Magnum-12B-Q4_K_S.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q4_K_S.gguf) | Q4_K_S | 7.12GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Lumimaid-Magnum-12B-Q4_0.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q4_0.gguf) | Q4_0 | 7.09GB | false | Legacy format, generally not worth using over similarly sized formats |
| [Lumimaid-Magnum-12B-Q4_0_8_8.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q4_0_8_8.gguf) | Q4_0_8_8 | 7.07GB | false | Optimized for ARM inference. Requires 'sve' support (see link below). |
| [Lumimaid-Magnum-12B-Q4_0_4_8.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q4_0_4_8.gguf) | Q4_0_4_8 | 7.07GB | false | Optimized for ARM inference. Requires 'i8mm' support (see link below). |
| [Lumimaid-Magnum-12B-Q4_0_4_4.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q4_0_4_4.gguf) | Q4_0_4_4 | 7.07GB | false | Optimized for ARM inference. Should work well on all ARM chips, pick this if you're unsure. |
| [Lumimaid-Magnum-12B-IQ4_XS.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-IQ4_XS.gguf) | IQ4_XS | 6.74GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Lumimaid-Magnum-12B-Q3_K_L.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q3_K_L.gguf) | Q3_K_L | 6.56GB | false | Lower quality but usable, good for low RAM availability. |
| [Lumimaid-Magnum-12B-Q3_K_M.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q3_K_M.gguf) | Q3_K_M | 6.08GB | false | Low quality. |
| [Lumimaid-Magnum-12B-IQ3_M.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-IQ3_M.gguf) | IQ3_M | 5.72GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Lumimaid-Magnum-12B-Q3_K_S.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q3_K_S.gguf) | Q3_K_S | 5.53GB | false | Low quality, not recommended. |
| [Lumimaid-Magnum-12B-Q2_K_L.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q2_K_L.gguf) | Q2_K_L | 5.45GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Lumimaid-Magnum-12B-IQ3_XS.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-IQ3_XS.gguf) | IQ3_XS | 5.31GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Lumimaid-Magnum-12B-Q2_K.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-Q2_K.gguf) | Q2_K | 4.79GB | false | Very low quality but surprisingly usable. |
| [Lumimaid-Magnum-12B-IQ2_M.gguf](https://huggingface.co/bartowski/Lumimaid-Magnum-12B-GGUF/blob/main/Lumimaid-Magnum-12B-IQ2_M.gguf) | IQ2_M | 4.44GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
Some say that this improves the quality, others don't notice any difference. If you use these models PLEASE COMMENT with your findings. I would like feedback that these are actually used and useful so I don't keep uploading quants no one is using.
Thanks!
## Downloading using huggingface-cli
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Lumimaid-Magnum-12B-GGUF --include "Lumimaid-Magnum-12B-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Lumimaid-Magnum-12B-GGUF --include "Lumimaid-Magnum-12B-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Lumimaid-Magnum-12B-Q8_0) or download them all in place (./)
## Q4_0_X_X
These are *NOT* for Metal (Apple) offloading, only ARM chips.
If you're using an ARM chip, the Q4_0_X_X quants will have a substantial speedup. Check out Q4_0_4_4 speed comparisons [on the original pull request](https://github.com/ggerganov/llama.cpp/pull/5780#pullrequestreview-21657544660)
To check which one would work best for your ARM chip, you can check [AArch64 SoC features](https://gpages.juszkiewicz.com.pl/arm-socs-table/arm-socs.html) (thanks EloyOn!).
## Which file should I choose?
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset
Thank you ZeroWw for the inspiration to experiment with embed/output
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}