bartowski/MN-12b-RP-Ink-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/MN-12b-RP-Ink-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-18 04:01:06 +08:00
commit 6beb2ddd71
28 changed files with 305 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
.gitattributes vendored Normal file
View File

@@ -0,0 +1,60 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink-f16.gguf filter=lfs diff=lfs merge=lfs -text
MN-12b-RP-Ink.imatrix filter=lfs diff=lfs merge=lfs -text

3
MN-12b-RP-Ink-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:14e4aa2ffe2b4245654a7034654537556882c397d911d254a62368d0b9fe58a7
size 4435026976

3
MN-12b-RP-Ink-IQ2_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d759b1d3baea4cefaed614517aa0e4d4fb8b23bce31c551a19d5bf36877fcc6e
size 4138476576

3
MN-12b-RP-Ink-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:39766b02f1bfa207137c034bb2ce4864ce1be295ed16d23eb27b03081af57d5e
size 5722235936

3
MN-12b-RP-Ink-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:29e33ff600be136522590f24e93b7a489af0c0aceb512434ff35fc7b57b4c72c
size 5306491936

3
MN-12b-RP-Ink-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5cc3618568ebdc0057bfb3250226a3bed577fbe4e08d2b96e28f26df208003ff
size 7097918496

3
MN-12b-RP-Ink-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:062a931660e2e1bfe37a2f5c924f4799dbdc6aa2198298c7817eadb6e2d92107
size 6742713376

3
MN-12b-RP-Ink-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7fbc46f8a2faf7569d9c7157fbbee5682f8b4db5755ca35c23531f4be749f3f4
size 4791051296

3
MN-12b-RP-Ink-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5951e2f219f8b3467bdbe4b934e0d0735a828906897e530bbd76705e7a4d7c4e
size 5446411296

3
MN-12b-RP-Ink-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:445c2cc3257e870ec3b0d0c6120818e62f09d071c92fd1ea870e64e507057fdf
size 6561506336

3
MN-12b-RP-Ink-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8d6dc3816fc57b6e2bd1dbef7ab3d33e0ceda588e54c1aa2464536844eb39441
size 6083093536

3
MN-12b-RP-Ink-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8c1a8eca95e66d4c2315d965154ca0152b7bb383f9e313a25c7be7934a8fda7c
size 5534229536

3
MN-12b-RP-Ink-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4707a20d159b2b90d867f36c9f1d15d812533842c9c458169f974718cdd274b7
size 7148708896

3
MN-12b-RP-Ink-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:94d0543c54b3eaa783ab82d77cd238cf36f572c66541ba4ae6fc13e84fef6eec
size 7094641696

3
MN-12b-RP-Ink-Q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:76daa49b276ff0cf63de75e259d81644ea1092b86f9e0d7b140aa65450dabbdd
size 7795221536

3
MN-12b-RP-Ink-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:85ac4524267b9508d4a405c809cf03416d3ed6a0463467474ad7ec34f023e031
size 7975281696

3
MN-12b-RP-Ink-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:de4bb9e1bdc3ef82106a807efc4bd30664fea2d41838d20d272c9e55a72d2960
size 7477208096

3
MN-12b-RP-Ink-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2138b96eabe8fa6f9fe68547f67307fead5f6f5073b765ac9a7e0a85096f17ac
size 7120200736

3
MN-12b-RP-Ink-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:109616214147bae755f05d917886563fc536129d64eb1a79f697ea16d17fae43
size 9141822496

3
MN-12b-RP-Ink-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:86367b1e5cf5244d1bd200805d358e6ea30a6a0eb0631bed21b5cfe66a233b1d
size 8727634976

3
MN-12b-RP-Ink-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cd16bd8880204bf95a9c425284f71ad0c6fe7c570fb80ed68d47e233cc2b5211
size 8518738976

3
MN-12b-RP-Ink-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4e01fc7fc34e445410f79c2f590a396c3a9ce0b4e581fb2b986b5609fb534624
size 10056213536

3
MN-12b-RP-Ink-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4574af55e64b2fe0ae9238641e07c8d349ea47055de0248db131dc3f8313579f
size 10381272096

3
MN-12b-RP-Ink-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f083a21017075337afa379ca4581a27dc4aed9d5ff3809e4af496c8a7d75e387
size 13022372896

3
MN-12b-RP-Ink-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dd8f1819182a0a9e9c813ccfcd23ff1460e17849521768f27c182c57c67c4e6f
size 24504279840

3
MN-12b-RP-Ink.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3f5b1971e9978b46a36260b7b7c1aa6d7b24f2afab69e34a959d36cda9e5d886
size 7054418

169
README.md Normal file
View File

@@ -0,0 +1,169 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
language:
- en
base_model: allura-org/MN-12b-RP-Ink
license: apache-2.0
tags:
- roleplay
- conversational
---
## Llamacpp imatrix Quantizations of MN-12b-RP-Ink
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b4381">b4381</a> for quantization.
Original model: https://huggingface.co/allura-org/MN-12b-RP-Ink
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<s>[INST]{prompt}[/INST]
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [MN-12b-RP-Ink-f16.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-f16.gguf) | f16 | 24.50GB | false | Full F16 weights. |
| [MN-12b-RP-Ink-Q8_0.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q8_0.gguf) | Q8_0 | 13.02GB | false | Extremely high quality, generally unneeded but max available quant. |
| [MN-12b-RP-Ink-Q6_K_L.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q6_K_L.gguf) | Q6_K_L | 10.38GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [MN-12b-RP-Ink-Q6_K.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q6_K.gguf) | Q6_K | 10.06GB | false | Very high quality, near perfect, *recommended*. |
| [MN-12b-RP-Ink-Q5_K_L.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q5_K_L.gguf) | Q5_K_L | 9.14GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [MN-12b-RP-Ink-Q5_K_M.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q5_K_M.gguf) | Q5_K_M | 8.73GB | false | High quality, *recommended*. |
| [MN-12b-RP-Ink-Q5_K_S.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q5_K_S.gguf) | Q5_K_S | 8.52GB | false | High quality, *recommended*. |
| [MN-12b-RP-Ink-Q4_K_L.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q4_K_L.gguf) | Q4_K_L | 7.98GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [MN-12b-RP-Ink-Q4_1.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q4_1.gguf) | Q4_1 | 7.80GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
| [MN-12b-RP-Ink-Q4_K_M.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q4_K_M.gguf) | Q4_K_M | 7.48GB | false | Good quality, default size for most use cases, *recommended*. |
| [MN-12b-RP-Ink-Q3_K_XL.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q3_K_XL.gguf) | Q3_K_XL | 7.15GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [MN-12b-RP-Ink-Q4_K_S.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q4_K_S.gguf) | Q4_K_S | 7.12GB | false | Slightly lower quality with more space savings, *recommended*. |
| [MN-12b-RP-Ink-IQ4_NL.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-IQ4_NL.gguf) | IQ4_NL | 7.10GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
| [MN-12b-RP-Ink-Q4_0.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q4_0.gguf) | Q4_0 | 7.09GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
| [MN-12b-RP-Ink-IQ4_XS.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-IQ4_XS.gguf) | IQ4_XS | 6.74GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [MN-12b-RP-Ink-Q3_K_L.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q3_K_L.gguf) | Q3_K_L | 6.56GB | false | Lower quality but usable, good for low RAM availability. |
| [MN-12b-RP-Ink-Q3_K_M.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q3_K_M.gguf) | Q3_K_M | 6.08GB | false | Low quality. |
| [MN-12b-RP-Ink-IQ3_M.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-IQ3_M.gguf) | IQ3_M | 5.72GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [MN-12b-RP-Ink-Q3_K_S.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q3_K_S.gguf) | Q3_K_S | 5.53GB | false | Low quality, not recommended. |
| [MN-12b-RP-Ink-Q2_K_L.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q2_K_L.gguf) | Q2_K_L | 5.45GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [MN-12b-RP-Ink-IQ3_XS.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-IQ3_XS.gguf) | IQ3_XS | 5.31GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [MN-12b-RP-Ink-Q2_K.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-Q2_K.gguf) | Q2_K | 4.79GB | false | Very low quality but surprisingly usable. |
| [MN-12b-RP-Ink-IQ2_M.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-IQ2_M.gguf) | IQ2_M | 4.44GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
| [MN-12b-RP-Ink-IQ2_S.gguf](https://huggingface.co/bartowski/MN-12b-RP-Ink-GGUF/blob/main/MN-12b-RP-Ink-IQ2_S.gguf) | IQ2_S | 4.14GB | false | Low quality, uses SOTA techniques to be usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/MN-12b-RP-Ink-GGUF --include "MN-12b-RP-Ink-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/MN-12b-RP-Ink-GGUF --include "MN-12b-RP-Ink-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (MN-12b-RP-Ink-Q8_0) or download them all in place (./)
</details>
## ARM/AVX information
Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.
Now, however, there is something called "online repacking" for weights. details in [this PR](https://github.com/ggerganov/llama.cpp/pull/9921). If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.
As of llama.cpp build [b4282](https://github.com/ggerganov/llama.cpp/releases/tag/b4282) you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.
Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to [this PR](https://github.com/ggerganov/llama.cpp/pull/10541) which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.
<details>
<summary>Click to view Q4_0_X_X information (deprecated</summary>
I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}