bartowski/OpenCoder-8B-Instruct-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/OpenCoder-8B-Instruct-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-22 23:04:07 +08:00
commit a1c79c19e1
28 changed files with 269 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
.gitattributes vendored Normal file
View File

@@ -0,0 +1,60 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q4_0_8_8.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q4_0_4_8.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q4_0_4_4.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct-f16.gguf filter=lfs diff=lfs merge=lfs -text
OpenCoder-8B-Instruct.imatrix filter=lfs diff=lfs merge=lfs -text

3
OpenCoder-8B-Instruct-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1b10fc7d1826110fdb89e8552d43cdba3498b2f218ab3d0feb115e7fc214cb83
size 2798004032

3
OpenCoder-8B-Instruct-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6f7aafbe44b903bc1cb5b8897cdd264853ba631c1cfeee89ffe7aad0e0320415
size 3617347392

3
OpenCoder-8B-Instruct-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8aeadaf187754496efe51bf2ca9f45bbd9911a3944f185b88bc8ce9afab92ac8
size 3351271232

3
OpenCoder-8B-Instruct-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d3445f4262418cdbad6e4503d34ec00e322495d95098f63c165ab0a0e99d00ea
size 4267034432

3
OpenCoder-8B-Instruct-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d5985fbd6e53e21be4bd3f2082177487f007c440716aeb9a178f6bb1b7de481e
size 3024807744

3
OpenCoder-8B-Instruct-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:303993410385e21e846ab7dfc830df3de8bcea20b5d5b7eb03bccad731e5e9cc
size 3411367744

3
OpenCoder-8B-Instruct-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a46ec89bf73addc7fe30c0e385036bd326db456de7c1da79bc22221ca853f090
size 4154480448

3
OpenCoder-8B-Instruct-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:54734487d308c4fd379f34a38693cfe51128ae30fbd4195fa53cf93fcec4d713
size 3851441984

3
OpenCoder-8B-Instruct-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3c27b0b0e8e63e823bf45fc11201fe13674cbf711ba85ef1926e1b8755628131
size 3497023296

3
OpenCoder-8B-Instruct-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:001b665e98b3593e625bf30001fc02d77ded22a7a7e9298db3d7c41ab51555d5
size 4500838208

3
OpenCoder-8B-Instruct-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:04d4b6b2fc55c6aaa351bdf4bdc33f893c5fccce1c5d565dec9dba1cf23f8a90
size 4491216704

3
OpenCoder-8B-Instruct-Q4_0_4_4.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:72f841e5533f691f16707c7a21eba25a3153e3f785f9d402c6fd2b8a80d68c61
size 4476536640

3
OpenCoder-8B-Instruct-Q4_0_4_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a8d3cf45c5d77e7c5b35ac592afef365245ba525369c30c36de020f269ff99fa
size 4476536640

3
OpenCoder-8B-Instruct-Q4_0_8_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8da73f348af95d592d99c725a2f859e270bc7547575cc7e21f7562a0c62955ac
size 4476536640

3
OpenCoder-8B-Instruct-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7143918092089b24452db31f8bc468467d8d6262af6550d0c3ce3124213e913e
size 5029844800

3
OpenCoder-8B-Instruct-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:25c7b89f02e2dbe389c31083bb2528ce5d590804376ee2ef2766e47f0d1f7c91
size 4736059200

3
OpenCoder-8B-Instruct-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f6ff07cfdf4d5db8aab95c64c85eac628bc56a538120b93fafcd4e6897a47dd7
size 4507993920

3
OpenCoder-8B-Instruct-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:49322809b66dea9b34e3b8c352247d4507f9229357b1207b8617e2592d9feb12
size 5776430912

3
OpenCoder-8B-Instruct-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5af251d6a7c1de672032045842e37298087a5b11eb62a2d759c19b66acbe8641
size 5532124992

3
OpenCoder-8B-Instruct-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:db40c3178dce89893231d198267a11de2bb60df576a43f12d4ca81e246f83a78
size 5398431552

3
OpenCoder-8B-Instruct-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:205540ed0727ad636ad151812f970320698dfe73c7c101fe8c493116cd3f01fc
size 6377944896

3
OpenCoder-8B-Instruct-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d38845bb78324f3c471c2c22e29a5016b2c201c0a54743e7c44b0fdd2722e342
size 6569678656

3
OpenCoder-8B-Instruct-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f39bf90d760ba0766eb681d65d4844dee8d13022a197da6549453869db27b911
size 8259983168

3
OpenCoder-8B-Instruct-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:58fbfae4aaeb581a29377622b7fe7303d67c6cbe750a3727e7cf79e34c3325a9
size 15545292320

3
OpenCoder-8B-Instruct.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7738e4fb54c2eaee916ff23b0da2806462a1a285f6222f1b16935c2c27bdf64c
size 4988170

133
README.md Normal file
View File

@@ -0,0 +1,133 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
license_name: inf
datasets:
- OpenCoder-LLM/opencoder-sft-stage1
- OpenCoder-LLM/opencoder-sft-stage2
language:
- en
- zh
license_link: https://huggingface.co/infly/OpenCoder-8B-Instruct/blob/main/LICENSE
base_model: infly/OpenCoder-8B-Instruct
license: other
---
## Llamacpp imatrix Quantizations of OpenCoder-8B-Instruct
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b4014">b4014</a> for quantization.
Original model: https://huggingface.co/infly/OpenCoder-8B-Instruct
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [OpenCoder-8B-Instruct-f16.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-f16.gguf) | f16 | 15.55GB | false | Full F16 weights. |
| [OpenCoder-8B-Instruct-Q8_0.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q8_0.gguf) | Q8_0 | 8.26GB | false | Extremely high quality, generally unneeded but max available quant. |
| [OpenCoder-8B-Instruct-Q6_K_L.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q6_K_L.gguf) | Q6_K_L | 6.57GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [OpenCoder-8B-Instruct-Q6_K.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q6_K.gguf) | Q6_K | 6.38GB | false | Very high quality, near perfect, *recommended*. |
| [OpenCoder-8B-Instruct-Q5_K_L.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q5_K_L.gguf) | Q5_K_L | 5.78GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [OpenCoder-8B-Instruct-Q5_K_M.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q5_K_M.gguf) | Q5_K_M | 5.53GB | false | High quality, *recommended*. |
| [OpenCoder-8B-Instruct-Q5_K_S.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q5_K_S.gguf) | Q5_K_S | 5.40GB | false | High quality, *recommended*. |
| [OpenCoder-8B-Instruct-Q4_K_L.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q4_K_L.gguf) | Q4_K_L | 5.03GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [OpenCoder-8B-Instruct-Q4_K_M.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q4_K_M.gguf) | Q4_K_M | 4.74GB | false | Good quality, default size for most use cases, *recommended*. |
| [OpenCoder-8B-Instruct-Q4_K_S.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q4_K_S.gguf) | Q4_K_S | 4.51GB | false | Slightly lower quality with more space savings, *recommended*. |
| [OpenCoder-8B-Instruct-Q3_K_XL.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q3_K_XL.gguf) | Q3_K_XL | 4.50GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [OpenCoder-8B-Instruct-Q4_0.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q4_0.gguf) | Q4_0 | 4.49GB | false | Legacy format, generally not worth using over similarly sized formats |
| [OpenCoder-8B-Instruct-Q4_0_8_8.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q4_0_8_8.gguf) | Q4_0_8_8 | 4.48GB | false | Optimized for ARM inference. Requires 'sve' support (see link below). *Don't use on Mac or Windows*. |
| [OpenCoder-8B-Instruct-Q4_0_4_8.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q4_0_4_8.gguf) | Q4_0_4_8 | 4.48GB | false | Optimized for ARM inference. Requires 'i8mm' support (see link below). *Don't use on Mac or Windows*. |
| [OpenCoder-8B-Instruct-Q4_0_4_4.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q4_0_4_4.gguf) | Q4_0_4_4 | 4.48GB | false | Optimized for ARM inference. Should work well on all ARM chips, pick this if you're unsure. *Don't use on Mac or Windows*. |
| [OpenCoder-8B-Instruct-IQ4_XS.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-IQ4_XS.gguf) | IQ4_XS | 4.27GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [OpenCoder-8B-Instruct-Q3_K_L.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q3_K_L.gguf) | Q3_K_L | 4.15GB | false | Lower quality but usable, good for low RAM availability. |
| [OpenCoder-8B-Instruct-Q3_K_M.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q3_K_M.gguf) | Q3_K_M | 3.85GB | false | Low quality. |
| [OpenCoder-8B-Instruct-IQ3_M.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-IQ3_M.gguf) | IQ3_M | 3.62GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [OpenCoder-8B-Instruct-Q3_K_S.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q3_K_S.gguf) | Q3_K_S | 3.50GB | false | Low quality, not recommended. |
| [OpenCoder-8B-Instruct-Q2_K_L.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q2_K_L.gguf) | Q2_K_L | 3.41GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [OpenCoder-8B-Instruct-IQ3_XS.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-IQ3_XS.gguf) | IQ3_XS | 3.35GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [OpenCoder-8B-Instruct-Q2_K.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-Q2_K.gguf) | Q2_K | 3.02GB | false | Very low quality but surprisingly usable. |
| [OpenCoder-8B-Instruct-IQ2_M.gguf](https://huggingface.co/bartowski/OpenCoder-8B-Instruct-GGUF/blob/main/OpenCoder-8B-Instruct-IQ2_M.gguf) | IQ2_M | 2.80GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
Some say that this improves the quality, others don't notice any difference. If you use these models PLEASE COMMENT with your findings. I would like feedback that these are actually used and useful so I don't keep uploading quants no one is using.
Thanks!
## Downloading using huggingface-cli
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/OpenCoder-8B-Instruct-GGUF --include "OpenCoder-8B-Instruct-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/OpenCoder-8B-Instruct-GGUF --include "OpenCoder-8B-Instruct-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (OpenCoder-8B-Instruct-Q8_0) or download them all in place (./)
## Q4_0_X_X
These are *NOT* for Metal (Apple) offloading, only ARM chips.
If you're using an ARM chip, the Q4_0_X_X quants will have a substantial speedup. Check out Q4_0_4_4 speed comparisons [on the original pull request](https://github.com/ggerganov/llama.cpp/pull/5780#pullrequestreview-21657544660)
To check which one would work best for your ARM chip, you can check [AArch64 SoC features](https://gpages.juszkiewicz.com.pl/arm-socs-table/arm-socs.html) (thanks EloyOn!).
## Which file should I choose?
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}