初始化项目,由ModelHub XC社区提供模型

Model: bartowski/L3-Pneuma-8B-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC committed 2026-11-11 02:36:12 +08:00
commit 26e3f7f6d3
27 files changed
+307

No files matched your search

+59
View File
@@ -0,0 +1,59 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B-f16.gguf filter=lfs diff=lfs merge=lfs -text
L3-Pneuma-8B.imatrix filter=lfs diff=lfs merge=lfs -text
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5f6026c07d39bf423c613ac489c982856a40f9f103afaee8e15778907f13b412
size 2948281536
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:01c0dc776e22a9482f586b2190faffa2a772bf08938a8e3ab7dc0a6b8782872f
size 3784824000
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5ed6c733fdbeb0512111ede032a059abc1910645601678d1130a2ad3c2369502
size 3518747840
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cba23af71f462745896036b61aae61fe8215d3323aad819732b99ce8c1b34d45
size 4677989568
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a80e4eedb280fe4a1e9b475a14f0d29f416a7476c00d5a2e8ca01b01c16b21e1
size 4447663296
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4b4e8f266a029cacd92799bed5a339fd3a4ac145527f159c2aa98d844e20334f
size 3179132096
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dd74356579a0b1e87d56b9de8ec46b5c4ea1a1756f6c066051fad186f5a397f2
size 3692156096
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0c2a688d4f9b1abdcc2526cc49bb59301d597ed7d2355529c5bb84fac4c7b505
size 4321957056
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1d8331399cf6ede773bdf961094d0bf5d8794c418bae22af05ee8e8033ac2b82
size 4018918592
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5340e3105d081f8e688405ab7792f96c1ba2ba1044dfb88ae0e6f35612bf21da
size 3664499904
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:079da7ed9dd8913403eb4356d45fd55213fb65f53ecc0073686edba537288233
size 4781626560
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a2461f54a2fb9518e5ad27078007c9053f2fc8695e89f5e97addd19956058f83
size 4675892416
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f49530224a96c96d489f37bc135f60a4950ef4b5e03f75873c7864932b7b7017
size 5130253504
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e35ca11527d6962fbe5548c2f3d84aa94717b8ec677ed97033aa0535faee9c13
size 5310633152
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2b1ef9589b1a84f6e592fca04a96c57cc55c2caeb44cc256def78c1716253ca0
size 4920734912
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3c2e89f28dbe1a35f86279938e11359376d8663ee4a99156cbc70164eeb2743c
size 4692669632
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fcdfa1c4e9c3bf419820613bb729f8ba2266b1c201c3e2e242851e25e46e6aee
size 6057219264
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:89662d0b854b2d93ae317e5faefa74e25a3d17243166b45e2b9d25c8ec64f184
size 5732988096
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:47ff4071d21d7ba56a125ce1683300182948cb08dd0e5d928b17c74f28c60215
size 5599294656
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2397e5a3e5681f49d8c110fbc38988d27374d8805f23516130ae751040147239
size 6596007104
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0899747e1d631107eee0dd520ef39ea79c4f14545665d7849a8e7cdef8ab09e7
size 6850467008
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bbecd8d7b3574a22c6095ec2f35685ef8ecc7ae30a4b8f7a6cacdae37491eb22
size 8540771520
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d4e53b5bdd8895531a5100473b2bbaba82ead616957d27d6d6aadc520d96c771
size 16068891584
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4bca960334ceb3416bf4487243813e69ea50a6d163cde3639a4ec803200f5a6e
size 4988170
+175
View File
@@ -0,0 +1,175 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
base_model: Replete-AI/L3-Pneuma-8B
license: llama3
tags:
- axolotl
- generated_from_trainer
model-index:
- name: L3-Pneuma-8B
results: []
---
## Llamacpp imatrix Quantizations of L3-Pneuma-8B
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b4404">b4404</a> for quantization.
Original model: https://huggingface.co/Replete-AI/L3-Pneuma-8B
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>
{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
<|eot_id|><|start_header_id|>pneuma<|end_header_id|>
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [L3-Pneuma-8B-f16.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-f16.gguf) | f16 | 16.07GB | false | Full F16 weights. |
| [L3-Pneuma-8B-Q8_0.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q8_0.gguf) | Q8_0 | 8.54GB | false | Extremely high quality, generally unneeded but max available quant. |
| [L3-Pneuma-8B-Q6_K_L.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q6_K_L.gguf) | Q6_K_L | 6.85GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [L3-Pneuma-8B-Q6_K.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q6_K.gguf) | Q6_K | 6.60GB | false | Very high quality, near perfect, *recommended*. |
| [L3-Pneuma-8B-Q5_K_L.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q5_K_L.gguf) | Q5_K_L | 6.06GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [L3-Pneuma-8B-Q5_K_M.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q5_K_M.gguf) | Q5_K_M | 5.73GB | false | High quality, *recommended*. |
| [L3-Pneuma-8B-Q5_K_S.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q5_K_S.gguf) | Q5_K_S | 5.60GB | false | High quality, *recommended*. |
| [L3-Pneuma-8B-Q4_K_L.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q4_K_L.gguf) | Q4_K_L | 5.31GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [L3-Pneuma-8B-Q4_1.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q4_1.gguf) | Q4_1 | 5.13GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
| [L3-Pneuma-8B-Q4_K_M.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q4_K_M.gguf) | Q4_K_M | 4.92GB | false | Good quality, default size for most use cases, *recommended*. |
| [L3-Pneuma-8B-Q3_K_XL.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q3_K_XL.gguf) | Q3_K_XL | 4.78GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [L3-Pneuma-8B-Q4_K_S.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q4_K_S.gguf) | Q4_K_S | 4.69GB | false | Slightly lower quality with more space savings, *recommended*. |
| [L3-Pneuma-8B-Q4_0.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q4_0.gguf) | Q4_0 | 4.68GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
| [L3-Pneuma-8B-IQ4_NL.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-IQ4_NL.gguf) | IQ4_NL | 4.68GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
| [L3-Pneuma-8B-IQ4_XS.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-IQ4_XS.gguf) | IQ4_XS | 4.45GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [L3-Pneuma-8B-Q3_K_L.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q3_K_L.gguf) | Q3_K_L | 4.32GB | false | Lower quality but usable, good for low RAM availability. |
| [L3-Pneuma-8B-Q3_K_M.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q3_K_M.gguf) | Q3_K_M | 4.02GB | false | Low quality. |
| [L3-Pneuma-8B-IQ3_M.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-IQ3_M.gguf) | IQ3_M | 3.78GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [L3-Pneuma-8B-Q2_K_L.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q2_K_L.gguf) | Q2_K_L | 3.69GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [L3-Pneuma-8B-Q3_K_S.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q3_K_S.gguf) | Q3_K_S | 3.66GB | false | Low quality, not recommended. |
| [L3-Pneuma-8B-IQ3_XS.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-IQ3_XS.gguf) | IQ3_XS | 3.52GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [L3-Pneuma-8B-Q2_K.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-Q2_K.gguf) | Q2_K | 3.18GB | false | Very low quality but surprisingly usable. |
| [L3-Pneuma-8B-IQ2_M.gguf](https://huggingface.co/bartowski/L3-Pneuma-8B-GGUF/blob/main/L3-Pneuma-8B-IQ2_M.gguf) | IQ2_M | 2.95GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/L3-Pneuma-8B-GGUF --include "L3-Pneuma-8B-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/L3-Pneuma-8B-GGUF --include "L3-Pneuma-8B-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (L3-Pneuma-8B-Q8_0) or download them all in place (./)
</details>
## ARM/AVX information
Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.
Now, however, there is something called "online repacking" for weights. details in [this PR](https://github.com/ggerganov/llama.cpp/pull/9921). If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.
As of llama.cpp build [b4282](https://github.com/ggerganov/llama.cpp/releases/tag/b4282) you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.
Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to [this PR](https://github.com/ggerganov/llama.cpp/pull/10541) which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.
<details>
<summary>Click to view Q4_0_X_X information (deprecated</summary>
I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski
+1
View File
@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}