bartowski/Falcon3-1B-Base-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Falcon3-1B-Base-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-09-29 21:49:13 +08:00
commit 16f6e79ac3
26 changed files with 297 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

58
.gitattributes vendored Normal file
View File

@@ -0,0 +1,58 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base-f16.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-1B-Base.imatrix filter=lfs diff=lfs merge=lfs -text

3
Falcon3-1B-Base-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:74a1f1e549474a7f8517039517de76dd3ade65d482a26ef1463843ab37386f8c
size 846688288

3
Falcon3-1B-Base-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:08279a158315999e6d7001a76e5100bb3935d412d5f68539e9b80e048e4bf424
size 801435680

3
Falcon3-1B-Base-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b29ebc5652c79ac4154470d3a452e63aad3d12a3a21b2bc4b98763c33de2df05
size 1013248032

3
Falcon3-1B-Base-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0edb8ded5e72bb16448ef5987c78a21f2f893e95b86805a78c36a2b9967023d5
size 969469984

3
Falcon3-1B-Base-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8cf781aa8361970279b7303b706a923e667d5f089bfdff188eeddaa38cd9e8f4
size 727085088

3
Falcon3-1B-Base-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e88e3af710c8405d50e4639b9cff93128b23c2130b274c4da0ef31cdefd34694
size 989229088

3
Falcon3-1B-Base-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8783ca9806d91eb8f6f0162b85767372c301f237ff068d2599190d6b3e0828bc
size 934244384

3
Falcon3-1B-Base-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:42539136236c2a67b3a762a30c6f85b97b4e9d184201b0d2cdefa7e72a30982f
size 884961312

3
Falcon3-1B-Base-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f24965bf9e7ad721d620e202e9f5022b713436699921aa24333a6f6907a09200
size 827191328

3
Falcon3-1B-Base-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7505746c660067cb7b5cdd270db81d84a1338c44881dbe136e74890ef41fe379
size 1169125408

3
Falcon3-1B-Base-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fbbbadc29e8cc1165306f735b7a1047b5998e099b5513d0bd9b74112f23ddc58
size 1015345184

3
Falcon3-1B-Base-Q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0938b03889735c2695b4a44f0757059071f7811f22eff7adcc09dc0c454fad9d
size 1100804128

3
Falcon3-1B-Base-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:08caf80b072a86d2b6f46b65b81019d42621a8e09b601748fa36d84afe1db0ff
size 1256271904

3
Falcon3-1B-Base-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:55b940e5d9ef6198f36c66625bb51cad6168ee43d2435418f6ec0966066b5854
size 1057042464

3
Falcon3-1B-Base-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:feffbcb0ff5e36f34e0901f9e49d459ec39475ad150d5162434c628da4f636d9
size 1018490912

3
Falcon3-1B-Base-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0075c25208114c41c7d6bc02332fc8869a0609b691294d418868bedb73d98174
size 1376596000

3
Falcon3-1B-Base-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:743c8e520f6bf567ab753c1fd42443dd4f74ec50678abf2e8b19c73f10bb468b
size 1210920992

3
Falcon3-1B-Base-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:866e4c81c6a0a4d4a5bf0cff6ac3ebd43ba933de10d0ef0a8a2b5ab5cc0a28d9
size 1188360224

3
Falcon3-1B-Base-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:31c1ebe9ddc844085a2e78e0192ea43b6ac2a8a248b2f1c512442e4c2156483c
size 1374416928

3
Falcon3-1B-Base-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:84c6e0b5919edb471c0d8c14c7df1b3af6c03049f87c248d3a2458a72b843ce1
size 1504440352

3
Falcon3-1B-Base-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f70c01a91e9e5981f859f396f96bce08e25cb830fecdebf6590cfdc68abcdbaf
size 1778708512

3
Falcon3-1B-Base-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4f92ca9e223744644a1e2e6908017ac3b0989a4c03e3459a0a5ec66fe0013ca5
size 3343707936

3
Falcon3-1B-Base.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8bc49d9936e5960017bf6edc8feae9d8228b3ac394a2be41afcad7f29eaa3e1f
size 1478732

169
README.md Normal file
View File

@@ -0,0 +1,169 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
license_name: falcon-llm-license
language:
- en
- fr
- es
- pt
license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
base_model: tiiuae/Falcon3-1B-Base
license: other
tags:
- falcon3
---
## Llamacpp imatrix Quantizations of Falcon3-1B-Base
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b4381">b4381</a> for quantization.
Original model: https://huggingface.co/tiiuae/Falcon3-1B-Base
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
No prompt format found, check original model page
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Falcon3-1B-Base-f16.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-f16.gguf) | f16 | 3.34GB | false | Full F16 weights. |
| [Falcon3-1B-Base-Q8_0.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q8_0.gguf) | Q8_0 | 1.78GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Falcon3-1B-Base-Q6_K_L.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q6_K_L.gguf) | Q6_K_L | 1.50GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Falcon3-1B-Base-Q5_K_L.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q5_K_L.gguf) | Q5_K_L | 1.38GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Falcon3-1B-Base-Q6_K.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q6_K.gguf) | Q6_K | 1.37GB | false | Very high quality, near perfect, *recommended*. |
| [Falcon3-1B-Base-Q4_K_L.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q4_K_L.gguf) | Q4_K_L | 1.26GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Falcon3-1B-Base-Q5_K_M.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q5_K_M.gguf) | Q5_K_M | 1.21GB | false | High quality, *recommended*. |
| [Falcon3-1B-Base-Q5_K_S.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q5_K_S.gguf) | Q5_K_S | 1.19GB | false | High quality, *recommended*. |
| [Falcon3-1B-Base-Q3_K_XL.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q3_K_XL.gguf) | Q3_K_XL | 1.17GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Falcon3-1B-Base-Q4_1.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q4_1.gguf) | Q4_1 | 1.10GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
| [Falcon3-1B-Base-Q4_K_M.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q4_K_M.gguf) | Q4_K_M | 1.06GB | false | Good quality, default size for most use cases, *recommended*. |
| [Falcon3-1B-Base-Q4_K_S.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q4_K_S.gguf) | Q4_K_S | 1.02GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Falcon3-1B-Base-Q4_0.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q4_0.gguf) | Q4_0 | 1.02GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
| [Falcon3-1B-Base-IQ4_NL.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-IQ4_NL.gguf) | IQ4_NL | 1.01GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
| [Falcon3-1B-Base-Q2_K_L.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q2_K_L.gguf) | Q2_K_L | 0.99GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Falcon3-1B-Base-IQ4_XS.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-IQ4_XS.gguf) | IQ4_XS | 0.97GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Falcon3-1B-Base-Q3_K_L.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q3_K_L.gguf) | Q3_K_L | 0.93GB | false | Lower quality but usable, good for low RAM availability. |
| [Falcon3-1B-Base-Q3_K_M.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q3_K_M.gguf) | Q3_K_M | 0.88GB | false | Low quality. |
| [Falcon3-1B-Base-IQ3_M.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-IQ3_M.gguf) | IQ3_M | 0.85GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Falcon3-1B-Base-Q3_K_S.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q3_K_S.gguf) | Q3_K_S | 0.83GB | false | Low quality, not recommended. |
| [Falcon3-1B-Base-IQ3_XS.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-IQ3_XS.gguf) | IQ3_XS | 0.80GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Falcon3-1B-Base-Q2_K.gguf](https://huggingface.co/bartowski/Falcon3-1B-Base-GGUF/blob/main/Falcon3-1B-Base-Q2_K.gguf) | Q2_K | 0.73GB | false | Very low quality but surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Falcon3-1B-Base-GGUF --include "Falcon3-1B-Base-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Falcon3-1B-Base-GGUF --include "Falcon3-1B-Base-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Falcon3-1B-Base-Q8_0) or download them all in place (./)
</details>
## ARM/AVX information
Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.
Now, however, there is something called "online repacking" for weights. details in [this PR](https://github.com/ggerganov/llama.cpp/pull/9921). If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.
As of llama.cpp build [b4282](https://github.com/ggerganov/llama.cpp/releases/tag/b4282) you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.
Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to [this PR](https://github.com/ggerganov/llama.cpp/pull/10541) which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.
<details>
<summary>Click to view Q4_0_X_X information (deprecated</summary>
I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}