bartowski/Falcon3-10B-Instruct-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Falcon3-10B-Instruct-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-06 06:33:11 +08:00
commit 2b96227355
26 changed files with 398 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
.gitattributes vendored Normal file
View File

@@ -0,0 +1,59 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-f16.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct-f32.gguf filter=lfs diff=lfs merge=lfs -text
Falcon3-10B-Instruct.imatrix filter=lfs diff=lfs merge=lfs -text

3
Falcon3-10B-Instruct-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6bfdb96d85d285fba5cce92e19d5a169fe5007c89402213c326a7bed14a7a98a
size 3592345280

3
Falcon3-10B-Instruct-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:de23fb3ab383d7cfc9cdae7b5e56515a9a3ba39da165d8532d4c656640caf981
size 4704986816

3
Falcon3-10B-Instruct-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:233ddddf817da2d2ecbf8938c842466c8a26bddd03539f23bf265219d684eda5
size 4368479936

3
Falcon3-10B-Instruct-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fa5c378fc72ef712351c57d398bd73406ca7d195640859e759b7029e6428dff4
size 5906347712

3
Falcon3-10B-Instruct-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:674c4a1abd4f2ba4735fbcf8758d9d261a03c852a8fcb77ee0b173b4ba249470
size 5596886720

3
Falcon3-10B-Instruct-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c1a557301c2033b0cc382cd0378427339145570b3167696cd5aad1ea3f88598a
size 3924047552

3
Falcon3-10B-Instruct-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7de3aee551f7d29e7a5bcd215858a410ac4ce9143e923a89a907219469cd75c2
size 4317263552

3
Falcon3-10B-Instruct-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d98144e6c0029150b9380432fc5fcf4ac7645c48f5a410778e85f6e269554c00
size 5450806976

3
Falcon3-10B-Instruct-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9cd54148a18fed7218248f1b061da28eb17b6c211a4dd104eefadaab78163df5
size 5052479168

3
Falcon3-10B-Instruct-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c2415a25684c23ab46f1487921c2283e6df240a4b24665d8a17f563139702081
size 4591138496

3
Falcon3-10B-Instruct-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:39a4b5c7eade80af8d5995accd7264b8da8fe77d0857e466a49a1ec2e8039b72
size 5803128512

3
Falcon3-10B-Instruct-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3fc9b922fa977d4c9a668259392b85d3debc9c0090c8ab69c7fd75e48c74c1aa
size 5928466112

3
Falcon3-10B-Instruct-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5bba0ab10458d9157ab2cce73cb5b80472d51c3abb30a8c30ce8c96f0f6cbac8
size 6586365632

3
Falcon3-10B-Instruct-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0a33327bd71e1788a8e9f17889824a17a65efd3f96a4b2a5e2bc6ff2f39b8241
size 6287521472

3
Falcon3-10B-Instruct-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0ee5cd9a6ca5229a1128efca439eb5a819de60494716701d2e8cf1f1fc4204d7
size 5952157376

3
Falcon3-10B-Instruct-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6a08593991072df35c06985421e35958a622756a239beaa47eabb97425c2ca4e
size 7589066432

3
Falcon3-10B-Instruct-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b517e3bb0c5cea162dc78d3ec6faf3050070b831dc62fc043185ee1a13d25b87
size 7340553920

3
Falcon3-10B-Instruct-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dc746e634a575d4408963c058518f76a919f8ff53e849eb932541cfd252bed43
size 7144191680

3
Falcon3-10B-Instruct-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:18238ce1f254925d523a5dc4a1368325cd281b6921058e888218177344abdb9a
size 8459400896

3
Falcon3-10B-Instruct-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a02adea7b8006cc89d1e9f5a7987267a1c051db1ddacae82d11a5ca17e29396c
size 8654436032

3
Falcon3-10B-Instruct-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e3f8f52d40f6e7fac068f7e99ce0ab3fe87fad927b13ef4bec14c1c2a451b363
size 10955241152

3
Falcon3-10B-Instruct-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:68f5dfc8589711205d954f2d281324c793fb6c8bd4ecf37711b89be1cdc4377d
size 20616557984

3
Falcon3-10B-Instruct.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:edcc3047e032777ac26b4d4de5dee85e67f2731a99e658f7f015931a7ac1c0c0
size 6644818

269
README.md Normal file
View File

@@ -0,0 +1,269 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
license_name: falcon-llm-license
tags:
- falcon3
license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
license: other
base_model: tiiuae/Falcon3-10B-Instruct
model-index:
- name: Falcon3-10B-Instruct
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 78.17
name: strict accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=tiiuae/Falcon3-10B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 44.82
name: normalized accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=tiiuae/Falcon3-10B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 25.91
name: exact match
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=tiiuae/Falcon3-10B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 10.51
name: acc_norm
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=tiiuae/Falcon3-10B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 13.61
name: acc_norm
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=tiiuae/Falcon3-10B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 38.1
name: accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=tiiuae/Falcon3-10B-Instruct
name: Open LLM Leaderboard
---
## Llamacpp imatrix Quantizations of Falcon3-10B-Instruct
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b4381">b4381</a> for quantization.
Original model: https://huggingface.co/tiiuae/Falcon3-10B-Instruct
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<|system|>
{system_prompt}
<|user|>
{prompt}
<|assistant|>
```
## What's new:
Fix tokenizer
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Falcon3-10B-Instruct-f16.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-f16.gguf) | f16 | 20.62GB | false | Full F16 weights. |
| [Falcon3-10B-Instruct-Q8_0.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q8_0.gguf) | Q8_0 | 10.96GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Falcon3-10B-Instruct-Q6_K_L.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q6_K_L.gguf) | Q6_K_L | 8.65GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Falcon3-10B-Instruct-Q6_K.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q6_K.gguf) | Q6_K | 8.46GB | false | Very high quality, near perfect, *recommended*. |
| [Falcon3-10B-Instruct-Q5_K_L.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q5_K_L.gguf) | Q5_K_L | 7.59GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Falcon3-10B-Instruct-Q5_K_M.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q5_K_M.gguf) | Q5_K_M | 7.34GB | false | High quality, *recommended*. |
| [Falcon3-10B-Instruct-Q5_K_S.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q5_K_S.gguf) | Q5_K_S | 7.14GB | false | High quality, *recommended*. |
| [Falcon3-10B-Instruct-Q4_K_L.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q4_K_L.gguf) | Q4_K_L | 6.59GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Falcon3-10B-Instruct-Q4_K_M.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q4_K_M.gguf) | Q4_K_M | 6.29GB | false | Good quality, default size for most use cases, *recommended*. |
| [Falcon3-10B-Instruct-Q4_K_S.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q4_K_S.gguf) | Q4_K_S | 5.95GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Falcon3-10B-Instruct-Q4_0.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q4_0.gguf) | Q4_0 | 5.93GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
| [Falcon3-10B-Instruct-IQ4_NL.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-IQ4_NL.gguf) | IQ4_NL | 5.91GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
| [Falcon3-10B-Instruct-Q3_K_XL.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q3_K_XL.gguf) | Q3_K_XL | 5.80GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Falcon3-10B-Instruct-IQ4_XS.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-IQ4_XS.gguf) | IQ4_XS | 5.60GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Falcon3-10B-Instruct-Q3_K_L.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q3_K_L.gguf) | Q3_K_L | 5.45GB | false | Lower quality but usable, good for low RAM availability. |
| [Falcon3-10B-Instruct-Q3_K_M.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q3_K_M.gguf) | Q3_K_M | 5.05GB | false | Low quality. |
| [Falcon3-10B-Instruct-IQ3_M.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-IQ3_M.gguf) | IQ3_M | 4.70GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Falcon3-10B-Instruct-Q3_K_S.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q3_K_S.gguf) | Q3_K_S | 4.59GB | false | Low quality, not recommended. |
| [Falcon3-10B-Instruct-IQ3_XS.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-IQ3_XS.gguf) | IQ3_XS | 4.37GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Falcon3-10B-Instruct-Q2_K_L.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q2_K_L.gguf) | Q2_K_L | 4.32GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Falcon3-10B-Instruct-Q2_K.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-Q2_K.gguf) | Q2_K | 3.92GB | false | Very low quality but surprisingly usable. |
| [Falcon3-10B-Instruct-IQ2_M.gguf](https://huggingface.co/bartowski/Falcon3-10B-Instruct-GGUF/blob/main/Falcon3-10B-Instruct-IQ2_M.gguf) | IQ2_M | 3.59GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Falcon3-10B-Instruct-GGUF --include "Falcon3-10B-Instruct-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Falcon3-10B-Instruct-GGUF --include "Falcon3-10B-Instruct-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Falcon3-10B-Instruct-Q8_0) or download them all in place (./)
</details>
## ARM/AVX information
Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.
Now, however, there is something called "online repacking" for weights. details in [this PR](https://github.com/ggerganov/llama.cpp/pull/9921). If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.
As of llama.cpp build [b4282](https://github.com/ggerganov/llama.cpp/releases/tag/b4282) you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.
Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to [this PR](https://github.com/ggerganov/llama.cpp/pull/10541) which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.
<details>
<summary>Click to view Q4_0_X_X information (deprecated</summary>
I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}