bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-10-16 14:32:21 +08:00
commit de1acf4597
29 changed files with 310 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
.gitattributes vendored Normal file
View File

@@ -0,0 +1,49 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.gguf* filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Delta-Vector_Austral-Xgen-9B-Winton.imatrix filter=lfs diff=lfs merge=lfs -text

3
Delta-Vector_Austral-Xgen-9B-Winton-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e9fd2677b9869fa2049ca55e1416120c296f89357ef02f08fc5e9576b6b87a8c
size 3777847456

3
Delta-Vector_Austral-Xgen-9B-Winton-IQ2_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:92a87400055a1a629a4d65e9befe02e71ddb89495ffd5461d9bfe62a75075a57
size 3509805216

3
Delta-Vector_Austral-Xgen-9B-Winton-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6bea3275fa40911dc95848a136451ad0c84f84ea07dd34a19d953ce41749c9fc
size 4910964896

3
Delta-Vector_Austral-Xgen-9B-Winton-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:59473da2271a0e04e59efedf469d3c2399b106093826a0dd4db6121506cb1f3d
size 4540293280

3
Delta-Vector_Austral-Xgen-9B-Winton-IQ3_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:374735c3a69b3d214213b0d80b1087df39ca77f548e3556129cb1ac329b9aad9
size 4241186976

3
Delta-Vector_Austral-Xgen-9B-Winton-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2a3bd8516fe69bce3090f2970f99536bd0ede91468c1d847dfa67c0b55e9da39
size 6129442976

3
Delta-Vector_Austral-Xgen-9B-Winton-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:755cd9118d2693508cc87f743a2a95e6038a8e3e4eb4ee7bcbe43d69c15df36b
size 5815525536

3
Delta-Vector_Austral-Xgen-9B-Winton-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:55ce5c04ed7489c80d00f16d7b685179f64a10e5f24c0174f338ce21fe74c4cf
size 4096516256

3
Delta-Vector_Austral-Xgen-9B-Winton-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1a05ddac96d75f843dd71f09100e0b97744e9bbbe0089877c4baddc0269de061
size 4506116256

3
Delta-Vector_Austral-Xgen-9B-Winton-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:92c1312e205f8d9b3d3ae9ec7ef9413271d4b079dcdef16ed5e0731bef7b5fd5
size 5671182496

3
Delta-Vector_Austral-Xgen-9B-Winton-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:aebde6bb8c91c5e1ce7b34bd1ec9d0b22027f43c41af451c8bb11d167d15f9a2
size 5238644896

3
Delta-Vector_Austral-Xgen-9B-Winton-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b19a2b393ff3a41911dd71b4effc4a4a39c0d1a1fd6bbd746c2db17063b6f8c6
size 4746633376

3
Delta-Vector_Austral-Xgen-9B-Winton-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7a75f0b753af96326e32e6e753c09bac71deb7b1c64aed64adfcce7d54db4ca7
size 6038184096

3
Delta-Vector_Austral-Xgen-9B-Winton-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:43dde56baca749081c0bbb5298097d96fa0dfd0dd47e65e6cbdbd50d5d976dbe
size 6124200096

3
Delta-Vector_Austral-Xgen-9B-Winton-Q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:829e969c41d94e4cc7b520cb9aca6f150092d2f429f4d7acc449e3d435db2dc2
size 6745481376

3
Delta-Vector_Austral-Xgen-9B-Winton-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:150af57ee4e926bd7193626ff920bfce05b075f6e51fa49fc035dca62b1643ff
size 6773989536

3
Delta-Vector_Austral-Xgen-9B-Winton-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:34665ecfbb3489e6c365b9b7e347db7c64c5dfd0346fd24c5f5eb37ae55a1a1c
size 6462693536

3
Delta-Vector_Austral-Xgen-9B-Winton-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc5326b5c99235423f0faa83514bc68f9104833c0515529c22e7cd8518d08887
size 6144647328

3
Delta-Vector_Austral-Xgen-9B-Winton-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c2eaea279c00d75d0f59f3be34558c3dbc2d10950f466863e71ddc1ad4138179
size 7827808416

3
Delta-Vector_Austral-Xgen-9B-Winton-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6aa87062db5c4dea7a781437a927f5a7e4437b44d72b324a9e9842c4ffa5a041
size 7568941216

3
Delta-Vector_Austral-Xgen-9B-Winton-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bbdfffa5c3e1ddd8835abffce872d3cad18647457a8b798a97fc84e59e4ed8d0
size 7385112736

3
Delta-Vector_Austral-Xgen-9B-Winton-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ce140d69a75db948dd99c98db9113ed10b4ce950b4358d0266a9ec1214013682
size 8744329376

3
Delta-Vector_Austral-Xgen-9B-Winton-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:508df54f22dbcf4766859c68aafd34a26df25e0de32ab9e145a963d61e3516dd
size 8947490976

3
Delta-Vector_Austral-Xgen-9B-Winton-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4d4e73bfe214d7b093713060eeee0e7b6be8d10929c51fa2b048432837c82e25
size 11324481696

3
Delta-Vector_Austral-Xgen-9B-Winton-bf16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d54723ce54fdca76b9e69d167036b324662b133310f4425b87e5cf515ebd0f6b
size 21312167776

3
Delta-Vector_Austral-Xgen-9B-Winton.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5c00fb3b3cf8c976f2cb39f6a6efee992fe94e11e3999567060947f7e91330a1
size 7014623

182
README.md Normal file
View File

@@ -0,0 +1,182 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
base_model: Delta-Vector/Austral-Xgen-9B-Winton
base_model_relation: quantized
license: apache-2.0
language:
- en
tags:
- roleplay
- finetune
- axolotl
- adventure
- creative-writing
- Mistral
- 24B
---
## Llamacpp imatrix Quantizations of Austral-Xgen-9B-Winton by Delta-Vector
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b5697">b5697</a> for quantization.
Original model: https://huggingface.co/Delta-Vector/Austral-Xgen-9B-Winton
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
Run them directly with [llama.cpp](https://github.com/ggerganov/llama.cpp), or any other llama.cpp based project
## Prompt format
```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Austral-Xgen-9B-Winton-bf16.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-bf16.gguf) | bf16 | 21.31GB | false | Full BF16 weights. |
| [Austral-Xgen-9B-Winton-Q8_0.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q8_0.gguf) | Q8_0 | 11.32GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Austral-Xgen-9B-Winton-Q6_K_L.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q6_K_L.gguf) | Q6_K_L | 8.95GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Austral-Xgen-9B-Winton-Q6_K.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q6_K.gguf) | Q6_K | 8.74GB | false | Very high quality, near perfect, *recommended*. |
| [Austral-Xgen-9B-Winton-Q5_K_L.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q5_K_L.gguf) | Q5_K_L | 7.83GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Austral-Xgen-9B-Winton-Q5_K_M.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q5_K_M.gguf) | Q5_K_M | 7.57GB | false | High quality, *recommended*. |
| [Austral-Xgen-9B-Winton-Q5_K_S.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q5_K_S.gguf) | Q5_K_S | 7.39GB | false | High quality, *recommended*. |
| [Austral-Xgen-9B-Winton-Q4_K_L.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q4_K_L.gguf) | Q4_K_L | 6.77GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Austral-Xgen-9B-Winton-Q4_1.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q4_1.gguf) | Q4_1 | 6.75GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
| [Austral-Xgen-9B-Winton-Q4_K_M.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q4_K_M.gguf) | Q4_K_M | 6.46GB | false | Good quality, default size for most use cases, *recommended*. |
| [Austral-Xgen-9B-Winton-Q4_K_S.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q4_K_S.gguf) | Q4_K_S | 6.14GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Austral-Xgen-9B-Winton-IQ4_NL.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-IQ4_NL.gguf) | IQ4_NL | 6.13GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
| [Austral-Xgen-9B-Winton-Q4_0.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q4_0.gguf) | Q4_0 | 6.12GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
| [Austral-Xgen-9B-Winton-Q3_K_XL.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q3_K_XL.gguf) | Q3_K_XL | 6.04GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Austral-Xgen-9B-Winton-IQ4_XS.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-IQ4_XS.gguf) | IQ4_XS | 5.82GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Austral-Xgen-9B-Winton-Q3_K_L.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q3_K_L.gguf) | Q3_K_L | 5.67GB | false | Lower quality but usable, good for low RAM availability. |
| [Austral-Xgen-9B-Winton-Q3_K_M.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q3_K_M.gguf) | Q3_K_M | 5.24GB | false | Low quality. |
| [Austral-Xgen-9B-Winton-IQ3_M.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-IQ3_M.gguf) | IQ3_M | 4.91GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Austral-Xgen-9B-Winton-Q3_K_S.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q3_K_S.gguf) | Q3_K_S | 4.75GB | false | Low quality, not recommended. |
| [Austral-Xgen-9B-Winton-IQ3_XS.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-IQ3_XS.gguf) | IQ3_XS | 4.54GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Austral-Xgen-9B-Winton-Q2_K_L.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q2_K_L.gguf) | Q2_K_L | 4.51GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Austral-Xgen-9B-Winton-IQ3_XXS.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-IQ3_XXS.gguf) | IQ3_XXS | 4.24GB | false | Lower quality, new method with decent performance, comparable to Q3 quants. |
| [Austral-Xgen-9B-Winton-Q2_K.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-Q2_K.gguf) | Q2_K | 4.10GB | false | Very low quality but surprisingly usable. |
| [Austral-Xgen-9B-Winton-IQ2_M.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-IQ2_M.gguf) | IQ2_M | 3.78GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
| [Austral-Xgen-9B-Winton-IQ2_S.gguf](https://huggingface.co/bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF/blob/main/Delta-Vector_Austral-Xgen-9B-Winton-IQ2_S.gguf) | IQ2_S | 3.51GB | false | Low quality, uses SOTA techniques to be usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF --include "Delta-Vector_Austral-Xgen-9B-Winton-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Delta-Vector_Austral-Xgen-9B-Winton-GGUF --include "Delta-Vector_Austral-Xgen-9B-Winton-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Delta-Vector_Austral-Xgen-9B-Winton-Q8_0) or download them all in place (./)
</details>
## ARM/AVX information
Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.
Now, however, there is something called "online repacking" for weights. details in [this PR](https://github.com/ggerganov/llama.cpp/pull/9921). If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.
As of llama.cpp build [b4282](https://github.com/ggerganov/llama.cpp/releases/tag/b4282) you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.
Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to [this PR](https://github.com/ggerganov/llama.cpp/pull/10541) which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.
<details>
<summary>Click to view Q4_0_X_X information (deprecated</summary>
I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Thank you to LM Studio for sponsoring my work.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}