bartowski/HelpingAI_Helpingai3-raw-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/HelpingAI_Helpingai3-raw-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-09-19 18:56:12 +08:00
commit 54d5b8774a
29 changed files with 310 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
.gitattributes vendored Normal file
View File

@@ -0,0 +1,49 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.gguf* filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
HelpingAI_Helpingai3-raw.imatrix filter=lfs diff=lfs merge=lfs -text

3
HelpingAI_Helpingai3-raw-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d687c71610b6201f83cd13eae8b3f04b55ae4bc9a2fdb2130a7b5d1f6cb33461
size 3592343488

3
HelpingAI_Helpingai3-raw-IQ2_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8e084c3ac9ad3857899b27256aff5d301c69f3ba73876942ba9e99a61799b3ce
size 3322253248

3
HelpingAI_Helpingai3-raw-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4220bec13075eae545f0532dbcd87b832abc3cc574cd81a08baae1de1a87a6a7
size 4704985024

3
HelpingAI_Helpingai3-raw-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:777d2b9a53bc6f7eacba4d487c237f8735c61bc7e38824fe74583e4789a68cf9
size 4368478144

3
HelpingAI_Helpingai3-raw-IQ3_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8ac05e5ab865c7ee520f2d33bbb9ef516dcf4fa409ed34492bae8ddc9a988990
size 4078579648

3
HelpingAI_Helpingai3-raw-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f011dfe45cde98b0b168938dec5ac02ce9d7ac10daaa77d09dcc9c7dc7496ada
size 5906345920

3
HelpingAI_Helpingai3-raw-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:55a38290fefdf84425e2ca57400371606e568beb84287650dfd1080a3422eac2
size 5596884928

3
HelpingAI_Helpingai3-raw-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2951690d808dfa8ba7f3b44e056bc8229954b4e9540dcc9e8235fd78447594d5
size 3924045760

3
HelpingAI_Helpingai3-raw-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b0be0aa285abf1ef43f346d516a90f95bd5ed017505b2cb372f4cf87dfa015c2
size 4317261760

3
HelpingAI_Helpingai3-raw-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:607da2daf9b135190b8a2a41f5a7b10ea885987276eca11512aec49b818122eb
size 5450805184

3
HelpingAI_Helpingai3-raw-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:32462156f6247c621fde4e5e54b70e0e48454c29e29ca5efae75dd49b74ba13f
size 5052477376

3
HelpingAI_Helpingai3-raw-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:582cbbc3150df7c8e0eab2863dfe5509d3e6af4f090c38c674acd06f03553b28
size 4591136704

3
HelpingAI_Helpingai3-raw-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eac9392429d1614cd6a0d253a02c61713f51aa3cc2faaeeb85f21c79a2f88c00
size 5803126720

3
HelpingAI_Helpingai3-raw-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:73a8df72c993a0a65d130440ebda6f25cac6f27b5f4523c910a5b4cbb569fa85
size 5928464320

3
HelpingAI_Helpingai3-raw-Q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0d61f1415a4a298d4b042fbf9d72a68ead05c0950f18822b17996f6460f6c1bd
size 6525267904

3
HelpingAI_Helpingai3-raw-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7002b499ce207aa9419666e188df1f1e76f5dbd03b3ca6d094f9845871fceaeb
size 6586363840

3
HelpingAI_Helpingai3-raw-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:de7a223ad397ba27c889dad08466de471166f1e76962b855c72cf6b779a7b857
size 6287519680

3
HelpingAI_Helpingai3-raw-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a21e7b876adac36f88f9bfb1dcaaa1b25e86f2db5c181c8eb289cedd776249c1
size 5952155584

3
HelpingAI_Helpingai3-raw-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:99c06f663e1704c501befa730e07b8a18e555ec75e65cbab5fc5e12f100c9dc1
size 7589064640

3
HelpingAI_Helpingai3-raw-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eafe59cb1b9de88c7182b2c5c5ad92bb0f39cdb638908bf663dde95b07f61ce4
size 7340552128

3
HelpingAI_Helpingai3-raw-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8a4036d121e13013146d13b939cfb9def28c2b4c6cb9a161c3200043bf40af14
size 7144189888

3
HelpingAI_Helpingai3-raw-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:58c89c6601763af3bae039e439f8b031387386dfa42c5362f78d23325ff88aa0
size 8459399104

3
HelpingAI_Helpingai3-raw-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ef6e20acf7953b2f5eb1b827d5fdc130386742162afcc2bf0e86a64e370b8a6f
size 8654434240

3
HelpingAI_Helpingai3-raw-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4588de616cec378d69eea05715537d5a9bdbb960e3b9d378cddcdcdcca526382
size 10955239360

3
HelpingAI_Helpingai3-raw-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fbec6f0e6c3ec436726dcacdd9eaba0d45051fa8fab33b830f8addfe9a122420
size 20616556192

3
HelpingAI_Helpingai3-raw.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fb33427317ff12ac45ac65a9e4b413ec8d179246385e540f246db8b414ff182b
size 6644818

182
README.md Normal file
View File

@@ -0,0 +1,182 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
license: other
base_model_relation: quantized
language:
- en
license_name: helpingai
base_model: HelpingAI/Helpingai3-raw
tags:
- HelpingAI
- Emotionally-Intelligent
- EQ-focused
- Conversational
- SLM
license_link: https://helpingai.co/license
---
## Llamacpp imatrix Quantizations of Helpingai3-raw by HelpingAI
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b4925">b4925</a> for quantization.
Original model: https://huggingface.co/HelpingAI/Helpingai3-raw
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
Run them directly with [llama.cpp](https://github.com/ggerganov/llama.cpp), or any other llama.cpp based project
## Prompt format
```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Helpingai3-raw-f16.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-f16.gguf) | f16 | 20.62GB | false | Full F16 weights. |
| [Helpingai3-raw-Q8_0.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q8_0.gguf) | Q8_0 | 10.96GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Helpingai3-raw-Q6_K_L.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q6_K_L.gguf) | Q6_K_L | 8.65GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Helpingai3-raw-Q6_K.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q6_K.gguf) | Q6_K | 8.46GB | false | Very high quality, near perfect, *recommended*. |
| [Helpingai3-raw-Q5_K_L.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q5_K_L.gguf) | Q5_K_L | 7.59GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Helpingai3-raw-Q5_K_M.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q5_K_M.gguf) | Q5_K_M | 7.34GB | false | High quality, *recommended*. |
| [Helpingai3-raw-Q5_K_S.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q5_K_S.gguf) | Q5_K_S | 7.14GB | false | High quality, *recommended*. |
| [Helpingai3-raw-Q4_K_L.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q4_K_L.gguf) | Q4_K_L | 6.59GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Helpingai3-raw-Q4_1.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q4_1.gguf) | Q4_1 | 6.53GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
| [Helpingai3-raw-Q4_K_M.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q4_K_M.gguf) | Q4_K_M | 6.29GB | false | Good quality, default size for most use cases, *recommended*. |
| [Helpingai3-raw-Q4_K_S.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q4_K_S.gguf) | Q4_K_S | 5.95GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Helpingai3-raw-Q4_0.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q4_0.gguf) | Q4_0 | 5.93GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
| [Helpingai3-raw-IQ4_NL.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-IQ4_NL.gguf) | IQ4_NL | 5.91GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
| [Helpingai3-raw-Q3_K_XL.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q3_K_XL.gguf) | Q3_K_XL | 5.80GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Helpingai3-raw-IQ4_XS.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-IQ4_XS.gguf) | IQ4_XS | 5.60GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Helpingai3-raw-Q3_K_L.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q3_K_L.gguf) | Q3_K_L | 5.45GB | false | Lower quality but usable, good for low RAM availability. |
| [Helpingai3-raw-Q3_K_M.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q3_K_M.gguf) | Q3_K_M | 5.05GB | false | Low quality. |
| [Helpingai3-raw-IQ3_M.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-IQ3_M.gguf) | IQ3_M | 4.70GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Helpingai3-raw-Q3_K_S.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q3_K_S.gguf) | Q3_K_S | 4.59GB | false | Low quality, not recommended. |
| [Helpingai3-raw-IQ3_XS.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-IQ3_XS.gguf) | IQ3_XS | 4.37GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Helpingai3-raw-Q2_K_L.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q2_K_L.gguf) | Q2_K_L | 4.32GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Helpingai3-raw-IQ3_XXS.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-IQ3_XXS.gguf) | IQ3_XXS | 4.08GB | false | Lower quality, new method with decent performance, comparable to Q3 quants. |
| [Helpingai3-raw-Q2_K.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-Q2_K.gguf) | Q2_K | 3.92GB | false | Very low quality but surprisingly usable. |
| [Helpingai3-raw-IQ2_M.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-IQ2_M.gguf) | IQ2_M | 3.59GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
| [Helpingai3-raw-IQ2_S.gguf](https://huggingface.co/bartowski/HelpingAI_Helpingai3-raw-GGUF/blob/main/HelpingAI_Helpingai3-raw-IQ2_S.gguf) | IQ2_S | 3.32GB | false | Low quality, uses SOTA techniques to be usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/HelpingAI_Helpingai3-raw-GGUF --include "HelpingAI_Helpingai3-raw-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/HelpingAI_Helpingai3-raw-GGUF --include "HelpingAI_Helpingai3-raw-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (HelpingAI_Helpingai3-raw-Q8_0) or download them all in place (./)
</details>
## ARM/AVX information
Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.
Now, however, there is something called "online repacking" for weights. details in [this PR](https://github.com/ggerganov/llama.cpp/pull/9921). If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.
As of llama.cpp build [b4282](https://github.com/ggerganov/llama.cpp/releases/tag/b4282) you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.
Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to [this PR](https://github.com/ggerganov/llama.cpp/pull/10541) which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.
<details>
<summary>Click to view Q4_0_X_X information (deprecated</summary>
I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Thank you to LM Studio for sponsoring my work.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}