bartowski/Crimson_Dawn-v0.2-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Crimson_Dawn-v0.2-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-25 17:17:12 +08:00
commit f35f7f9eff
28 changed files with 278 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
.gitattributes vendored Normal file
View File

@@ -0,0 +1,60 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q4_0_8_8.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q4_0_4_8.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q4_0_4_4.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2-f16.gguf filter=lfs diff=lfs merge=lfs -text
Crimson_Dawn-v0.2.imatrix filter=lfs diff=lfs merge=lfs -text

3
Crimson_Dawn-v0.2-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f26f9830a25f3c09d0f731a3bb2f474191008462df6f4a096ca9e3c81bc4e9c1
size 4435023616

3
Crimson_Dawn-v0.2-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0b1dfecc3336e4ccd32a414d1f6d06439f481d87e8c74829e938393359250149
size 5722232576

3
Crimson_Dawn-v0.2-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c788fc2e63b441af139f8ca28a8d5594f9d21bde8c8f087ea2c62121cfecdecc
size 5306488576

3
Crimson_Dawn-v0.2-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8c81d4fdedb38edf7002eb9b9ff95104b1bedbe5da01f5aca9c8625b81384bbb
size 6742710016

3
Crimson_Dawn-v0.2-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e896c55ab00e8281e1cc63c6c34266c3d3d4f01b25cbf8ff952259e5942345e6
size 4791047936

3
Crimson_Dawn-v0.2-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a3b1bfde277ed111925a731ca16ce776710802d2c4528a8093f4e1fc9715db4b
size 5446407936

3
Crimson_Dawn-v0.2-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6f91661996cc15e3410492587c90033aee9b054d8c40c057f074cb8c9c2a6b33
size 6561502976

3
Crimson_Dawn-v0.2-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4c3f5b1a4cace7050fae73fc86aaa15c0cbf4028445bc41f811bbd0efa7e6b26
size 6083090176

3
Crimson_Dawn-v0.2-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3bd7ff4c14e79ab4cbf01cfa8472b3ecbc7b189fe77cc32f6b6b4cd9289f5b93
size 5534226176

3
Crimson_Dawn-v0.2-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:644589fa7cecc4e8fa39215bf757da255407b3d2c5c87a939df55b4c6c2a9c3c
size 7148705536

3
Crimson_Dawn-v0.2-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d4056c39c00dde833bb2fa62280a841b056431abd9688ed384342ad5dcbf23ff
size 7094638336

3
Crimson_Dawn-v0.2-Q4_0_4_4.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b8166b55698b399aad95c21fcc891b88c049d4cb602d91920afed7e6c62d8b03
size 7071700736

3
Crimson_Dawn-v0.2-Q4_0_4_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5c59d5782adc016e7d378404166e3ed6203ccc38cb8f12b2e13ecae1be741ed1
size 7071700736

3
Crimson_Dawn-v0.2-Q4_0_8_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:406e5354f21b03b6b28c0b690bbca555bf572ef52f5c2007b44af0a5a8418736
size 7071700736

3
Crimson_Dawn-v0.2-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:61b21155be086de2592b4a699182f8693fd18ebbce1ee78e5f1416adb1a7d668
size 7975278336

3
Crimson_Dawn-v0.2-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4cd1c12413f54910a596cc2fbdf3743036c76acd00bb0f3a6ab5d8d43be0bda9
size 7477204736

3
Crimson_Dawn-v0.2-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f5b8e51e421be73b179b4e428a916c8039feff29c0ded80e451d7de4a1c1f526
size 7120197376

3
Crimson_Dawn-v0.2-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:68b7aa81b75ce4f4c5a54d210672cb35808875ade0361311ff1c6d2eeae7e14b
size 9141819136

3
Crimson_Dawn-v0.2-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:72b14dc9686f76e2605f8ba248fbc6176d0066f9cde59dfba87c2b2eec239648
size 8727631616

3
Crimson_Dawn-v0.2-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e4fda11406e19605edc27b174c6039d049142bd23607e380d4cea9b1f2d530cb
size 8518735616

3
Crimson_Dawn-v0.2-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f8b6a1ec365152af970ad8c328052f18fd90da79b8999e5fe6c9e8f610d3c9e3
size 10056210176

3
Crimson_Dawn-v0.2-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9ba31facd665b4ddc93a5a6fef0030385e9f795401f5c4a72022c7faa5b060c1
size 10381268736

3
Crimson_Dawn-v0.2-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9d2b45246370660f59ddf9a97900c645e4adc5d538894735880eccddb032f2f8
size 13022369536

3
Crimson_Dawn-v0.2-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9a201196026a9e76ba69fe7a3dfd809b92134630804fb8d6a0174d6b9789737d
size 24504276448

3
Crimson_Dawn-v0.2.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:afc6e36558b80e581e2156c499ec44b751fbb97f223dfe6438c1b910ed4666a2
size 7054418

142
README.md Normal file
View File

@@ -0,0 +1,142 @@
---
base_model: Epiculous/Crimson_Dawn-v0.2
datasets:
- Epiculous/SynthRP-Gens-v1.1-Filtered-n-Cleaned
- anthracite-org/stheno-filtered-v1.1
- PJMixers/hieunguyenminh_roleplay-deduped-ShareGPT
- Gryphe/Sonnet3.5-Charcard-Roleplay
- Epiculous/Synthstruct-Gens-v1.1-Filtered-n-Cleaned
- anthracite-org/kalo-opus-instruct-22k-no-refusal
- anthracite-org/nopm_claude_writing_fixed
- anthracite-org/kalo_opus_misc_240827
language:
- en
- fr
- de
- es
- it
- pt
- ru
- zh
- ja
license: apache-2.0
pipeline_tag: text-generation
quantized_by: bartowski
---
## Llamacpp imatrix Quantizations of Crimson_Dawn-v0.2
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b3658">b3658</a> for quantization.
Original model: https://huggingface.co/Epiculous/Crimson_Dawn-v0.2
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Crimson_Dawn-v0.2-f16.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-f16.gguf) | f16 | 24.50GB | false | Full F16 weights. |
| [Crimson_Dawn-v0.2-Q8_0.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q8_0.gguf) | Q8_0 | 13.02GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Crimson_Dawn-v0.2-Q6_K_L.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q6_K_L.gguf) | Q6_K_L | 10.38GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Crimson_Dawn-v0.2-Q6_K.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q6_K.gguf) | Q6_K | 10.06GB | false | Very high quality, near perfect, *recommended*. |
| [Crimson_Dawn-v0.2-Q5_K_L.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q5_K_L.gguf) | Q5_K_L | 9.14GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Crimson_Dawn-v0.2-Q5_K_M.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q5_K_M.gguf) | Q5_K_M | 8.73GB | false | High quality, *recommended*. |
| [Crimson_Dawn-v0.2-Q5_K_S.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q5_K_S.gguf) | Q5_K_S | 8.52GB | false | High quality, *recommended*. |
| [Crimson_Dawn-v0.2-Q4_K_L.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q4_K_L.gguf) | Q4_K_L | 7.98GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Crimson_Dawn-v0.2-Q4_K_M.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q4_K_M.gguf) | Q4_K_M | 7.48GB | false | Good quality, default size for must use cases, *recommended*. |
| [Crimson_Dawn-v0.2-Q3_K_XL.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q3_K_XL.gguf) | Q3_K_XL | 7.15GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Crimson_Dawn-v0.2-Q4_K_S.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q4_K_S.gguf) | Q4_K_S | 7.12GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Crimson_Dawn-v0.2-Q4_0.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q4_0.gguf) | Q4_0 | 7.09GB | false | Legacy format, generally not worth using over similarly sized formats |
| [Crimson_Dawn-v0.2-Q4_0_8_8.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q4_0_8_8.gguf) | Q4_0_8_8 | 7.07GB | false | Optimized for ARM inference. Requires 'sve' support (see link below). |
| [Crimson_Dawn-v0.2-Q4_0_4_8.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q4_0_4_8.gguf) | Q4_0_4_8 | 7.07GB | false | Optimized for ARM inference. Requires 'i8mm' support (see link below). |
| [Crimson_Dawn-v0.2-Q4_0_4_4.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q4_0_4_4.gguf) | Q4_0_4_4 | 7.07GB | false | Optimized for ARM inference. Should work well on all ARM chips, pick this if you're unsure. |
| [Crimson_Dawn-v0.2-IQ4_XS.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-IQ4_XS.gguf) | IQ4_XS | 6.74GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Crimson_Dawn-v0.2-Q3_K_L.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q3_K_L.gguf) | Q3_K_L | 6.56GB | false | Lower quality but usable, good for low RAM availability. |
| [Crimson_Dawn-v0.2-Q3_K_M.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q3_K_M.gguf) | Q3_K_M | 6.08GB | false | Low quality. |
| [Crimson_Dawn-v0.2-IQ3_M.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-IQ3_M.gguf) | IQ3_M | 5.72GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Crimson_Dawn-v0.2-Q3_K_S.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q3_K_S.gguf) | Q3_K_S | 5.53GB | false | Low quality, not recommended. |
| [Crimson_Dawn-v0.2-Q2_K_L.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q2_K_L.gguf) | Q2_K_L | 5.45GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Crimson_Dawn-v0.2-IQ3_XS.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-IQ3_XS.gguf) | IQ3_XS | 5.31GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Crimson_Dawn-v0.2-Q2_K.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-Q2_K.gguf) | Q2_K | 4.79GB | false | Very low quality but surprisingly usable. |
| [Crimson_Dawn-v0.2-IQ2_M.gguf](https://huggingface.co/bartowski/Crimson_Dawn-v0.2-GGUF/blob/main/Crimson_Dawn-v0.2-IQ2_M.gguf) | IQ2_M | 4.44GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
Some say that this improves the quality, others don't notice any difference. If you use these models PLEASE COMMENT with your findings. I would like feedback that these are actually used and useful so I don't keep uploading quants no one is using.
Thanks!
## Downloading using huggingface-cli
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Crimson_Dawn-v0.2-GGUF --include "Crimson_Dawn-v0.2-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Crimson_Dawn-v0.2-GGUF --include "Crimson_Dawn-v0.2-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Crimson_Dawn-v0.2-Q8_0) or download them all in place (./)
## Q4_0_X_X
If you're using an ARM chip, the Q4_0_X_X quants will have a substantial speedup. Check out Q4_0_4_4 speed comparisons [on the original pull request](https://github.com/ggerganov/llama.cpp/pull/5780#pullrequestreview-21657544660)
To check which one would work best for your ARM chip, you can check [AArch64 SoC features](https://gpages.juszkiewicz.com.pl/arm-socs-table/arm-socs.html) (thanks EloyOn!).
## Which file should I choose?
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset
Thank you ZeroWw for the inspiration to experiment with embed/output
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}