bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-05 05:51:12 +08:00
commit 6ced17f493
28 changed files with 264 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
.gitattributes vendored Normal file
View File

@@ -0,0 +1,60 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_8_8.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_4_8.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_4_4.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-f16.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base.imatrix filter=lfs diff=lfs merge=lfs -text

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0910a270d32e90e8e674c500acffebd783e2eafb192a63d4d9b405cd8828c494
size 2948281504

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1d38f123b57fba68878c06fc57b4fb03e6ca82aa323fd16c6884844f7541f225
size 3784823968

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:676a8a684a8da9e3426f5bac57215b9e3fc11c15c2c45e2a37fb0c3afdeb978b
size 3518747808

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:53bb82668b36aebaf849408af78ecb3a9ab88d167c38a710e7adcef821e43429
size 4447663264

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eddca70eba63299bbec8323fe8386b1a0005327ba9c2d588fd08394a93a87ac2
size 3179132064

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:586b4dca73d28b3f6135c0ba03146ee7efb01ae0eb932a25cbf6689e179988fe
size 3692156064

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9c82ce0bbfb2e691252b8b85fdf6145b123e656d178d09381614eeef1a4d57f7
size 4321957024

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e4a3a3db563383aaf524d659369a1c096f97d58e872398cc1f799433381d2d49
size 4018918560

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f7f28ac836f783ef4302236bf0b08a5acd90c5186f08f250ce62a37f66028c45
size 3664499872

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:962b1336507a81238e5b5d43ed6a1d373a4734d97eadcd97c153a6650a973936
size 4781626528

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c1d499b6531654e5836ffb2ff461575748b75bcf35ebbe9994fff44c0cc80b4a
size 4675892384

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_4_4.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1ac0253dace4a952178680e63063ae028e9b715145fa3dc77188d16a024ac193
size 4661212320

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_4_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5f88839adf9666d525352308d66c15a5abf5ace19c819b6f8a5474f984ec1bb1
size 4661212320

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_8_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6f852a5693ee39343aa4a39819615bdcf5999599e5a0dde0bdc243bcfacb5499
size 4661212320

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:024b2e4711256d2f8824afc2772dc58e55f4c4da45f44afc997491325f785a32
size 5310633120

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bef9154764b1ab5ec415d94411e8afa1e952619d46c2c4d38c6b2d696ed9b50d
size 4920734880

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e56faebaf02c1f004c1c4b01608c960e0a3545b397535c2281904ba32a4b3813
size 4692669600

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2db9deb93c472ade8b6b94358b85808ba0a6a2c1dd7d44938185a0d11c1fc8f7
size 6057219232

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:91aa9d2378f396a350808709c1745335b6befd6d8e19ad891bb5e8362d0720f8
size 5732988064

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cb18d28c5796a10fefa23acb8c2e56ec5f21140d828d39b643be619e21232ccb
size 5599294624

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b33f8665e115a906425525f11d8a95b70823674f9628c553ed78d6b7d425dd31
size 6596007072

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ae1e5b6467221a44022ca1bd1f6f88c2f06c9b9525149622666a08cb5f4c72dd
size 6850466976

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6d6910c38c3d8717a8483bd39a0368d81d82bc52b9ed4d044c4627752064cb4b
size 8540771488

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:79f231251a2694961011699daaf1c37749397edda7be11433c20926eb288d0bc
size 16068891488

3
Llama-3.1-SuperNova-8B-Lite_TIES_with_Base.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dfc8d9a924415ebd418aa04b54bb66b81790cce3216570ccd517d6eaff222a8d
size 4988170

128
README.md Normal file
View File

@@ -0,0 +1,128 @@
---
base_model: Joseph717171/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base
license: llama3.1
pipeline_tag: text-generation
tags:
- mergekit
- merge
quantized_by: bartowski
---
## Llamacpp imatrix Quantizations of Llama-3.1-SuperNova-8B-Lite_TIES_with_Base
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b4014">b4014</a> for quantization.
Original model: https://huggingface.co/Joseph717171/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>
{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-f16.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-f16.gguf) | f16 | 16.07GB | false | Full F16 weights. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q8_0.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q8_0.gguf) | Q8_0 | 8.54GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q6_K_L.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q6_K_L.gguf) | Q6_K_L | 6.85GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q6_K.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q6_K.gguf) | Q6_K | 6.60GB | false | Very high quality, near perfect, *recommended*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_L.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_L.gguf) | Q5_K_L | 6.06GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_M.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_M.gguf) | Q5_K_M | 5.73GB | false | High quality, *recommended*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_S.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q5_K_S.gguf) | Q5_K_S | 5.60GB | false | High quality, *recommended*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_L.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_L.gguf) | Q4_K_L | 5.31GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_M.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_M.gguf) | Q4_K_M | 4.92GB | false | Good quality, default size for must use cases, *recommended*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_XL.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_XL.gguf) | Q3_K_XL | 4.78GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_S.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_S.gguf) | Q4_K_S | 4.69GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0.gguf) | Q4_0 | 4.68GB | false | Legacy format, generally not worth using over similarly sized formats |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_8_8.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_8_8.gguf) | Q4_0_8_8 | 4.66GB | false | Optimized for ARM inference. Requires 'sve' support (see link below). *Don't use on Mac or Windows*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_4_8.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_4_8.gguf) | Q4_0_4_8 | 4.66GB | false | Optimized for ARM inference. Requires 'i8mm' support (see link below). *Don't use on Mac or Windows*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_4_4.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_0_4_4.gguf) | Q4_0_4_4 | 4.66GB | false | Optimized for ARM inference. Should work well on all ARM chips, pick this if you're unsure. *Don't use on Mac or Windows*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ4_XS.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ4_XS.gguf) | IQ4_XS | 4.45GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_L.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_L.gguf) | Q3_K_L | 4.32GB | false | Lower quality but usable, good for low RAM availability. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_M.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_M.gguf) | Q3_K_M | 4.02GB | false | Low quality. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ3_M.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ3_M.gguf) | IQ3_M | 3.78GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q2_K_L.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q2_K_L.gguf) | Q2_K_L | 3.69GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_S.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q3_K_S.gguf) | Q3_K_S | 3.66GB | false | Low quality, not recommended. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ3_XS.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ3_XS.gguf) | IQ3_XS | 3.52GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q2_K.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q2_K.gguf) | Q2_K | 3.18GB | false | Very low quality but surprisingly usable. |
| [Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ2_M.gguf](https://huggingface.co/bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF/blob/main/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-IQ2_M.gguf) | IQ2_M | 2.95GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
Some say that this improves the quality, others don't notice any difference. If you use these models PLEASE COMMENT with your findings. I would like feedback that these are actually used and useful so I don't keep uploading quants no one is using.
Thanks!
## Downloading using huggingface-cli
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF --include "Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-GGUF --include "Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Llama-3.1-SuperNova-8B-Lite_TIES_with_Base-Q8_0) or download them all in place (./)
## Q4_0_X_X
These are *NOT* for Metal (Apple) offloading, only ARM chips.
If you're using an ARM chip, the Q4_0_X_X quants will have a substantial speedup. Check out Q4_0_4_4 speed comparisons [on the original pull request](https://github.com/ggerganov/llama.cpp/pull/5780#pullrequestreview-21657544660)
To check which one would work best for your ARM chip, you can check [AArch64 SoC features](https://gpages.juszkiewicz.com.pl/arm-socs-table/arm-socs.html) (thanks EloyOn!).
## Which file should I choose?
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset
Thank you ZeroWw for the inspiration to experiment with embed/output
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}