bartowski/Llama3.1-8B-ShiningValiant2-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Llama3.1-8B-ShiningValiant2-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-09-23 03:41:13 +08:00
commit 02e96e371a
28 changed files with 294 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
.gitattributes vendored Normal file
View File

@@ -0,0 +1,60 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q4_0_4_8.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q4_0_4_4.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-f16.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q4_0_8_8.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama3.1-8B-ShiningValiant2.imatrix filter=lfs diff=lfs merge=lfs -text

3
Llama3.1-8B-ShiningValiant2-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:582c3441d9bc9f072c22c28913899550afcbefaa25d63a8f146563ab2211972b
size 2948286624

3
Llama3.1-8B-ShiningValiant2-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:96e415513047595736c55cd663a21b6cc2c7f9a6e50aa60903e6e86fdc033865
size 3784829088

3
Llama3.1-8B-ShiningValiant2-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:57c65ae1711f6c6673b79292b84f7e5e871f3c564723d3d8c7640b41189da250
size 3518752928

3
Llama3.1-8B-ShiningValiant2-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3fed70a56129cf6a552595c75446ff85f4fa6f21a2cab227d30be48d74e5c952
size 4447668384

3
Llama3.1-8B-ShiningValiant2-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ba33d9d4b2596f45c6454130cdd4b392aa43deb3b4c29517817dcd07fedab758
size 3179137184

3
Llama3.1-8B-ShiningValiant2-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:496081dbd4b05899a770f51b85b30bfc577786a5ae521937d153da1ec375c2fd
size 3692161184

3
Llama3.1-8B-ShiningValiant2-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:937c6fe46ccd670750d453d39434fa1ec1f7e8b1b07f36a01dfe587f8f6fd4d4
size 4321962144

3
Llama3.1-8B-ShiningValiant2-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f4831e8440c6b1b223d2ea5a637bd2a5b687a5f5a38a069676b45489ab84d5aa
size 4018923680

3
Llama3.1-8B-ShiningValiant2-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0bc63518cdc5a5bd077270699e94fe1085f8ecab520e20272aa8e613cf90081c
size 3664504992

3
Llama3.1-8B-ShiningValiant2-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d71827f82368a6f60998eea4f383594868e34b65f4dbf3dcc456661976a1686e
size 4781631648

3
Llama3.1-8B-ShiningValiant2-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e723aac57a451d7d1f931b69624c066ef082bb1788548f280988bbef4f8de264
size 4675897504

3
Llama3.1-8B-ShiningValiant2-Q4_0_4_4.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:da797b7cc1e8c7d5caa135dc848d81655abd3cd532e58ad5b7dea88a5d4d2942
size 4661217440

3
Llama3.1-8B-ShiningValiant2-Q4_0_4_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc3445632a199870d349cbdf8f4aa6ca9704a9ca5290da22d5a39c963d2ee14c
size 4661217440

3
Llama3.1-8B-ShiningValiant2-Q4_0_8_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:80cf466b26634725318c7acca6bef3149484fcf60630bfce4c697944129a54a6
size 4661217440

3
Llama3.1-8B-ShiningValiant2-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fc99d05a72e096fff1a6c6fa92e4d38470928d9d0086cf64f0c2935ffe5da59f
size 5310638240

3
Llama3.1-8B-ShiningValiant2-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9369eb97922a9f01e4eae610e3d7aaeca30762d78d9239884179451d60bdbdd2
size 4920740000

3
Llama3.1-8B-ShiningValiant2-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e33db816fc57ae5ec47cd6b9fe856bbce78fce4d54b8a78a58bd6065e5ee7f41
size 4692674720

3
Llama3.1-8B-ShiningValiant2-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1fa1abc3b7aef9bc9a9b06d3a4fa5bdba87c4d955a7ec05c34efeb988e223846
size 6057224352

3
Llama3.1-8B-ShiningValiant2-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9bf96377c3b02c9fdeb579dacdef387eb5c8240f6245f5d5f0931b615ec1d1f0
size 5732993184

3
Llama3.1-8B-ShiningValiant2-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8757644260c96a3ae2c6d50d0d32284ac883b93c6bd59fdcb2f829c59434472a
size 5599299744

3
Llama3.1-8B-ShiningValiant2-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:34eb727898bfec26f0ef956b8a5d8a45c82a44e5352ab9b9731706cc472d24ef
size 6596012192

3
Llama3.1-8B-ShiningValiant2-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:95241b1108d9ecc0f5ae87d61d2eb5f9928f3e6efc5013f38124a161716e2bee
size 6850472096

3
Llama3.1-8B-ShiningValiant2-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0ae169c9c2d08bb61fad0d525cfd6f1131aa3b9f19b47d6703a2f2c880fad1ec
size 8540776608

3
Llama3.1-8B-ShiningValiant2-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9ae5af0e427df99d61ab244eac7fb0bd5d67fb3b55b002ec381c611a1c56cb5d
size 16068896640

3
Llama3.1-8B-ShiningValiant2.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2e98c7e93eb241347cb84b0be152232335de36908b46f70153c2ffe0a3ad1522
size 4988170

158
README.md Normal file
View File

@@ -0,0 +1,158 @@
---
base_model: ValiantLabs/Llama3.1-8B-ShiningValiant2
datasets:
- sequelbox/Celestia
- sequelbox/Supernova
language:
- en
license: llama3.1
pipeline_tag: text-generation
tags:
- shining-valiant
- shining-valiant-2
- valiant
- valiant-labs
- llama
- llama-3.1
- llama-3.1-instruct
- llama-3.1-instruct-8b
- llama-3
- llama-3-instruct
- llama-3-instruct-8b
- 8b
- science
- physics
- biology
- chemistry
- compsci
- computer-science
- engineering
- technical
- conversational
- chat
- instruct
quantized_by: bartowski
model_type: llama
---
## Llamacpp imatrix Quantizations of Llama3.1-8B-ShiningValiant2
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b3772">b3772</a> for quantization.
Original model: https://huggingface.co/ValiantLabs/Llama3.1-8B-ShiningValiant2
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024
{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>
{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Llama3.1-8B-ShiningValiant2-f16.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-f16.gguf) | f16 | 16.07GB | false | Full F16 weights. |
| [Llama3.1-8B-ShiningValiant2-Q8_0.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q8_0.gguf) | Q8_0 | 8.54GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Llama3.1-8B-ShiningValiant2-Q6_K_L.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q6_K_L.gguf) | Q6_K_L | 6.85GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Llama3.1-8B-ShiningValiant2-Q6_K.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q6_K.gguf) | Q6_K | 6.60GB | false | Very high quality, near perfect, *recommended*. |
| [Llama3.1-8B-ShiningValiant2-Q5_K_L.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q5_K_L.gguf) | Q5_K_L | 6.06GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Llama3.1-8B-ShiningValiant2-Q5_K_M.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q5_K_M.gguf) | Q5_K_M | 5.73GB | false | High quality, *recommended*. |
| [Llama3.1-8B-ShiningValiant2-Q5_K_S.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q5_K_S.gguf) | Q5_K_S | 5.60GB | false | High quality, *recommended*. |
| [Llama3.1-8B-ShiningValiant2-Q4_K_L.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q4_K_L.gguf) | Q4_K_L | 5.31GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Llama3.1-8B-ShiningValiant2-Q4_K_M.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q4_K_M.gguf) | Q4_K_M | 4.92GB | false | Good quality, default size for must use cases, *recommended*. |
| [Llama3.1-8B-ShiningValiant2-Q3_K_XL.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q3_K_XL.gguf) | Q3_K_XL | 4.78GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Llama3.1-8B-ShiningValiant2-Q4_K_S.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q4_K_S.gguf) | Q4_K_S | 4.69GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Llama3.1-8B-ShiningValiant2-Q4_0.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q4_0.gguf) | Q4_0 | 4.68GB | false | Legacy format, generally not worth using over similarly sized formats |
| [Llama3.1-8B-ShiningValiant2-Q4_0_8_8.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q4_0_8_8.gguf) | Q4_0_8_8 | 4.66GB | false | Optimized for ARM inference. Requires 'sve' support (see link below). |
| [Llama3.1-8B-ShiningValiant2-Q4_0_4_8.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q4_0_4_8.gguf) | Q4_0_4_8 | 4.66GB | false | Optimized for ARM inference. Requires 'i8mm' support (see link below). |
| [Llama3.1-8B-ShiningValiant2-Q4_0_4_4.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q4_0_4_4.gguf) | Q4_0_4_4 | 4.66GB | false | Optimized for ARM inference. Should work well on all ARM chips, pick this if you're unsure. |
| [Llama3.1-8B-ShiningValiant2-IQ4_XS.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-IQ4_XS.gguf) | IQ4_XS | 4.45GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Llama3.1-8B-ShiningValiant2-Q3_K_L.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q3_K_L.gguf) | Q3_K_L | 4.32GB | false | Lower quality but usable, good for low RAM availability. |
| [Llama3.1-8B-ShiningValiant2-Q3_K_M.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q3_K_M.gguf) | Q3_K_M | 4.02GB | false | Low quality. |
| [Llama3.1-8B-ShiningValiant2-IQ3_M.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-IQ3_M.gguf) | IQ3_M | 3.78GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Llama3.1-8B-ShiningValiant2-Q2_K_L.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q2_K_L.gguf) | Q2_K_L | 3.69GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Llama3.1-8B-ShiningValiant2-Q3_K_S.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q3_K_S.gguf) | Q3_K_S | 3.66GB | false | Low quality, not recommended. |
| [Llama3.1-8B-ShiningValiant2-IQ3_XS.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-IQ3_XS.gguf) | IQ3_XS | 3.52GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Llama3.1-8B-ShiningValiant2-Q2_K.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-Q2_K.gguf) | Q2_K | 3.18GB | false | Very low quality but surprisingly usable. |
| [Llama3.1-8B-ShiningValiant2-IQ2_M.gguf](https://huggingface.co/bartowski/Llama3.1-8B-ShiningValiant2-GGUF/blob/main/Llama3.1-8B-ShiningValiant2-IQ2_M.gguf) | IQ2_M | 2.95GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
Some say that this improves the quality, others don't notice any difference. If you use these models PLEASE COMMENT with your findings. I would like feedback that these are actually used and useful so I don't keep uploading quants no one is using.
Thanks!
## Downloading using huggingface-cli
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Llama3.1-8B-ShiningValiant2-GGUF --include "Llama3.1-8B-ShiningValiant2-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Llama3.1-8B-ShiningValiant2-GGUF --include "Llama3.1-8B-ShiningValiant2-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Llama3.1-8B-ShiningValiant2-Q8_0) or download them all in place (./)
## Q4_0_X_X
These are *NOT* for Metal (Apple) offloading, only ARM chips.
If you're using an ARM chip, the Q4_0_X_X quants will have a substantial speedup. Check out Q4_0_4_4 speed comparisons [on the original pull request](https://github.com/ggerganov/llama.cpp/pull/5780#pullrequestreview-21657544660)
To check which one would work best for your ARM chip, you can check [AArch64 SoC features](https://gpages.juszkiewicz.com.pl/arm-socs-table/arm-socs.html) (thanks EloyOn!).
## Which file should I choose?
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset
Thank you ZeroWw for the inspiration to experiment with embed/output
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}