bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-13 14:57:07 +08:00
commit 39d7fb3106
28 changed files with 254 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
.gitattributes vendored Normal file
View File

@@ -0,0 +1,60 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_8_8.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_4_8.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_4_4.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B-f16.gguf filter=lfs diff=lfs merge=lfs -text
Lyra_Gutenbergs-Twilight_Magnum-12B.imatrix filter=lfs diff=lfs merge=lfs -text

3
Lyra_Gutenbergs-Twilight_Magnum-12B-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0e983730da7a00d37ab940969c238ed8d94b6683986cf96f9a2c03df5d68c3d4
size 4435023200

3
Lyra_Gutenbergs-Twilight_Magnum-12B-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a35ea5b51cd5c48d53942c92654103c1c66e72677ff282efd5aa87c1d332bbb3
size 5722232160

3
Lyra_Gutenbergs-Twilight_Magnum-12B-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:aef8523a53119df18df809cdba66c8c1a581249ae30b6fec63016bbce74ac950
size 5306488160

3
Lyra_Gutenbergs-Twilight_Magnum-12B-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5c7d3d99b51b85c0f7389c365ae0d7f6e0413d22d4979de42478ac3c18b65a75
size 6742709600

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f2b0e6d11119f3c02b084e6e3e39049c8389cc793d30e4d5ee6388bd4c7fd798
size 4791047520

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c5fa1e9b4d10dcc201902c44e293da37d8b950de4f3a0e2708df813da16ffbd0
size 5446407520

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:76248c6c1bc3968d97e696a70a318e2ab47f878e25a653e5630c375140f1acca
size 6561502560

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3c8aecc4edbe2a00919b0e3855f33136bcd9d9efc22aafadde17b12c5a8658f4
size 6083089760

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d26fb9dc411d1fb284374efe17e272b03bc3c081d7bf5a66ceb7b8230720b34e
size 5534225760

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8967e85c2e17410eb6d453f29637030276170d60fb1d9ff820561c571e6be1f7
size 7148705120

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d16c9506dade20c182ec1c665217fa1dc06f9a196607fa29e0e13aaa20795544
size 7094637920

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_4_4.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b1aa7642d4da7270b20a49cd97d501ccbf25e19f5180c3ac0193a5e1c1002de0
size 7071700320

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_4_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9297e5aa14191a009fe7f982ee55c8383d1889890915c7dd1724561e0d9d737e
size 7071700320

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_8_8.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f789fcb5588bf94b9fffe482ba4b9ad22285ff600ba12fad4ca2d45d9e1f7f0a
size 7071700320

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6c1385f38ed7a96e9f85c31fa103184400aa98c8184a2f159fa9590081b343be
size 7975277920

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f879694b491f43d89f49780f2108e2b4d70ae06d9e08d55ad80c9676e2dcf280
size 7477204320

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c99a450d8ae82c413e9116a2b6e22d7f950b147af731a651c3dbeefffe0556cf
size 7120196960

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:22b9cad563529cf2a9876b939a6b32949ea06e471c04f455343385f37ed21d16
size 9141818720

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1fcda4757a03c9bb382595d2d6e030e587bc3186bf2f6ef9c1445b16708af8d1
size 8727631200

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e4fbd8d5e4d1ba080e69d4f30258f8234c87b1f043facbd83aba0f2b7f8eff18
size 8518735200

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9d197e5d91e61dca461421041c43c1b69b20f03ae508b05b436f699d846e8c1f
size 10056209760

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5ef7caac3a6a86a4b2f60fdcf5825a166d02c5db39b422de46496dfb0548a236
size 10381268320

3
Lyra_Gutenbergs-Twilight_Magnum-12B-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:edee8e608d52da7b1bf9f4b5d5da2a5ac2179461e8e972374e0904a345645b39
size 13022369120

3
Lyra_Gutenbergs-Twilight_Magnum-12B-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:de936cbd12084935a665e5a30ae3213a535ff9a34407689a84a18ab76d8386a4
size 24504276000

3
Lyra_Gutenbergs-Twilight_Magnum-12B.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b4c3ce4d5a416700351cc1ad7f0da826ca2e28de9d01990e69432c3c39490558
size 7054418

118
README.md Normal file
View File

@@ -0,0 +1,118 @@
---
base_model: Nitral-AI/Lyra_Gutenbergs-Twilight_Magnum-12B
language:
- en
license: other
pipeline_tag: text-generation
quantized_by: bartowski
---
## Llamacpp imatrix Quantizations of Lyra_Gutenbergs-Twilight_Magnum-12B
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b3631">b3631</a> for quantization.
Original model: https://huggingface.co/Nitral-AI/Lyra_Gutenbergs-Twilight_Magnum-12B
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8)
Run them in [LM Studio](https://lmstudio.ai/)
## Prompt format
```
<s>[INST] {prompt} [/INST]
```
Note that this model does not support a System prompt.
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-f16.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-f16.gguf) | f16 | 24.50GB | false | Full F16 weights. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q8_0.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q8_0.gguf) | Q8_0 | 13.02GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q6_K_L.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q6_K_L.gguf) | Q6_K_L | 10.38GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q6_K.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q6_K.gguf) | Q6_K | 10.06GB | false | Very high quality, near perfect, *recommended*. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_L.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_L.gguf) | Q5_K_L | 9.14GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_M.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_M.gguf) | Q5_K_M | 8.73GB | false | High quality, *recommended*. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_S.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q5_K_S.gguf) | Q5_K_S | 8.52GB | false | High quality, *recommended*. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_L.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_L.gguf) | Q4_K_L | 7.98GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_M.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_M.gguf) | Q4_K_M | 7.48GB | false | Good quality, default size for must use cases, *recommended*. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_XL.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_XL.gguf) | Q3_K_XL | 7.15GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_S.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_S.gguf) | Q4_K_S | 7.12GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0.gguf) | Q4_0 | 7.09GB | false | Legacy format, generally not worth using over similarly sized formats |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_8_8.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_8_8.gguf) | Q4_0_8_8 | 7.07GB | false | Optimized for ARM and CPU inference, much faster than Q4_0 at similar quality. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_4_8.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_4_8.gguf) | Q4_0_4_8 | 7.07GB | false | Optimized for ARM and CPU inference, much faster than Q4_0 at similar quality. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_4_4.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_0_4_4.gguf) | Q4_0_4_4 | 7.07GB | false | Optimized for ARM and CPU inference, much faster than Q4_0 at similar quality. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-IQ4_XS.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-IQ4_XS.gguf) | IQ4_XS | 6.74GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_L.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_L.gguf) | Q3_K_L | 6.56GB | false | Lower quality but usable, good for low RAM availability. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_M.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_M.gguf) | Q3_K_M | 6.08GB | false | Low quality. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-IQ3_M.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-IQ3_M.gguf) | IQ3_M | 5.72GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_S.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q3_K_S.gguf) | Q3_K_S | 5.53GB | false | Low quality, not recommended. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q2_K_L.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q2_K_L.gguf) | Q2_K_L | 5.45GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-IQ3_XS.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-IQ3_XS.gguf) | IQ3_XS | 5.31GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-Q2_K.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-Q2_K.gguf) | Q2_K | 4.79GB | false | Very low quality but surprisingly usable. |
| [Lyra_Gutenbergs-Twilight_Magnum-12B-IQ2_M.gguf](https://huggingface.co/bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF/blob/main/Lyra_Gutenbergs-Twilight_Magnum-12B-IQ2_M.gguf) | IQ2_M | 4.44GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
Some say that this improves the quality, others don't notice any difference. If you use these models PLEASE COMMENT with your findings. I would like feedback that these are actually used and useful so I don't keep uploading quants no one is using.
Thanks!
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset
Thank you ZeroWw for the inspiration to experiment with embed/output
## Downloading using huggingface-cli
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF --include "Lyra_Gutenbergs-Twilight_Magnum-12B-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Lyra_Gutenbergs-Twilight_Magnum-12B-GGUF --include "Lyra_Gutenbergs-Twilight_Magnum-12B-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (Lyra_Gutenbergs-Twilight_Magnum-12B-Q8_0) or download them all in place (./)
## Which file should I choose?
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}