bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

Llamacpp quants

Browse Source
This commit is contained in:
ai-modelscope
2024-11-28 21:14:15 +08:00
parent ef4df92c10
commit 909b516337
27 changed files with 203 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files

56
.gitattributes vendored
View File

@@ -1,47 +1,59 @@
*.7z filter=lfs diff=lfs merge=lfs -text *.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text *.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text *.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text *.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text *.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text *.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text *.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text *.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text *.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text *.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text *.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text *.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text *.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text *.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text *.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text *.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text *.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text *.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text *.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text *.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text *.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text *.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text *.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.gguf* filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text *.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1-f16.gguf filter=lfs diff=lfs merge=lfs -text
Llama-3-Alpha-Centauri-v0.1.imatrix filter=lfs diff=lfs merge=lfs -text

3
Llama-3-Alpha-Centauri-v0.1-IQ1_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:36d252dae7a54022989d1695644b83217a62126d10275b43f1e5f69e72c3b9d5
size 2161971808

3
Llama-3-Alpha-Centauri-v0.1-IQ1_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1a3f7d27481b8a941388921fa2db4cdf189beaf3c31c865dec5ecba513a7eb2e
size 2019627616

3
Llama-3-Alpha-Centauri-v0.1-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4ce442a99ece8c69845d2f709149ba21441e5b6dfe6c543091391bdc04fd8177
size 2948280928

3
Llama-3-Alpha-Centauri-v0.1-IQ2_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1df1929eebb1fb631f3b920be137358366b4a643b84bc348216ff1f4cc915fa1
size 2758488672

3
Llama-3-Alpha-Centauri-v0.1-IQ2_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b8e5b86b9c898cda17531b74c6073c3df8db3316e4469380bebf3e62bc187e1f
size 2605781600

3
Llama-3-Alpha-Centauri-v0.1-IQ2_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b6e359a53134a35b39beafbc26d1ff1214542634bf50d638570f30a4159af49a
size 2399212128

3
Llama-3-Alpha-Centauri-v0.1-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5f95c29512e20c5d33a31439fe0aabe2b31f041de027f0e337355a42831c5712
size 3784823392

3
Llama-3-Alpha-Centauri-v0.1-IQ3_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:aeb02fce06cecc19f81bbbf98be97cb3ac2666743b7af4b885c01c9272e3d3a0
size 3682325088

3
Llama-3-Alpha-Centauri-v0.1-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3969997dfbe8f9e730812da9cf4182e16b4cc6675c194716a12c555c4cc98889
size 3518747232

3
Llama-3-Alpha-Centauri-v0.1-IQ3_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:97cdeffe2ea415d1eb9b895f9a7fd7d5f8d4ba9acef334ac2fe4c726b9b20cf4
size 3274912352

3
Llama-3-Alpha-Centauri-v0.1-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2a6e422299a51bc18696d38d7c9cccfa8f782cfbcbc4bf4e57e00f20abbf357d
size 4677988960

3
Llama-3-Alpha-Centauri-v0.1-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f3483b13c4bc5ee1e01d53ddf72257c2e4fee5c4ea704fb0fa6570e1763f7a99
size 4447662688

3
Llama-3-Alpha-Centauri-v0.1-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3a364b7a8d30cfdbac758835c67cbdebff56b0a95b8be25c0bf2b7afa56ebef2
size 3179131488

3
Llama-3-Alpha-Centauri-v0.1-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:942f50dc0e47e7725aaafc8dde5fd267894da81f81ca5b7ef91c8dc1d8de7fc1
size 4321956448

3
Llama-3-Alpha-Centauri-v0.1-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:52a92006ae09a3a2961aa2833bd50b2d72ca6ebf327b7d5f4cf362adf5ed0cae
size 4018917984

3
Llama-3-Alpha-Centauri-v0.1-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ad256241576974ac59b10d8f0823d0dee470b2cffdacddedb8f353c44f4a5e4b
size 3664499296

3
Llama-3-Alpha-Centauri-v0.1-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f5dc674765d5ca3fa7e16a69a11e7a26b6002acc6294b02f4da82eb47f13396c
size 4920734304

3
Llama-3-Alpha-Centauri-v0.1-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:137b2a5b6704b04fbe80621a597ca2a30ce8cc493ae20f434dd290aa0c052e31
size 4692669024

3
Llama-3-Alpha-Centauri-v0.1-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d6f031f8ebd8258a509340da43c8879e1f8c8087fd98cb414bfc15e34413c215
size 5732987488

3
Llama-3-Alpha-Centauri-v0.1-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:45e6fcccac02df88381141f03abfcb33e4e3b98c7b3574ad583b81c3f2f1f568
size 5599294048

3
Llama-3-Alpha-Centauri-v0.1-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6671c5bba7a0d602d8b038dc79256a9e217dbb9fa365b2446cbbb81e1e1b90b4
size 6596006496

3
Llama-3-Alpha-Centauri-v0.1-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ad37a066069fb96453f411474024136b48a5fe895bf3ce206d8ec44aa1bc01f8
size 8540770912

3
Llama-3-Alpha-Centauri-v0.1-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:32fa2c650d4a90422c0fb889a3afbac6b292f9f7a03e8f4bfceb64b0f973d0e9
size 16068890944

3
Llama-3-Alpha-Centauri-v0.1.imatrix Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:db38ca63f43be5c703c63246eda252894d551fd5f4d71e417f156ba73d6e78cb
size 4988169

137
README.md
View File

@@ -1,47 +1,102 @@
--- ---
license: Apache License 2.0 license: llama3
datasets:
#model-type: - NobodyExistsOnTheInternet/ToxicQAFinal
##如 gpt、phi、llama、chatglm、baichuan 等 quantized_by: bartowski
#- gpt pipeline_tag: text-generation
#domain:
##如 nlp、cv、audio、multi-modal
#- nlp
#language:
##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa
#- cn
#metrics:
##如 CIDEr、Blue、ROUGE 等
#- CIDEr
#tags:
##各种自定义，包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他
#- pretrained
#tools:
##如 vllm、fastchat、llamacpp、AdaSeq 等
#- vllm
--- ---
### 当前模型的贡献者未提供更加详细的模型介绍。模型文件和权重，可浏览“模型文件”页面获取。
#### 您可以通过如下git clone命令，或者ModelScope SDK来下载模型
SDK下载 ## Llamacpp imatrix Quantizations of Llama-3-Alpha-Centauri-v0.1
```bash
#安装ModelScope Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b3001">b3001</a> for quantization.
pip install modelscope
Original model: https://huggingface.co/fearlessdots/Llama-3-Alpha-Centauri-v0.1
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/b6ac44691e994344625687afe3263b3a)
## Prompt format
``` ```
```python <|begin_of_text|><|start_header_id|>system<|end_header_id|>
#SDK模型下载
from modelscope import snapshot_download {system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>
model_dir = snapshot_download('bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF')
``` {prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Git下载
```
#Git模型下载
git clone https://www.modelscope.cn/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF.git
``` ```
<p style="color: lightgrey;">如果您是本模型的贡献者，我们邀请您根据<a href="https://modelscope.cn/docs/ModelScope%E6%A8%A1%E5%9E%8B%E6%8E%A5%E5%85%A5%E6%B5%81%E7%A8%8B%E6%A6%82%E8%A7%88" style="color: lightgrey; text-decoration: underline;">模型贡献文档</a>，及时完善模型卡片内容。</p> ## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Description |
| -------- | ---------- | --------- | ----------- |
| [Llama-3-Alpha-Centauri-v0.1-Q8_0.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-Q8_0.gguf) | Q8_0 | 8.54GB | Extremely high quality, generally unneeded but max available quant. |
| [Llama-3-Alpha-Centauri-v0.1-Q6_K.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-Q6_K.gguf) | Q6_K | 6.59GB | Very high quality, near perfect, *recommended*. |
| [Llama-3-Alpha-Centauri-v0.1-Q5_K_M.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-Q5_K_M.gguf) | Q5_K_M | 5.73GB | High quality, *recommended*. |
| [Llama-3-Alpha-Centauri-v0.1-Q5_K_S.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-Q5_K_S.gguf) | Q5_K_S | 5.59GB | High quality, *recommended*. |
| [Llama-3-Alpha-Centauri-v0.1-Q4_K_M.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-Q4_K_M.gguf) | Q4_K_M | 4.92GB | Good quality, uses about 4.83 bits per weight, *recommended*. |
| [Llama-3-Alpha-Centauri-v0.1-Q4_K_S.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-Q4_K_S.gguf) | Q4_K_S | 4.69GB | Slightly lower quality with more space savings, *recommended*. |
| [Llama-3-Alpha-Centauri-v0.1-IQ4_NL.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ4_NL.gguf) | IQ4_NL | 4.67GB | Decent quality, slightly smaller than Q4_K_S with similar performance *recommended*. |
| [Llama-3-Alpha-Centauri-v0.1-IQ4_XS.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ4_XS.gguf) | IQ4_XS | 4.44GB | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Llama-3-Alpha-Centauri-v0.1-Q3_K_L.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-Q3_K_L.gguf) | Q3_K_L | 4.32GB | Lower quality but usable, good for low RAM availability. |
| [Llama-3-Alpha-Centauri-v0.1-Q3_K_M.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-Q3_K_M.gguf) | Q3_K_M | 4.01GB | Even lower quality. |
| [Llama-3-Alpha-Centauri-v0.1-IQ3_M.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ3_M.gguf) | IQ3_M | 3.78GB | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Llama-3-Alpha-Centauri-v0.1-IQ3_S.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ3_S.gguf) | IQ3_S | 3.68GB | Lower quality, new method with decent performance, recommended over Q3_K_S quant, same size with better performance. |
| [Llama-3-Alpha-Centauri-v0.1-Q3_K_S.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-Q3_K_S.gguf) | Q3_K_S | 3.66GB | Low quality, not recommended. |
| [Llama-3-Alpha-Centauri-v0.1-IQ3_XS.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ3_XS.gguf) | IQ3_XS | 3.51GB | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Llama-3-Alpha-Centauri-v0.1-IQ3_XXS.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ3_XXS.gguf) | IQ3_XXS | 3.27GB | Lower quality, new method with decent performance, comparable to Q3 quants. |
| [Llama-3-Alpha-Centauri-v0.1-Q2_K.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-Q2_K.gguf) | Q2_K | 3.17GB | Very low quality but surprisingly usable. |
| [Llama-3-Alpha-Centauri-v0.1-IQ2_M.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ2_M.gguf) | IQ2_M | 2.94GB | Very low quality, uses SOTA techniques to also be surprisingly usable. |
| [Llama-3-Alpha-Centauri-v0.1-IQ2_S.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ2_S.gguf) | IQ2_S | 2.75GB | Very low quality, uses SOTA techniques to be usable. |
| [Llama-3-Alpha-Centauri-v0.1-IQ2_XS.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ2_XS.gguf) | IQ2_XS | 2.60GB | Very low quality, uses SOTA techniques to be usable. |
| [Llama-3-Alpha-Centauri-v0.1-IQ2_XXS.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ2_XXS.gguf) | IQ2_XXS | 2.39GB | Lower quality, uses SOTA techniques to be usable. |
| [Llama-3-Alpha-Centauri-v0.1-IQ1_M.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ1_M.gguf) | IQ1_M | 2.16GB | Extremely low quality, *not* recommended. |
| [Llama-3-Alpha-Centauri-v0.1-IQ1_S.gguf](https://huggingface.co/bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF/blob/main/Llama-3-Alpha-Centauri-v0.1-IQ1_S.gguf) | IQ1_S | 2.01GB | Extremely low quality, *not* recommended. |
## Downloading using huggingface-cli
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF --include "Llama-3-Alpha-Centauri-v0.1-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/Llama-3-Alpha-Centauri-v0.1-GGUF --include "Llama-3-Alpha-Centauri-v0.1-Q8_0.gguf/*" --local-dir Llama-3-Alpha-Centauri-v0.1-Q8_0
```
You can either specify a new local-dir (Llama-3-Alpha-Centauri-v0.1-Q8_0) or download them all in place (./)
## Which file should I choose?
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU and Apple Metal, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
The I-quants are *not* compatible with Vulcan, which is also AMD, so if you have an AMD card double check if you're using the rocBLAS build or the Vulcan build. At the time of writing this, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}