bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-22 01:45:12 +08:00
commit f4f176f843
28 changed files with 346 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

86
.gitattributes vendored Normal file
View File

@@ -0,0 +1,86 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-bf16.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-imatrix.gguf filter=lfs diff=lfs merge=lfs -text
AvelonLabs_OpenClaude-1.7B-Merged-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text

3
AvelonLabs_OpenClaude-1.7B-Merged-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9910a264cfaaaab3d5d064885f0cce839c7f3e671e6a6f8e359b4dbe150aee02
size 769729504

3
AvelonLabs_OpenClaude-1.7B-Merged-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:43f52870a8eadbe0280105f02412c2dcf163e890ccc531860f8364503402cf06
size 925940704

3
AvelonLabs_OpenClaude-1.7B-Merged-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f426a847a3cac2733a0d3a6a9853bf89879e2bf44bcbfc5eefb029f60a26dace
size 865418208

3
AvelonLabs_OpenClaude-1.7B-Merged-IQ3_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1a448c558aee5eaf0e36b914bef26c0780c62bbea80ab235ccb4b7d7cdc8baf0
size 791061472

3
AvelonLabs_OpenClaude-1.7B-Merged-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cffb13228ade48e33898b1f80a8263d53e8f1aac25ba756973435b430a910363
size 1069562848

3
AvelonLabs_OpenClaude-1.7B-Merged-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4a1a3c1306beea02c27b1e1a18fb1213bf1ec9e600552807c59d53e0f8f34277
size 1027357664

3
AvelonLabs_OpenClaude-1.7B-Merged-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dcaa61fb4e391840fe2c16c03d5175686eb62807573287f30349adb7057ffcf0
size 806697952

3
AvelonLabs_OpenClaude-1.7B-Merged-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7aefe5d65c3f53de5d355a080080b1713e3dd4dbc5767b2ef2df42d39e299155
size 882058208

3
AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4923ec67f519af769a644a3bd9189a4669fa9c2d7c948c71147a001187057a23
size 1011760096

3
AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:14e3a678e6ed52eb73f23957ad12030161dec4d46c5e0a64a9931b501cea9415
size 965622752

3
AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8670b87430f23bf25cc54b94745323e46e1ded1c2c30c39daf3f1f1a6bf4207a
size 890190816

3
AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3c6baa04963c93185cb9c1b635633399bd7815fadb5d41a6a94aeb3494890ce3
size 1087120352

3
AvelonLabs_OpenClaude-1.7B-Merged-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:61677d252474b14e9aa4b412a6889262afa7c47f53e772137551336aaf8ed01a
size 1071922144

3
AvelonLabs_OpenClaude-1.7B-Merged-Q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d81c1a17ad5980049d7ddaef8f4e7570df5d5ecfc5fbce99bd2c422ed30a7bd2
size 1153973216

3
AvelonLabs_OpenClaude-1.7B-Merged-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6f8345da61f1601c84247a43934ee14ddd888d1d3dba801c2549c85f0065cafd
size 1219929056

3
AvelonLabs_OpenClaude-1.7B-Merged-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bf3aaff3d06b7bac47e07baaff9a4246f655421464620fe18c04ff22d4f5b6af
size 1144568800

3
AvelonLabs_OpenClaude-1.7B-Merged-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d4f529556f0113332041332690d130647a663419163a8dffbd77dfc46487d43e
size 1077951456

3
AvelonLabs_OpenClaude-1.7B-Merged-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:efec5d9e88bca3676219628b0bbe25fd188546a732a4389bf6754b71eeb9b56e
size 1364435936

3
AvelonLabs_OpenClaude-1.7B-Merged-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:651add504cb57a216b999f3c89fdac4f01a9ab2624352f26fd6d829390b17270
size 1289075680

3
AvelonLabs_OpenClaude-1.7B-Merged-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2924049d48d7e8851eba0658b68800cd6396f75ad2e4070df59808794a318543
size 1238383584

3
AvelonLabs_OpenClaude-1.7B-Merged-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8ba596d1a1d961ffbb8f7ec7a4dd0fc4db2f7ea4540945dd21a6f8b836ac7b92
size 1481751520

3
AvelonLabs_OpenClaude-1.7B-Merged-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c0c909a012ab31df619a9580bf7190db528b98100111845b9603e8ab28a5eeb0
size 1557111776

3
AvelonLabs_OpenClaude-1.7B-Merged-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:88e84f6b8aa61246bd95b9d257c66b4d637d4fdecb56b3d924ddf9da32ee8a1e
size 1834427360

3
AvelonLabs_OpenClaude-1.7B-Merged-bf16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0c40811a439d66dc9ebc0fbeef9174d827a504ecfa82b0915908030dedfe74a4
size 3447349952

3
AvelonLabs_OpenClaude-1.7B-Merged-imatrix.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:85db99c0364c7a69064f8697656e8f54ef64f3d1bf03b74c921b006d98b3f7dd
size 2094560

184
README.md Normal file
View File

@@ -0,0 +1,184 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
base_model_relation: quantized
language:
- en
license: apache-2.0
datasets:
- angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k
- lordx64/reasoning-distill-claude-opus-4-7-max
base_model: AvelonLabs/OpenClaude-1.7B-Merged
---
## Llamacpp imatrix Quantizations of OpenClaude-1.7B-Merged by AvelonLabs
Using <a href="https://github.com/ggml-org/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggml-org/llama.cpp/releases/tag/b9518">b9518</a> for quantization.
Original model: https://huggingface.co/AvelonLabs/OpenClaude-1.7B-Merged
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/82ae9b520227f57d79ba04add13d0d0d)
Run them in your choice of tools:
- [llama.cpp](https://github.com/ggml-org/llama.cpp)
- [ramalama](https://github.com/containers/ramalama)
- [LM Studio](https://lmstudio.ai/)
- [koboldcpp](https://github.com/LostRuins/koboldcpp)
- [Jan AI](https://www.jan.ai/)
- [Text Generation Web UI](https://github.com/oobabooga/text-generation-webui)
- [LoLLMs](https://github.com/ParisNeo/lollms)
Note: if it's a newly supported model, you may need to wait for an update from the developers.
## Prompt format
```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [OpenClaude-1.7B-Merged-bf16.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-bf16.gguf) | bf16 | 3.45GB | false | Full BF16 weights. |
| [OpenClaude-1.7B-Merged-Q8_0.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q8_0.gguf) | Q8_0 | 1.83GB | false | Extremely high quality, generally unneeded but max available quant. |
| [OpenClaude-1.7B-Merged-Q6_K_L.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q6_K_L.gguf) | Q6_K_L | 1.56GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [OpenClaude-1.7B-Merged-Q6_K.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q6_K.gguf) | Q6_K | 1.48GB | false | Very high quality, near perfect, *recommended*. |
| [OpenClaude-1.7B-Merged-Q5_K_L.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q5_K_L.gguf) | Q5_K_L | 1.36GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [OpenClaude-1.7B-Merged-Q5_K_M.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q5_K_M.gguf) | Q5_K_M | 1.29GB | false | High quality, *recommended*. |
| [OpenClaude-1.7B-Merged-Q5_K_S.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q5_K_S.gguf) | Q5_K_S | 1.24GB | false | High quality, *recommended*. |
| [OpenClaude-1.7B-Merged-Q4_K_L.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q4_K_L.gguf) | Q4_K_L | 1.22GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [OpenClaude-1.7B-Merged-Q4_1.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q4_1.gguf) | Q4_1 | 1.15GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
| [OpenClaude-1.7B-Merged-Q4_K_M.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q4_K_M.gguf) | Q4_K_M | 1.14GB | false | Good quality, default size for most use cases, *recommended*. |
| [OpenClaude-1.7B-Merged-Q3_K_XL.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_XL.gguf) | Q3_K_XL | 1.09GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [OpenClaude-1.7B-Merged-Q4_K_S.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q4_K_S.gguf) | Q4_K_S | 1.08GB | false | Slightly lower quality with more space savings, *recommended*. |
| [OpenClaude-1.7B-Merged-Q4_0.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q4_0.gguf) | Q4_0 | 1.07GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
| [OpenClaude-1.7B-Merged-IQ4_NL.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-IQ4_NL.gguf) | IQ4_NL | 1.07GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
| [OpenClaude-1.7B-Merged-IQ4_XS.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-IQ4_XS.gguf) | IQ4_XS | 1.03GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [OpenClaude-1.7B-Merged-Q3_K_L.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_L.gguf) | Q3_K_L | 1.01GB | false | Lower quality but usable, good for low RAM availability. |
| [OpenClaude-1.7B-Merged-Q3_K_M.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_M.gguf) | Q3_K_M | 0.97GB | false | Low quality. |
| [OpenClaude-1.7B-Merged-IQ3_M.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-IQ3_M.gguf) | IQ3_M | 0.93GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [OpenClaude-1.7B-Merged-Q3_K_S.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q3_K_S.gguf) | Q3_K_S | 0.89GB | false | Low quality, not recommended. |
| [OpenClaude-1.7B-Merged-Q2_K_L.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q2_K_L.gguf) | Q2_K_L | 0.88GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [OpenClaude-1.7B-Merged-IQ3_XS.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-IQ3_XS.gguf) | IQ3_XS | 0.87GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [OpenClaude-1.7B-Merged-Q2_K.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-Q2_K.gguf) | Q2_K | 0.81GB | false | Very low quality but surprisingly usable. |
| [OpenClaude-1.7B-Merged-IQ3_XXS.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-IQ3_XXS.gguf) | IQ3_XXS | 0.79GB | false | Lower quality, new method with decent performance, comparable to Q3 quants. |
| [OpenClaude-1.7B-Merged-IQ2_M.gguf](https://huggingface.co/bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF/blob/main/AvelonLabs_OpenClaude-1.7B-Merged-IQ2_M.gguf) | IQ2_M | 0.77GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF --include "AvelonLabs_OpenClaude-1.7B-Merged-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/AvelonLabs_OpenClaude-1.7B-Merged-GGUF --include "AvelonLabs_OpenClaude-1.7B-Merged-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (AvelonLabs_OpenClaude-1.7B-Merged-Q8_0) or download them all in place (./)
</details>
## ARM/AVX information
Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.
Now, however, there is something called "online repacking" for weights. details in [this PR](https://github.com/ggml-org/llama.cpp/pull/9921). If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.
As of llama.cpp build [b4282](https://github.com/ggml-org/llama.cpp/releases/tag/b4282) you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.
Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to [this PR](https://github.com/ggml-org/llama.cpp/pull/10541) which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.
<details>
<summary>Click to view Q4_0_X_X information (deprecated</summary>
I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggml-org/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Thank you to LM Studio for sponsoring my work.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}