bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-11 21:43:14 +08:00
commit 0ed8f2e5ef
28 changed files with 319 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

73
.gitattributes vendored Normal file
View File

@@ -0,0 +1,73 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-bf16.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-imatrix.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:39ad8eb3a5fc1084119470ecc6c769a0e4053c59709a97765c14299fdcdecd0b
size 1512984064

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:82ef00d26b87c257b111cee9cf10dc7e23008bb76f8cb718e711bf0e8df2562b
size 1962896384

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e55eee669539334cdc7ec0cc5c67d4253e598535463771d51eeb2866a0ea9f1e
size 1814375424

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ3_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b4033162344229d0b842ae1ffb92ec02253642868b678d1e9010350d1d036fc4
size 1670188544

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a62dc1b175cff8ca61b9d160cd7a78d2a21027812cc52cc65aa35fd87957c1ed
size 2381343744

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3a3e81554316fea9dd633009af41781d9e994562891dfc1423f586919291ddba
size 2270751744

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:325c854ca96901bfaf292843813b61656738d82e8eda9cf0decea79bcdbb4a92
size 1669499904

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f3fa30f30caae9f31bdb66e27408d670230074abb802f51895425b1f0d7d440f
size 1763700224

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:10db123bad7393d5941905b8d1654870767bf6ec89c242f9fbeb391b220e46d5
size 2239785984

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3f3596deb12f59fcb062737b089e100374bf471878cf4bb35aa9bec4e0052b28
size 2075618304

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fac96be857161c9b8efdda1cf496317d366d72d178524ca19dad050300401ed6
size 1886997504

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7c026f1091f3feb295b2620b9ec4dd0de3fb56c28bea40c220e0e211c9375b9f
size 2333986304

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e094b78771fab218b847175a0fbc8aeef94b39c052663a6003dacef5c7170d28
size 2375773184

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:408bbb47e9e25d1b708fd8e72dbe092408223b781a7675dd0b571c59523d01a0
size 2596629504

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cb72c8421c373970e652ab6599b3a30523236d24ab16b4b2dd0bd9378a72fb76
size 2591481344

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d8179cac47579ede2c32723d25a6bf773ce3dcda4c70453a662e2aec4ce4d3e8
size 2497281024

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ff302ca1e680c11b2a0bbe89faf7daf9f0bdcc92138bf97f6ffa81af03b31a8c
size 2383309824

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1b7cfbef013d9c01ebeee2f52beb4226210eb4311fc242b6379d13e038fd9be5
size 2983714304

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4eb6c4ed029496eddf565db44043c5268effbca60cf14ac792be0c26865d05bc
size 2889513984

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9a3621d84a300a27081208a8a758aac562cd225ef7aa210edcb44f5852cd43b8
size 2823711744

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:56ccfc274cee2eda6d05a0f7b65285e8760d32182a9505d7e222ad116360640e
size 3306261504

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:97ddc21af1ed9c4d1547ad03eecc41e7d2f47dfd82423a216b2c9259ff085f55
size 3400461824

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cf4972a80bcbe80aea2caf241525481e60613664c39ba419100a42785f010520
size 4280405504

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-bf16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:23c10c54aa0154c5de7cf6952dd941124c25a16c891a9418cc26ed2c9f41abe3
size 8051285184

3
AllThingsIntel_Apollo-V0.1-4B-Thinking-imatrix.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e3212439be0c8365636b468ed4cb05f855bb7ddc2a0aacf1cc214cc89880fe6c
size 3872640

170
README.md Normal file
View File

@@ -0,0 +1,170 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
base_model_relation: quantized
base_model: AllThingsIntel/Apollo-V0.1-4B-Thinking
---
## Llamacpp imatrix Quantizations of Apollo-V0.1-4B-Thinking by AllThingsIntel
Using <a href="https://github.com/ggml-org/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggml-org/llama.cpp/releases/tag/b6907">b6907</a> for quantization.
Original model: https://huggingface.co/AllThingsIntel/Apollo-V0.1-4B-Thinking
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8) combined with a subset of combined_all_small.parquet from Ed Addario [here](https://huggingface.co/datasets/eaddario/imatrix-calibration/blob/main/combined_all_small.parquet)
Run them in [LM Studio](https://lmstudio.ai/)
Run them directly with [llama.cpp](https://github.com/ggml-org/llama.cpp), or any other llama.cpp based project
## Prompt format
```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [Apollo-V0.1-4B-Thinking-bf16.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-bf16.gguf) | bf16 | 8.05GB | false | Full BF16 weights. |
| [Apollo-V0.1-4B-Thinking-Q8_0.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q8_0.gguf) | Q8_0 | 4.28GB | false | Extremely high quality, generally unneeded but max available quant. |
| [Apollo-V0.1-4B-Thinking-Q6_K_L.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q6_K_L.gguf) | Q6_K_L | 3.40GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [Apollo-V0.1-4B-Thinking-Q6_K.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q6_K.gguf) | Q6_K | 3.31GB | false | Very high quality, near perfect, *recommended*. |
| [Apollo-V0.1-4B-Thinking-Q5_K_L.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q5_K_L.gguf) | Q5_K_L | 2.98GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [Apollo-V0.1-4B-Thinking-Q5_K_M.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q5_K_M.gguf) | Q5_K_M | 2.89GB | false | High quality, *recommended*. |
| [Apollo-V0.1-4B-Thinking-Q5_K_S.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q5_K_S.gguf) | Q5_K_S | 2.82GB | false | High quality, *recommended*. |
| [Apollo-V0.1-4B-Thinking-Q4_1.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_1.gguf) | Q4_1 | 2.60GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
| [Apollo-V0.1-4B-Thinking-Q4_K_L.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_K_L.gguf) | Q4_K_L | 2.59GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [Apollo-V0.1-4B-Thinking-Q4_K_M.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_K_M.gguf) | Q4_K_M | 2.50GB | false | Good quality, default size for most use cases, *recommended*. |
| [Apollo-V0.1-4B-Thinking-Q4_K_S.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_K_S.gguf) | Q4_K_S | 2.38GB | false | Slightly lower quality with more space savings, *recommended*. |
| [Apollo-V0.1-4B-Thinking-Q4_0.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_0.gguf) | Q4_0 | 2.38GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
| [Apollo-V0.1-4B-Thinking-IQ4_NL.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ4_NL.gguf) | IQ4_NL | 2.38GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
| [Apollo-V0.1-4B-Thinking-Q3_K_XL.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_XL.gguf) | Q3_K_XL | 2.33GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [Apollo-V0.1-4B-Thinking-IQ4_XS.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ4_XS.gguf) | IQ4_XS | 2.27GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [Apollo-V0.1-4B-Thinking-Q3_K_L.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_L.gguf) | Q3_K_L | 2.24GB | false | Lower quality but usable, good for low RAM availability. |
| [Apollo-V0.1-4B-Thinking-Q3_K_M.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_M.gguf) | Q3_K_M | 2.08GB | false | Low quality. |
| [Apollo-V0.1-4B-Thinking-IQ3_M.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ3_M.gguf) | IQ3_M | 1.96GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [Apollo-V0.1-4B-Thinking-Q3_K_S.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q3_K_S.gguf) | Q3_K_S | 1.89GB | false | Low quality, not recommended. |
| [Apollo-V0.1-4B-Thinking-IQ3_XS.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ3_XS.gguf) | IQ3_XS | 1.81GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [Apollo-V0.1-4B-Thinking-Q2_K_L.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q2_K_L.gguf) | Q2_K_L | 1.76GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [Apollo-V0.1-4B-Thinking-IQ3_XXS.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ3_XXS.gguf) | IQ3_XXS | 1.67GB | false | Lower quality, new method with decent performance, comparable to Q3 quants. |
| [Apollo-V0.1-4B-Thinking-Q2_K.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-Q2_K.gguf) | Q2_K | 1.67GB | false | Very low quality but surprisingly usable. |
| [Apollo-V0.1-4B-Thinking-IQ2_M.gguf](https://huggingface.co/bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF/blob/main/AllThingsIntel_Apollo-V0.1-4B-Thinking-IQ2_M.gguf) | IQ2_M | 1.51GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF --include "AllThingsIntel_Apollo-V0.1-4B-Thinking-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/AllThingsIntel_Apollo-V0.1-4B-Thinking-GGUF --include "AllThingsIntel_Apollo-V0.1-4B-Thinking-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (AllThingsIntel_Apollo-V0.1-4B-Thinking-Q8_0) or download them all in place (./)
</details>
## ARM/AVX information
Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.
Now, however, there is something called "online repacking" for weights. details in [this PR](https://github.com/ggml-org/llama.cpp/pull/9921). If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.
As of llama.cpp build [b4282](https://github.com/ggml-org/llama.cpp/releases/tag/b4282) you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.
Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to [this PR](https://github.com/ggml-org/llama.cpp/pull/10541) which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.
<details>
<summary>Click to view Q4_0_X_X information (deprecated</summary>
I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggml-org/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Thank you to LM Studio for sponsoring my work.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}