bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-27 08:18:06 +08:00
commit a800e83b50
30 changed files with 366 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

75
.gitattributes vendored Normal file
View File

@@ -0,0 +1,75 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-bf16.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q2_K_L.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q4_K_L.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q5_K_L.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q3_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-imatrix.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q6_K_L.gguf filter=lfs diff=lfs merge=lfs -text
utter-project_EuroLLM-22B-Instruct-2512-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text

209
README.md Normal file
View File

@@ -0,0 +1,209 @@
---
quantized_by: bartowski
pipeline_tag: text-generation
base_model_relation: quantized
base_model: utter-project/EuroLLM-22B-Instruct-2512
license: apache-2.0
language:
- en
- de
- es
- fr
- it
- pt
- pl
- nl
- tr
- sv
- cs
- el
- hu
- ro
- fi
- uk
- sl
- sk
- da
- lt
- lv
- et
- bg
- 'no'
- ca
- hr
- ga
- mt
- gl
- zh
- ru
- ko
- ja
- ar
- hi
---
## Llamacpp imatrix Quantizations of EuroLLM-22B-Instruct-2512 by utter-project
Using <a href="https://github.com/ggml-org/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggml-org/llama.cpp/releases/tag/b7406">b7406</a> for quantization.
Original model: https://huggingface.co/utter-project/EuroLLM-22B-Instruct-2512
All quants made using imatrix option with dataset from [here](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8) combined with a subset of combined_all_small.parquet from Ed Addario [here](https://huggingface.co/datasets/eaddario/imatrix-calibration/blob/main/combined_all_small.parquet)
Run them in [LM Studio](https://lmstudio.ai/)
Run them directly with [llama.cpp](https://github.com/ggml-org/llama.cpp), or any other llama.cpp based project
## Prompt format
```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
## Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
| -------- | ---------- | --------- | ----- | ----------- |
| [EuroLLM-22B-Instruct-2512-bf16.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-bf16.gguf) | bf16 | 45.28GB | false | Full BF16 weights. |
| [EuroLLM-22B-Instruct-2512-Q8_0.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q8_0.gguf) | Q8_0 | 24.06GB | false | Extremely high quality, generally unneeded but max available quant. |
| [EuroLLM-22B-Instruct-2512-Q6_K_L.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q6_K_L.gguf) | Q6_K_L | 18.96GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
| [EuroLLM-22B-Instruct-2512-Q6_K.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q6_K.gguf) | Q6_K | 18.57GB | false | Very high quality, near perfect, *recommended*. |
| [EuroLLM-22B-Instruct-2512-Q5_K_L.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q5_K_L.gguf) | Q5_K_L | 16.53GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
| [EuroLLM-22B-Instruct-2512-Q5_K_M.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q5_K_M.gguf) | Q5_K_M | 16.04GB | false | High quality, *recommended*. |
| [EuroLLM-22B-Instruct-2512-Q5_K_S.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q5_K_S.gguf) | Q5_K_S | 15.67GB | false | High quality, *recommended*. |
| [EuroLLM-22B-Instruct-2512-Q4_1.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q4_1.gguf) | Q4_1 | 14.31GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
| [EuroLLM-22B-Instruct-2512-Q4_K_L.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q4_K_L.gguf) | Q4_K_L | 14.24GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
| [EuroLLM-22B-Instruct-2512-Q4_K_M.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q4_K_M.gguf) | Q4_K_M | 13.66GB | false | Good quality, default size for most use cases, *recommended*. |
| [EuroLLM-22B-Instruct-2512-Q4_K_S.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q4_K_S.gguf) | Q4_K_S | 13.02GB | false | Slightly lower quality with more space savings, *recommended*. |
| [EuroLLM-22B-Instruct-2512-Q4_0.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q4_0.gguf) | Q4_0 | 12.98GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
| [EuroLLM-22B-Instruct-2512-IQ4_NL.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-IQ4_NL.gguf) | IQ4_NL | 12.98GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
| [EuroLLM-22B-Instruct-2512-Q3_K_XL.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q3_K_XL.gguf) | Q3_K_XL | 12.74GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| [EuroLLM-22B-Instruct-2512-IQ4_XS.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-IQ4_XS.gguf) | IQ4_XS | 12.31GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
| [EuroLLM-22B-Instruct-2512-Q3_K_L.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q3_K_L.gguf) | Q3_K_L | 12.05GB | false | Lower quality but usable, good for low RAM availability. |
| [EuroLLM-22B-Instruct-2512-Q3_K_M.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q3_K_M.gguf) | Q3_K_M | 11.12GB | false | Low quality. |
| [EuroLLM-22B-Instruct-2512-IQ3_M.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-IQ3_M.gguf) | IQ3_M | 10.44GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| [EuroLLM-22B-Instruct-2512-Q3_K_S.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q3_K_S.gguf) | Q3_K_S | 10.04GB | false | Low quality, not recommended. |
| [EuroLLM-22B-Instruct-2512-IQ3_XS.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-IQ3_XS.gguf) | IQ3_XS | 9.59GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| [EuroLLM-22B-Instruct-2512-Q2_K_L.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q2_K_L.gguf) | Q2_K_L | 9.43GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| [EuroLLM-22B-Instruct-2512-IQ3_XXS.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-IQ3_XXS.gguf) | IQ3_XXS | 8.96GB | false | Lower quality, new method with decent performance, comparable to Q3 quants. |
| [EuroLLM-22B-Instruct-2512-Q2_K.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-Q2_K.gguf) | Q2_K | 8.66GB | false | Very low quality but surprisingly usable. |
| [EuroLLM-22B-Instruct-2512-IQ2_M.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-IQ2_M.gguf) | IQ2_M | 8.00GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
| [EuroLLM-22B-Instruct-2512-IQ2_S.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-IQ2_S.gguf) | IQ2_S | 7.44GB | false | Low quality, uses SOTA techniques to be usable. |
| [EuroLLM-22B-Instruct-2512-IQ2_XS.gguf](https://huggingface.co/bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF/blob/main/utter-project_EuroLLM-22B-Instruct-2512-IQ2_XS.gguf) | IQ2_XS | 7.01GB | false | Low quality, uses SOTA techniques to be usable. |
## Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
## Downloading using huggingface-cli
<details>
<summary>Click to view download instructions</summary>
First, make sure you have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Then, you can target the specific file you want:
```
huggingface-cli download bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF --include "utter-project_EuroLLM-22B-Instruct-2512-Q4_K_M.gguf" --local-dir ./
```
If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download bartowski/utter-project_EuroLLM-22B-Instruct-2512-GGUF --include "utter-project_EuroLLM-22B-Instruct-2512-Q8_0/*" --local-dir ./
```
You can either specify a new local-dir (utter-project_EuroLLM-22B-Instruct-2512-Q8_0) or download them all in place (./)
</details>
## ARM/AVX information
Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.
Now, however, there is something called "online repacking" for weights. details in [this PR](https://github.com/ggml-org/llama.cpp/pull/9921). If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.
As of llama.cpp build [b4282](https://github.com/ggml-org/llama.cpp/releases/tag/b4282) you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.
Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to [this PR](https://github.com/ggml-org/llama.cpp/pull/10541) which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.
<details>
<summary>Click to view Q4_0_X_X information (deprecated</summary>
I'm keeping this section to show the potential theoretical uplift in performance from using the Q4_0 with online repacking.
<details>
<summary>Click to view benchmarks on an AVX2 system (EPYC7702)</summary>
| model | size | params | backend | threads | test | t/s | % (vs Q4_0) |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |-------------: |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp512 | 204.03 ± 1.03 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp1024 | 282.92 ± 0.19 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | pp2048 | 259.49 ± 0.44 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg128 | 39.12 ± 0.27 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg256 | 39.31 ± 0.69 | 100% |
| qwen2 3B Q4_0 | 1.70 GiB | 3.09 B | CPU | 64 | tg512 | 40.52 ± 0.03 | 100% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp512 | 301.02 ± 1.74 | 147% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp1024 | 287.23 ± 0.20 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | pp2048 | 262.77 ± 1.81 | 101% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg128 | 18.80 ± 0.99 | 48% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg256 | 24.46 ± 3.04 | 83% |
| qwen2 3B Q4_K_M | 1.79 GiB | 3.09 B | CPU | 64 | tg512 | 36.32 ± 3.59 | 90% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp512 | 271.71 ± 3.53 | 133% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp1024 | 279.86 ± 45.63 | 100% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | pp2048 | 320.77 ± 5.00 | 124% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg128 | 43.51 ± 0.05 | 111% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg256 | 43.35 ± 0.09 | 110% |
| qwen2 3B Q4_0_8_8 | 1.69 GiB | 3.09 B | CPU | 64 | tg512 | 42.60 ± 0.31 | 105% |
Q4_0_8_8 offers a nice bump to prompt processing and a small bump to text generation
</details>
</details>
## Which file should I choose?
<details>
<summary>Click here for details</summary>
A great write up with charts showing various performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)
The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.
If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.
If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.
Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.
If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.
If you want to get more into the weeds, you can check out this extremely useful feature chart:
[llama.cpp feature matrix](https://github.com/ggml-org/llama.cpp/wiki/Feature-matrix)
But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.
These I-quants can also be used on CPU, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.
</details>
## Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Thank you to LM Studio for sponsoring my work.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}

3
utter-project_EuroLLM-22B-Instruct-2512-IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e1596084cb005bfe876e672c0ff757a9aafe1af5b9f509bcb6272a54f72c0370
size 8002655520

3
utter-project_EuroLLM-22B-Instruct-2512-IQ2_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d8d9809849a3873aa0112a92a54990b1e5df9402fce9f44ae53ff304b999db5a
size 7437604128

3
utter-project_EuroLLM-22B-Instruct-2512-IQ2_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7eabc0398ba9db948077fc7217e753d02298c625f958c79c2055c1c36dc914b9
size 7009735968

3
utter-project_EuroLLM-22B-Instruct-2512-IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2c3e330a3ef8d0fb0cfeb074c468710daeb9a75bc75510a56b8bc4e84a10e2d7
size 10435728672

3
utter-project_EuroLLM-22B-Instruct-2512-IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d4f81474158dc432265d60b77d0ef78388000b26563f989ebfb2c84647fb08c4
size 9586382112

3
utter-project_EuroLLM-22B-Instruct-2512-IQ3_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:11bde6a20ef0043e66bca9ee6ad4c6d85f69aa7f31bee73bd5f806445a17ee44
size 8955811104

3
utter-project_EuroLLM-22B-Instruct-2512-IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:356f609ff9b8afa81a8d300c0d41242e5fe63e1ab0134268556ec7d7ac5d7374
size 12984112416

3
utter-project_EuroLLM-22B-Instruct-2512-IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4f97376bd177f9f5a687c7ed106cbd2f4f896603f156e097a70db03f0ba7da2a
size 12311909664

3
utter-project_EuroLLM-22B-Instruct-2512-Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0ceb9bd4f0c0f762f01ece23d6a5b1a35d706671364978746d281f717c08e7d6
size 8659227936

3
utter-project_EuroLLM-22B-Instruct-2512-Q2_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c1334695044dae0835c2ed250634fc3f6ff198e4fe59504487d6970081bf1e34
size 9427227936

3
utter-project_EuroLLM-22B-Instruct-2512-Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e2e433890beb21fd7beb3a2747016053dcd57462b8925cf27db9ac5aea73207c
size 12054205728

3
utter-project_EuroLLM-22B-Instruct-2512-Q3_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d77e974487a848e0a1286dc64f93364266d0fc12b957755b9f5f602a2401a330
size 11116778784

3
utter-project_EuroLLM-22B-Instruct-2512-Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f0ced50d89e34934964f2482f024a65e7e9658c6ed0cca94296b0b63e36cf5b8
size 10039661856

3
utter-project_EuroLLM-22B-Instruct-2512-Q3_K_XL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:92212bdd9d77ac1c9ebc730c749f03e777887d63371cca10c198aa31ad35b941
size 12742333728

3
utter-project_EuroLLM-22B-Instruct-2512-Q4_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b7e0f7085c41ca53d95120cc0abd6d1c24158b50f258b28c49c526b327842fff
size 12979393824

3
utter-project_EuroLLM-22B-Instruct-2512-Q4_1.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:275a07e81bf0999b0fc19e3122a6143e6755278d3ce458808e3c83a49e22e41f
size 14307284256

3
utter-project_EuroLLM-22B-Instruct-2512-Q4_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:057c986ed297584b4420147d4e47ef366cbed99217b3433cfed1a141226d4ee6
size 14242256160

3
utter-project_EuroLLM-22B-Instruct-2512-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2a222374c4adacd55b55795e2f9dca42a2f100d5a2d5858442f928c4c8bdf5e7
size 13658576160

3
utter-project_EuroLLM-22B-Instruct-2512-Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:42270babbd9a0fd9b601bb9583b076710fb6ee4e8bd48ce377f7d73fd3fd5194
size 13020288288

3
utter-project_EuroLLM-22B-Instruct-2512-Q5_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a21445c02e0987c80c64f3d2d551147d61bae7faee65801f017f169a9b4934d5
size 16527627552

3
utter-project_EuroLLM-22B-Instruct-2512-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:01f99634b10acfa8c4c5356e88b06b9c559bb32d41409be9d99d39e169b01bb1
size 16042251552

3
utter-project_EuroLLM-22B-Instruct-2512-Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9eded21b1e4d59e9ca04badb02b6a2a1ff34ab8b53adde5a103d3029c75d1cb3
size 15672923424

3
utter-project_EuroLLM-22B-Instruct-2512-Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:11015044dc0d531b7f75a8f0714e377e4bfd7f86ef6ed3b5a1b10a47ab0080d5
size 18574906656

3
utter-project_EuroLLM-22B-Instruct-2512-Q6_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ea9aa37842fa18f319930332c869044f4f39be69f316e7639f95686abed28cfd
size 18955834656

3
utter-project_EuroLLM-22B-Instruct-2512-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e8d67bdc6d98fa7c7cf2ee0be094f5b57cfa227d28b31bfa7ffbf88c69beb90d
size 24057222432

3
utter-project_EuroLLM-22B-Instruct-2512-bf16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1d6b916d4f9042d18a2a1e618b0a0ee4ea40945c16ac01319ce0e6284f3a8b38
size 45279089632

3
utter-project_EuroLLM-22B-Instruct-2512-imatrix.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b7d05131284b4d8fea8d6885f41cd987e96dbdd47a9f5a0f2fe17a8c47b83612
size 11559712