commit f58e7cf1266a8085b841818907030a39055fd62c Author: ModelHub XC Date: Sat May 9 20:26:10 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: mradermacher/Athena-gemma-2-9b-it-i1-GGUF Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..40ec38b --- /dev/null +++ b/.gitattributes @@ -0,0 +1,60 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +imatrix.dat filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q4_0_4_4.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q4_0_4_8.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-Q4_0_8_8.gguf filter=lfs diff=lfs merge=lfs -text +Athena-gemma-2-9b-it.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text diff --git a/Athena-gemma-2-9b-it.i1-IQ1_M.gguf b/Athena-gemma-2-9b-it.i1-IQ1_M.gguf new file mode 100644 index 0000000..1f7443e --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-IQ1_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ad83ea9c89ac2ab4c0b42b8c2a02f294ed27dbc57734ff0b17339386138690c6 +size 2545952800 diff --git a/Athena-gemma-2-9b-it.i1-IQ1_S.gguf b/Athena-gemma-2-9b-it.i1-IQ1_S.gguf new file mode 100644 index 0000000..2414f7d --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-IQ1_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7b0b31aaa5969acceb743c91337fc35cc1aa4d94a9827bb05ae4efca60f75731 +size 2378565664 diff --git a/Athena-gemma-2-9b-it.i1-IQ2_M.gguf b/Athena-gemma-2-9b-it.i1-IQ2_M.gguf new file mode 100644 index 0000000..466002b --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-IQ2_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4a47f1afc1e83b894d54b94de76e3c68a28ff51bf815ef3e292f1e002018947f +size 3434670112 diff --git a/Athena-gemma-2-9b-it.i1-IQ2_S.gguf b/Athena-gemma-2-9b-it.i1-IQ2_S.gguf new file mode 100644 index 0000000..3eed3f0 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-IQ2_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6b81037833596fe82ff48c06a481b6bad279f4c6b33cfe80f10ac1241668e595 +size 3211487264 diff --git a/Athena-gemma-2-9b-it.i1-IQ2_XS.gguf b/Athena-gemma-2-9b-it.i1-IQ2_XS.gguf new file mode 100644 index 0000000..192f09b --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-IQ2_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:675e1fcefa19e694e051483bf0f8a95e29915f3c54d35e41763df63233fbd3d2 +size 3067381792 diff --git a/Athena-gemma-2-9b-it.i1-IQ2_XXS.gguf b/Athena-gemma-2-9b-it.i1-IQ2_XXS.gguf new file mode 100644 index 0000000..29bd282 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-IQ2_XXS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:664fd5af4a3b112c8b6814db5b542e3312299cfd90b5cf1a88c4305e748810a1 +size 2824931360 diff --git a/Athena-gemma-2-9b-it.i1-IQ3_M.gguf b/Athena-gemma-2-9b-it.i1-IQ3_M.gguf new file mode 100644 index 0000000..1f4e593 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-IQ3_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a843d09e0e2d42d522a93aea2191cc776023206a21657a2a71ab2aa6fe811f54 +size 4494616608 diff --git a/Athena-gemma-2-9b-it.i1-IQ3_S.gguf b/Athena-gemma-2-9b-it.i1-IQ3_S.gguf new file mode 100644 index 0000000..3affa55 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-IQ3_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6d75bf37357e4f4385326b0b3628f88f00e593c7d913ad0bc7b8a644016d5219 +size 4337666080 diff --git a/Athena-gemma-2-9b-it.i1-IQ3_XS.gguf b/Athena-gemma-2-9b-it.i1-IQ3_XS.gguf new file mode 100644 index 0000000..d993ca8 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-IQ3_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:472f83e66a257be37118a7ef88e6718efe1bfbbb6bb03bccd9a5ec9e794ee4e0 +size 4144990240 diff --git a/Athena-gemma-2-9b-it.i1-IQ3_XXS.gguf b/Athena-gemma-2-9b-it.i1-IQ3_XXS.gguf new file mode 100644 index 0000000..618bc31 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-IQ3_XXS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:928c504950084badaf710d8a1cd3bce14f365451a552f7a721cc0b052130b432 +size 3796740128 diff --git a/Athena-gemma-2-9b-it.i1-IQ4_XS.gguf b/Athena-gemma-2-9b-it.i1-IQ4_XS.gguf new file mode 100644 index 0000000..d565ab1 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-IQ4_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a1084c2d36ccbf6e05828a665810d12d958ef7ce98022db5299d4969ad9a2e93 +size 5183031328 diff --git a/Athena-gemma-2-9b-it.i1-Q2_K.gguf b/Athena-gemma-2-9b-it.i1-Q2_K.gguf new file mode 100644 index 0000000..5c43406 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q2_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8c228cf2520c033e16d06acb1b7ec3955f63328d19ae08529449f1bd2d2dac85 +size 3805399072 diff --git a/Athena-gemma-2-9b-it.i1-Q3_K_L.gguf b/Athena-gemma-2-9b-it.i1-Q3_K_L.gguf new file mode 100644 index 0000000..61bbc8a --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q3_K_L.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2ebc954f0eb43121f3ef2c00f692b5a4a95b6b72d04dab749629828a364083a9 +size 5132453920 diff --git a/Athena-gemma-2-9b-it.i1-Q3_K_M.gguf b/Athena-gemma-2-9b-it.i1-Q3_K_M.gguf new file mode 100644 index 0000000..f611be0 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q3_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5cad00f9d50e64825ebce2b0fe2cf96da3e38abc211966542e88128aee12d1d2 +size 4761782304 diff --git a/Athena-gemma-2-9b-it.i1-Q3_K_S.gguf b/Athena-gemma-2-9b-it.i1-Q3_K_S.gguf new file mode 100644 index 0000000..238777b --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q3_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:28643d4d91bd35269b214928a68f418880a67df3fa790db3bcbc1625fcdaad03 +size 4337666080 diff --git a/Athena-gemma-2-9b-it.i1-Q4_0.gguf b/Athena-gemma-2-9b-it.i1-Q4_0.gguf new file mode 100644 index 0000000..2876608 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q4_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:64fc567ebcd05b78cdc8db3e57754975294f4b350a60d900a3e07a0dab259013 +size 5459200032 diff --git a/Athena-gemma-2-9b-it.i1-Q4_0_4_4.gguf b/Athena-gemma-2-9b-it.i1-Q4_0_4_4.gguf new file mode 100644 index 0000000..02b5e14 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q4_0_4_4.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9e3b204c48431a993c5b80cb17dac995858892354645cc928254bd4334ccc047 +size 5443143712 diff --git a/Athena-gemma-2-9b-it.i1-Q4_0_4_8.gguf b/Athena-gemma-2-9b-it.i1-Q4_0_4_8.gguf new file mode 100644 index 0000000..ecb0885 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q4_0_4_8.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c0ba1af3ccd6316d16243dc408b66f7c9389e9de21a41e1642f4a886d8774234 +size 5443143712 diff --git a/Athena-gemma-2-9b-it.i1-Q4_0_8_8.gguf b/Athena-gemma-2-9b-it.i1-Q4_0_8_8.gguf new file mode 100644 index 0000000..c05bb8b --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q4_0_8_8.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:659e6776ef83d42cdba40dbd7550572b9b6fd0bfe97245575011e6a8c49c9f0d +size 5443143712 diff --git a/Athena-gemma-2-9b-it.i1-Q4_K_M.gguf b/Athena-gemma-2-9b-it.i1-Q4_K_M.gguf new file mode 100644 index 0000000..a85e62e --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:eeb642a66c1b61931b0f3d9fd1afbeaa013ff874a597b9d5222e69691a5411b3 +size 5761058848 diff --git a/Athena-gemma-2-9b-it.i1-Q4_K_S.gguf b/Athena-gemma-2-9b-it.i1-Q4_K_S.gguf new file mode 100644 index 0000000..d816128 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q4_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0ec0b0d5068e197c09a868eb7a2da2decbe60b81f6f49387c4c0cc60e9a309d1 +size 5478926368 diff --git a/Athena-gemma-2-9b-it.i1-Q5_K_M.gguf b/Athena-gemma-2-9b-it.i1-Q5_K_M.gguf new file mode 100644 index 0000000..ce52c7b --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5c58715be024453c2aaea54e1d6b6d792fadc4d6670f64d9ce3057528fe91437 +size 6647367712 diff --git a/Athena-gemma-2-9b-it.i1-Q5_K_S.gguf b/Athena-gemma-2-9b-it.i1-Q5_K_S.gguf new file mode 100644 index 0000000..8d92c50 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q5_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9dec9f572bfe66c24bb4bfb058381311e4112b6d7a6ef9c14676b3d420c2c895 +size 6483593248 diff --git a/Athena-gemma-2-9b-it.i1-Q6_K.gguf b/Athena-gemma-2-9b-it.i1-Q6_K.gguf new file mode 100644 index 0000000..a94ba52 --- /dev/null +++ b/Athena-gemma-2-9b-it.i1-Q6_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6003536e165c9f79e00c94c978973c949d28056e4a32d8875c7d64714f9aba93 +size 7589070880 diff --git a/README.md b/README.md new file mode 100644 index 0000000..002c3b0 --- /dev/null +++ b/README.md @@ -0,0 +1,82 @@ +--- +base_model: EpistemeAI/Athena-gemma-2-9b-it +language: +- en +library_name: transformers +license: apache-2.0 +quantized_by: mradermacher +tags: +- text-generation-inference +- transformers +- unsloth +- gemma2 +- trl +--- +## About + + + + + + +weighted/imatrix quants of https://huggingface.co/EpistemeAI/Athena-gemma-2-9b-it + + +static quants are available at https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-GGUF +## Usage + +If you are unsure how to use GGUF files, refer to one of [TheBloke's +READMEs](https://huggingface.co/TheBloke/KafkaLM-70B-German-V0.1-GGUF) for +more details, including on how to concatenate multi-part files. + +## Provided Quants + +(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants) + +| Link | Type | Size/GB | Notes | +|:-----|:-----|--------:|:------| +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-IQ1_S.gguf) | i1-IQ1_S | 2.5 | for the desperate | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-IQ1_M.gguf) | i1-IQ1_M | 2.6 | mostly desperate | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-IQ2_XXS.gguf) | i1-IQ2_XXS | 2.9 | | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-IQ2_XS.gguf) | i1-IQ2_XS | 3.2 | | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-IQ2_S.gguf) | i1-IQ2_S | 3.3 | | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-IQ2_M.gguf) | i1-IQ2_M | 3.5 | | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-IQ3_XXS.gguf) | i1-IQ3_XXS | 3.9 | lower quality | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q2_K.gguf) | i1-Q2_K | 3.9 | IQ3_XXS probably better | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-IQ3_XS.gguf) | i1-IQ3_XS | 4.2 | | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-IQ3_S.gguf) | i1-IQ3_S | 4.4 | beats Q3_K* | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q3_K_S.gguf) | i1-Q3_K_S | 4.4 | IQ3_XS probably better | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-IQ3_M.gguf) | i1-IQ3_M | 4.6 | | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q3_K_M.gguf) | i1-Q3_K_M | 4.9 | IQ3_S probably better | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q3_K_L.gguf) | i1-Q3_K_L | 5.2 | IQ3_M probably better | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-IQ4_XS.gguf) | i1-IQ4_XS | 5.3 | | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q4_0_4_4.gguf) | i1-Q4_0_4_4 | 5.5 | fast on arm, low quality | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q4_0_4_8.gguf) | i1-Q4_0_4_8 | 5.5 | fast on arm+i8mm, low quality | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q4_0_8_8.gguf) | i1-Q4_0_8_8 | 5.5 | fast on arm+sve, low quality | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q4_0.gguf) | i1-Q4_0 | 5.6 | fast, low quality | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q4_K_S.gguf) | i1-Q4_K_S | 5.6 | optimal size/speed/quality | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q4_K_M.gguf) | i1-Q4_K_M | 5.9 | fast, recommended | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q5_K_S.gguf) | i1-Q5_K_S | 6.6 | | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q5_K_M.gguf) | i1-Q5_K_M | 6.7 | | +| [GGUF](https://huggingface.co/mradermacher/Athena-gemma-2-9b-it-i1-GGUF/resolve/main/Athena-gemma-2-9b-it.i1-Q6_K.gguf) | i1-Q6_K | 7.7 | practically like static Q6_K | + +Here is a handy graph by ikawrakow comparing some lower-quality quant +types (lower is better): + +![image.png](https://www.nethype.de/huggingface_embed/quantpplgraph.png) + +And here are Artefact2's thoughts on the matter: +https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9 + +## FAQ / Model Request + +See https://huggingface.co/mradermacher/model_requests for some answers to +questions you might have and/or if you want some other model quantized. + +## Thanks + +I thank my company, [nethype GmbH](https://www.nethype.de/), for letting +me use its servers and providing upgrades to my workstation to enable +this work in my free time. Additional thanks to [@nicoboss](https://huggingface.co/nicoboss) for giving me access to his private supercomputer, enabling me to provide many more imatrix quants, at much higher quality, than I would otherwise be able to. + + diff --git a/imatrix.dat b/imatrix.dat new file mode 100644 index 0000000..029947e --- /dev/null +++ b/imatrix.dat @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1767e55d35a5ba59f23ef148cbf6d2d9f5a759b3724c947aa1d991cb5107414c +size 6116887