commit d02affab6d892ea76c802c41d7bb65d57471556f Author: ModelHub XC Date: Sun May 17 04:26:48 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..e3cce61 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,58 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +imatrix.dat filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text +Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ1_M.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ1_M.gguf new file mode 100644 index 0000000..2dcc636 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ1_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:db8244b3b5fe7cdf02af7c8502df08e9aeafe6c70a7e92671bd2362ecc8b63c9 +size 924188064 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ1_S.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ1_S.gguf new file mode 100644 index 0000000..7995bdc --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ1_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:02b06099128648e4ee5382163dfb057553e27ae204957b732ca021dddadf9ac0 +size 868154784 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_M.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_M.gguf new file mode 100644 index 0000000..8fc74e9 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6c31e63bda55dca457624ce05ad88b58d838fd8eff520abfa57ab7f3cdc5b84f +size 1229028768 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_S.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_S.gguf new file mode 100644 index 0000000..e89560e --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e03643feb48713fd962252d7707d3fc99cc083a94816d80055b534862940cb5c +size 1154317728 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_XS.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_XS.gguf new file mode 100644 index 0000000..f842aca --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9f5b2ddf84301d78dbc297616ed28f18b5d82d52a54a0253c92038451e90d1cb +size 1100545440 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_XXS.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_XXS.gguf new file mode 100644 index 0000000..0e12268 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_XXS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ebc16bdc5c51a6dc013eafd2ede3677d3a739bdb9878c272ce5056b8f12ae8a1 +size 1017576864 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_M.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_M.gguf new file mode 100644 index 0000000..f1862e5 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a82af823de410f867453a0bcee194c989fcd76fd6f9cc6b8b02a1ccd7097282b +size 1599665568 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_S.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_S.gguf new file mode 100644 index 0000000..685c3c0 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:de95a7e32afc7c0bd26c70c39dcb04b42242e2c0eb71d14b7ce62e0232d3f363 +size 1542845856 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_XS.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_XS.gguf new file mode 100644 index 0000000..9bc234c --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:848f6ff5f238a3bc6fd83cd51625718231880cca441f73731d5464e5fe9efa6d +size 1476785568 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_XXS.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_XXS.gguf new file mode 100644 index 0000000..98c9085 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_XXS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8045309a65804c756c51b5fea3577aa82f43e65ab2b6090a5b5e859dd3e9dea1 +size 1348763040 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ4_XS.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ4_XS.gguf new file mode 100644 index 0000000..d23df18 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ4_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:081009eb55d5bc36e851d18336b13c27d37397e2184a063f25e9049b97f6f126 +size 1829107104 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-Q2_K.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q2_K.gguf new file mode 100644 index 0000000..98e5782 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q2_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:86f6473bfa10f05c0b4efb11692445be792ea121ab199e394225dae6244b1d2f +size 1363932576 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-Q2_K_S.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q2_K_S.gguf new file mode 100644 index 0000000..bc62efb --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q2_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:54728f2473063bebaab1efc7f66857e41e13a8adcddb0f5104e3425947127688 +size 1274279328 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_L.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_L.gguf new file mode 100644 index 0000000..8bc8947 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_L.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b11de70dc122a76899aa71900eb4451e8fe403b2f297536e2bcb834549b6ffb2 +size 1815344544 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_M.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_M.gguf new file mode 100644 index 0000000..55c6757 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6c0c4d65546fd002270f98f2d690c2f63d73ee0159ae54d9e84da0eefd4ef1bd +size 1687156128 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_S.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_S.gguf new file mode 100644 index 0000000..6f7e76f --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:603a7bcdf2d7178ba0ba29ee9106e9df93f96f874e7c2ee80c732308a8dd5ca9 +size 1542845856 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_0.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_0.gguf new file mode 100644 index 0000000..1bb039e --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7ee467c21c68963e183c9d217cec743e3aa93674d118c00c7bbe5ec66c397d05 +size 1921906080 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_K_M.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_K_M.gguf new file mode 100644 index 0000000..3a49df3 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d3dff662e6ba2ced41b9447e9b9d94af00ba112a0d4d5cfe70b5c29fd9702394 +size 2019374496 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_K_S.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_K_S.gguf new file mode 100644 index 0000000..10ae405 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7ffb6d727ee10d4c92d8abed7860d0e64686ca1281d8b5d40642d99d14151f15 +size 1928197536 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-Q5_K_M.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q5_K_M.gguf new file mode 100644 index 0000000..e77b834 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b3249808e9719fa85a5adfca28bfd9055b9faacb86d0b0d65c8824a187112059 +size 2322150816 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-Q5_K_S.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q5_K_S.gguf new file mode 100644 index 0000000..0431151 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q5_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:dd40c8f901a535443018b00c6ce3a3bbb2787ccb5049787501e4020b08817070 +size 2269509024 diff --git a/Hermes-3-Llama-3.2-3B-abliterated.i1-Q6_K.gguf b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q6_K.gguf new file mode 100644 index 0000000..165aac8 --- /dev/null +++ b/Hermes-3-Llama-3.2-3B-abliterated.i1-Q6_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3e8a8576902c9d9a261cdcb25981c4d0ef79cfeff2e6988b13e7af81d11a57fb +size 2643850656 diff --git a/README.md b/README.md new file mode 100644 index 0000000..75af7ba --- /dev/null +++ b/README.md @@ -0,0 +1,88 @@ +--- +base_model: lunahr/Hermes-3-Llama-3.2-3B-abliterated +language: +- en +library_name: transformers +license: llama3 +quantized_by: mradermacher +tags: +- Llama-3 +- instruct +- finetune +- chatml +- gpt4 +- synthetic data +- distillation +- function calling +- json mode +- axolotl +- roleplaying +- chat +- abliterated +--- +## About + + + + + + +weighted/imatrix quants of https://huggingface.co/lunahr/Hermes-3-Llama-3.2-3B-abliterated + + +static quants are available at https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-GGUF +## Usage + +If you are unsure how to use GGUF files, refer to one of [TheBloke's +READMEs](https://huggingface.co/TheBloke/KafkaLM-70B-German-V0.1-GGUF) for +more details, including on how to concatenate multi-part files. + +## Provided Quants + +(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants) + +| Link | Type | Size/GB | Notes | +|:-----|:-----|--------:|:------| +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ1_S.gguf) | i1-IQ1_S | 1.0 | for the desperate | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ1_M.gguf) | i1-IQ1_M | 1.0 | mostly desperate | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_XXS.gguf) | i1-IQ2_XXS | 1.1 | | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_XS.gguf) | i1-IQ2_XS | 1.2 | | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_S.gguf) | i1-IQ2_S | 1.3 | | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ2_M.gguf) | i1-IQ2_M | 1.3 | | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-Q2_K_S.gguf) | i1-Q2_K_S | 1.4 | very low quality | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_XXS.gguf) | i1-IQ3_XXS | 1.4 | lower quality | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-Q2_K.gguf) | i1-Q2_K | 1.5 | IQ3_XXS probably better | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_XS.gguf) | i1-IQ3_XS | 1.6 | | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_S.gguf) | i1-IQ3_S | 1.6 | beats Q3_K* | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_S.gguf) | i1-Q3_K_S | 1.6 | IQ3_XS probably better | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ3_M.gguf) | i1-IQ3_M | 1.7 | | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_M.gguf) | i1-Q3_K_M | 1.8 | IQ3_S probably better | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-Q3_K_L.gguf) | i1-Q3_K_L | 1.9 | IQ3_M probably better | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-IQ4_XS.gguf) | i1-IQ4_XS | 1.9 | | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_0.gguf) | i1-Q4_0 | 2.0 | fast, low quality | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_K_S.gguf) | i1-Q4_K_S | 2.0 | optimal size/speed/quality | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-Q4_K_M.gguf) | i1-Q4_K_M | 2.1 | fast, recommended | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-Q5_K_S.gguf) | i1-Q5_K_S | 2.4 | | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-Q5_K_M.gguf) | i1-Q5_K_M | 2.4 | | +| [GGUF](https://huggingface.co/mradermacher/Hermes-3-Llama-3.2-3B-abliterated-i1-GGUF/resolve/main/Hermes-3-Llama-3.2-3B-abliterated.i1-Q6_K.gguf) | i1-Q6_K | 2.7 | practically like static Q6_K | + +Here is a handy graph by ikawrakow comparing some lower-quality quant +types (lower is better): + +![image.png](https://www.nethype.de/huggingface_embed/quantpplgraph.png) + +And here are Artefact2's thoughts on the matter: +https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9 + +## FAQ / Model Request + +See https://huggingface.co/mradermacher/model_requests for some answers to +questions you might have and/or if you want some other model quantized. + +## Thanks + +I thank my company, [nethype GmbH](https://www.nethype.de/), for letting +me use its servers and providing upgrades to my workstation to enable +this work in my free time. Additional thanks to [@nicoboss](https://huggingface.co/nicoboss) for giving me access to his private supercomputer, enabling me to provide many more imatrix quants, at much higher quality, than I would otherwise be able to. + + diff --git a/imatrix.dat b/imatrix.dat new file mode 100644 index 0000000..d1181b3 --- /dev/null +++ b/imatrix.dat @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:014369ceff9ad9de2cd1f1bbab1eaf1c1f8c05ea4cac0ad64adbe40ab568aa2b +size 2988377