commit f6d0fc9a98fd60965c9d392843c5f4ec1828cdfe Author: ModelHub XC Date: Fri Jun 5 04:30:16 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..d8d1d51 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,60 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +imatrix.dat filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text +Llama3-Toxic-8B-Float16.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text diff --git a/Llama3-Toxic-8B-Float16.i1-IQ1_M.gguf b/Llama3-Toxic-8B-Float16.i1-IQ1_M.gguf new file mode 100644 index 0000000..1cf33ae --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ1_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4eee97d4c621553605f4430d771425f4231b59dcffe879e15002f6be84a2a36c +size 2161972512 diff --git a/Llama3-Toxic-8B-Float16.i1-IQ1_S.gguf b/Llama3-Toxic-8B-Float16.i1-IQ1_S.gguf new file mode 100644 index 0000000..035be36 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ1_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f1de30890bd9647bd0fd52df80e01853d0945d40b9fbbea907fa59bcda1b616d +size 2019628320 diff --git a/Llama3-Toxic-8B-Float16.i1-IQ2_M.gguf b/Llama3-Toxic-8B-Float16.i1-IQ2_M.gguf new file mode 100644 index 0000000..b11949a --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ2_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:28c2d486d502c41c61e93422610922636e7b236d618368b50cf60d390f6e6cb9 +size 2948281632 diff --git a/Llama3-Toxic-8B-Float16.i1-IQ2_S.gguf b/Llama3-Toxic-8B-Float16.i1-IQ2_S.gguf new file mode 100644 index 0000000..069109b --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ2_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:55b47d23acd082171bcecdad86bb22134eaccb7de4ec1047a96f956382eea29f +size 2758489376 diff --git a/Llama3-Toxic-8B-Float16.i1-IQ2_XS.gguf b/Llama3-Toxic-8B-Float16.i1-IQ2_XS.gguf new file mode 100644 index 0000000..da3cafc --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ2_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:849828171a61f67117e255aeea0d1dba0eb333b80bb13c82265a4ecf7df5fb76 +size 2605782304 diff --git a/Llama3-Toxic-8B-Float16.i1-IQ2_XXS.gguf b/Llama3-Toxic-8B-Float16.i1-IQ2_XXS.gguf new file mode 100644 index 0000000..7f0edac --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ2_XXS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0b08333f2e8aa7af4a0e93027041078c84d20c7ec46b483f745cc98c2cbd2659 +size 2399212832 diff --git a/Llama3-Toxic-8B-Float16.i1-IQ3_M.gguf b/Llama3-Toxic-8B-Float16.i1-IQ3_M.gguf new file mode 100644 index 0000000..19453d9 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ3_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4851632a85def4156f7584a0fa26fd9b3fd89c03847c588fdfff4fb5d9937508 +size 3784824096 diff --git a/Llama3-Toxic-8B-Float16.i1-IQ3_S.gguf b/Llama3-Toxic-8B-Float16.i1-IQ3_S.gguf new file mode 100644 index 0000000..803aee2 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ3_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9a362a98bb825b2bd7a62dff350e47d191d727fcb4afe07fb9540b58d0d2d578 +size 3682325792 diff --git a/Llama3-Toxic-8B-Float16.i1-IQ3_XS.gguf b/Llama3-Toxic-8B-Float16.i1-IQ3_XS.gguf new file mode 100644 index 0000000..8a17f56 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ3_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6b756d9357edc644ba96b486aa5bf04a8bb88dfbc5896ed0f31a2c485a6caa15 +size 3518747936 diff --git a/Llama3-Toxic-8B-Float16.i1-IQ3_XXS.gguf b/Llama3-Toxic-8B-Float16.i1-IQ3_XXS.gguf new file mode 100644 index 0000000..fc859fb --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ3_XXS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3ebc7a630a2563abc9c246e2c5a3d47e9af911a7d00d1e8d3896b546a4111640 +size 3274913056 diff --git a/Llama3-Toxic-8B-Float16.i1-IQ4_NL.gguf b/Llama3-Toxic-8B-Float16.i1-IQ4_NL.gguf new file mode 100644 index 0000000..5cd42a8 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ4_NL.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:704c0ef7f4bc88e4ea9e6536cd41764e756bb0010576789167f0b778f6e37ef2 +size 4677989664 diff --git a/Llama3-Toxic-8B-Float16.i1-IQ4_XS.gguf b/Llama3-Toxic-8B-Float16.i1-IQ4_XS.gguf new file mode 100644 index 0000000..c1617a2 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-IQ4_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:69efc80c6b88f0e02c920a42e7282145e466439aaabf662ce6a7e3b45b34051a +size 4447663392 diff --git a/Llama3-Toxic-8B-Float16.i1-Q2_K.gguf b/Llama3-Toxic-8B-Float16.i1-Q2_K.gguf new file mode 100644 index 0000000..a5fb6e6 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q2_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f352356693b9f4694e37da85af3c8fa03833779a39f570f7ac850715690d4bd9 +size 3179132192 diff --git a/Llama3-Toxic-8B-Float16.i1-Q2_K_S.gguf b/Llama3-Toxic-8B-Float16.i1-Q2_K_S.gguf new file mode 100644 index 0000000..cdff914 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q2_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c58dbd05cb5e0c9e489a96b05c61b5aafad19936af3e1abda707e0049a0ae8d2 +size 2988815648 diff --git a/Llama3-Toxic-8B-Float16.i1-Q3_K_L.gguf b/Llama3-Toxic-8B-Float16.i1-Q3_K_L.gguf new file mode 100644 index 0000000..c7f6a86 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q3_K_L.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e81fdfc2c831b4e279526378a10409a8c385ad1c8a53eb73e45c2e2ec204a99d +size 4321957152 diff --git a/Llama3-Toxic-8B-Float16.i1-Q3_K_M.gguf b/Llama3-Toxic-8B-Float16.i1-Q3_K_M.gguf new file mode 100644 index 0000000..fc93dae --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q3_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:319ca942303ff7cfe5355f0485670d90f189a962f85d403e9617aa8c708a9578 +size 4018918688 diff --git a/Llama3-Toxic-8B-Float16.i1-Q3_K_S.gguf b/Llama3-Toxic-8B-Float16.i1-Q3_K_S.gguf new file mode 100644 index 0000000..399e544 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q3_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3762f9897f56864949aaea7e7089cf933e444de8a8ba83e11609808b78bab2b5 +size 3664500000 diff --git a/Llama3-Toxic-8B-Float16.i1-Q4_0.gguf b/Llama3-Toxic-8B-Float16.i1-Q4_0.gguf new file mode 100644 index 0000000..e38feb7 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q4_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:61946188cababa80548536891dd458e65f328baa6e39c688698f47ac4facda42 +size 4675892512 diff --git a/Llama3-Toxic-8B-Float16.i1-Q4_1.gguf b/Llama3-Toxic-8B-Float16.i1-Q4_1.gguf new file mode 100644 index 0000000..f2ef0b3 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q4_1.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6a3c549424d5a92909d5fb523d1a8d0945343c1be95dcb75e5eb09a0de393fef +size 5130253600 diff --git a/Llama3-Toxic-8B-Float16.i1-Q4_K_M.gguf b/Llama3-Toxic-8B-Float16.i1-Q4_K_M.gguf new file mode 100644 index 0000000..f896bf4 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8a4f20fb67007b27e792fe5eb40237dde6d471b7208d0a3b6727ed7e6e250b7f +size 4920735008 diff --git a/Llama3-Toxic-8B-Float16.i1-Q4_K_S.gguf b/Llama3-Toxic-8B-Float16.i1-Q4_K_S.gguf new file mode 100644 index 0000000..daaaa98 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q4_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e4ff57a7f2443efa673338718fb310ce14a474a22fa398e91fde35c93560d75c +size 4692669728 diff --git a/Llama3-Toxic-8B-Float16.i1-Q5_K_M.gguf b/Llama3-Toxic-8B-Float16.i1-Q5_K_M.gguf new file mode 100644 index 0000000..73839db --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:dc0510f2222cf6f5feaf26035887769dff2f5f31ba0a9b1bac924fa76755af27 +size 5732988192 diff --git a/Llama3-Toxic-8B-Float16.i1-Q5_K_S.gguf b/Llama3-Toxic-8B-Float16.i1-Q5_K_S.gguf new file mode 100644 index 0000000..b60b669 --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q5_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:548ac0448084ebf1611b0ab37fda8bf9a305326a62ff44a0b6913dc0bcfb43ab +size 5599294752 diff --git a/Llama3-Toxic-8B-Float16.i1-Q6_K.gguf b/Llama3-Toxic-8B-Float16.i1-Q6_K.gguf new file mode 100644 index 0000000..6d564bd --- /dev/null +++ b/Llama3-Toxic-8B-Float16.i1-Q6_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:312ca9010824c9e2fe5a7840a79cc9741361297b8a562974eaa0a1753579cce5 +size 6596007200 diff --git a/README.md b/README.md new file mode 100644 index 0000000..24c56fb --- /dev/null +++ b/README.md @@ -0,0 +1,83 @@ +--- +base_model: theminji/Llama3-Toxic-8B-Float16 +language: +- en +library_name: transformers +license: apache-2.0 +quantized_by: mradermacher +tags: +- text-generation-inference +- transformers +- unsloth +- llama +- trl +- sft +--- +## About + + + + + + +weighted/imatrix quants of https://huggingface.co/theminji/Llama3-Toxic-8B-Float16 + + +static quants are available at https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-GGUF +## Usage + +If you are unsure how to use GGUF files, refer to one of [TheBloke's +READMEs](https://huggingface.co/TheBloke/KafkaLM-70B-German-V0.1-GGUF) for +more details, including on how to concatenate multi-part files. + +## Provided Quants + +(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants) + +| Link | Type | Size/GB | Notes | +|:-----|:-----|--------:|:------| +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ1_S.gguf) | i1-IQ1_S | 2.1 | for the desperate | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ1_M.gguf) | i1-IQ1_M | 2.3 | mostly desperate | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ2_XXS.gguf) | i1-IQ2_XXS | 2.5 | | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ2_XS.gguf) | i1-IQ2_XS | 2.7 | | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ2_S.gguf) | i1-IQ2_S | 2.9 | | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ2_M.gguf) | i1-IQ2_M | 3.0 | | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q2_K_S.gguf) | i1-Q2_K_S | 3.1 | very low quality | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q2_K.gguf) | i1-Q2_K | 3.3 | IQ3_XXS probably better | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ3_XXS.gguf) | i1-IQ3_XXS | 3.4 | lower quality | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ3_XS.gguf) | i1-IQ3_XS | 3.6 | | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q3_K_S.gguf) | i1-Q3_K_S | 3.8 | IQ3_XS probably better | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ3_S.gguf) | i1-IQ3_S | 3.8 | beats Q3_K* | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ3_M.gguf) | i1-IQ3_M | 3.9 | | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q3_K_M.gguf) | i1-Q3_K_M | 4.1 | IQ3_S probably better | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q3_K_L.gguf) | i1-Q3_K_L | 4.4 | IQ3_M probably better | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ4_XS.gguf) | i1-IQ4_XS | 4.5 | | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q4_0.gguf) | i1-Q4_0 | 4.8 | fast, low quality | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-IQ4_NL.gguf) | i1-IQ4_NL | 4.8 | prefer IQ4_XS | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q4_K_S.gguf) | i1-Q4_K_S | 4.8 | optimal size/speed/quality | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q4_K_M.gguf) | i1-Q4_K_M | 5.0 | fast, recommended | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q4_1.gguf) | i1-Q4_1 | 5.2 | | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q5_K_S.gguf) | i1-Q5_K_S | 5.7 | | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q5_K_M.gguf) | i1-Q5_K_M | 5.8 | | +| [GGUF](https://huggingface.co/mradermacher/Llama3-Toxic-8B-Float16-i1-GGUF/resolve/main/Llama3-Toxic-8B-Float16.i1-Q6_K.gguf) | i1-Q6_K | 6.7 | practically like static Q6_K | + +Here is a handy graph by ikawrakow comparing some lower-quality quant +types (lower is better): + +![image.png](https://www.nethype.de/huggingface_embed/quantpplgraph.png) + +And here are Artefact2's thoughts on the matter: +https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9 + +## FAQ / Model Request + +See https://huggingface.co/mradermacher/model_requests for some answers to +questions you might have and/or if you want some other model quantized. + +## Thanks + +I thank my company, [nethype GmbH](https://www.nethype.de/), for letting +me use its servers and providing upgrades to my workstation to enable +this work in my free time. Additional thanks to [@nicoboss](https://huggingface.co/nicoboss) for giving me access to his private supercomputer, enabling me to provide many more imatrix quants, at much higher quality, than I would otherwise be able to. + + diff --git a/imatrix.dat b/imatrix.dat new file mode 100644 index 0000000..33f23e1 --- /dev/null +++ b/imatrix.dat @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1cb7a906919df1b72a3bcbfa33b799cf7877c91cdc9ffb4fdadc1bbab948d82e +size 4988157