commit 92b6bf0da62226eef9c64d27533123e473621ed3 Author: ModelHub XC Date: Mon May 11 22:06:35 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..3521f07 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,60 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +imatrix.dat filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_4_4.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_4_8.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_8_8.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text +Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ1_M.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ1_M.gguf new file mode 100644 index 0000000..b246eee --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ1_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b739b8491c7c4ae9685ee7c56c38de9c3bc692edd874c55afa98024b1b6b0540 +size 2161973184 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ1_S.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ1_S.gguf new file mode 100644 index 0000000..4a3ca45 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ1_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a792cb3d3e4a9f01d57ffb77239a5bc9509da8b8bd688d713453cce6b1bf48e7 +size 2019628992 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_M.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_M.gguf new file mode 100644 index 0000000..217cf5f --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:12ba4b5412da5f4a6d9761229797a39060398c659d68dd6c81f8d309ab193e39 +size 2948282304 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_S.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_S.gguf new file mode 100644 index 0000000..b695efb --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c8f8d9a7a4c68b7af89a6f4a3ad0682313143c71c5dd749c78e9b870e95c7f42 +size 2758490048 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_XS.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_XS.gguf new file mode 100644 index 0000000..b5bf76c --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9b4825a40b7a63c2ad5b0e572bab4292516558fd0d769b05835a78a8ba7d47bf +size 2605782976 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_XXS.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_XXS.gguf new file mode 100644 index 0000000..cc08c63 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_XXS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:24e42dba12b0a6110feadb39084cd2367f2dae38ddaadc4bb20218c2bd89cf49 +size 2399213504 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_M.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_M.gguf new file mode 100644 index 0000000..150ce53 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:52d7edaee01cc09fc06afd2b810d9e7740cafc8f4c4d6787bc54cf937c854b77 +size 3784824768 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_S.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_S.gguf new file mode 100644 index 0000000..be0aaed --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:41c455c3ecfe9e345bb54e1305463fc42542a1a5acc431279f9798f9c79630b3 +size 3682326464 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_XS.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_XS.gguf new file mode 100644 index 0000000..f2d2bbb --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:128ab5a823a047ed87bc4348081d8d17b4098411878ebf8340588c1a02bb19cf +size 3518748608 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_XXS.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_XXS.gguf new file mode 100644 index 0000000..0d79135 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_XXS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9d63c12ae0d29cc8b01a42508474926ccd2332aeeb507e32745f5a7e60c6593d +size 3274913728 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ4_XS.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ4_XS.gguf new file mode 100644 index 0000000..30b96a9 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ4_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:dbfd9fa20ae7fe88b9b3798e818b3cde3c7511965572f118a7a47733f2545852 +size 4447664064 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q2_K.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q2_K.gguf new file mode 100644 index 0000000..f16046e --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q2_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:665b27957d70efad699dd55f00285e0e26414d0823dea819ce6b2eaeb6cf7f8e +size 3179132864 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_L.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_L.gguf new file mode 100644 index 0000000..a1e5187 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_L.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b15aa57b597c454b5e7eeda89f16eac82bec4b331ad4ea4fa545d03f97968191 +size 4321957824 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_M.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_M.gguf new file mode 100644 index 0000000..f1a97a4 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:41856398853c13c5a712afc5c9a12e580463331f58472c6f05414e2140011240 +size 4018919360 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_S.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_S.gguf new file mode 100644 index 0000000..c087d89 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:83889445bf17fe33ca4a0e53f8cae08425b36825402915a64123d63718dfa349 +size 3664500672 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0.gguf new file mode 100644 index 0000000..8b5ef0c --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:61e11845f494369d41eb8437a20bed14d2d787707ad37a49d41ccfb577d1cb33 +size 4675893184 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_4_4.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_4_4.gguf new file mode 100644 index 0000000..9ecbd32 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_4_4.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3666967e1abc979c5b651e25ff7829a79f8d1546534d71da7ce1e838ad83f958 +size 4661213120 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_4_8.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_4_8.gguf new file mode 100644 index 0000000..10fa8bf --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_4_8.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b5553fed0bd921f053360236c80f270133c4fe2f141c4aecb6c92efbf1158634 +size 4661213120 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_8_8.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_8_8.gguf new file mode 100644 index 0000000..9013011 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_8_8.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:43d6b9da50adf8bf22c7673e3180026dfcf658632def461837efbf1730130074 +size 4661213120 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_K_M.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_K_M.gguf new file mode 100644 index 0000000..ef80d92 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:44aa7f50ddbb6350baabff84a761f3fc27860a017eca703eb7aa08e8311120e9 +size 4920735680 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_K_S.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_K_S.gguf new file mode 100644 index 0000000..6ba32d8 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:95a62344636b72ea881dc3b384d23282eac6b96146b0732fccdb2ee8f5841db7 +size 4692670400 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q5_K_M.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q5_K_M.gguf new file mode 100644 index 0000000..3545055 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b68ca921c284e572f042b8d9cf9a6b8a28857c1a0e2c1535e9d9c13016873df8 +size 5732988864 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q5_K_S.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q5_K_S.gguf new file mode 100644 index 0000000..8f4793a --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q5_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:49cc8b3896600403f87fc5046284805b665916667bb65f37b950c971a821938a +size 5599295424 diff --git a/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q6_K.gguf b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q6_K.gguf new file mode 100644 index 0000000..32bb890 --- /dev/null +++ b/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q6_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0ce2db12baad0261727d274feddb3fac64e6fb2cc65207637f6e6da15dba0595 +size 6596007872 diff --git a/README.md b/README.md new file mode 100644 index 0000000..f4b5ffb --- /dev/null +++ b/README.md @@ -0,0 +1,86 @@ +--- +base_model: tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.2 +datasets: +- lmsys/lmsys-chat-1m +- tokyotech-llm/lmsys-chat-1m-synth +- argilla/magpie-ultra-v0.1 +- tokyotech-llm/swallow-magpie-ultra-v0.1 +- tokyotech-llm/swallow-gemma-magpie-v0.1 +language: +- en +- ja +library_name: transformers +license: +- llama3.1 +- gemma +model_type: llama +quantized_by: mradermacher +--- +## About + + + + + + +weighted/imatrix quants of https://huggingface.co/tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.2 + + +static quants are available at https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-GGUF +## Usage + +If you are unsure how to use GGUF files, refer to one of [TheBloke's +READMEs](https://huggingface.co/TheBloke/KafkaLM-70B-German-V0.1-GGUF) for +more details, including on how to concatenate multi-part files. + +## Provided Quants + +(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants) + +| Link | Type | Size/GB | Notes | +|:-----|:-----|--------:|:------| +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ1_S.gguf) | i1-IQ1_S | 2.1 | for the desperate | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ1_M.gguf) | i1-IQ1_M | 2.3 | mostly desperate | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_XXS.gguf) | i1-IQ2_XXS | 2.5 | | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_XS.gguf) | i1-IQ2_XS | 2.7 | | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_S.gguf) | i1-IQ2_S | 2.9 | | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ2_M.gguf) | i1-IQ2_M | 3.0 | | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q2_K.gguf) | i1-Q2_K | 3.3 | IQ3_XXS probably better | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_XXS.gguf) | i1-IQ3_XXS | 3.4 | lower quality | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_XS.gguf) | i1-IQ3_XS | 3.6 | | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_S.gguf) | i1-Q3_K_S | 3.8 | IQ3_XS probably better | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_S.gguf) | i1-IQ3_S | 3.8 | beats Q3_K* | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ3_M.gguf) | i1-IQ3_M | 3.9 | | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_M.gguf) | i1-Q3_K_M | 4.1 | IQ3_S probably better | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q3_K_L.gguf) | i1-Q3_K_L | 4.4 | IQ3_M probably better | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-IQ4_XS.gguf) | i1-IQ4_XS | 4.5 | | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_4_4.gguf) | i1-Q4_0_4_4 | 4.8 | fast on arm, low quality | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_4_8.gguf) | i1-Q4_0_4_8 | 4.8 | fast on arm+i8mm, low quality | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0_8_8.gguf) | i1-Q4_0_8_8 | 4.8 | fast on arm+sve, low quality | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_0.gguf) | i1-Q4_0 | 4.8 | fast, low quality | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_K_S.gguf) | i1-Q4_K_S | 4.8 | optimal size/speed/quality | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q4_K_M.gguf) | i1-Q4_K_M | 5.0 | fast, recommended | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q5_K_S.gguf) | i1-Q5_K_S | 5.7 | | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q5_K_M.gguf) | i1-Q5_K_M | 5.8 | | +| [GGUF](https://huggingface.co/mradermacher/Llama-3.1-Swallow-8B-Instruct-v0.2-i1-GGUF/resolve/main/Llama-3.1-Swallow-8B-Instruct-v0.2.i1-Q6_K.gguf) | i1-Q6_K | 6.7 | practically like static Q6_K | + +Here is a handy graph by ikawrakow comparing some lower-quality quant +types (lower is better): + +![image.png](https://www.nethype.de/huggingface_embed/quantpplgraph.png) + +And here are Artefact2's thoughts on the matter: +https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9 + +## FAQ / Model Request + +See https://huggingface.co/mradermacher/model_requests for some answers to +questions you might have and/or if you want some other model quantized. + +## Thanks + +I thank my company, [nethype GmbH](https://www.nethype.de/), for letting +me use its servers and providing upgrades to my workstation to enable +this work in my free time. Additional thanks to [@nicoboss](https://huggingface.co/nicoboss) for giving me access to his private supercomputer, enabling me to provide many more imatrix quants, at much higher quality, than I would otherwise be able to. + + diff --git a/imatrix.dat b/imatrix.dat new file mode 100644 index 0000000..e917d79 --- /dev/null +++ b/imatrix.dat @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:77249c492ee8954e0812be99ce5efc0fed952c717840458264b60a5547b57c66 +size 4988157