commit 4a480699ea73cfa95fb438e7190a4ce29c6419ea Author: ModelHub XC Date: Sun Sep 27 09:27:16 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: mradermacher/Falcon3-7B-Instruct-i1-GGUF Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..8fee7ed --- /dev/null +++ b/.gitattributes @@ -0,0 +1,58 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +imatrix.dat filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text +Falcon3-7B-Instruct.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text diff --git a/Falcon3-7B-Instruct.i1-IQ1_M.gguf b/Falcon3-7B-Instruct.i1-IQ1_M.gguf new file mode 100644 index 0000000..78cd773 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-IQ1_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6a4705c86f1aea84841683427a452c2a74b9840eb16a12dd2f6bc14ba30f865f +size 1912273440 diff --git a/Falcon3-7B-Instruct.i1-IQ1_S.gguf b/Falcon3-7B-Instruct.i1-IQ1_S.gguf new file mode 100644 index 0000000..ac5726a --- /dev/null +++ b/Falcon3-7B-Instruct.i1-IQ1_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6d259560ff86b96fe82774796d7f96432b70123e108077145bef46e7d42b723a +size 1769646624 diff --git a/Falcon3-7B-Instruct.i1-IQ2_M.gguf b/Falcon3-7B-Instruct.i1-IQ2_M.gguf new file mode 100644 index 0000000..2b41931 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-IQ2_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:037e5e2a45364fb0f59553701e8c4e9cee7a497996a19589891ccf06bf50be04 +size 2647144992 diff --git a/Falcon3-7B-Instruct.i1-IQ2_S.gguf b/Falcon3-7B-Instruct.i1-IQ2_S.gguf new file mode 100644 index 0000000..2f2bb60 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-IQ2_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0bcba8b118482c5a487a31800691fc361d48fefa213d72adc57e3ad33e20c3d8 +size 2456975904 diff --git a/Falcon3-7B-Instruct.i1-IQ2_XS.gguf b/Falcon3-7B-Instruct.i1-IQ2_XS.gguf new file mode 100644 index 0000000..3476f49 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-IQ2_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:aed4ee5cdf61d57f2403e8b674e80babbe0c6f38527be8a091312445de5e091b +size 2348411424 diff --git a/Falcon3-7B-Instruct.i1-IQ2_XXS.gguf b/Falcon3-7B-Instruct.i1-IQ2_XXS.gguf new file mode 100644 index 0000000..70cfd95 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-IQ2_XXS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:541343f99bfa4e24c9dd69251a79f162e5f903b9608630ffd37fa2ef6695089a +size 2149984800 diff --git a/Falcon3-7B-Instruct.i1-IQ3_M.gguf b/Falcon3-7B-Instruct.i1-IQ3_M.gguf new file mode 100644 index 0000000..d3239b3 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-IQ3_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bd6b3b808d503186d4a2dac48c8a965662a9a5fae05eaf01afbbee68e6ff26e0 +size 3441207840 diff --git a/Falcon3-7B-Instruct.i1-IQ3_S.gguf b/Falcon3-7B-Instruct.i1-IQ3_S.gguf new file mode 100644 index 0000000..4249267 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-IQ3_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:69aea7d64eb68a816b1b7ba3e13a2d5a07b66994ff585f233cc99f8411e7643e +size 3366214176 diff --git a/Falcon3-7B-Instruct.i1-IQ3_XS.gguf b/Falcon3-7B-Instruct.i1-IQ3_XS.gguf new file mode 100644 index 0000000..4ffac50 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-IQ3_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:97ae685523ba555e500ba27aa01c92846716d43df23007bb413858e12f5e1d04 +size 3210353184 diff --git a/Falcon3-7B-Instruct.i1-IQ3_XXS.gguf b/Falcon3-7B-Instruct.i1-IQ3_XXS.gguf new file mode 100644 index 0000000..b489320 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-IQ3_XXS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:19bddde673d92e7c629b0a11073331b72313f2b7f879d5425b104f5fae103b8b +size 2991381024 diff --git a/Falcon3-7B-Instruct.i1-IQ4_XS.gguf b/Falcon3-7B-Instruct.i1-IQ4_XS.gguf new file mode 100644 index 0000000..b6de8f2 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-IQ4_XS.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d08a9de24d96b0ea9c4b66d67c890e31f33a9c2de755e555adae9e56a1582e54 +size 4082506272 diff --git a/Falcon3-7B-Instruct.i1-Q2_K.gguf b/Falcon3-7B-Instruct.i1-Q2_K.gguf new file mode 100644 index 0000000..3d0da01 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-Q2_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bdc81d4a601d9d406cca6bbe0f7d347a420dd19a3cc92877de1e2924c040277e +size 2886982176 diff --git a/Falcon3-7B-Instruct.i1-Q2_K_S.gguf b/Falcon3-7B-Instruct.i1-Q2_K_S.gguf new file mode 100644 index 0000000..ab7f43f --- /dev/null +++ b/Falcon3-7B-Instruct.i1-Q2_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b3527cd08fee1b188617a3fff30ce51319019a6796ec4b39d92e5f598353ce38 +size 2699688480 diff --git a/Falcon3-7B-Instruct.i1-Q3_K_L.gguf b/Falcon3-7B-Instruct.i1-Q3_K_L.gguf new file mode 100644 index 0000000..ef62735 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-Q3_K_L.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:10b9e2fcf9dbfe58dfe9034af53d510791f8c15e37fc6e94055b0776b340db67 +size 3967982112 diff --git a/Falcon3-7B-Instruct.i1-Q3_K_M.gguf b/Falcon3-7B-Instruct.i1-Q3_K_M.gguf new file mode 100644 index 0000000..a44a581 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-Q3_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1255d4b8691a7254bd0c73bc557aa614b291ade9731130d86c4858fcf4869d89 +size 3685849632 diff --git a/Falcon3-7B-Instruct.i1-Q3_K_S.gguf b/Falcon3-7B-Instruct.i1-Q3_K_S.gguf new file mode 100644 index 0000000..d04e5a5 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-Q3_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a2cb3920387a7be0dc865bde0908e2231610dc93bd9a429db1372c565b7291d9 +size 3366214176 diff --git a/Falcon3-7B-Instruct.i1-Q4_0.gguf b/Falcon3-7B-Instruct.i1-Q4_0.gguf new file mode 100644 index 0000000..5168abf --- /dev/null +++ b/Falcon3-7B-Instruct.i1-Q4_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fa63a995349449689197ea1ae13715b027d51825fd9a8cc72d4dcc44a1a7fe17 +size 4316174880 diff --git a/Falcon3-7B-Instruct.i1-Q4_K_M.gguf b/Falcon3-7B-Instruct.i1-Q4_K_M.gguf new file mode 100644 index 0000000..236aa1a --- /dev/null +++ b/Falcon3-7B-Instruct.i1-Q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7b8ebd6fe753c75246a88110e74f0ce26f21ba4e4a663cd91bb00f1a27cafd51 +size 4569725472 diff --git a/Falcon3-7B-Instruct.i1-Q4_K_S.gguf b/Falcon3-7B-Instruct.i1-Q4_K_S.gguf new file mode 100644 index 0000000..e60c0e3 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-Q4_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4f73a0e2473053d723c6693ea71f7a6a375d09f52aba5aa88161a39f20c2c061 +size 4331018784 diff --git a/Falcon3-7B-Instruct.i1-Q5_K_M.gguf b/Falcon3-7B-Instruct.i1-Q5_K_M.gguf new file mode 100644 index 0000000..7342bdd --- /dev/null +++ b/Falcon3-7B-Instruct.i1-Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3cf22ad167f86b8b3b5785d6cf57b04b00f6220187936552f7eb18c9472451fa +size 5321947680 diff --git a/Falcon3-7B-Instruct.i1-Q5_K_S.gguf b/Falcon3-7B-Instruct.i1-Q5_K_S.gguf new file mode 100644 index 0000000..5490523 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-Q5_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3d5e85e98fba2abbf666678d08411f3f6dc6a43d54ca30a8a8155e14267b6d54 +size 5184494112 diff --git a/Falcon3-7B-Instruct.i1-Q6_K.gguf b/Falcon3-7B-Instruct.i1-Q6_K.gguf new file mode 100644 index 0000000..db6e487 --- /dev/null +++ b/Falcon3-7B-Instruct.i1-Q6_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:653aa8717e9403d03078401e347fcfd5e73e57be419134982613b41daf907f4b +size 6121183776 diff --git a/README.md b/README.md new file mode 100644 index 0000000..e6b71f8 --- /dev/null +++ b/README.md @@ -0,0 +1,81 @@ +--- +base_model: tiiuae/Falcon3-7B-Instruct +language: +- en +- fr +- es +- pt +library_name: transformers +license: other +license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html +license_name: falcon-llm-license +quantized_by: mradermacher +tags: +- falcon3 +--- +## About + + + + + + +weighted/imatrix quants of https://huggingface.co/tiiuae/Falcon3-7B-Instruct + + +static quants are available at https://huggingface.co/mradermacher/Falcon3-7B-Instruct-GGUF +## Usage + +If you are unsure how to use GGUF files, refer to one of [TheBloke's +READMEs](https://huggingface.co/TheBloke/KafkaLM-70B-German-V0.1-GGUF) for +more details, including on how to concatenate multi-part files. + +## Provided Quants + +(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants) + +| Link | Type | Size/GB | Notes | +|:-----|:-----|--------:|:------| +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-IQ1_S.gguf) | i1-IQ1_S | 1.9 | for the desperate | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-IQ1_M.gguf) | i1-IQ1_M | 2.0 | mostly desperate | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-IQ2_XXS.gguf) | i1-IQ2_XXS | 2.2 | | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-IQ2_XS.gguf) | i1-IQ2_XS | 2.4 | | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-IQ2_S.gguf) | i1-IQ2_S | 2.6 | | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-IQ2_M.gguf) | i1-IQ2_M | 2.7 | | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-Q2_K_S.gguf) | i1-Q2_K_S | 2.8 | very low quality | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-Q2_K.gguf) | i1-Q2_K | 3.0 | IQ3_XXS probably better | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-IQ3_XXS.gguf) | i1-IQ3_XXS | 3.1 | lower quality | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-IQ3_XS.gguf) | i1-IQ3_XS | 3.3 | | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-IQ3_S.gguf) | i1-IQ3_S | 3.5 | beats Q3_K* | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-Q3_K_S.gguf) | i1-Q3_K_S | 3.5 | IQ3_XS probably better | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-IQ3_M.gguf) | i1-IQ3_M | 3.5 | | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-Q3_K_M.gguf) | i1-Q3_K_M | 3.8 | IQ3_S probably better | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-Q3_K_L.gguf) | i1-Q3_K_L | 4.1 | IQ3_M probably better | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-IQ4_XS.gguf) | i1-IQ4_XS | 4.2 | | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-Q4_0.gguf) | i1-Q4_0 | 4.4 | fast, low quality | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-Q4_K_S.gguf) | i1-Q4_K_S | 4.4 | optimal size/speed/quality | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-Q4_K_M.gguf) | i1-Q4_K_M | 4.7 | fast, recommended | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-Q5_K_S.gguf) | i1-Q5_K_S | 5.3 | | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-Q5_K_M.gguf) | i1-Q5_K_M | 5.4 | | +| [GGUF](https://huggingface.co/mradermacher/Falcon3-7B-Instruct-i1-GGUF/resolve/main/Falcon3-7B-Instruct.i1-Q6_K.gguf) | i1-Q6_K | 6.2 | practically like static Q6_K | + +Here is a handy graph by ikawrakow comparing some lower-quality quant +types (lower is better): + +![image.png](https://www.nethype.de/huggingface_embed/quantpplgraph.png) + +And here are Artefact2's thoughts on the matter: +https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9 + +## FAQ / Model Request + +See https://huggingface.co/mradermacher/model_requests for some answers to +questions you might have and/or if you want some other model quantized. + +## Thanks + +I thank my company, [nethype GmbH](https://www.nethype.de/), for letting +me use its servers and providing upgrades to my workstation to enable +this work in my free time. Additional thanks to [@nicoboss](https://huggingface.co/nicoboss) for giving me access to his private supercomputer, enabling me to provide many more imatrix quants, at much higher quality, than I would otherwise be able to. + + diff --git a/imatrix.dat b/imatrix.dat new file mode 100644 index 0000000..d8bb746 --- /dev/null +++ b/imatrix.dat @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f42c03d2777203822b3861fc8497e945de691f3a7c5638dd3de14531e867918c +size 4651353