commit ef3f795bf96dd149dd540f9adaf2b23dc605bb3b Author: ModelHub XC Date: Mon Jul 13 05:18:09 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: about0/qwen-chat-GGUF-14B Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..c1ccb78 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,36 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gguf filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000..12a2d60 --- /dev/null +++ b/README.md @@ -0,0 +1,69 @@ +--- +language: +- zh +- en +tags: +- qwen +- chat +- 中文 +model_name: Qwen Chat 14B +model_type: qwen +pipeline_tag: text-generation +quantized_by: about0 +--- + +# Qwen Chat 14B - GGUF + +Here are the llama.cpp-compatible GGUF converted and/or quantized models for [Qwen 14B Chat](https://huggingface.co/Qwen/Qwen-14B-Chat). + +## Explanation of quantization methods +
+ Click to see details + +Methods: +* type-0 (Q4_0, Q5_0, Q8_0) - weights w are obtained from quants q using w = d * q, where d is the block scale. +* type-1 (Q4_1, Q5_1) - weights are given by w = d * q + m, where m is the block minimum + +The new methods available are: +* GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw) +* GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This ends up using 3.4375 bpw. +* GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw. +* GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw +* GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw +* GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type. + +This is exposed via llama.cpp quantization types that define various "quantization mixes" as follows: + +* LLAMA_FTYPE_MOSTLY_Q2_K - uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors. +* LLAMA_FTYPE_MOSTLY_Q3_K_S - uses GGML_TYPE_Q3_K for all tensors +* LLAMA_FTYPE_MOSTLY_Q3_K_M - uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K +* LLAMA_FTYPE_MOSTLY_Q3_K_L - uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K +* LLAMA_FTYPE_MOSTLY_Q4_K_S - uses GGML_TYPE_Q4_K for all tensors +* LLAMA_FTYPE_MOSTLY_Q4_K_M - uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K +* LLAMA_FTYPE_MOSTLY_Q5_K_S - uses GGML_TYPE_Q5_K for all tensors +* LLAMA_FTYPE_MOSTLY_Q5_K_M - uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K +* LLAMA_FTYPE_MOSTLY_Q6_K- uses 6-bit quantization (GGML_TYPE_Q8_K) for all tensors +
+ +## Provided files + +| Name | Quant method | Bits | Size | Max RAM required | Use case | +| ---- | ---- | ---- | ---- | ---- | ----- | +| [qwen-chat-14B-Q2_K.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q2_K.gguf) | Q2_K | 2 | 6.2 GB| 9.1 GB | smallest, significant quality-loss - not recommended for most purposes | +| [qwen-chat-14B-Q3_K_S.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q3_K_S.gguf) | Q3_K_S | 3 | 6.5 GB | 9.4 GB | very small, high quality-loss | +| [qwen-chat-14B-Q3_K_M.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q3_K_M.gguf) | Q3_K_M | 3 | 7.2 GB | 10.1 GB | very small, high quality-loss | +| [qwen-chat-14B-Q3_K_L.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q3_K_L.gguf) | Q3_K_L | 3 | 7.5 GB| 10.4 GB | small, substantial quality-loss | +| [qwen-chat-14B-Q4_0.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q4_0.gguf) | Q4_0 | 4 | 7.7 GB| 10.6 GB | legacy; small, very high quality-loss - prefer using Q3_K_L | +| [qwen-chat-14B-Q4_1.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q4_1.gguf) | Q4_1 | 4 | 8.4 GB| 11.3 GB | legacy; small, very high quality-loss - prefer using Q4_K_S | +| [qwen-chat-14B-Q4_K_S.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q4_K_S.gguf) | Q4_K_S | 4 | 8.0 GB| 10.9 GB | small, greater quality-loss | +| [qwen-chat-14B-Q4_K_M.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q4_K_M.gguf) | Q4_K_M | 4 | 8.9 GB| 11.8 GB | medium, balanced quality - recommended | +| [qwen-chat-14B-Q5_0.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q5_0.gguf) | Q5_0 | 5 | 9.2 GB| 12.1 GB | legacy; medium, balanced quality - prefer using Q5_K_M | +| [qwen-chat-14B-Q5_1.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q5_1.gguf) | Q5_1 | 5 | 10 GB| 12.9 GB | legacy; medium, balanced quality - prefer using Q5_K_M | +| [qwen-chat-14B-Q5_K_S.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q5_K_S.gguf) | Q5_K_S | 5 | 9.4 GB | 12.3 GB | large, low quality-loss - recommended | +| [qwen-chat-14B-Q5_K_M.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q5_K_M.gguf) | Q5_K_M | 5 | 11 GB | 13.9 GB | large, very low quality-loss - recommended | +| [qwen-chat-14B-Q6_K.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q6_K.gguf) | Q6_K | 6 | 12 GB| 14.9 GB | very large, extremely low quality-loss | +| [qwen-chat-14B-Q8_0.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-Q8_0.gguf) | Q8_0 | 8 | 15 GB| 17.9 GB | very large, extremely low quality-loss - not recommended | +| [qwen-chat-14B-f16.gguf](https://huggingface.co/about0/qwen-chat-GGUF-14B/blob/main/qwen-chat-14B-f16.gguf) | f16 | 16 | 27 GB| 29.9 GB | very large, no quality-loss - not recommended | + +### Model Sources +- **Repository:** [Qwen-14B-Chat](https://huggingface.co/Qwen/Qwen-14B-Chat) diff --git a/qwen-chat-14B-Q2_K.gguf b/qwen-chat-14B-Q2_K.gguf new file mode 100644 index 0000000..97bc06a --- /dev/null +++ b/qwen-chat-14B-Q2_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4f4308858a22a4b9b9cf07a5a1c090209da62517110a252da0a1f034ba4cac2b +size 6550539008 diff --git a/qwen-chat-14B-Q3_K_L.gguf b/qwen-chat-14B-Q3_K_L.gguf new file mode 100644 index 0000000..c9e827f --- /dev/null +++ b/qwen-chat-14B-Q3_K_L.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c7407828318b9c6730485124bb982875f530a1fba3a0752b9a954200c429ccdd +size 7987845888 diff --git a/qwen-chat-14B-Q3_K_M.gguf b/qwen-chat-14B-Q3_K_M.gguf new file mode 100644 index 0000000..86aa8be --- /dev/null +++ b/qwen-chat-14B-Q3_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:54b1a581c392030c31d7df12219bda62c647aad88397e92a4e3513406c7c1873 +size 7690230528 diff --git a/qwen-chat-14B-Q3_K_S.gguf b/qwen-chat-14B-Q3_K_S.gguf new file mode 100644 index 0000000..d63722b --- /dev/null +++ b/qwen-chat-14B-Q3_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:41392e1e965a61fb547ea9f4f57aeee70ae9a448ad30e034c28e66902880c003 +size 6949100288 diff --git a/qwen-chat-14B-Q4_0.gguf b/qwen-chat-14B-Q4_0.gguf new file mode 100644 index 0000000..5e9c86c --- /dev/null +++ b/qwen-chat-14B-Q4_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:16721e4f4f0073c0ccbc3740623dcc542f3ded218a512b672c9e722fa1a71eff +size 8179313408 diff --git a/qwen-chat-14B-Q4_1.gguf b/qwen-chat-14B-Q4_1.gguf new file mode 100644 index 0000000..7c3c0bd --- /dev/null +++ b/qwen-chat-14B-Q4_1.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2b9239fbb57002b24d21f63335174e5f22ee7726f6da6d732412ff33bf9033b3 +size 9016044288 diff --git a/qwen-chat-14B-Q4_K_M.gguf b/qwen-chat-14B-Q4_K_M.gguf new file mode 100644 index 0000000..0f6f0d9 --- /dev/null +++ b/qwen-chat-14B-Q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fa0b8114669a7e9904d5a33f8f7529f4104b37b57b6ae7c44c5b198f11ac52c1 +size 9449073408 diff --git a/qwen-chat-14B-Q4_K_S.gguf b/qwen-chat-14B-Q4_K_S.gguf new file mode 100644 index 0000000..7495759 --- /dev/null +++ b/qwen-chat-14B-Q4_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:00a332130215c485bc123acee462b9c6aa5789b963fd2e36f7afb0694bd41c22 +size 8547461888 diff --git a/qwen-chat-14B-Q5_0.gguf b/qwen-chat-14B-Q5_0.gguf new file mode 100644 index 0000000..ba1f37a --- /dev/null +++ b/qwen-chat-14B-Q5_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d95f41978b9b734a6125926ff68ab2bb13663e8460517caebb77a97e8fdd2925 +size 9852775168 diff --git a/qwen-chat-14B-Q5_1.gguf b/qwen-chat-14B-Q5_1.gguf new file mode 100644 index 0000000..67a8d13 --- /dev/null +++ b/qwen-chat-14B-Q5_1.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ef870bea4ef3648f223801d8d166f2dc6153357c655a3e7691b606103e64bad1 +size 10689506048 diff --git a/qwen-chat-14B-Q5_K_M.gguf b/qwen-chat-14B-Q5_K_M.gguf new file mode 100644 index 0000000..5bb704f --- /dev/null +++ b/qwen-chat-14B-Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:342ef4900ef424fc39e4b0705b1a8a4d74f2589604ad207e9e158a0fce70ff32 +size 10884147968 diff --git a/qwen-chat-14B-Q5_K_S.gguf b/qwen-chat-14B-Q5_K_S.gguf new file mode 100644 index 0000000..190deb5 --- /dev/null +++ b/qwen-chat-14B-Q5_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f15f2ca97d79c169eba2629a8d6997412db4ad422fac010b4515c2d08c6aaf5e +size 10028083968 diff --git a/qwen-chat-14B-Q6_K.gguf b/qwen-chat-14B-Q6_K.gguf new file mode 100644 index 0000000..e2ed68a --- /dev/null +++ b/qwen-chat-14B-Q6_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5197eaa8097f43870a3d2c6865498de5a1fbdf7fd3ba8ed0757aed51135fb3ab +size 12310149888 diff --git a/qwen-chat-14B-Q8_0.gguf b/qwen-chat-14B-Q8_0.gguf new file mode 100644 index 0000000..e023ecc --- /dev/null +++ b/qwen-chat-14B-Q8_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c57771eecec4106455d6c447bc7d2a558c8809ed86709de50588f4ddac547e98 +size 15061719808 diff --git a/qwen-chat-14B-f16.gguf b/qwen-chat-14B-f16.gguf new file mode 100644 index 0000000..d43c1c6 --- /dev/null +++ b/qwen-chat-14B-f16.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:aa61e8a46a0efaa00929498a0226a1a8dab188564303c6c2e6c26cd1d675ec61 +size 28342590144