40 lines
1.4 KiB
Markdown
40 lines
1.4 KiB
Markdown
---
|
|
license: mit
|
|
language:
|
|
- zh
|
|
- en
|
|
pipeline_tag: text-generation
|
|
library_name: transformers
|
|
base_model:
|
|
- THUDM/GLM-Z1-9B-0414
|
|
---
|
|
# Melvin56/GLM-Z1-9B-0414-GGUF
|
|
|
|
Original Model : [THUDM/GLM-Z1-9B-0414](https://huggingface.co/THUDM/GLM-Z1-9B-0414)
|
|
|
|
Llama.cpp build: ced44be3 (5199)
|
|
|
|
I used imatrix to create all these quants using this [Dataset](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8).
|
|
|
|
---
|
|
# Update-02
|
|
* [Fixed Quant] Re-quantized all quants to fix this issue. [#13099](https://github.com/ggml-org/llama.cpp/pull/13099) [#13140](https://github.com/ggml-org/llama.cpp/pull/13140)
|
|
|
|
---
|
|
|
|
| | CPU (AVX2) | CPU (ARM NEON) | Metal | cuBLAS | rocBLAS | SYCL | CLBlast | Vulkan | Kompute |
|
|
| :------------ | :---------: | :------------: | :---: | :----: | :-----: | :---: | :------: | :----: | :------: |
|
|
| K-quants | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ 🐢5 | ✅ 🐢5 | ❌ |
|
|
| I-quants | ✅ 🐢4 | ✅ 🐢4 | ✅ 🐢4 | ✅ | ✅ | Partial¹ | ❌ | ❌ | ❌ |
|
|
```
|
|
✅: feature works
|
|
🚫: feature does not work
|
|
❓: unknown, please contribute if you can test it youself
|
|
🐢: feature is slow
|
|
¹: IQ3_S and IQ1_S, see #5886
|
|
²: Only with -ngl 0
|
|
³: Inference is 50% slower
|
|
⁴: Slower than K-quants of comparable size
|
|
⁵: Slower than cuBLAS/rocBLAS on similar cards
|
|
⁶: Only q8_0 and iq4_nl
|
|
``` |