EngineX/xc-llm-kunlun
3
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

[Feature] support compressed-tensors w4a16 quantization (#154)

Browse Source 
- native int4 kimi model inference is supported

Signed-off-by: Li Wei <liwei.109@outlook.com>
This commit is contained in:
Li Wei
2026-01-27 19:56:22 +08:00
committed by GitHub
parent 0711c1abfa
commit 71bd70ad6c
9 changed files with 369 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docs/source/user_guide/feature_guide/quantization.md
View File

@@ -8,22 +8,23 @@ Like vLLM, we now support quantization methods such as compressed-tensors, AWQ,
<table border="1" style="border-collapse: collapse; width: auto; margin: 0 0 0 0; text-align: center;">
<thead>
<tr>
<td colspan="2" style="padding: 10px; font-weight: bold; border: 1px solid #000;">Compressed-Tensor (w8a8)</td>
<td colspan="2" style="padding: 10px; font-weight: bold; border: 1px solid #000;">Compressed-Tensors (w8a8-Int8)</td>
<td colspan="4" style="padding: 10px; font-weight: bold; border: 1px solid #000;">Weight only (w4a16/w8a16)</td>
</tr>
<tr>
<td style="padding: 10px; border: 1px solid #000;">Dynamic</td>
<td style="padding: 10px; border: 1px solid #000;">Static</td>
<td colspan="2" style="padding: 10px; border: 1px solid #000;">AWQ (w4a16)</td>
<td colspan="1" style="padding: 10px; border: 1px solid #000;">AWQ (w4a16)</td>
<td colspan="2" style="padding: 10px; border: 1px solid #000;">GPTQ (w4a16/w8a16)</td>
<td colspan="1" style="padding: 10px; border: 1px solid #000;">Compressed-Tensors (w4a16)</td>
</tr>
<tr>
<td style="padding: 10px; border: 1px solid #000;">Dense/MoE</td>
<td style="padding: 10px; border: 1px solid #000;">Dense/MoE</td>
<td style="padding: 10px; border: 1px solid #000;">Dense/MoE</td>
<td style="padding: 10px; border: 1px solid #000;">Dense</td>
<td style="padding: 10px; border: 1px solid #000;">MoE</td>
<td style="padding: 10px; border: 1px solid #000;">Dense</td>
<td style="padding: 10px; border: 1px solid #000;">MoE</td>
<td style="padding: 10px; border: 1px solid #000;">Dense/MoE</td>
</tr>
</thead>
<tbody>
@@ -32,14 +33,16 @@ Like vLLM, we now support quantization methods such as compressed-tensors, AWQ,
<td style="padding: 10px; border: 1px solid #000;">✅</td>
<td style="padding: 10px; border: 1px solid #000;">✅</td>
<td style="padding: 10px; border: 1px solid #000;">✅</td>
<td style="padding: 10px; border: 1px solid #000;">✅</td>
<td style="padding: 10px; border: 1px solid #000;">WIP</td>
<td style="padding: 10px; border: 1px solid #000;">✅</td>
</tr>
</tbody>
</table>
+ W8A8 dynamic and static quantization are now supported for all LLMs and VLMs.
+ AWQ/GPTQ quantization is supported for all dense models.
+ Compressed-Tensors w8a8-Int8 dynamic and static quantization are supported for all LLMs and VLMs.
+ Compressed-Tensors w4a16 are supported for all LLMs and VLMs.
+ AWQ(w4a16) quantization is supported for all LLMs and VLMs.
+ GPTQ (w4a16/w8a16) quantization is supported for all dense models.
## Usages