EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

[doc] fixed dpsk quant faq (#3865)

Browse Source
This commit is contained in:
Shenggui Li
2025-02-26 11:40:47 +08:00
committed by GitHub
parent 06427dfab1
commit 3dc9ff3ce8
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/references/deepseek.md
View File

@@ -90,8 +90,8 @@ Overall, with these optimizations, we have achieved up to a 7x acceleration in o
2. **Question**: How to use quantized DeepSeek models?
**Answer**: DeepSeek's MLA does not have support for quantization. You need to add the `--disable-mla` flag to run the quantized model successfully. Meanwhile, AWQ does not support BF16, so add the `--dtype half` flag if AWQ is used for quantization. One example is as follows:
**Answer**: AWQ does not support BF16, so add the `--dtype half` flag if AWQ is used for quantization. One example is as follows:
```bash
python3 -m sglang.launch_server --model cognitivecomputations/DeepSeek-R1-AWQ --tp 8 --trust-remote-code --dtype half --disable-mla
python3 -m sglang.launch_server --model cognitivecomputations/DeepSeek-R1-AWQ --tp 8 --trust-remote-code --dtype half
```