From 660305c38aed68437a16ac3acc383a9a3612f44e Mon Sep 17 00:00:00 2001 From: Zhan Lu <51200935+lausannel@users.noreply.github.com> Date: Fri, 14 Mar 2025 18:30:55 +0000 Subject: [PATCH] [Doc] fix wrong flag in deepseek documentation (#4427) --- benchmark/deepseek_v3/README.md | 4 ++-- docs/references/deepseek.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/benchmark/deepseek_v3/README.md b/benchmark/deepseek_v3/README.md index b64a8aaf5..0a41ceae1 100644 --- a/benchmark/deepseek_v3/README.md +++ b/benchmark/deepseek_v3/README.md @@ -197,11 +197,11 @@ Assuming that master node IP is `MASTER_IP`, checkpoint path is `/path/to/DeepSe #master python3 -m sglang.launch_server \ --model meituan/DeepSeek-R1-Block-INT8 --tp 16 --dist-init-addr \ - MASTER_IP:5000 --nnodes 2 --node-rank 0 --trust-remote --enable-torch-compile --torch-compile-max-bs 8 + MASTER_IP:5000 --nnodes 2 --node-rank 0 --trust-remote-code --enable-torch-compile --torch-compile-max-bs 8 #cluster python3 -m sglang.launch_server \ --model meituan/DeepSeek-R1-Block-INT8 --tp 16 --dist-init-addr \ - MASTER_IP:5000 --nnodes 2 --node-rank 1 --trust-remote --enable-torch-compile --torch-compile-max-bs 8 + MASTER_IP:5000 --nnodes 2 --node-rank 1 --trust-remote-code --enable-torch-compile --torch-compile-max-bs 8 ``` > **Note that the launch command here enables `torch.compile` Optimization**. For optimal performance, please refer to the command options in [Performance Optimization Options](#option_args). diff --git a/docs/references/deepseek.md b/docs/references/deepseek.md index d163ed8cf..a80aab9dd 100644 --- a/docs/references/deepseek.md +++ b/docs/references/deepseek.md @@ -141,7 +141,7 @@ With data parallelism attention enabled, we have achieved up to **1.9x** decodin Add arguments `--speculative-algorithm`, `--speculative-draft-model-path`, `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` to enable this feature. For example: ``` -python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/DeepSeek-V3-NextN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 --trust-remote --tp 8 +python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/DeepSeek-V3-NextN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 --trust-remote-code --tp 8 ``` - The draft model are available at huggingface: [lmsys/DeepSeek-V3-NextN](https://huggingface.co/lmsys/DeepSeek-V3-NextN), [lmsys/DeepSeek-R1-NextN](https://huggingface.co/lmsys/DeepSeek-R1-NextN). It can also be exported from original DeepSeek-V3/R1 model with [export_deepseek_nextn.py](https://github.com/sgl-project/sglang/blob/main/scripts/export_deepseek_nextn.py) script. - The best configuratin for `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` can be searched with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py) script for given batch size. The minimum configuration is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`, which can achieve speedup for larger batch sizes.