1755 lines
64 KiB
Markdown
1755 lines
64 KiB
Markdown
# DeepSeek-V4-Pro
|
||
|
||
## 1 Introduction
|
||
|
||
DeepSeek-V4 introduces several key upgrades over DeepSeek-V3:
|
||
|
||
- The Manifold-Constrained Hyper-Connections (mHC) to strengthen conventional residual connections.
|
||
- A hybrid attention architecture, which greatly improves long-context efficiency through Compress-4-Attention and Compress-128-Attention. For the Mixture-of-Experts (MoE) components, it still adopts the DeepSeekMoE architecture, with only minor adjustments.
|
||
|
||
DeepSeek-V4-Pro, the maximum reasoning effort mode of DeepSeek-V4, significantly advances the knowledge capabilities of open-source models, firmly establishing itself as the best open-source model available today. It achieves top-tier performance in coding benchmarks and significantly bridges the gap with leading closed-source models on reasoning and agentic tasks.
|
||
|
||
This document will show the main verification steps of the model, including supported features, feature configuration, environment preparation, single-node and multi-node deployment, accuracy and performance evaluation.
|
||
|
||
## 2 Supported Features
|
||
|
||
Refer to [supported features](../../user_guide/support_matrix/supported_models.md) to get the model's supported feature matrix.
|
||
|
||
Refer to [feature guide](../../user_guide/feature_guide/index.md) to get the feature's configuration.
|
||
|
||
## 3 Prerequisites
|
||
|
||
### 3.1 Model Weight
|
||
|
||
- `DeepSeek-V4-Pro-w4a8-mtp` (Quantized version): requires 2 Atlas 800 A3 (128GB × 8) nodes or 4 Atlas 800 A2 (64GB × 8) nodes. [Download model weight](https://www.modelscope.cn/models/Eco-Tech/DeepSeek-V4-Pro-w4a8-mtp)
|
||
- For Ascend 950DT servers, use the original [`DeepSeek-V4-Pro`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) weights released by DeepSeek on Hugging Face. The Attention weights use MXFP8, while the MoE weights use MXFP4 with 4-bit weights and 8-bit activation computation (W4A8). No Ascend-specific quantization or weight conversion is required. The mixed deployment in Section 5.1 uses 2 Ascend 950DT servers (96GB × 8), while the 1P1D example in Section 5.2.3 uses 4 Prefill servers and 4 Decode servers.
|
||
|
||
It is recommended to download the model weight to the shared directory of multiple nodes, such as `/root/.cache/`.
|
||
|
||
### 3.2 Verify Multi-node Communication (Optional)
|
||
|
||
If you want to deploy a multi-node environment, you need to verify multi-node communication according to [verify multi-node communication environment](../../installation.md#verify-multi-node-communication).
|
||
|
||
## 4 Installation
|
||
|
||
### 4.1 Docker Image Installation
|
||
|
||
Select an image based on your machine type and start the docker image on your node, refer to [using docker](../../installation.md#set-up-using-docker).
|
||
|
||
:::::{tab-set}
|
||
:sync-group: install
|
||
|
||
::::{tab-item} Ascend 950DT series
|
||
:sync: Ascend 950DT
|
||
|
||
Start the docker image on your each node.
|
||
|
||
```{code-block} bash
|
||
:substitutions:
|
||
|
||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a5
|
||
export NAME=vllm-ascend
|
||
|
||
docker run --rm \
|
||
--name $NAME \
|
||
--net=host \
|
||
--shm-size=512g \
|
||
--device /dev/davinci0 \
|
||
--device /dev/davinci1 \
|
||
--device /dev/davinci2 \
|
||
--device /dev/davinci3 \
|
||
--device /dev/davinci4 \
|
||
--device /dev/davinci5 \
|
||
--device /dev/davinci6 \
|
||
--device /dev/davinci7 \
|
||
--device /dev/davinci_manager \
|
||
--device /dev/hisi_hdc \
|
||
--device /dev/ummu \
|
||
--device /dev/uburma \
|
||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||
-v /etc/hccl_rootinfo.json:/etc/hccl_rootinfo.json \
|
||
-v /etc/hixlep/:/etc/hixlep/ \
|
||
-v /root/.cache:/root/.cache \
|
||
-v /usr/local/sbin:/usr/local/sbin \
|
||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
|
||
-itd $IMAGE bash
|
||
```
|
||
|
||
::::
|
||
::::{tab-item} A3 series
|
||
:sync: A3
|
||
|
||
Start the docker image on each node.
|
||
|
||
```{code-block} bash
|
||
:substitutions:
|
||
|
||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3
|
||
docker run --rm \
|
||
--name vllm-ascend \
|
||
--shm-size=512g \
|
||
--net=host \
|
||
--device /dev/davinci0 \
|
||
--device /dev/davinci1 \
|
||
--device /dev/davinci2 \
|
||
--device /dev/davinci3 \
|
||
--device /dev/davinci4 \
|
||
--device /dev/davinci5 \
|
||
--device /dev/davinci6 \
|
||
--device /dev/davinci7 \
|
||
--device /dev/davinci8 \
|
||
--device /dev/davinci9 \
|
||
--device /dev/davinci10 \
|
||
--device /dev/davinci11 \
|
||
--device /dev/davinci12 \
|
||
--device /dev/davinci13 \
|
||
--device /dev/davinci14 \
|
||
--device /dev/davinci15 \
|
||
--device /dev/davinci_manager \
|
||
--device /dev/devmm_svm \
|
||
--device /dev/hisi_hdc \
|
||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
|
||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
|
||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||
-v /etc/hccn.conf:/etc/hccn.conf \
|
||
-it $IMAGE bash
|
||
```
|
||
|
||
::::
|
||
::::{tab-item} A2 series
|
||
:sync: A2
|
||
|
||
Start the docker image on each node.
|
||
|
||
```{code-block} bash
|
||
:substitutions:
|
||
|
||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||
docker run --rm \
|
||
--name vllm-ascend \
|
||
--shm-size=512g \
|
||
--net=host \
|
||
--device /dev/davinci0 \
|
||
--device /dev/davinci1 \
|
||
--device /dev/davinci2 \
|
||
--device /dev/davinci3 \
|
||
--device /dev/davinci4 \
|
||
--device /dev/davinci5 \
|
||
--device /dev/davinci6 \
|
||
--device /dev/davinci7 \
|
||
--device /dev/davinci_manager \
|
||
--device /dev/devmm_svm \
|
||
--device /dev/hisi_hdc \
|
||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
|
||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
|
||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||
-v /etc/hccn.conf:/etc/hccn.conf \
|
||
-it $IMAGE bash
|
||
```
|
||
|
||
::::
|
||
:::::
|
||
|
||
After a successful docker run, you can verify the running container service by executing the `docker ps` command.
|
||
|
||
### 4.2 Source Code Installation
|
||
|
||
If you don't want to use the docker image as above, you can also build all from source:
|
||
|
||
- Install `vllm-ascend` from source, refer to [installation](../../installation.md).
|
||
|
||
If you want to deploy a multi-node environment, you need to set up the environment on each node.
|
||
|
||
## 5 Online Service Deployment
|
||
|
||
:::{note}
|
||
In this tutorial, we suppose you downloaded the model weight to `/root/.cache/`. Feel free to change it to your own path.
|
||
:::
|
||
|
||
### 5.1 Multi-Node Online Deployment
|
||
|
||
The quantized model `DeepSeek-V4-Pro-w4a8-mtp` requires at least 2 Atlas 800 A3 (128GB × 8) nodes or 4 Atlas 800 A2 (64GB × 8) nodes. The mixed-deployment example for Ascend 950DT servers uses 2 servers (96GB × 8). Run the following scripts on each server respectively.
|
||
|
||
:::::{tab-set}
|
||
:sync-group: install
|
||
|
||
::::{tab-item} A2 series
|
||
:sync: A2
|
||
|
||
**Node0**
|
||
|
||
```bash
|
||
local_ip="xxx"
|
||
node0_ip="xxxx"
|
||
|
||
export HCCL_IF_IP=$local_ip
|
||
export IFNAME="xxx"
|
||
export GLOO_SOCKET_IFNAME="$IFNAME"
|
||
export TP_SOCKET_IFNAME="$IFNAME"
|
||
export HCCL_SOCKET_IFNAME="$IFNAME"
|
||
export HCCL_BUFFSIZE=512
|
||
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export ACL_OP_INIT_MODE=1
|
||
export VLLM_ENGINE_READY_TIMEOUT_S=3600
|
||
export HCCL_OP_EXPANSION_MODE="AIV"
|
||
|
||
export TASK_QUEUE_ENABLE=1
|
||
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
|
||
|
||
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
|
||
|
||
export HCCL_CONNECT_TIMEOUT=7200
|
||
export ASCEND_CONNECT_TIMEOUT=10000
|
||
export ASCEND_TRANSFER_TIMEOUT=10000
|
||
export VLLM_RPC_TIMEOUT=1800000
|
||
|
||
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
|
||
--host 0.0.0.0 \
|
||
--port 10010 \
|
||
--max-model-len 135000 \
|
||
--max-num-batched-tokens 4096 \
|
||
--served-model-name dsv4 \
|
||
--gpu-memory-utilization 0.9 \
|
||
--max-num-seqs 16 \
|
||
--data-parallel-size 4 \
|
||
--tensor-parallel-size 8 \
|
||
--data-parallel-size-local 1 \
|
||
--data-parallel-start-rank 0 \
|
||
--data-parallel-address $node0_ip \
|
||
--enable-expert-parallel \
|
||
--quantization ascend \
|
||
--no-enable-prefix-caching \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--async-scheduling \
|
||
--safetensors-load-strategy 'prefetch' \
|
||
--block-size 128 \
|
||
--speculative-config '{
|
||
"num_speculative_tokens": 1,
|
||
"method": "mtp",
|
||
"enforce_eager": true
|
||
}' \
|
||
--additional-config '{
|
||
"ascend_compilation_config":{
|
||
"enable_npugraph_ex":true,
|
||
"enable_static_kernel":false
|
||
},
|
||
"enable_cpu_binding": true,
|
||
"enable_shared_expert_dp": true,
|
||
"multistream_overlap_shared_expert":true
|
||
}' \
|
||
--compilation-config '{
|
||
"cudagraph_mode":"FULL_DECODE_ONLY"
|
||
}' \
|
||
--model-loader-extra-config '{
|
||
"enable_multithread_load": "true",
|
||
"num_threads": 128
|
||
}'
|
||
```
|
||
|
||
**Node1-Node3**
|
||
|
||
```bash
|
||
local_ip="xxx"
|
||
node0_ip="xxxx"
|
||
|
||
export HCCL_IF_IP=$local_ip
|
||
export IFNAME="xxx"
|
||
export GLOO_SOCKET_IFNAME="$IFNAME"
|
||
export TP_SOCKET_IFNAME="$IFNAME"
|
||
export HCCL_SOCKET_IFNAME="$IFNAME"
|
||
export HCCL_BUFFSIZE=512
|
||
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export ACL_OP_INIT_MODE=1
|
||
export VLLM_ENGINE_READY_TIMEOUT_S=3600
|
||
export HCCL_OP_EXPANSION_MODE="AIV"
|
||
|
||
export TASK_QUEUE_ENABLE=1
|
||
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
|
||
|
||
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
|
||
|
||
export HCCL_CONNECT_TIMEOUT=7200
|
||
export ASCEND_CONNECT_TIMEOUT=10000
|
||
export ASCEND_TRANSFER_TIMEOUT=10000
|
||
export VLLM_RPC_TIMEOUT=1800000
|
||
|
||
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
|
||
--host 0.0.0.0 \
|
||
--port 10010 \
|
||
--max-model-len 135000 \
|
||
--max-num-batched-tokens 4096 \
|
||
--served-model-name dsv4 \
|
||
--gpu-memory-utilization 0.9 \
|
||
--max-num-seqs 16 \
|
||
--data-parallel-size 4 \
|
||
--tensor-parallel-size 8 \
|
||
--data-parallel-size-local 1 \
|
||
--data-parallel-start-rank 1 \
|
||
--data-parallel-address $node0_ip \
|
||
--enable-expert-parallel \
|
||
--quantization ascend \
|
||
--no-enable-prefix-caching \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--async-scheduling \
|
||
--safetensors-load-strategy 'prefetch' \
|
||
--block-size 128 \
|
||
--headless \
|
||
--speculative-config '{
|
||
"num_speculative_tokens": 1,
|
||
"method": "mtp",
|
||
"enforce_eager": true
|
||
}' \
|
||
--additional-config '{
|
||
"ascend_compilation_config":{
|
||
"enable_npugraph_ex":true,
|
||
"enable_static_kernel":false
|
||
},
|
||
"enable_cpu_binding": true,
|
||
"enable_shared_expert_dp": true,
|
||
"multistream_overlap_shared_expert":true
|
||
}' \
|
||
--compilation-config '{
|
||
"cudagraph_mode":"FULL_DECODE_ONLY"
|
||
}' \
|
||
--model-loader-extra-config '{
|
||
"enable_multithread_load": "true",
|
||
"num_threads": 128
|
||
}'
|
||
```
|
||
|
||
::::
|
||
::::{tab-item} A3 series
|
||
:sync: A3
|
||
|
||
**Node0**
|
||
|
||
```bash
|
||
# this obtained through ifconfig
|
||
# nic_name is the network interface name corresponding to local_ip of the current node
|
||
nic_name="xxx"
|
||
local_ip="xxx"
|
||
|
||
# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
|
||
node0_ip="xxxx"
|
||
|
||
export HCCL_OP_EXPANSION_MODE="AIV"
|
||
export HCCL_IF_IP=$local_ip
|
||
export GLOO_SOCKET_IFNAME=$nic_name
|
||
export TP_SOCKET_IFNAME=$nic_name
|
||
export HCCL_SOCKET_IFNAME=$nic_name
|
||
export HCCL_BUFFSIZE=2048
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export TASK_QUEUE_ENABLE=1
|
||
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
|
||
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
|
||
|
||
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
|
||
--safetensors-load-strategy 'prefetch' \
|
||
--max-model-len 135000 \
|
||
--max-num-batched-tokens 4096 \
|
||
--served-model-name dsv4 \
|
||
--gpu-memory-utilization 0.9 \
|
||
--max-num-seqs 32 \
|
||
--data-parallel-size 2 \
|
||
--data-parallel-size-local 1 \
|
||
--data-parallel-start-rank 0 \
|
||
--data-parallel-address $node0_ip \
|
||
--data-parallel-rpc-port 13399 \
|
||
--tensor-parallel-size 16 \
|
||
--enable-expert-parallel \
|
||
--quantization ascend \
|
||
--port 8900 \
|
||
--host 0.0.0.0 \
|
||
--block-size 128 \
|
||
--async-scheduling \
|
||
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
|
||
--additional-config '
|
||
{"ascend_compilation_config":{
|
||
"enable_npugraph_ex":true,
|
||
"enable_static_kernel":false
|
||
},
|
||
"enable_cpu_binding": true,
|
||
"multistream_overlap_shared_expert":true}'
|
||
```
|
||
|
||
**Node1**
|
||
|
||
```bash
|
||
# this obtained through ifconfig
|
||
# nic_name is the network interface name corresponding to local_ip of the current node
|
||
nic_name="xxx"
|
||
local_ip="xxx"
|
||
|
||
# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
|
||
node0_ip="xxxx"
|
||
|
||
export HCCL_OP_EXPANSION_MODE="AIV"
|
||
export HCCL_IF_IP=$local_ip
|
||
export GLOO_SOCKET_IFNAME=$nic_name
|
||
export TP_SOCKET_IFNAME=$nic_name
|
||
export HCCL_SOCKET_IFNAME=$nic_name
|
||
export HCCL_BUFFSIZE=2048
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export TASK_QUEUE_ENABLE=1
|
||
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
|
||
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
|
||
|
||
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
|
||
--safetensors-load-strategy 'prefetch' \
|
||
--max-model-len 135000 \
|
||
--max-num-batched-tokens 4096 \
|
||
--served-model-name dsv4 \
|
||
--gpu-memory-utilization 0.9 \
|
||
--max-num-seqs 32 \
|
||
--data-parallel-size 2 \
|
||
--data-parallel-size-local 1 \
|
||
--data-parallel-start-rank 1 \
|
||
--data-parallel-address $node0_ip \
|
||
--data-parallel-rpc-port 13399 \
|
||
--tensor-parallel-size 16 \
|
||
--enable-expert-parallel \
|
||
--quantization ascend \
|
||
--port 8900 \
|
||
--host 0.0.0.0 \
|
||
--block-size 128 \
|
||
--async-scheduling \
|
||
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
|
||
--additional-config '
|
||
{"ascend_compilation_config":{
|
||
"enable_npugraph_ex":true,
|
||
"enable_static_kernel":false
|
||
},
|
||
"enable_cpu_binding": true,
|
||
"multistream_overlap_shared_expert":true}'
|
||
```
|
||
|
||
::::
|
||
::::{tab-item} Ascend 950DT series
|
||
:sync: Ascend 950DT
|
||
|
||
Set `node0_ip` to the service IP of Node0 on both nodes.
|
||
|
||
**Node0**
|
||
|
||
```bash
|
||
source /root/.bashrc
|
||
|
||
nic_name="xxx"
|
||
local_ip="xx.xx.xx.1"
|
||
node0_ip="xx.xx.xx.1"
|
||
|
||
export HCCL_IF_IP=$local_ip
|
||
export GLOO_SOCKET_IFNAME=$nic_name
|
||
export TP_SOCKET_IFNAME=$nic_name
|
||
export HCCL_SOCKET_IFNAME=$nic_name
|
||
export HCCL_ALGO=level0:fullmesh
|
||
export VLLM_RPC_TIMEOUT=3600000
|
||
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
|
||
export HCCL_EXEC_TIMEOUT=2040
|
||
export HCCL_CONNECT_TIMEOUT=1200
|
||
export HCCL_BUFFSIZE=512
|
||
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||
|
||
vllm serve /root/.cache/DeepSeek-V4-Pro \
|
||
--host $local_ip \
|
||
--port 8900 \
|
||
--data-parallel-size 2 \
|
||
--data-parallel-size-local 1 \
|
||
--data-parallel-start-rank 0 \
|
||
--data-parallel-address $node0_ip \
|
||
--data-parallel-rpc-port 6987 \
|
||
--tensor-parallel-size 8 \
|
||
--max-model-len 200000 \
|
||
--max-num-batched-tokens 8192 \
|
||
--served-model-name dsv4 \
|
||
--gpu-memory-utilization 0.94 \
|
||
--enable-expert-parallel \
|
||
--async-scheduling \
|
||
--max-num-seqs 40 \
|
||
--block-size 128 \
|
||
--api-server-count 1 \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--safetensors-load-strategy prefetch \
|
||
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
|
||
--speculative-config '{"num_speculative_tokens": 3, "method": "mtp", "enforce_eager": true}' \
|
||
--additional-config '{"enable_cpu_binding": true, "multistream_overlap_shared_expert": true, "multistream_dsa_preprocess": false, "dp_allreduce_on_npu": false, "enable_flashcomm1": true, "enable_dsa_cp": true}'
|
||
```
|
||
|
||
**Node1**
|
||
|
||
```bash
|
||
source /root/.bashrc
|
||
|
||
nic_name="xxx"
|
||
local_ip="xx.xx.xx.2"
|
||
node0_ip="xx.xx.xx.1"
|
||
|
||
export HCCL_IF_IP=$local_ip
|
||
export GLOO_SOCKET_IFNAME=$nic_name
|
||
export TP_SOCKET_IFNAME=$nic_name
|
||
export HCCL_SOCKET_IFNAME=$nic_name
|
||
export HCCL_ALGO=level0:fullmesh
|
||
export VLLM_RPC_TIMEOUT=3600000
|
||
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
|
||
export HCCL_EXEC_TIMEOUT=2040
|
||
export HCCL_CONNECT_TIMEOUT=1200
|
||
export HCCL_BUFFSIZE=512
|
||
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||
|
||
vllm serve /root/.cache/DeepSeek-V4-Pro \
|
||
--data-parallel-size 2 \
|
||
--data-parallel-size-local 1 \
|
||
--data-parallel-start-rank 1 \
|
||
--data-parallel-address $node0_ip \
|
||
--data-parallel-rpc-port 6987 \
|
||
--tensor-parallel-size 8 \
|
||
--max-model-len 200000 \
|
||
--max-num-batched-tokens 8192 \
|
||
--served-model-name dsv4 \
|
||
--gpu-memory-utilization 0.94 \
|
||
--enable-expert-parallel \
|
||
--async-scheduling \
|
||
--max-num-seqs 40 \
|
||
--block-size 128 \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--safetensors-load-strategy prefetch \
|
||
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
|
||
--speculative-config '{"num_speculative_tokens": 3, "method": "mtp", "enforce_eager": true}' \
|
||
--additional-config '{"enable_cpu_binding": true, "multistream_overlap_shared_expert": true, "multistream_dsa_preprocess": false, "dp_allreduce_on_npu": false, "enable_flashcomm1": true, "enable_dsa_cp": true}' \
|
||
--headless
|
||
```
|
||
|
||
::::
|
||
:::::
|
||
|
||
Key Parameter Descriptions:
|
||
|
||
- `--data-parallel-size` sets the global number of data parallel ranks, and `--data-parallel-size-local` sets the number of DP ranks on the current node.
|
||
- `--data-parallel-start-rank` specifies the starting data parallel rank of the current node. Each node must be set to a unique value (e.g., Node0 = 0, Node1 = 1).
|
||
- `--data-parallel-address` specifies the IP address of the data parallel master node (Node0). It must be consistent across all nodes.
|
||
- `--data-parallel-rpc-port` is the DP RPC port. Use the same value on all nodes and ensure the port is available.
|
||
- `--tensor-parallel-size` sets the tensor parallel size within each DP rank. Configure it together with the DP sizes according to the deployment topology and available NPUs.
|
||
- On Ascend 950DT, DeepSeek-V4 does not currently support standalone tensor parallelism (TP-only). TP partitions only the `wq_b`, `wo_a`, and `wo_b` linear layers, so data parallelism is recommended. When TP is required, use it together with DSA-CP and enable FlashComm (`enable_flashcomm1`) and DSA-CP (`enable_dsa_cp`), as in the two-server DeepSeek-V4-Pro mixed deployment.
|
||
- `--enable-expert-parallel` enables expert parallelism for MoE layers. Do not mix MoE tensor parallelism and expert parallelism in the same MoE layer.
|
||
- `--headless` (used on non-master nodes) disables the API server on the node, since only the master node serves requests.
|
||
- `--max-model-len` specifies the maximum context length. Adjust it according to your actual scenario.
|
||
- `--max-num-seqs` indicates the maximum number of requests that each DP group is allowed to process. If the number of requests sent to the service exceeds this limit, the excess requests will remain in a waiting state and will not be scheduled. Note that the time spent in the waiting state is also counted in metrics such as TTFT and TPOT. Therefore, when testing performance, it is generally recommended that `--max-num-seqs` * `--data-parallel-size` >= the actual total concurrency.
|
||
- `--max-num-batched-tokens` is the maximum number of tokens processed in one scheduler step. A larger value can improve prefill efficiency but consumes more activation memory.
|
||
- `--no-enable-prefix-caching` indicates that prefix caching is disabled. To enable it, remove this option.
|
||
- `--block-size` sets the KV cache block size. To enable the experimental 4K prefix cache hit support, change it from `128` to `32`.
|
||
- `--quantization ascend` enables Ascend quantization for the W4A8 model.
|
||
- `--speculative-config` configures the MTP (Multi-Token Prediction) speculative decoding to accelerate inference.
|
||
- `--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'` enables full ACL graph execution in the decode phase to reduce scheduling latency.
|
||
- `--async-scheduling` enables asynchronous scheduling to overlap CPU scheduling with NPU computation.
|
||
- `--additional-config` enables Ascend-specific optimizations. `enable_npugraph_ex` enables enhanced ACL graph execution, `enable_static_kernel: false` keeps static-kernel compilation disabled, `enable_cpu_binding` enables Ascend-native CPU binding, `enable_shared_expert_dp` enables data parallelism for shared experts, and `multistream_overlap_shared_expert` overlaps shared expert computation for better MoE throughput.
|
||
- `VLLM_ASCEND_ENABLE_FLASHCOMM1=1` enables the FlashComm communication optimization.
|
||
|
||
Common Issues Tip: If you encounter issues, please refer to the [Public FAQ](https://docs.vllm.ai/projects/ascend/en/latest/faqs.html) for troubleshooting.
|
||
|
||
Service Verification:
|
||
|
||
```shell
|
||
curl http://<node0_ip>:8900/v1/chat/completions \
|
||
-H "Content-Type: application/json" \
|
||
-d '{
|
||
"model": "dsv4",
|
||
"messages": [
|
||
{
|
||
"role": "user",
|
||
"content": "Who are you?"
|
||
}
|
||
],
|
||
"max_tokens": 256,
|
||
"temperature": 0
|
||
}'
|
||
```
|
||
|
||
Expected Result:
|
||
|
||
The service returns HTTP 200 OK with a JSON response containing the `choices` field.
|
||
|
||
### 5.2 Multi-Node PD Separation Deployment
|
||
|
||
We recommend using Mooncake for deployment: [Mooncake](../features/pd_disaggregation_mooncake_multi_node.md).
|
||
|
||
In the standard deployment mode, Prefill (prompt processing) and Decode (token generation) tasks run on the same set of NPUs. PD (Prefill-Decode) separation addresses this by running Prefill and Decode on dedicated node groups, each configured independently. This architecture is recommended for production deployments with concurrent multi-user workloads, where stable latency and high throughput are both required.
|
||
|
||
The following sections describe PD separation deployment in multi-node environments using Atlas 800 A3 nodes (128GB × 8), Atlas 800 A2 nodes (64GB × 8), or Ascend 950DT servers (96GB × 8).
|
||
|
||
#### 5.2.1 A3 Series PD Separation Deployment
|
||
|
||
This section shows the deployment guide of DeepSeek-V4-Pro on Atlas 800 A3 (128GB × 8) multi-node environment with 1P1D for better performance.
|
||
|
||
Before you start, please:
|
||
|
||
1. Prepare the script `launch_online_dp.py` on each node.
|
||
|
||
```python
|
||
import argparse
|
||
import multiprocessing
|
||
import os
|
||
import subprocess
|
||
import sys
|
||
|
||
def parse_args():
|
||
parser = argparse.ArgumentParser()
|
||
parser.add_argument(
|
||
"--dp-size",
|
||
type=int,
|
||
required=True,
|
||
help="Data parallel size."
|
||
)
|
||
parser.add_argument(
|
||
"--tp-size",
|
||
type=int,
|
||
default=1,
|
||
help="Tensor parallel size."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-size-local",
|
||
type=int,
|
||
default=-1,
|
||
help="Local data parallel size."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-rank-start",
|
||
type=int,
|
||
default=0,
|
||
help="Starting rank for data parallel."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-address",
|
||
type=str,
|
||
required=True,
|
||
help="IP address for data parallel master node."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-rpc-port",
|
||
type=str,
|
||
default=12345,
|
||
help="Port for data parallel master node."
|
||
)
|
||
parser.add_argument(
|
||
"--vllm-start-port",
|
||
type=int,
|
||
default=9000,
|
||
help="Starting port for the engine."
|
||
)
|
||
return parser.parse_args()
|
||
|
||
args = parse_args()
|
||
dp_size = args.dp_size
|
||
tp_size = args.tp_size
|
||
dp_size_local = args.dp_size_local
|
||
if dp_size_local == -1:
|
||
dp_size_local = dp_size
|
||
dp_rank_start = args.dp_rank_start
|
||
dp_address = args.dp_address
|
||
dp_rpc_port = args.dp_rpc_port
|
||
vllm_start_port = args.vllm_start_port
|
||
|
||
def run_command(visible_devices, dp_rank, vllm_engine_port):
|
||
command = [
|
||
"bash",
|
||
"./run_dp_template.sh",
|
||
visible_devices,
|
||
str(vllm_engine_port),
|
||
str(dp_size),
|
||
str(dp_rank),
|
||
dp_address,
|
||
dp_rpc_port,
|
||
str(tp_size),
|
||
]
|
||
subprocess.run(command, check=True)
|
||
|
||
if __name__ == "__main__":
|
||
template_path = "./run_dp_template.sh"
|
||
if not os.path.exists(template_path):
|
||
print(f"Template file {template_path} does not exist.")
|
||
sys.exit(1)
|
||
|
||
processes = []
|
||
num_cards = dp_size_local * tp_size
|
||
for i in range(dp_size_local):
|
||
dp_rank = dp_rank_start + i
|
||
vllm_engine_port = vllm_start_port + i
|
||
visible_devices = ",".join(str(x) for x in range(i * tp_size, (i + 1) * tp_size))
|
||
process = multiprocessing.Process(target=run_command,
|
||
args=(visible_devices, dp_rank,
|
||
vllm_engine_port))
|
||
processes.append(process)
|
||
process.start()
|
||
|
||
for process in processes:
|
||
process.join()
|
||
```
|
||
|
||
Parameter descriptions:
|
||
|
||
|Parameter|Type|Required|Default|Description|
|
||
|---------|----|--------|-------|-----------|
|
||
|`--dp-size`|int|Yes|-|Data parallel size (total number of DP ranks across all nodes).|
|
||
|`--tp-size`|int|No|1|Tensor parallel size within each DP rank.|
|
||
|`--dp-size-local`|int|No|(same as `--dp-size`)|Number of DP ranks on the current node. If not set, defaults to `--dp-size`.|
|
||
|`--dp-rank-start`|int|No|0|Starting rank offset for data parallel ranks on this node.|
|
||
|`--dp-address`|str|Yes|-|IP address of the data parallel master node.|
|
||
|`--dp-rpc-port`|str|No|12345|RPC port for data parallel master communication.|
|
||
|`--vllm-start-port`|int|No|9000|Starting port for each vLLM engine instance on this node.|
|
||
|
||
2. Prepare the script `run_dp_template.sh` on each node.
|
||
|
||
1. Prefill node 0
|
||
|
||
```shell
|
||
nic_name="xxxx" # change to your own nic name
|
||
local_ip=xx.xx.xx.1 # change to your own ip
|
||
|
||
export HCCL_OP_EXPANSION_MODE="AIV"
|
||
export HCCL_IF_IP=$local_ip
|
||
export GLOO_SOCKET_IFNAME=$nic_name
|
||
export TP_SOCKET_IFNAME=$nic_name
|
||
export HCCL_SOCKET_IFNAME=$nic_name
|
||
export VLLM_RPC_TIMEOUT=3600000
|
||
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
|
||
export HCCL_EXEC_TIMEOUT=204
|
||
export HCCL_CONNECT_TIMEOUT=120
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export HCCL_BUFFSIZE=1024
|
||
export TASK_QUEUE_ENABLE=1
|
||
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
|
||
export ASCEND_RT_VISIBLE_DEVICES=$1
|
||
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
|
||
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
|
||
|
||
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
|
||
--host 0.0.0.0 \
|
||
--port $2 \
|
||
--data-parallel-size $3 \
|
||
--data-parallel-rank $4 \
|
||
--data-parallel-address $5 \
|
||
--data-parallel-rpc-port $6 \
|
||
--tensor-parallel-size $7 \
|
||
--enable-expert-parallel \
|
||
--seed 1024 \
|
||
--served-model-name auto \
|
||
--max-model-len 131072 \
|
||
--max-num-batched-tokens 4096 \
|
||
--max-num-seqs 16 \
|
||
--no-disable-hybrid-kv-cache-manager \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--safetensors-load-strategy 'prefetch' \
|
||
--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
|
||
--trust-remote-code \
|
||
--gpu-memory-utilization 0.92 \
|
||
--quantization ascend \
|
||
--block-size 128 \
|
||
--enforce-eager \
|
||
--speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
|
||
--additional-config '{"enable_cpu_binding": true, "enable_dsa_cp": true}' \
|
||
--kv-transfer-config \
|
||
'{"kv_connector": "MooncakeHybridConnector",
|
||
"kv_role": "kv_producer",
|
||
"kv_port": "30200",
|
||
"engine_id": "1",
|
||
"kv_connector_extra_config": {
|
||
"prefill": {
|
||
"dp_size": 2,
|
||
"tp_size": 16
|
||
},
|
||
"decode": {
|
||
"dp_size": 16,
|
||
"tp_size": 2
|
||
}
|
||
}
|
||
}'
|
||
```
|
||
|
||
2. Prefill node 1
|
||
|
||
```shell
|
||
nic_name="xxxx" # change to your own nic name
|
||
local_ip=xx.xx.xx.2 # change to your own ip
|
||
|
||
export HCCL_OP_EXPANSION_MODE="AIV"
|
||
export HCCL_IF_IP=$local_ip
|
||
export GLOO_SOCKET_IFNAME=$nic_name
|
||
export TP_SOCKET_IFNAME=$nic_name
|
||
export HCCL_SOCKET_IFNAME=$nic_name
|
||
export VLLM_RPC_TIMEOUT=3600000
|
||
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
|
||
export HCCL_EXEC_TIMEOUT=204
|
||
export HCCL_CONNECT_TIMEOUT=120
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export HCCL_BUFFSIZE=1024
|
||
export TASK_QUEUE_ENABLE=1
|
||
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
|
||
export ASCEND_RT_VISIBLE_DEVICES=$1
|
||
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
|
||
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
|
||
|
||
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
|
||
--host 0.0.0.0 \
|
||
--port $2 \
|
||
--data-parallel-size $3 \
|
||
--data-parallel-rank $4 \
|
||
--data-parallel-address $5 \
|
||
--data-parallel-rpc-port $6 \
|
||
--tensor-parallel-size $7 \
|
||
--enable-expert-parallel \
|
||
--seed 1024 \
|
||
--served-model-name auto \
|
||
--max-model-len 131072 \
|
||
--max-num-batched-tokens 4096 \
|
||
--max-num-seqs 16 \
|
||
--no-disable-hybrid-kv-cache-manager \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--safetensors-load-strategy 'prefetch' \
|
||
--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
|
||
--trust-remote-code \
|
||
--gpu-memory-utilization 0.92 \
|
||
--quantization ascend \
|
||
--block-size 128 \
|
||
--enforce-eager \
|
||
--speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
|
||
--additional-config '{"enable_cpu_binding": true, "enable_dsa_cp": true}' \
|
||
--kv-transfer-config \
|
||
'{"kv_connector": "MooncakeHybridConnector",
|
||
"kv_role": "kv_producer",
|
||
"kv_port": "30200",
|
||
"engine_id": "1",
|
||
"kv_connector_extra_config": {
|
||
"prefill": {
|
||
"dp_size": 2,
|
||
"tp_size": 16
|
||
},
|
||
"decode": {
|
||
"dp_size": 16,
|
||
"tp_size": 2
|
||
}
|
||
}
|
||
}'
|
||
```
|
||
|
||
3. Decode node (Same as another D node)
|
||
|
||
```shell
|
||
nic_name="xxxx" # change to your own nic name
|
||
local_ip=xx.xx.xx.3/4 # change to your own ip
|
||
|
||
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
|
||
export HCCL_OP_EXPANSION_MODE="AIV"
|
||
export TASK_QUEUE_ENABLE=1
|
||
export VLLM_RPC_TIMEOUT=3600000
|
||
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
|
||
export HCCL_EXEC_TIMEOUT=2000
|
||
export HCCL_CONNECT_TIMEOUT=1200
|
||
export HCCL_IF_IP=$local_ip
|
||
export GLOO_SOCKET_IFNAME=$nic_name
|
||
export TP_SOCKET_IFNAME=$nic_name
|
||
export HCCL_SOCKET_IFNAME=$nic_name
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export HCCL_BUFFSIZE=1024
|
||
export ASCEND_RT_VISIBLE_DEVICES=$1
|
||
|
||
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
|
||
--host 0.0.0.0 \
|
||
--port $2 \
|
||
--data-parallel-size $3 \
|
||
--data-parallel-rank $4 \
|
||
--data-parallel-address $5 \
|
||
--data-parallel-rpc-port $6 \
|
||
--tensor-parallel-size $7 \
|
||
--enable-expert-parallel \
|
||
--seed 1024 \
|
||
--served-model-name auto \
|
||
--max-model-len 131072 \
|
||
--max-num-batched-tokens 120 \
|
||
--max-num-seqs 60 \
|
||
--async-scheduling \
|
||
--block-size 128 \
|
||
--no-enable-prefix-caching \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--no-disable-hybrid-kv-cache-manager \
|
||
--safetensors-load-strategy 'prefetch' \
|
||
--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
|
||
--trust-remote-code \
|
||
--gpu-memory-utilization 0.9 \
|
||
--quantization ascend \
|
||
--speculative-config '{"num_speculative_tokens": 1, "method":"mtp", "enforce_eager": true}' \
|
||
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
|
||
--kv-transfer-config \
|
||
'{"kv_connector": "MooncakeHybridConnector",
|
||
"kv_role": "kv_consumer",
|
||
"kv_port": "30800",
|
||
"engine_id": "8",
|
||
"kv_connector_extra_config": {
|
||
"prefill": {
|
||
"dp_size": 2,
|
||
"tp_size": 16
|
||
},
|
||
"decode": {
|
||
"dp_size": 16,
|
||
"tp_size": 2
|
||
}
|
||
}
|
||
}' \
|
||
--additional-config '{
|
||
"ascend_compilation_config":{
|
||
"enable_npugraph_ex":true,
|
||
"enable_static_kernel":false
|
||
},
|
||
"enable_cpu_binding":true,
|
||
"recompute_scheduler_enable":true
|
||
}'
|
||
```
|
||
|
||
3. Start the server with the following command on each node.
|
||
|
||
1. Prefill node 0
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 2 --tp-size 16 --dp-size-local 1 --dp-rank-start 0 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
2. Prefill node 1
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 2 --tp-size 16 --dp-size-local 1 --dp-rank-start 1 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
3. Decode node 0
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 16 --tp-size 2 --dp-size-local 8 --dp-rank-start 0 --dp-address xx.xx.xx.3 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
4. Decode node 1
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 16 --tp-size 2 --dp-size-local 8 --dp-rank-start 8 --dp-address xx.xx.xx.3 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
4. Deploy the P-D disaggregation proxy.
|
||
|
||
Refer to [Prefill-Decode Disaggregation (Deepseek)](../features/pd_disaggregation_mooncake_multi_node.md) to deploy the P-D disaggregation proxy.
|
||
|
||
#### 5.2.2 A2 Series PD Separation Deployment
|
||
|
||
This section shows the deployment guide of DeepSeek-V4-Pro on Atlas 800 A2 (64GB × 8) multi-node environment with 1P1D for better performance.
|
||
|
||
Before you start, please:
|
||
|
||
1. Prepare the script `launch_online_dp.py` on each node.
|
||
|
||
```python
|
||
import argparse
|
||
import multiprocessing
|
||
import os
|
||
import subprocess
|
||
import sys
|
||
|
||
def parse_args():
|
||
parser = argparse.ArgumentParser()
|
||
parser.add_argument(
|
||
"--dp-size",
|
||
type=int,
|
||
required=True,
|
||
help="Data parallel size."
|
||
)
|
||
parser.add_argument(
|
||
"--tp-size",
|
||
type=int,
|
||
default=1,
|
||
help="Tensor parallel size."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-size-local",
|
||
type=int,
|
||
default=-1,
|
||
help="Local data parallel size."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-rank-start",
|
||
type=int,
|
||
default=0,
|
||
help="Starting rank for data parallel."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-address",
|
||
type=str,
|
||
required=True,
|
||
help="IP address for data parallel master node."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-rpc-port",
|
||
type=str,
|
||
default=12345,
|
||
help="Port for data parallel master node."
|
||
)
|
||
parser.add_argument(
|
||
"--vllm-start-port",
|
||
type=int,
|
||
default=9000,
|
||
help="Starting port for the engine."
|
||
)
|
||
return parser.parse_args()
|
||
|
||
args = parse_args()
|
||
dp_size = args.dp_size
|
||
tp_size = args.tp_size
|
||
dp_size_local = args.dp_size_local
|
||
if dp_size_local == -1:
|
||
dp_size_local = dp_size
|
||
dp_rank_start = args.dp_rank_start
|
||
dp_address = args.dp_address
|
||
dp_rpc_port = args.dp_rpc_port
|
||
vllm_start_port = args.vllm_start_port
|
||
|
||
def run_command(visible_devices, dp_rank, vllm_engine_port):
|
||
command = [
|
||
"bash",
|
||
"./run_dp_template.sh",
|
||
visible_devices,
|
||
str(vllm_engine_port),
|
||
str(dp_size),
|
||
str(dp_rank),
|
||
dp_address,
|
||
dp_rpc_port,
|
||
str(tp_size),
|
||
]
|
||
subprocess.run(command, check=True)
|
||
|
||
if __name__ == "__main__":
|
||
template_path = "./run_dp_template.sh"
|
||
if not os.path.exists(template_path):
|
||
print(f"Template file {template_path} does not exist.")
|
||
sys.exit(1)
|
||
|
||
processes = []
|
||
num_cards = dp_size_local * tp_size
|
||
for i in range(dp_size_local):
|
||
dp_rank = dp_rank_start + i
|
||
vllm_engine_port = vllm_start_port + i
|
||
visible_devices = ",".join(str(x) for x in range(i * tp_size, (i + 1) * tp_size))
|
||
process = multiprocessing.Process(target=run_command,
|
||
args=(visible_devices, dp_rank,
|
||
vllm_engine_port))
|
||
processes.append(process)
|
||
process.start()
|
||
|
||
for process in processes:
|
||
process.join()
|
||
```
|
||
|
||
2. Prepare the script `run_dp_template.sh` on each node.
|
||
|
||
1. Prefill node (4 P nodes share the same script)
|
||
|
||
```shell
|
||
nic_name="xxxx" # change to your own nic name
|
||
local_ip=xx.xx.xx.1/2/3/4 # change to your own ip
|
||
|
||
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
|
||
export HCCL_OP_EXPANSION_MODE="AIV"
|
||
export TASK_QUEUE_ENABLE=1
|
||
|
||
export VLLM_RPC_TIMEOUT=3600000
|
||
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
|
||
export HCCL_EXEC_TIMEOUT=204
|
||
export HCCL_CONNECT_TIMEOUT=1200
|
||
|
||
export HCCL_IF_IP=$local_ip
|
||
export GLOO_SOCKET_IFNAME=$nic_name
|
||
export TP_SOCKET_IFNAME=$nic_name
|
||
export HCCL_SOCKET_IFNAME=$nic_name
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export HCCL_BUFFSIZE=1024
|
||
|
||
sysctl -w vm.swappiness=0
|
||
sysctl -w kernel.numa_balancing=0
|
||
sysctl kernel.sched_migration_cost_ns=50000
|
||
|
||
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
|
||
export ASCEND_RT_VISIBLE_DEVICES=$1
|
||
|
||
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
|
||
--host 0.0.0.0 \
|
||
--port $2 \
|
||
--data-parallel-size $3 \
|
||
--data-parallel-rank $4 \
|
||
--data-parallel-address $5 \
|
||
--data-parallel-rpc-port $6 \
|
||
--tensor-parallel-size $7 \
|
||
--enable-expert-parallel \
|
||
--seed 1024 \
|
||
--served-model-name dsv4 \
|
||
--max-model-len 133072 \
|
||
--max-num-batched-tokens 4096 \
|
||
--max-num-seqs 16 \
|
||
--no-disable-hybrid-kv-cache-manager \
|
||
--trust-remote-code \
|
||
--gpu-memory-utilization 0.9 \
|
||
--quantization ascend \
|
||
--safetensors-load-strategy 'prefetch' \
|
||
--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--enforce-eager \
|
||
--no-enable-prefix-caching \
|
||
--speculative-config '{"num_speculative_tokens": 1, "method":"mtp", "enforce_eager": true}' \
|
||
--additional-config '{"enable_cpu_binding": true, "enable_shared_expert_dp": true, "enable_dsa_cp": true}' \
|
||
--kv-transfer-config \
|
||
'{"kv_connector": "MooncakeHybridConnector",
|
||
"kv_role": "kv_producer",
|
||
"kv_port": "30000",
|
||
"engine_id": "0",
|
||
"kv_connector_extra_config": {
|
||
"prefill": {
|
||
"dp_size": 4,
|
||
"tp_size": 8
|
||
},
|
||
"decode": {
|
||
"dp_size": 8,
|
||
"tp_size": 4
|
||
}
|
||
}
|
||
}'
|
||
```
|
||
|
||
2. Decode node (4 D nodes share the same script)
|
||
|
||
```shell
|
||
nic_name="xxxx" # change to your own nic name
|
||
local_ip=xx.xx.xx.5/6/7/8 # change to your own ip
|
||
|
||
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
|
||
export HCCL_OP_EXPANSION_MODE="AIV"
|
||
export TASK_QUEUE_ENABLE=1
|
||
|
||
export VLLM_RPC_TIMEOUT=3600000
|
||
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
|
||
export HCCL_EXEC_TIMEOUT=204
|
||
export HCCL_CONNECT_TIMEOUT=1200
|
||
|
||
export HCCL_IF_IP=$local_ip
|
||
export GLOO_SOCKET_IFNAME=$nic_name
|
||
export TP_SOCKET_IFNAME=$nic_name
|
||
export HCCL_SOCKET_IFNAME=$nic_name
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export HCCL_BUFFSIZE=1024
|
||
|
||
sysctl -w vm.swappiness=0
|
||
sysctl -w kernel.numa_balancing=0
|
||
sysctl kernel.sched_migration_cost_ns=50000
|
||
|
||
export ASCEND_RT_VISIBLE_DEVICES=$1
|
||
|
||
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
|
||
--host 0.0.0.0 \
|
||
--port $2 \
|
||
--data-parallel-size $3 \
|
||
--data-parallel-rank $4 \
|
||
--data-parallel-address $5 \
|
||
--data-parallel-rpc-port $6 \
|
||
--tensor-parallel-size $7 \
|
||
--enable-expert-parallel \
|
||
--seed 1024 \
|
||
--served-model-name dsv4 \
|
||
--max-model-len 133072 \
|
||
--max-num-batched-tokens 120 \
|
||
--max-num-seqs 60 \
|
||
--async-scheduling \
|
||
--block-size 128 \
|
||
--no-disable-hybrid-kv-cache-manager \
|
||
--trust-remote-code \
|
||
--gpu-memory-utilization 0.9 \
|
||
--quantization ascend \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--safetensors-load-strategy 'prefetch' \
|
||
--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
|
||
--no-enable-prefix-caching \
|
||
--speculative-config '{"num_speculative_tokens": 1, "method":"mtp", "enforce_eager": true}' \
|
||
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
|
||
--kv-transfer-config \
|
||
'{"kv_connector": "MooncakeHybridConnector",
|
||
"kv_role": "kv_consumer",
|
||
"kv_port": "30100",
|
||
"engine_id": "1",
|
||
"kv_connector_extra_config": {
|
||
"prefill": {
|
||
"dp_size": 4,
|
||
"tp_size": 8
|
||
},
|
||
"decode": {
|
||
"dp_size": 8,
|
||
"tp_size": 4
|
||
}
|
||
}
|
||
}' \
|
||
--additional-config '{"ascend_compilation_config":{"enable_npugraph_ex":true,"enable_static_kernel":false}, "enable_cpu_binding":true, "recompute_scheduler_enable":true}'
|
||
```
|
||
|
||
3. Start the server with the following command on each node.
|
||
|
||
1. Prefill node 0
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 0 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
2. Prefill node 1
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 1 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
3. Prefill node 2
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 2 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
4. Prefill node 3
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 3 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
5. Decode node 0
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 0 --dp-address xx.xx.xx.2 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
6. Decode node 1
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 2 --dp-address xx.xx.xx.2 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
7. Decode node 2
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 4 --dp-address xx.xx.xx.2 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
8. Decode node 3
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 6 --dp-address xx.xx.xx.2 --dp-rpc-port 12321 --vllm-start-port 7100
|
||
```
|
||
|
||
4. Deploy the P-D disaggregation proxy.
|
||
|
||
Refer to [Prefill-Decode Disaggregation (Deepseek)](../features/pd_disaggregation_mooncake_multi_node.md) to deploy the P-D disaggregation proxy.
|
||
|
||
#### 5.2.3 Ascend 950DT Series PD Separation Deployment
|
||
|
||
This section shows a logical 1P1D deployment on Ascend 950DT servers. The Prefill group uses 4 servers and the Decode group uses 4 servers. Each group contains 32 DP ranks with `DP32/TP1`; every server starts 8 local DP ranks. The `prefill` and `decode` settings in `--kv-transfer-config` must therefore both use `dp_size: 32` and `tp_size: 1`.
|
||
|
||
Before you start, mount `/etc/hixlep/` into the container and complete the following steps:
|
||
|
||
1. Prepare the script `launch_online_dp.py` on each node.
|
||
|
||
```python
|
||
import argparse
|
||
import multiprocessing
|
||
import os
|
||
import subprocess
|
||
import sys
|
||
|
||
def parse_args():
|
||
parser = argparse.ArgumentParser()
|
||
parser.add_argument(
|
||
"--dp-size",
|
||
type=int,
|
||
required=True,
|
||
help="Data parallel size."
|
||
)
|
||
parser.add_argument(
|
||
"--tp-size",
|
||
type=int,
|
||
default=1,
|
||
help="Tensor parallel size."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-size-local",
|
||
type=int,
|
||
default=-1,
|
||
help="Local data parallel size."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-rank-start",
|
||
type=int,
|
||
default=0,
|
||
help="Starting rank for data parallel."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-address",
|
||
type=str,
|
||
required=True,
|
||
help="IP address for data parallel master node."
|
||
)
|
||
parser.add_argument(
|
||
"--dp-rpc-port",
|
||
type=str,
|
||
default=12345,
|
||
help="Port for data parallel master node."
|
||
)
|
||
parser.add_argument(
|
||
"--vllm-start-port",
|
||
type=int,
|
||
default=9000,
|
||
help="Starting port for the engine."
|
||
)
|
||
return parser.parse_args()
|
||
|
||
args = parse_args()
|
||
dp_size = args.dp_size
|
||
tp_size = args.tp_size
|
||
dp_size_local = args.dp_size_local
|
||
if dp_size_local == -1:
|
||
dp_size_local = dp_size
|
||
dp_rank_start = args.dp_rank_start
|
||
dp_address = args.dp_address
|
||
dp_rpc_port = args.dp_rpc_port
|
||
vllm_start_port = args.vllm_start_port
|
||
|
||
def run_command(visible_devices, dp_rank, vllm_engine_port):
|
||
command = [
|
||
"bash",
|
||
"./run_dp_template.sh",
|
||
visible_devices,
|
||
str(vllm_engine_port),
|
||
str(dp_size),
|
||
str(dp_rank),
|
||
dp_address,
|
||
dp_rpc_port,
|
||
str(tp_size),
|
||
]
|
||
subprocess.run(command, check=True)
|
||
|
||
if __name__ == "__main__":
|
||
template_path = "./run_dp_template.sh"
|
||
if not os.path.exists(template_path):
|
||
print(f"Template file {template_path} does not exist.")
|
||
sys.exit(1)
|
||
|
||
processes = []
|
||
num_cards = dp_size_local * tp_size
|
||
for i in range(dp_size_local):
|
||
dp_rank = dp_rank_start + i
|
||
vllm_engine_port = vllm_start_port + i
|
||
visible_devices = ",".join(str(x) for x in range(i * tp_size, (i + 1) * tp_size))
|
||
process = multiprocessing.Process(target=run_command,
|
||
args=(visible_devices, dp_rank,
|
||
vllm_engine_port))
|
||
processes.append(process)
|
||
process.start()
|
||
|
||
for process in processes:
|
||
process.join()
|
||
```
|
||
|
||
2. Prepare `run_dp_template.sh` on each Prefill node.
|
||
|
||
```bash
|
||
#!/usr/bin/env bash
|
||
source /root/.bashrc
|
||
|
||
nic_name="xxx"
|
||
local_ip="xx.xx.xx.x"
|
||
|
||
export HCCL_IF_IP=$local_ip
|
||
export GLOO_SOCKET_IFNAME=$nic_name
|
||
export TP_SOCKET_IFNAME=$nic_name
|
||
export HCCL_SOCKET_IFNAME=$nic_name
|
||
export HCCL_ALGO=level0:fullmesh
|
||
export VLLM_RPC_TIMEOUT=3600000
|
||
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
|
||
export HCCL_EXEC_TIMEOUT=2040
|
||
export HCCL_CONNECT_TIMEOUT=1200
|
||
export HCCL_BUFFSIZE=512
|
||
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export ASCEND_RT_VISIBLE_DEVICES=$1
|
||
|
||
vllm serve /root/.cache/DeepSeek-V4-Pro \
|
||
--host $local_ip \
|
||
--port $2 \
|
||
--data-parallel-size $3 \
|
||
--data-parallel-rank $4 \
|
||
--data-parallel-address $5 \
|
||
--data-parallel-rpc-port $6 \
|
||
--tensor-parallel-size $7 \
|
||
--max-model-len 1048576 \
|
||
--max-num-batched-tokens 4096 \
|
||
--served-model-name dsv4 \
|
||
--gpu-memory-utilization 0.85 \
|
||
--enable-expert-parallel \
|
||
--async-scheduling \
|
||
--max-num-seqs 8 \
|
||
--block-size 32 \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--trust-remote-code \
|
||
--enforce-eager \
|
||
--no-disable-hybrid-kv-cache-manager \
|
||
--speculative-config '{"num_speculative_tokens": 1, "method": "mtp"}' \
|
||
--kv-transfer-config \
|
||
'{"kv_connector": "MooncakeHybridConnector",
|
||
"kv_role": "kv_producer",
|
||
"kv_port": "36010",
|
||
"engine_id": "1",
|
||
"kv_connector_extra_config": {
|
||
"prefill": {
|
||
"dp_size": 32,
|
||
"tp_size": 1
|
||
},
|
||
"decode": {
|
||
"dp_size": 32,
|
||
"tp_size": 1
|
||
},
|
||
"ascend_local_comm_res_path": "/etc/hixlep"
|
||
}
|
||
}' \
|
||
--additional-config '{"enable_cpu_binding": true}'
|
||
```
|
||
|
||
3. Prepare `run_dp_template.sh` on each Decode node.
|
||
|
||
```bash
|
||
#!/usr/bin/env bash
|
||
source /root/.bashrc
|
||
|
||
nic_name="xxx"
|
||
local_ip="xx.xx.xx.x"
|
||
|
||
export HCCL_IF_IP=$local_ip
|
||
export GLOO_SOCKET_IFNAME=$nic_name
|
||
export TP_SOCKET_IFNAME=$nic_name
|
||
export HCCL_SOCKET_IFNAME=$nic_name
|
||
export HCCL_ALGO=level0:fullmesh
|
||
export VLLM_RPC_TIMEOUT=3600000
|
||
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
|
||
export HCCL_EXEC_TIMEOUT=2040
|
||
export HCCL_CONNECT_TIMEOUT=1200
|
||
export HCCL_BUFFSIZE=1024
|
||
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
|
||
export OMP_PROC_BIND=false
|
||
export OMP_NUM_THREADS=10
|
||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||
export ASCEND_RT_VISIBLE_DEVICES=$1
|
||
|
||
vllm serve /root/.cache/DeepSeek-V4-Pro \
|
||
--host $local_ip \
|
||
--port $2 \
|
||
--data-parallel-size $3 \
|
||
--data-parallel-rank $4 \
|
||
--data-parallel-address $5 \
|
||
--data-parallel-rpc-port $6 \
|
||
--tensor-parallel-size $7 \
|
||
--max-model-len 1048576 \
|
||
--max-num-batched-tokens 256 \
|
||
--served-model-name dsv4 \
|
||
--gpu-memory-utilization 0.92 \
|
||
--enable-expert-parallel \
|
||
--async-scheduling \
|
||
--max-num-seqs 32 \
|
||
--block-size 32 \
|
||
--no-enable-prefix-caching \
|
||
--tokenizer-mode deepseek_v4 \
|
||
--tool-call-parser deepseek_v4 \
|
||
--enable-auto-tool-choice \
|
||
--reasoning-parser deepseek_v4 \
|
||
--trust-remote-code \
|
||
--no-disable-hybrid-kv-cache-manager \
|
||
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
|
||
--speculative-config '{"num_speculative_tokens": 3, "method": "mtp", "enforce_eager": false}' \
|
||
--kv-transfer-config \
|
||
'{"kv_connector": "MooncakeHybridConnector",
|
||
"kv_role": "kv_consumer",
|
||
"kv_port": "36010",
|
||
"engine_id": "1",
|
||
"kv_connector_extra_config": {
|
||
"prefill": {
|
||
"dp_size": 32,
|
||
"tp_size": 1
|
||
},
|
||
"decode": {
|
||
"dp_size": 32,
|
||
"tp_size": 1
|
||
},
|
||
"ascend_local_comm_res_path": "/etc/hixlep"
|
||
}
|
||
}' \
|
||
--additional-config '{"enable_cpu_binding": true, "recompute_scheduler_enable": true, "multistream_overlap_shared_expert": true}'
|
||
```
|
||
|
||
4. Start the server with the following command on each node. Set `local_ip` in `run_dp_template.sh` to the current node IP before starting the service.
|
||
|
||
1. Prefill node 0
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 32 --tp-size 1 --dp-size-local 8 --dp-rank-start 0 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 8000
|
||
```
|
||
|
||
2. Prefill node 1
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 32 --tp-size 1 --dp-size-local 8 --dp-rank-start 8 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 8000
|
||
```
|
||
|
||
3. Prefill node 2
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 32 --tp-size 1 --dp-size-local 8 --dp-rank-start 16 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 8000
|
||
```
|
||
|
||
4. Prefill node 3
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 32 --tp-size 1 --dp-size-local 8 --dp-rank-start 24 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 8000
|
||
```
|
||
|
||
5. Decode node 0
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 32 --tp-size 1 --dp-size-local 8 --dp-rank-start 0 --dp-address xx.xx.xx.5 --dp-rpc-port 12325 --vllm-start-port 8000
|
||
```
|
||
|
||
6. Decode node 1
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 32 --tp-size 1 --dp-size-local 8 --dp-rank-start 8 --dp-address xx.xx.xx.5 --dp-rpc-port 12325 --vllm-start-port 8000
|
||
```
|
||
|
||
7. Decode node 2
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 32 --tp-size 1 --dp-size-local 8 --dp-rank-start 16 --dp-address xx.xx.xx.5 --dp-rpc-port 12325 --vllm-start-port 8000
|
||
```
|
||
|
||
8. Decode node 3
|
||
|
||
```shell
|
||
# change ip to your own
|
||
python launch_online_dp.py --dp-size 32 --tp-size 1 --dp-size-local 8 --dp-rank-start 24 --dp-address xx.xx.xx.5 --dp-rpc-port 12325 --vllm-start-port 8000
|
||
```
|
||
|
||
5. Deploy the P-D disaggregation proxy.
|
||
|
||
Refer to [Prefill-Decode Disaggregation (Deepseek)](../features/pd_disaggregation_mooncake_multi_node.md) and register all Prefill and Decode engine endpoints started above. Each node uses ports 8000 through 8007.
|
||
|
||
Key Parameter Descriptions:
|
||
|
||
- `--no-disable-hybrid-kv-cache-manager` keeps the hybrid KV cache manager enabled. DeepSeek-V4 KV Pool deployments require this flag; otherwise, the service may OOM during startup.
|
||
- `--enforce-eager` forces eager execution on prefill nodes instead of graph compilation.
|
||
- `enable_dsa_cp: true` enables DSA context parallelism on prefill nodes. DSA-CP and FlashComm1 must be enabled separately when both are required.
|
||
- `kv_connector_extra_config.prefill.dp_size/tp_size` and `decode.dp_size/tp_size` must match the actual global DP and TP layout on the prefill and decode sides.
|
||
- `VLLM_ASCEND_ENABLE_FLASHCOMM1=1`: enables the communication optimization function on the prefill nodes.
|
||
- `VLLM_ASCEND_ENABLE_FUSED_MC2=1`: enables the Fused MC2 fusion operator to accelerate communication on prefill nodes (A3 series).
|
||
- `recompute_scheduler_enable: true`: enables the recomputation scheduler. When the KV Cache of the decode node is insufficient, requests will be sent to the prefill node to recompute the KV Cache. In the PD separation scenario, enable this configuration only on decode nodes.
|
||
- `MooncakeHybridConnector`: the KV transfer connector used for PD separation, transferring KV Cache between prefill and decode nodes.
|
||
- `enable_shared_expert_dp: true`: enables data parallelism for shared experts, applicable to MoE models.
|
||
- On Ascend 950DT, `ascend_local_comm_res_path` specifies the local communication resource directory used by the KV connector. The directory must be available at the same path in the container.
|
||
|
||
Deployment Verification:
|
||
|
||
After the PD separation service is fully started, send a request through the proxy port on the prefill master node to verify that Prefill and Decode nodes are working correctly together. Refer to [Prefill-Decode Disaggregation (Deepseek)](../features/pd_disaggregation_mooncake_multi_node.md) for the proxy verification method.
|
||
|
||
Common Issues Tip: If you encounter issues with PD separation deployment, please refer to the [Public FAQ](https://docs.vllm.ai/projects/ascend/en/latest/faqs.html) for troubleshooting.
|
||
|
||
## 6 Functional Verification
|
||
|
||
Once your server is started, you can query the model with input prompts:
|
||
|
||
In <node0_ip>:<port>, use the IP address and port number of the primary node. If the primary and standby nodes are separated, use the IP address and port number of the proxy node.
|
||
|
||
```shell
|
||
curl http://<node0_ip>:<port>/v1/chat/completions \
|
||
-H "Content-Type: application/json" \
|
||
-d '{
|
||
"model": "dsv4",
|
||
"messages": [
|
||
{
|
||
"role": "user",
|
||
"content": "Who are you?"
|
||
}
|
||
],
|
||
"max_tokens": 256,
|
||
"temperature": 0
|
||
}'
|
||
```
|
||
|
||
Expected Result:
|
||
|
||
The service returns HTTP 200 OK with a JSON response containing the `choices` field.
|
||
|
||
## 7 Accuracy Evaluation
|
||
|
||
Here are two accuracy evaluation methods.
|
||
|
||
### Using AISBench
|
||
|
||
1. Refer to [Using AISBench](../../developer_guide/evaluation/using_ais_bench.md) for details.
|
||
|
||
2. After execution, you can get the result.
|
||
|
||
| dataset | version | metric | mode | vllm-api-general-chat | note |
|
||
| ----- | ----- | ----- | ----- | ----- | ----- |
|
||
| GPQA | - | accuracy | gen | 89.90 | 1 Atlas 800 A3 (128GB × 8) |
|
||
| GSM8K | - | accuracy | gen | 96.21 | 1 Atlas 800 A3 (128GB × 8) |
|
||
|
||
## 8 Performance Evaluation
|
||
|
||
### Using AISBench
|
||
|
||
Refer to [Using AISBench for performance evaluation](../../developer_guide/evaluation/using_ais_bench.md#execute-performance-evaluation) for details.
|
||
|
||
### Using vLLM Benchmark
|
||
|
||
Refer to [vllm benchmark](https://docs.vllm.ai/en/latest/benchmarking/) for more details.
|
||
|
||
## 9 Performance Tuning
|
||
|
||
### 9.1 Recommended Configurations
|
||
|
||
> **Note**: The following configurations are validated in specific test environments and are for reference only. The optimal configuration depends on factors such as maximum input/output length, prefix cache hit rate, precision requirements, and deployment machine ratios. It is recommended to refer to Section 9.2 for tuning based on actual conditions.
|
||
|
||
#### Table 1: Scenario Overview
|
||
|
||
> `*Total NPUs` indicates the total number of NPUs used across all nodes.
|
||
|
||
|Scenario|Deployment Mode|*Total NPUs|Weight Version|Key Considerations|
|
||
|--------|---------------|-----------|---------------|-------------------|
|
||
|High Throughput|Single-Node Mixed|32 (A3)|DeepSeek-V4-Pro-w4a8-mtp|Use dp2 tp16 to balance memory capacity and compute efficiency|
|
||
|High Throughput|1P1D deployment|64 (A3)|DeepSeek-V4-Pro-w4a8-mtp|dp16 tp2 or dp2 tp16, depending on memory and concurrency|
|
||
|Long Context (1M)|Single-Node Mixed|32 (A3)|DeepSeek-V4-Pro-w4a8-mtp|Use dp2 tp16 to balance memory capacity and compute efficiency|
|
||
|Long Context (1M)|1P1D deployment|64 (A3)|DeepSeek-V4-Pro-w4a8-mtp|dp2 tp16 on both P and D nodes; balanced latency and throughput|
|
||
|High Throughput|Multi-Node Mixed|16 (Ascend 950DT)|DeepSeek-V4-Pro|Use dp2 tp8 across two Ascend 950DT servers|
|
||
|Long Context (1M)|1P1D deployment|64 (Ascend 950DT)|DeepSeek-V4-Pro|Use dp32 tp1 on both P and D groups|
|
||
|
||
#### Table 2: Detailed Node Configuration
|
||
|
||
|Scenario|Configuration|NPUs|TP|DP|Max Num Seqs|Max Num Batched Tokens|Max Model Len|MTP Speculation Num|
|
||
|--------|-------------|-----|--|--|------------|----------------------|--------------|--------------------|
|
||
|Multi-Node (A3)|Node0 / Node1|8|16|2|32|4096|135000|1|
|
||
|PD Separation (A3)|Prefill Node|8|16|2|16|4096|131072|1|
|
||
|PD Separation (A3)|Decode Node|8|2|16|60|120|131072|1|
|
||
|Multi-Node (Ascend 950DT)|Node0 / Node1|16|8|2|40|8192|200000|3|
|
||
|PD Separation (Ascend 950DT)|Prefill Group|32|1|32|8|4096|1048576|1|
|
||
|PD Separation (Ascend 950DT)|Decode Group|32|1|32|32|256|1048576|3|
|
||
|
||
> For complete startup commands and parameter descriptions, please refer to the deployment examples in [Chapter 5](#5-online-service-deployment).
|
||
|
||
### 9.2 Tuning Guidelines
|
||
|
||
#### 9.2.1 General Tuning Reference
|
||
|
||
Please refer to the [Public Performance Tuning Documentation](../../developer_guide/performance_and_debug/optimization_and_tuning.md) for tuning methods.
|
||
|
||
Please refer to the [Feature Guide](../../user_guide/support_matrix/feature_matrix.md) for detailed feature descriptions.
|
||
|
||
## 10 FAQ
|
||
|
||
For common environment, installation, and general parameter issues, please refer to the [Public FAQ](https://docs.vllm.ai/projects/ascend/en/latest/faqs.html); this chapter only covers model-specific issues.
|