EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

Docs/CI: Enable Fake Finish for Docs Only PR (#3350)

Browse Source
This commit is contained in:
Chayenne
2025-02-06 19:33:31 -08:00
committed by GitHub
parent cdae77b03d
commit 76ca91dff2
5 changed files with 114 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
docs/references/multi_node.md Normal file
View File

@@ -0,0 +1,25 @@
# Run Multi-Node Inference
## Llama 3.1 405B
**Run 405B (fp16) on Two Nodes**
```bash
# replace 172.16.4.52:20000 with your own node ip address and port of the first node
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct --tp 16 --dist-init-addr 172.16.4.52:20000 --nnodes 2 --node-rank 0
# replace 172.18.45.52:20000 with your own node ip address and port of the second node
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct --tp 16 --dist-init-addr 172.18.45.52:20000 --nnodes 2 --node-rank 1
```
Note that LLama 405B (fp8) can also be launched on a single node.
```bash
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tp 8
```
## DeepSeek V3/R1
Please refer to [DeepSeek documents for reference.](https://docs.sglang.ai/references/deepseek.html#running-examples-on-multi-node).