EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

Docs：Update the style of llma 3.1 405B docs (#2789)

Browse Source
This commit is contained in:
Chayenne
2025-01-08 01:07:54 -08:00
committed by GitHub
parent 977f785dad
commit 2e6346fc2e
3 changed files with 12 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docs/references/llama_405B.md
View File

@@ -1,16 +1,19 @@
# Example: Run Llama 3.1 405B
# Run Llama 3.1 405B
## Run 405B (fp8) on a Single Node
```bash
# Run 405B (fp8) on a single node
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tp 8
```
## Run 405B (fp16) on Two Nodes
```bash
# Run 405B (fp16) on two nodes
## on the first node, replace the `172.16.4.52:20000` with your own first node ip address and port
# on the first node, replace 172.16.4.52:20000 with your own node ip address and port
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct --tp 16 --nccl-init-addr 172.16.4.52:20000 --nnodes 2 --node-rank 0
## on the first node, replace the `172.16.4.52:20000` with your own first node ip address and port
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct --tp 16 --nccl-init-addr 172.16.4.52:20000 --nnodes 2 --node-rank 1
```
# on the second node, replace 172.18.45.52:20000 with your own node ip address and port
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct --tp 16 --nccl-init-addr 172.18.45.52:20000 --nnodes 2 --node-rank 1
```