EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

Docs: improve link to docs (#3860)

Browse Source 
Co-authored-by: Chayenne <zhaochen20@outlook.com>
This commit is contained in:
simveit
2025-02-26 19:29:25 +01:00
committed by GitHub
parent c9fc4a9d26
commit 44a2c4bd56
3 changed files with 18 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
docs/router/router.md
View File

@@ -27,11 +27,13 @@ The router supports two working modes:
This will be a drop-in replacement for the existing `--dp-size` argument of SGLang Runtime. Under the hood, it uses multi-processes to launch multiple workers, wait for them to be ready, then connect the router to all workers.
```bash
$ python -m sglang_router.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --dp-size 1
python -m sglang_router.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --dp-size 4
```
After the server is ready, you can directly send requests to the router as the same way as sending requests to each single worker.
Please adjust the batchsize accordingly to archieve maximum throughput.
```python
import requests
@@ -47,7 +49,7 @@ print(response.json())
This is useful for multi-node DP. First, launch workers on multiple nodes, then launch a router on the main node, and connect the router to all workers.
```bash
$ python -m sglang_router.launch_router --worker-urls http://worker_url_1 http://worker_url_2
python -m sglang_router.launch_router --worker-urls http://worker_url_1 http://worker_url_2
```
## Dynamic Scaling APIs
@@ -59,15 +61,17 @@ We offer `/add_worker` and `/remove_worker` APIs to dynamically add or remove wo
Usage:
```bash
$ curl -X POST http://localhost:30000/add_worker?url=http://worker_url_1
curl -X POST http://localhost:30000/add_worker?url=http://worker_url_1
```
Example:
```bash
$ python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --port 30001
$ curl -X POST http://localhost:30000/add_worker?url=http://127.0.0.1:30001
Successfully added worker: http://127.0.0.1:30001
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --port 30001
curl -X POST http://localhost:30000/add_worker?url=http://127.0.0.1:30001
# Successfully added worker: http://127.0.0.1:30001
```
- `/remove_worker`
@@ -75,14 +79,15 @@ Successfully added worker: http://127.0.0.1:30001
Usage:
```bash
$ curl -X POST http://localhost:30000/remove_worker?url=http://worker_url_1
curl -X POST http://localhost:30000/remove_worker?url=http://worker_url_1
```
Example:
```bash
$ curl -X POST http://localhost:30000/remove_worker?url=http://127.0.0.1:30001
Successfully removed worker: http://127.0.0.1:30001
curl -X POST http://localhost:30000/remove_worker?url=http://127.0.0.1:30001
# Successfully removed worker: http://127.0.0.1:30001
```
Note: