EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

[router] Add Rust CLI flags for queue size, timeout, and rate limit for token bucket rate limiter (#11483)

Browse Source 
Co-authored-by: Simo Lin <linsimo.mark@gmail.com>
This commit is contained in:
Jonah Bernard
2025-10-13 14:08:48 -04:00
committed by GitHub
parent 5e3f7e7fa9
commit f4aa78801e
3 changed files with 35 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
sgl-router/README.md
View File

@@ -426,6 +426,11 @@ curl -X POST http://localhost:8080/add_worker?url=http://worker3:8000&api_key=wo
#### Authentication
- `--api-key`: API key for router authentication (clients must provide this as Bearer token)
#### Concurrency and Rate Limiting
- `--queue-size`: Size of the pending-request queue when concurrency limits are reached (default: 100; set to 0 to disable queuing)
- `--queue-timeout-secs`: Maximum time a request may wait in the queue before timing out (default: 60; must be > 0 when queue is enabled)
- `--rate-limit-tokens-per-second`: Override token bucket refill rate for rate limiting (defaults to `--max-concurrent-requests` when omitted)
## Development
### Build Process