EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

[Doc]Add instruction for profiling with bench_one_batch (#5581)

Browse Source
This commit is contained in:
Baizhou Zhang
2025-04-20 14:05:36 -07:00
committed by GitHub
parent bca832c7c6
commit b54b5a96e4
2 changed files with 7 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
docs/references/benchmark_and_profiling.md
View File

@@ -41,9 +41,14 @@
Please make sure that the `SGLANG_TORCH_PROFILER_DIR` should be set at both server and client side, otherwise the trace file cannot be generated correctly . A secure way will be setting `SGLANG_TORCH_PROFILER_DIR` in the `.*rc` file of shell (e.g. `~/.bashrc` for bash shells). Please make sure that the `SGLANG_TORCH_PROFILER_DIR` should be set at both server and client side, otherwise the trace file cannot be generated correctly . A secure way will be setting `SGLANG_TORCH_PROFILER_DIR` in the `.*rc` file of shell (e.g. `~/.bashrc` for bash shells).
- To profile offline - To profile offline
```bash ```bash
export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log
# profile one batch with bench_one_batch.py
# batch size can be controlled with --batch argument
python3 -m sglang.bench_one_batch --model-path meta-llama/Llama-3.1-8B-Instruct --batch 32 --input-len 1024 --output-len 10 --profile
# profile multiple batches with bench_offline_throughput.py
python -m sglang.bench_offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8 python -m sglang.bench_offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8
``` ```

2
python/sglang/bench_one_batch.py
View File

@@ -396,7 +396,7 @@ def latency_test_run_once(
decode_latencies.append(latency) decode_latencies.append(latency)
if i < 5: if i < 5:
rank_print( rank_print(
f"Decode. latency: {latency:6.5f} s, throughput: {throughput:9.2f} token/s" f"Decode. Batch size: {batch_size}, latency: {latency:6.5f} s, throughput: {throughput:9.2f} token/s"
) )
if profile: if profile: