icer/qwen36_01
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

log seq2 benchmark experiments

Browse Source
This commit is contained in:
the magician
2026-07-14 15:27:52 +08:00
parent 61e169fe18
commit b41e7cd899
1 changed files with 68 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
worklogs/2026-07-13-initial-run.md
View File

@@ -176,3 +176,71 @@
- Current serving parameter `--max-num-seqs 1` prevents useful concurrent batching and likely caps weighted throughput. - Current serving parameter `--max-num-seqs 1` prevents useful concurrent batching and likely caps weighted throughput.
- Decode path is also slow: request-level output TPS only `5.9-8.5 tok/s`. - Decode path is also slow: request-level output TPS only `5.9-8.5 tok/s`.
- Next experiment should restart service with higher `--max-num-seqs`, higher `--max-num-batched-tokens`, and possibly `gpu-memory-utilization 0.95`, then rerun the same cumulative dataset. - Next experiment should restart service with higher `--max-num-seqs`, higher `--max-num-batched-tokens`, and possibly `gpu-memory-utilization 0.95`, then rerun the same cumulative dataset.
## 2026-07-14 Serving Parameter Experiment: seq2 / batched8192
- Restarted service from `/root` with:
- `--gpu-memory-utilization 0.95`
- `--max-num-seqs 2`
- `--max-num-batched-tokens 8192`
- same model, tensor parallel `4`, prefix cache, chunked prefill, and qwen3 parsers.
- Service started successfully:
- Log file: `/root/work/logs/server_exp_seq2_b8192.log`
- Process id observed: `2710`
- GPU blocks: `21100`, CPU blocks: `6553`
### Concurrency 2 result
- Benchmark:
- Dataset: `/root/work/logs/synthetic_cumulative_8_s3_cap40k.jsonl`
- Result file: `/root/work/logs/perf_cumulative8_s3_cap40k_exp_seq2_c2_r4.json`
- Concurrency `2`, requests `4`, max output tokens `64`.
- Results:
- Success rate: `100%`.
- TTFT P90: `58.51s`.
- Output TPS P10 per request: `0.87 tok/s`.
- Aggregate output TPS: `1.53 tok/s`.
- Cache hit rate: `43.22%`.
- Weighted token throughput: `1076.71`.
- Interpretation:
- Raising concurrency to `2` harms per-request decode speed and does not improve TTFT on this small long-context test.
- Cache hit rate also drops below target because overlapping requests disturb the ideal same-session sequential cache path.
### Concurrency 1 result
- Benchmark:
- Dataset: `/root/work/logs/synthetic_cumulative_8_s3_cap40k.jsonl`
- Result file: `/root/work/logs/perf_cumulative8_s3_cap40k_exp_seq2_c1_r4.json`
- Concurrency `1`, requests `4`, max output tokens `64`.
- Results:
- Success rate: `100%`.
- TTFT P90: `3.34s`.
- Output TPS P10 per request: `6.10 tok/s`.
- Aggregate output TPS: `5.50 tok/s`.
- Cache hit rate: `99.97%`.
- Cached tokens: `96304 / 96333`.
- Weighted token throughput: `1252.05`.
- Interpretation:
- With warm/exact prefix cache reuse, TTFT and cache-hit targets can pass.
- Output TPS is still far below the `>=20` target, so the next optimization area is decode throughput.
- Weighted throughput remains far below `8000`; this requires either much higher input/cache throughput under realistic scheduling or substantially faster decode.
## 2026-07-14 Failed Decode Scheduler Experiment: multi-step
- Investigated available CoreX/vLLM parameters from `/root/work/logs/api_server_help.txt`.
- Tried adding:
- `--num-scheduler-steps 4`
- existing `--enable-chunked-prefill`
- Result:
- Service failed during startup.
- Log file: `/root/work/logs/server_exp_sched4_seq2_b8192.log`
- Error: `Multi-Step + Chunked-Prefill not supported for attention backend: xformers`.
- The runtime suggests `flash-attn`, but current service logs show `Using XFormers backend`.
- Recovery:
- Restarted the previously working `seq2/b8192` service without `--num-scheduler-steps`.
- Recovery log file: `/root/work/logs/server_exp_seq2_b8192_recover.log`
- `/health` returned `200`.
- Server log recorded a short `/v1/chat/completions` request returning `200 OK`.
- Interpretation:
- Multi-step scheduling is not available with the current xFormers + chunked-prefill path.
- Next decode optimization should focus on either enabling a compatible attention backend, checking whether chunked prefill can be disabled for a decode-focused variant, or optimizing parser/reasoning/tool overhead before deeper code changes.