icer/qwen36_01
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

add decode throughput analysis

Browse Source
This commit is contained in:
the magician
2026-07-14 17:11:28 +08:00
parent b41e7cd899
commit 465a9c8995
11 changed files with 2960 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
worklogs/2026-07-13-initial-run.md
View File

@@ -244,3 +244,27 @@
- Interpretation:
- Multi-step scheduling is not available with the current xFormers + chunked-prefill path.
- Next decode optimization should focus on either enabling a compatible attention backend, checking whether chunked prefill can be disabled for a decode-focused variant, or optimizing parser/reasoning/tool overhead before deeper code changes.
## 2026-07-14 Decode Microbenchmark Round
- Added local and remote script:
- Local: `worklogs/decode_microbench.py`
- Remote: `/root/work/decode_microbench.py`
- Synced raw result JSON files to:
- `worklogs/remote_results/2026-07-14-decode/`
- Wrote Chinese analysis report:
- `worklogs/decode_analysis_report_2026-07-14.md`
- Key results:
- Full parser, short prompt, concurrency 1, 256 output tokens: Output TPS P10 `8.74 tok/s`, TTFT P90 `1.85s`.
- Full parser with 29 tools, concurrency 1, 128 output tokens: Output TPS P10 `8.70 tok/s`, TTFT P90 `4.98s`.
- Parser-off, short prompt, concurrency 1, 256 output tokens: Output TPS P10 `8.30 tok/s`, TTFT P90 `3.18s`.
- Parser-off concurrency curve with 128 output tokens:
- c1: aggregate Output TPS `8.00`, per-request P10 `8.30`.
- c2: aggregate Output TPS `7.23`, per-request P10 `3.68`.
- c4: aggregate Output TPS `5.08`, per-request P10 `2.57`, TTFT P90 `51.60s`.
- c2 with `ixsmi` sampling: average GPU utilization `28.91%`, max GPU utilization `100%`, average power `49.67W`.
- Interpretation:
- Tool/reasoning parser is not the decode throughput bottleneck.
- Increasing concurrency does not improve aggregate decode throughput and worsens per-request TPS.
- Low average GPU utilization suggests scheduling, TP synchronization, MoE kernel, paged attention, or xFormers backend limitations before raw compute saturation.
- Next focused experiment: disable chunked prefill and retry `--num-scheduler-steps 4` as a decode-only diagnostic variant.