icer/qwen36_01
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

add cumulative prefix benchmark generator

Browse Source
This commit is contained in:
the magician
2026-07-14 14:21:23 +08:00
parent c8cc9401a4
commit 61e169fe18
2 changed files with 168 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
worklogs/2026-07-13-initial-run.md
View File

@@ -147,3 +147,32 @@
- Long-context uncached prefill is the immediate bottleneck.
- Current synthetic generator must be improved to make same-session adjacent requests share exact prefix, otherwise cache behavior is not representative of the official 65.6% cached-token workload.
- Current runtime also needs template compatibility work for multiple/irregular system messages if official requests include them unmodified.
## 2026-07-14 Cumulative Prefix Synthetic Benchmark Round 2
- Added `--cumulative` mode to `generate_official_like_dataset.py`.
- New mode shares exact system/tool prefix and grows each session history in place, so same-session later requests actually reuse earlier token prefixes.
- Generated dataset:
- `/root/work/logs/synthetic_cumulative_8_s3_cap40k.jsonl`
- 8 requests, 3 sessions, approximate prompt average `27452.75`.
- Prompt cap `40000` to keep test runtime manageable under current `--max-model-len 100000`.
- Benchmark:
- Result file: `/root/work/logs/perf_cumulative8_s3_cap40k_c1_r4.json`.
- Concurrency `1`, requests `4`, max output tokens `64`.
- Results:
- Success rate: `100%` (`4/4`).
- TTFT P90: `58.49s`.
- Output TPS P10 per request: `5.94 tok/s`.
- Aggregate output TPS: `1.55 tok/s`.
- Aggregate uncached input TPS: `284.96 tok/s`.
- Aggregate cache TPS: `296.87 tok/s`.
- Cached tokens: `49152 / 96333`, cache hit rate `51.02%`.
- Weighted token throughput: `989.82`.
- Target comparison:
- Cache hit rate and success rate pass.
- TTFT, Output TPS P10, and weighted throughput are far below target.
- Interpretation:
- Prefix cache is now being exercised correctly, but cached prefill is still too slow at this configuration.
- Current serving parameter `--max-num-seqs 1` prevents useful concurrent batching and likely caps weighted throughput.
- Decode path is also slow: request-level output TPS only `5.9-8.5 tok/s`.
- Next experiment should restart service with higher `--max-num-seqs`, higher `--max-num-batched-tokens`, and possibly `gpu-memory-utilization 0.95`, then rerun the same cumulative dataset.