From b41e7cd899eb941f3209b8b5cdbec75f3fb5e0f9 Mon Sep 17 00:00:00 2001
From: the magician <82004885@qq.com>
Date: Tue, 14 Jul 2026 15:27:52 +0800
Subject: [PATCH] log seq2 benchmark experiments

---
 worklogs/2026-07-13-initial-run.md | 68 ++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)

diff --git a/worklogs/2026-07-13-initial-run.md b/worklogs/2026-07-13-initial-run.md
index ec42998..dca0b9d 100644
--- a/worklogs/2026-07-13-initial-run.md
+++ b/worklogs/2026-07-13-initial-run.md
@@ -176,3 +176,71 @@
   - Current serving parameter `--max-num-seqs 1` prevents useful concurrent batching and likely caps weighted throughput.
   - Decode path is also slow: request-level output TPS only `5.9-8.5 tok/s`.
   - Next experiment should restart service with higher `--max-num-seqs`, higher `--max-num-batched-tokens`, and possibly `gpu-memory-utilization 0.95`, then rerun the same cumulative dataset.
+
+## 2026-07-14 Serving Parameter Experiment: seq2 / batched8192
+
+- Restarted service from `/root` with:
+  - `--gpu-memory-utilization 0.95`
+  - `--max-num-seqs 2`
+  - `--max-num-batched-tokens 8192`
+  - same model, tensor parallel `4`, prefix cache, chunked prefill, and qwen3 parsers.
+- Service started successfully:
+  - Log file: `/root/work/logs/server_exp_seq2_b8192.log`
+  - Process id observed: `2710`
+  - GPU blocks: `21100`, CPU blocks: `6553`
+
+### Concurrency 2 result
+
+- Benchmark:
+  - Dataset: `/root/work/logs/synthetic_cumulative_8_s3_cap40k.jsonl`
+  - Result file: `/root/work/logs/perf_cumulative8_s3_cap40k_exp_seq2_c2_r4.json`
+  - Concurrency `2`, requests `4`, max output tokens `64`.
+- Results:
+  - Success rate: `100%`.
+  - TTFT P90: `58.51s`.
+  - Output TPS P10 per request: `0.87 tok/s`.
+  - Aggregate output TPS: `1.53 tok/s`.
+  - Cache hit rate: `43.22%`.
+  - Weighted token throughput: `1076.71`.
+- Interpretation:
+  - Raising concurrency to `2` harms per-request decode speed and does not improve TTFT on this small long-context test.
+  - Cache hit rate also drops below target because overlapping requests disturb the ideal same-session sequential cache path.
+
+### Concurrency 1 result
+
+- Benchmark:
+  - Dataset: `/root/work/logs/synthetic_cumulative_8_s3_cap40k.jsonl`
+  - Result file: `/root/work/logs/perf_cumulative8_s3_cap40k_exp_seq2_c1_r4.json`
+  - Concurrency `1`, requests `4`, max output tokens `64`.
+- Results:
+  - Success rate: `100%`.
+  - TTFT P90: `3.34s`.
+  - Output TPS P10 per request: `6.10 tok/s`.
+  - Aggregate output TPS: `5.50 tok/s`.
+  - Cache hit rate: `99.97%`.
+  - Cached tokens: `96304 / 96333`.
+  - Weighted token throughput: `1252.05`.
+- Interpretation:
+  - With warm/exact prefix cache reuse, TTFT and cache-hit targets can pass.
+  - Output TPS is still far below the `>=20` target, so the next optimization area is decode throughput.
+  - Weighted throughput remains far below `8000`; this requires either much higher input/cache throughput under realistic scheduling or substantially faster decode.
+
+## 2026-07-14 Failed Decode Scheduler Experiment: multi-step
+
+- Investigated available CoreX/vLLM parameters from `/root/work/logs/api_server_help.txt`.
+- Tried adding:
+  - `--num-scheduler-steps 4`
+  - existing `--enable-chunked-prefill`
+- Result:
+  - Service failed during startup.
+  - Log file: `/root/work/logs/server_exp_sched4_seq2_b8192.log`
+  - Error: `Multi-Step + Chunked-Prefill not supported for attention backend: xformers`.
+  - The runtime suggests `flash-attn`, but current service logs show `Using XFormers backend`.
+- Recovery:
+  - Restarted the previously working `seq2/b8192` service without `--num-scheduler-steps`.
+  - Recovery log file: `/root/work/logs/server_exp_seq2_b8192_recover.log`
+  - `/health` returned `200`.
+  - Server log recorded a short `/v1/chat/completions` request returning `200 OK`.
+- Interpretation:
+  - Multi-step scheduling is not available with the current xFormers + chunked-prefill path.
+  - Next decode optimization should focus on either enabling a compatible attention backend, checking whether chunked prefill can be disabled for a decode-focused variant, or optimizing parser/reasoning/tool overhead before deeper code changes.