icer/qwen36_01
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

add decode throughput analysis

Browse Source
This commit is contained in:
the magician
2026-07-14 17:11:28 +08:00
parent b41e7cd899
commit 465a9c8995
11 changed files with 2960 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
worklogs/2026-07-13-initial-run.md
View File

@@ -244,3 +244,27 @@
- Interpretation:
- Multi-step scheduling is not available with the current xFormers + chunked-prefill path.
- Next decode optimization should focus on either enabling a compatible attention backend, checking whether chunked prefill can be disabled for a decode-focused variant, or optimizing parser/reasoning/tool overhead before deeper code changes.
## 2026-07-14 Decode Microbenchmark Round
- Added local and remote script:
- Local: `worklogs/decode_microbench.py`
- Remote: `/root/work/decode_microbench.py`
- Synced raw result JSON files to:
- `worklogs/remote_results/2026-07-14-decode/`
- Wrote Chinese analysis report:
- `worklogs/decode_analysis_report_2026-07-14.md`
- Key results:
- Full parser, short prompt, concurrency 1, 256 output tokens: Output TPS P10 `8.74 tok/s`, TTFT P90 `1.85s`.
- Full parser with 29 tools, concurrency 1, 128 output tokens: Output TPS P10 `8.70 tok/s`, TTFT P90 `4.98s`.
- Parser-off, short prompt, concurrency 1, 256 output tokens: Output TPS P10 `8.30 tok/s`, TTFT P90 `3.18s`.
- Parser-off concurrency curve with 128 output tokens:
- c1: aggregate Output TPS `8.00`, per-request P10 `8.30`.
- c2: aggregate Output TPS `7.23`, per-request P10 `3.68`.
- c4: aggregate Output TPS `5.08`, per-request P10 `2.57`, TTFT P90 `51.60s`.
- c2 with `ixsmi` sampling: average GPU utilization `28.91%`, max GPU utilization `100%`, average power `49.67W`.
- Interpretation:
- Tool/reasoning parser is not the decode throughput bottleneck.
- Increasing concurrency does not improve aggregate decode throughput and worsens per-request TPS.
- Low average GPU utilization suggests scheduling, TP synchronization, MoE kernel, paged attention, or xFormers backend limitations before raw compute saturation.
- Next focused experiment: disable chunked prefill and retry `--num-scheduler-steps 4` as a decode-only diagnostic variant.

299
worklogs/decode_analysis_report_2026-07-14.md Normal file
View File

@@ -0,0 +1,299 @@
# Decode 吞吐专项实验分析报告
日期：2026-07-14
## 一、结论摘要
本轮实验确认：当前服务的主要短板是 decode 阶段吞吐，而不是 prompt prefill、tool parser 或 reasoning parser。
核心证据：
- 短 prompt、单并发、256 token 输出时，Output TPS P10 只有 `8.3-8.7 tok/s`，明显低于官方目标 `>=20 tok/s`。
- 去掉 `--enable-auto-tool-choice`、`--tool-call-parser`、`--reasoning-parser` 后，decode TPS 没有提升，反而略低。
- 加并发后没有获得 batch 增益：并发 1 聚合约 `8.00 tok/s`，并发 2 聚合约 `7.23 tok/s`，并发 4 聚合约 `5.08 tok/s`。
- 并发 2 时硬件采样显示平均 GPU-Util 只有 `28.9%`，平均功耗约 `49.7W / 250W`，说明 GPU 没有被持续打满。
初步判断：瓶颈更像是 decode 路径中的调度、TP 同步、MoE kernel/专家路由、paged attention/kernel launch 开销，或 xFormers 后端限制，而不是 API parser 层。
## 二、实验环境
远端服务：
- 模型：`/root/public-storage/models/Qwen/Qwen3.6-35B-A3B`
- API：`vllm.entrypoints.openai.api_server`
- GPU：4 x Iluvatar BI-V100, 32GB
- 当前主要实验配置：
- `-tp 4`
- `--gpu-memory-utilization 0.95`
- `--max-num-seqs 2`
- `--max-num-batched-tokens 8192`
- `--enable-chunked-prefill`
- `--enable-prefix-caching`
本地脚本：
- `worklogs/decode_microbench.py`
远端脚本：
- `/root/work/decode_microbench.py`
本地原始结果：
- `worklogs/remote_results/2026-07-14-decode/`
## 三、实验结果
### 1. 纯短 prompt decode 基线
完整 parser 配置，短 prompt，单并发，3 请求，每请求 `max_tokens=256`。
结果文件：
- 远端：`/root/work/logs/decode_full_parser_short_c1_t256_r3.json`
- 本地：`worklogs/remote_results/2026-07-14-decode/decode_full_parser_short_c1_t256_r3.json`
指标：
| 指标 | 数值 |
|---|---:|
| 成功率 | 100% |
| TTFT P90 | 1.85s |
| Output TPS P10 | 8.74 tok/s |
| Output TPS P50 | 8.74 tok/s |
| 聚合 Output TPS | 8.34 tok/s |
| 总 completion tokens | 768 |
| reasoning tokens | 768 |
解释：
短 prompt 下 TTFT 已经较低，但 decode 速度仍只有约 `8.7 tok/s`。这说明长上下文不是唯一问题，短输出生成本身就偏慢。
### 2. Tool/parser 触发场景
完整 parser 配置，携带 29 个 tools，单并发，3 请求，每请求 `max_tokens=128`。
结果文件：
- 远端：`/root/work/logs/decode_full_parser_tool_c1_t128_r3.json`
- 本地：`worklogs/remote_results/2026-07-14-decode/decode_full_parser_tool_c1_t128_r3.json`
指标：
| 指标 | 数值 |
|---|---:|
| 成功率 | 100% |
| TTFT P90 | 4.98s |
| Output TPS P10 | 8.70 tok/s |
| Output TPS P50 | 8.70 tok/s |
| 聚合 Output TPS | 7.38 tok/s |
| prompt tokens | 6639 |
| cached tokens | 4416 |
| completion tokens | 384 |
单请求细节：
- 第 1 个请求：TTFT `5.97s`，cached tokens `0`
- 第 2/3 个请求：TTFT 约 `1.0s`，cached tokens `2208`
解释：
工具 schema 会明显影响首次 prefill/TTFT，但前缀缓存命中后 TTFT 恢复。Output TPS 仍约 `8.7 tok/s`，与纯短 prompt 基本一致，所以 tool parser 不是 decode TPS 主瓶颈。
### 3. Parser-off 对照
重启服务，去掉：
- `--enable-auto-tool-choice`
- `--tool-call-parser qwen3_coder`
- `--reasoning-parser qwen3`
其余参数保持一致。
短 prompt，单并发，3 请求，每请求 `max_tokens=256`。
结果文件：
- 远端：`/root/work/logs/decode_noparser_short_c1_t256_r3.json`
- 本地：`worklogs/remote_results/2026-07-14-decode/decode_noparser_short_c1_t256_r3.json`
指标：
| 指标 | 完整 parser | parser-off |
|---|---:|---:|
| 成功率 | 100% | 100% |
| TTFT P90 | 1.85s | 3.18s |
| Output TPS P10 | 8.74 | 8.30 |
| Output TPS P50 | 8.74 | 8.33 |
| 聚合 Output TPS | 8.34 | 7.87 |
解释：
关闭 parser 没有改善 decode。parser/reasoning/tool 相关启动项不是当前 decode 吞吐低的主因。
### 4. 并发曲线
parser-off 服务，短 prompt，每请求 `max_tokens=128`。
结果文件：
- `decode_noparser_short_c1_t128_r4.json`
- `decode_noparser_short_c2_t128_r4.json`
- `decode_noparser_short_c4_t128_r4.json`
指标：
| 并发 | 请求数 | 成功率 | TTFT P90 | Output TPS P10/请求 | 聚合 Output TPS |
|---:|---:|---:|---:|---:|---:|
| 1 | 4 | 100% | 0.79s | 8.30 | 8.00 |
| 2 | 4 | 100% | 0.92s | 3.68 | 7.23 |
| 4 | 4 | 100% | 51.60s | 2.57 | 5.08 |
解释：
并发提高后没有形成有效 batch 增益。并发 2 时单请求 TPS 近似减半，聚合 TPS 也略降；并发 4 时出现明显排队，TTFT P90 被拉到 `51.6s`。
这说明当前配置下 decode 并发能力很弱。由于服务参数 `--max-num-seqs=2`，并发 4 的后两个请求排队符合预期；但并发 2 聚合吞吐仍不提升，说明 decode 内部没有把双请求 batch 变成更高硬件利用率。
### 5. 硬件利用率采样
parser-off 服务，并发 2，4 请求，每请求 `max_tokens=128`，同时采样 `ixsmi`。
结果文件：
- 远端：`/root/work/logs/decode_noparser_short_c2_t128_r4_monitor.json`
- 远端：`/root/work/logs/ixsmi_noparser_short_c2_t128_r4_monitor.json`
- 本地同名文件位于：`worklogs/remote_results/2026-07-14-decode/`
指标：
| 指标 | 数值 |
|---|---:|
| 成功率 | 100% |
| TTFT P90 | 0.92s |
| Output TPS P10 | 3.79 tok/s |
| 聚合 Output TPS | 7.40 tok/s |
| ixsmi records | 52 |
| parsed GPU samples | 208 |
| 平均 GPU-Util | 28.91% |
| 最大 GPU-Util | 100% |
| 平均显存 | 30477 MiB |
| 最大显存 | 30555 MiB |
| 平均功耗 | 49.67 W |
| 最大功耗 | 51 W |
解释：
GPU 利用率和功耗都偏低。虽然瞬时 GPU-Util 能到 100%，但平均只有约 29%，功耗长期接近空载到轻载水平。这说明 decode 过程中存在大量空泡、同步等待或小 kernel 启动开销，GPU 算力没有被持续喂满。
## 四、瓶颈判断
当前最可能的瓶颈排序：
1. Decode 调度/后端限制
- `--num-scheduler-steps 4` 曾尝试失败。
- 报错：`Multi-Step + Chunked-Prefill not supported for attention backend: xformers`。
- 当前服务日志显示使用 `XFormers backend`。
2. TP 通信或同步开销
- 模型使用 `-tp 4`。
- 短 decode 每步都可能涉及多卡同步。
- 并发 2 聚合 TPS 不升反降，符合小 batch 多卡同步效率差的特征。
3. MoE decode kernel/专家路由效率
- Qwen3.6-35B-A3B 是 MoE 模型。
- decode batch 小时，专家路由和 fused MoE kernel 可能难以形成高利用率。
4. Paged attention / attention backend 每 token 开销
- 当前 attention 后端为 xFormers。
- multi-step decode 被 xFormers + chunked prefill 组合限制。
5. API/parser 层
- 本轮实验基本排除其为主瓶颈。
- tool/schema 影响首次 TTFT，但不显著影响 decode TPS。
## 五、下一步建议
建议下一步不要直接改大段 kernel，而是先做两个能明确指向代码修改方向的服务变体实验。
### 实验 A：decode-only multi-step 变体
目的：验证 multi-step scheduling 是否能明显提升 decode。
做法：
- 暂时关闭 `--enable-chunked-prefill`
- 加回 `--num-scheduler-steps 4`
- 保持 `max_num_seqs=2`
- 跑短 prompt decode 并发 1/2 曲线
判断：
- 如果 Output TPS 明显提升，说明 decode 调度是关键方向。
- 后续要研究如何让 multi-step 与长上下文 chunked prefill 共存，或按场景切换。
风险：
- 官方长上下文负载仍需要 chunked prefill，所以这不是最终配置，只是定位实验。
### 实验 B：attention backend 变体
目的：确认是否可以启用兼容 multi-step 的 attention backend。
做法：
- 尝试设置 `VLLM_ATTENTION_BACKEND=FLASH_ATTN`
- 或检查 CoreX/xFormers flash attention 能否被 vLLM selector 选中
- 如果能启动，再测试 multi-step + chunked prefill
判断：
- 如果 flash attention 能启动且 multi-step 可用，优先走 backend 配置/适配路线。
- 如果不能启动，需要看 `vllm/attention/selector.py`、`vllm/attention/backends/*` 和 CoreX xFormers 补丁。
### 实验 C：代码级 profiling
目的：把低 GPU 利用率归因到具体模块。
建议插桩位置：
- `vllm/worker/model_runner.py`
- `vllm/worker/multi_step_model_runner.py`
- `vllm/model_executor/models/qwen3_moe.py`
- `vllm/model_executor/layers/fused_moe/*`
- `attention.py`
- `paged_attn.py`
记录每步：
- model forward 耗时
- attention 耗时
- MoE/MLP 耗时
- sampler 耗时
- 每步前后同步耗时
### 实验 D：服务参数小网格
目的：确认当前 `max_num_seqs=2` 是否已经是最优。
建议组合：
| 参数 | 候选 |
|---|---|
| `max_num_seqs` | 1, 2, 4 |
| `max_num_batched_tokens` | 4096, 8192 |
| `chunked_prefill` | on/off |
| `num_scheduler_steps` | 1, 4 |
优先只在短 prompt decode 上跑，快速筛掉无效组合。
## 六、当前推荐行动
下一步优先跑：
1. `chunked_prefill=off + num_scheduler_steps=4`
2. 若能启动，跑 decode c1/c2/c4 曲线和 ixsmi 采样
3. 若 decode TPS 明显提升，再研究如何兼容官方长上下文 prefill
4. 若没有提升，进入 qwen3_moe / fused_moe / attention 的代码级 profiling
本轮最重要的事实是：GPU 平均利用率只有约 `29%`，所以先不要把问题简单归因为“卡算不动”。更像是当前 decode 执行路径没有把四张卡持续喂满。

314
worklogs/decode_microbench.py Normal file
View File

@@ -0,0 +1,314 @@
import argparse
import concurrent.futures
import json
import math
import os
import re
import subprocess
import threading
import time
import urllib.error
import urllib.request
from dataclasses import asdict, dataclass
from datetime import datetime
from pathlib import Path
@dataclass
class RequestResult:
ok: bool
elapsed_sec: float
ttft_sec: float | None
completion_tokens: int
prompt_tokens: int
cached_tokens: int
reasoning_tokens: int
output_tps: float | None
chars: int
error: str | None = None
def percentile(values, pct):
values = sorted(v for v in values if v is not None)
if not values:
return None
if len(values) == 1:
return values[0]
pos = (len(values) - 1) * pct / 100.0
lo = math.floor(pos)
hi = math.ceil(pos)
if lo == hi:
return values[lo]
return values[lo] * (hi - pos) + values[hi] * (pos - lo)
def make_tools(count):
tools = []
for i in range(count):
tools.append({
"type": "function",
"function": {
"name": f"tool_{i:02d}_exec",
"description": "Run a deterministic diagnostic action.",
"parameters": {
"type": "object",
"properties": {
"command": {"type": "string"},
"path": {"type": "string"},
},
"required": ["command"],
},
},
})
return tools
def make_request(args, idx):
if args.prompt_mode == "short":
user = (
"Do not explain. Output a comma-separated sequence of four digit "
"numbers starting at 0001. Continue until the token limit stops you."
)
messages = [{"role": "user", "content": user}]
elif args.prompt_mode == "tool":
messages = [
{"role": "system", "content": "You are a coding agent. Return concise tool-call-like JSON text."},
{"role": "user", "content": "Create a shell command to list Python files and print the answer as JSON."},
]
else:
raise ValueError(f"unknown prompt_mode: {args.prompt_mode}")
payload = {
"model": args.model,
"messages": messages,
"max_tokens": args.max_tokens,
"temperature": 0,
"stream": True,
"stream_options": {"include_usage": True},
}
if args.with_tools:
payload["tools"] = make_tools(args.tool_count)
payload["tool_choice"] = "auto"
return payload
def post_stream(url, payload, timeout):
req = urllib.request.Request(
url.rstrip("/") + "/v1/chat/completions",
data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
headers={"Content-Type": "application/json"},
method="POST",
)
start = time.perf_counter()
first_at = None
usage = {}
chars = 0
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
for raw in resp:
line = raw.decode("utf-8", errors="replace").strip()
if not line or not line.startswith("data:"):
continue
data = line[5:].strip()
if data == "[DONE]":
break
obj = json.loads(data)
if obj.get("usage"):
usage = obj["usage"]
for choice in obj.get("choices") or []:
delta = choice.get("delta") or {}
parts = [
delta.get("content") or "",
delta.get("reasoning_content") or "",
]
for tool_call in delta.get("tool_calls") or []:
fn = tool_call.get("function") or {}
parts.append(fn.get("name") or "")
parts.append(fn.get("arguments") or "")
added = sum(len(p) for p in parts)
if added and first_at is None:
first_at = time.perf_counter()
chars += added
elapsed = time.perf_counter() - start
ttft = first_at - start if first_at is not None else None
details = usage.get("prompt_tokens_details") or {}
completion_tokens = int(usage.get("completion_tokens") or 0)
decode_sec = elapsed - ttft if ttft is not None else elapsed
return RequestResult(
ok=True,
elapsed_sec=elapsed,
ttft_sec=ttft,
completion_tokens=completion_tokens,
prompt_tokens=int(usage.get("prompt_tokens") or 0),
cached_tokens=int(details.get("cached_tokens") or 0),
reasoning_tokens=int(usage.get("reasoning_tokens") or 0),
output_tps=completion_tokens / decode_sec if completion_tokens and decode_sec > 0 else None,
chars=chars,
)
except urllib.error.HTTPError as exc:
body = exc.read().decode("utf-8", errors="replace")[:1000]
return RequestResult(False, time.perf_counter() - start, None, 0, 0, 0, 0, None, chars, f"HTTP {exc.code}: {body}")
except Exception as exc:
return RequestResult(False, time.perf_counter() - start, None, 0, 0, 0, 0, None, chars, f"{type(exc).__name__}: {exc}")
def parse_ixsmi(raw):
samples = []
for line in raw.splitlines():
m = re.search(r"\|\s*\d+%\s+\d+C\s+\S+\s+(\d+)W\s*/\s*(\d+)W\s*\|\s*(\d+)MiB\s*/\s*(\d+)MiB\s*\|\s*(\d+)%", line)
if m:
samples.append({
"power_w": int(m.group(1)),
"power_cap_w": int(m.group(2)),
"mem_mib": int(m.group(3)),
"mem_total_mib": int(m.group(4)),
"util_pct": int(m.group(5)),
})
return samples
def monitor_ixsmi(stop_event, out_path, interval):
records = []
env = dict(os.environ)
env["LD_LIBRARY_PATH"] = ":".join([
"/usr/local/corex/lib64",
"/usr/local/corex/lib",
"/usr/local/iluvatar/lib64",
env.get("LD_LIBRARY_PATH", ""),
])
candidates = [
"/usr/local/corex/bin/ixsmi",
"/usr/local/iluvatar/bin/ixsmi",
"ixsmi",
]
while not stop_event.is_set():
ts = datetime.now().isoformat(timespec="seconds")
try:
last_error = None
raw = None
for binary in candidates:
try:
raw = subprocess.check_output([binary], text=True, stderr=subprocess.STDOUT, timeout=10, env=env)
break
except Exception as exc:
last_error = exc
if raw is None:
raise last_error or RuntimeError("ixsmi not found")
records.append({"ts": ts, "ok": True, "raw": raw, "parsed": parse_ixsmi(raw)})
except Exception as exc:
records.append({"ts": ts, "ok": False, "error": repr(exc)})
stop_event.wait(interval)
Path(out_path).write_text(json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8")
def summarize_monitor(path):
if not path or not Path(path).exists():
return None
records = json.loads(Path(path).read_text(encoding="utf-8"))
util = []
mem = []
power = []
for rec in records:
for gpu in rec.get("parsed") or []:
util.append(gpu["util_pct"])
mem.append(gpu["mem_mib"])
power.append(gpu["power_w"])
if not util:
return {"records": len(records), "parsed_samples": 0}
return {
"records": len(records),
"parsed_samples": len(util),
"avg_gpu_util_pct": sum(util) / len(util),
"max_gpu_util_pct": max(util),
"avg_mem_mib": sum(mem) / len(mem),
"max_mem_mib": max(mem),
"avg_power_w": sum(power) / len(power),
"max_power_w": max(power),
}
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--url", default="http://127.0.0.1:1111")
parser.add_argument("--model", default="llm")
parser.add_argument("--label", required=True)
parser.add_argument("--prompt-mode", choices=["short", "tool"], default="short")
parser.add_argument("--with-tools", action="store_true")
parser.add_argument("--tool-count", type=int, default=16)
parser.add_argument("--concurrency", type=int, default=1)
parser.add_argument("--requests", type=int, default=4)
parser.add_argument("--max-tokens", type=int, default=256)
parser.add_argument("--timeout", type=int, default=900)
parser.add_argument("--monitor-out")
parser.add_argument("--monitor-interval", type=float, default=1.0)
parser.add_argument("--out", required=True)
args = parser.parse_args()
stop_event = threading.Event()
monitor_thread = None
if args.monitor_out:
monitor_thread = threading.Thread(
target=monitor_ixsmi,
args=(stop_event, args.monitor_out, args.monitor_interval),
daemon=True,
)
monitor_thread.start()
started = time.perf_counter()
results = []
try:
with concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency) as pool:
futures = [
pool.submit(post_stream, args.url, make_request(args, i), args.timeout)
for i in range(args.requests)
]
for fut in concurrent.futures.as_completed(futures):
result = fut.result()
results.append(result)
print(f"done {len(results)}/{args.requests} ok={result.ok} tps={result.output_tps}", flush=True)
finally:
stop_event.set()
if monitor_thread:
monitor_thread.join(timeout=15)
wall = time.perf_counter() - started
ok = [r for r in results if r.ok]
tps_values = [r.output_tps for r in ok if r.output_tps is not None]
ttft_values = [r.ttft_sec for r in ok if r.ttft_sec is not None]
completion = sum(r.completion_tokens for r in ok)
prompt = sum(r.prompt_tokens for r in ok)
cached = sum(r.cached_tokens for r in ok)
summary = {
"created_at": datetime.now().isoformat(timespec="seconds"),
"label": args.label,
"url": args.url,
"model": args.model,
"prompt_mode": args.prompt_mode,
"with_tools": args.with_tools,
"tool_count": args.tool_count if args.with_tools else 0,
"concurrency": args.concurrency,
"requests": args.requests,
"max_tokens": args.max_tokens,
"wall_sec": wall,
"success_rate": len(ok) / len(results) if results else 0,
"ttft_p50_sec": percentile(ttft_values, 50),
"ttft_p90_sec": percentile(ttft_values, 90),
"output_tps_p10_per_request": percentile(tps_values, 10),
"output_tps_p50_per_request": percentile(tps_values, 50),
"aggregate_output_tps": completion / wall if wall > 0 else 0,
"prompt_tokens": prompt,
"cached_tokens": cached,
"completion_tokens": completion,
"reasoning_tokens": sum(r.reasoning_tokens for r in ok),
"chars": sum(r.chars for r in ok),
"monitor": summarize_monitor(args.monitor_out) if args.monitor_out else None,
"results": [asdict(r) for r in results],
}
Path(args.out).parent.mkdir(parents=True, exist_ok=True)
Path(args.out).write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps({k: v for k, v in summary.items() if k != "results"}, ensure_ascii=False, indent=2))
print("RESULT_FILE", args.out)
if __name__ == "__main__":
main()

66
worklogs/remote_results/2026-07-14-decode/decode_full_parser_short_c1_t256_r3.json Normal file
View File

@@ -0,0 +1,66 @@
{
"created_at": "2026-07-14T08:00:25",
"label": "full_parser_short_c1_t256_r3",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 3,
"max_tokens": 256,
"wall_sec": 92.0443243663758,
"success_rate": 1.0,
"ttft_p50_sec": 1.0909903924912214,
"ttft_p90_sec": 1.8471774261444807,
"output_tps_p10_per_request": 8.736438485385,
"output_tps_p50_per_request": 8.738006408510326,
"aggregate_output_tps": 8.343806153033741,
"prompt_tokens": 117,
"cached_tokens": 64,
"completion_tokens": 768,
"reasoning_tokens": 768,
"chars": 2635,
"monitor": {
"records": 92,
"parsed_samples": 0
},
"results": [
{
"ok": true,
"elapsed_sec": 31.292513709515333,
"ttft_sec": 2.0362241845577955,
"completion_tokens": 256,
"prompt_tokens": 39,
"cached_tokens": 0,
"reasoning_tokens": 256,
"output_tps": 8.750255215433768,
"chars": 916,
"error": null
},
{
"ok": true,
"elapsed_sec": 30.353378538042307,
"ttft_sec": 1.0560779832303524,
"completion_tokens": 256,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 256,
"output_tps": 8.738006408510326,
"chars": 856,
"error": null
},
{
"ok": true,
"elapsed_sec": 30.394863702356815,
"ttft_sec": 1.0909903924912214,
"completion_tokens": 256,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 256,
"output_tps": 8.736046504603667,
"chars": 863,
"error": null
}
]
}

63
worklogs/remote_results/2026-07-14-decode/decode_full_parser_tool_c1_t128_r3.json Normal file
View File

@@ -0,0 +1,63 @@
{
"created_at": "2026-07-14T08:11:04",
"label": "full_parser_tool_c1_t128_r3",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "tool",
"with_tools": true,
"tool_count": 29,
"concurrency": 1,
"requests": 3,
"max_tokens": 128,
"wall_sec": 52.042631950229406,
"success_rate": 1.0,
"ttft_p50_sec": 1.0516742002218962,
"ttft_p90_sec": 4.98421496860683,
"output_tps_p10_per_request": 8.700721568035489,
"output_tps_p50_per_request": 8.704939531751927,
"aggregate_output_tps": 7.378566102637461,
"prompt_tokens": 6639,
"cached_tokens": 4416,
"completion_tokens": 384,
"reasoning_tokens": 384,
"chars": 1309,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 20.671645294874907,
"ttft_sec": 5.967350160703063,
"completion_tokens": 128,
"prompt_tokens": 2213,
"cached_tokens": 0,
"reasoning_tokens": 128,
"output_tps": 8.704939531751927,
"chars": 445,
"error": null
},
{
"ok": true,
"elapsed_sec": 15.636568604037166,
"ttft_sec": 1.0516742002218962,
"completion_tokens": 128,
"prompt_tokens": 2213,
"cached_tokens": 2208,
"reasoning_tokens": 128,
"output_tps": 8.776203409914055,
"chars": 456,
"error": null
},
{
"ok": true,
"elapsed_sec": 15.731618992984295,
"ttft_sec": 1.018412284553051,
"completion_tokens": 128,
"prompt_tokens": 2213,
"cached_tokens": 2208,
"reasoning_tokens": 128,
"output_tps": 8.699667077106378,
"chars": 408,
"error": null
}
]
}

75
worklogs/remote_results/2026-07-14-decode/decode_noparser_short_c1_t128_r4.json Normal file
View File

@@ -0,0 +1,75 @@
{
"created_at": "2026-07-14T08:34:26",
"label": "noparser_short_c1_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 4,
"max_tokens": 128,
"wall_sec": 64.02519051544368,
"success_rate": 1.0,
"ttft_p50_sec": 0.7723927367478609,
"ttft_p90_sec": 0.7913718132302165,
"output_tps_p10_per_request": 8.304385542911493,
"output_tps_p50_per_request": 8.42998946937195,
"aggregate_output_tps": 7.99685242446095,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 0,
"chars": 1827,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 16.272426838055253,
"ttft_sec": 0.7992097493261099,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 8.272358570683828,
"chars": 474,
"error": null
},
{
"ok": true,
"elapsed_sec": 15.848933763802052,
"ttft_sec": 0.7730832956731319,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 8.490399945966447,
"chars": 449,
"error": null
},
{
"ok": true,
"elapsed_sec": 16.047778205946088,
"ttft_sec": 0.7717021778225899,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 8.379115144776051,
"chars": 452,
"error": null
},
{
"ok": true,
"elapsed_sec": 15.853784639388323,
"ttft_sec": 0.7609824072569609,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 8.480863793967849,
"chars": 452,
"error": null
}
]
}

63
worklogs/remote_results/2026-07-14-decode/decode_noparser_short_c1_t256_r3.json Normal file
View File

@@ -0,0 +1,63 @@
{
"created_at": "2026-07-14T08:23:01",
"label": "noparser_short_c1_t256_r3",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 3,
"max_tokens": 256,
"wall_sec": 97.52940320037305,
"success_rate": 1.0,
"ttft_p50_sec": 0.8136040437966585,
"ttft_p90_sec": 3.1817516405135393,
"output_tps_p10_per_request": 8.295711129981452,
"output_tps_p50_per_request": 8.327121469734012,
"aggregate_output_tps": 7.874548339254703,
"prompt_tokens": 117,
"cached_tokens": 64,
"completion_tokens": 768,
"reasoning_tokens": 0,
"chars": 2635,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 34.66234661638737,
"ttft_sec": 3.7737885396927595,
"completion_tokens": 256,
"prompt_tokens": 39,
"cached_tokens": 0,
"reasoning_tokens": 0,
"output_tps": 8.287858545043312,
"chars": 916,
"error": null
},
{
"ok": true,
"elapsed_sec": 31.533973263576627,
"ttft_sec": 0.7910567671060562,
"completion_tokens": 256,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 8.327121469734012,
"chars": 856,
"error": null
},
{
"ok": true,
"elapsed_sec": 31.330975525081158,
"ttft_sec": 0.8136040437966585,
"completion_tokens": 256,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 8.388664802176624,
"chars": 863,
"error": null
}
]
}

75
worklogs/remote_results/2026-07-14-decode/decode_noparser_short_c2_t128_r4.json Normal file
View File

@@ -0,0 +1,75 @@
{
"created_at": "2026-07-14T08:35:37",
"label": "noparser_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 70.80830597691238,
"success_rate": 1.0,
"ttft_p50_sec": 0.9245951194316149,
"ttft_p90_sec": 0.9249729935079813,
"output_tps_p10_per_request": 3.6832897295816034,
"output_tps_p50_per_request": 3.7126825201316764,
"aggregate_output_tps": 7.230790130284175,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 0,
"chars": 1816,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 35.12894960306585,
"ttft_sec": 0.9234285913407803,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 3.742085961974495,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 35.12996072135866,
"ttft_sec": 0.9242945611476898,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 3.7420700827891884,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 35.67656988278031,
"ttft_sec": 0.925006128847599,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 3.683287489056221,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 35.67638896778226,
"ttft_sec": 0.9248956777155399,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 3.683294957474164,
"chars": 454,
"error": null
}
]
}

84
worklogs/remote_results/2026-07-14-decode/decode_noparser_short_c2_t128_r4_monitor.json Normal file
View File

@@ -0,0 +1,84 @@
{
"created_at": "2026-07-14T08:56:17",
"label": "noparser_short_c2_t128_r4_monitor",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 69.160331716761,
"success_rate": 1.0,
"ttft_p50_sec": 0.9177342765033245,
"ttft_p90_sec": 0.9243953077122569,
"output_tps_p10_per_request": 3.7938414669033445,
"output_tps_p50_per_request": 3.8028299895382536,
"aggregate_output_tps": 7.4030876846693445,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 0,
"chars": 1816,
"monitor": {
"records": 52,
"parsed_samples": 208,
"avg_gpu_util_pct": 28.91346153846154,
"max_gpu_util_pct": 100,
"avg_mem_mib": 30477.0,
"max_mem_mib": 30555,
"avg_power_w": 49.67307692307692,
"max_power_w": 51
},
"results": [
{
"ok": true,
"elapsed_sec": 34.663240656256676,
"ttft_sec": 0.9245693255215883,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 3.7938660578905252,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 34.66297300904989,
"ttft_sec": 0.9239892661571503,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 3.793830927908839,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 34.49104499258101,
"ttft_sec": 0.9110554531216621,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 3.811793921185982,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 34.491438234224916,
"ttft_sec": 0.9114792868494987,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 3.811797393814395,
"chars": 454,
"error": null
}
]
}

75
worklogs/remote_results/2026-07-14-decode/decode_noparser_short_c4_t128_r4.json Normal file
View File

@@ -0,0 +1,75 @@
{
"created_at": "2026-07-14T08:37:18",
"label": "noparser_short_c4_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 4,
"requests": 4,
"max_tokens": 128,
"wall_sec": 100.87516433186829,
"success_rate": 1.0,
"ttft_p50_sec": 26.261823972687125,
"ttft_p90_sec": 51.59956250209361,
"output_tps_p10_per_request": 2.5736346925937763,
"output_tps_p50_per_request": 2.585721667979742,
"aggregate_output_tps": 5.075580331305096,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 0,
"chars": 1846,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 50.6585742700845,
"ttft_sec": 0.9234585296362638,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 2.573634304341249,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 50.65938341990113,
"ttft_sec": 0.9242926891893148,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 2.5736355985163404,
"chars": 486,
"error": null
},
{
"ok": true,
"elapsed_sec": 100.87196587957442,
"ttft_sec": 51.59965132176876,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 2.597807737443144,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 100.8716309145093,
"ttft_sec": 51.599355256184936,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 0,
"output_tps": 2.597809788360666,
"chars": 452,
"error": null
}
]
}

1822
worklogs/remote_results/2026-07-14-decode/ixsmi_noparser_short_c2_t128_r4_monitor.json Normal file
View File

File diff suppressed because it is too large Load Diff