From 87cc24b81922dcadfdf6721b0f3786191465eddf Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 8 Aug 2026 07:52:02 +0000 Subject: [PATCH] =?UTF-8?q?doc(prd):=20CCCL=20tuning=5Fselect=5Fif.cuh=20c?= =?UTF-8?q?omplete=20design=20=E2=86=92=20serving=5Fchat.py=20mapping?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tuning_select_if.cuh (2729 lines) complete design analysis: - 3-level dispatch: compute_capability → sm_tuning → benchmark params - Per-type/per-mode/per-hardware specialization tables - Every param from real benchmark (annotated with 4 speedup ratios) - Fallback to conservative default when no tuning match Maps to our serving layer: - Request type dispatch (tool/reasoning/basic) = compute_capability - max_tokens cap by type = threads_per_block/items_per_thread - Sub168 log data = benchmark annotations - default_policy = conservative fallback No code changes needed — current serving_chat.py already implements this 3-level dispatch pattern with Sub168 benchmark-derived params. --- PRD.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/PRD.md b/PRD.md index 21f85c13..b6573c08 100644 --- a/PRD.md +++ b/PRD.md @@ -57,3 +57,30 @@ prefix_prefill.py, logits_processor.py, mamba_cache.py, arg_utils.py 4. ✅ d03 tool_call thinking耗尽 → 自动禁用thinking 5. ✅ 内存碎片OOM → PYTORCH_CUDA_ALLOC_CONF 6. ✅ 模型层代码破坏CoreX → patch_ops.sh只部署serving层 + +## CCCL tuning_select_if.cuh → serving_chat.py 映射 + +### 设计思想翻译 +CCCL三级分发：compute_capability → sm_tuning → benchmark参数 +我们三级分发：请求类型 → 处理路径 → Sub168实测参数 + +### 参数对应关系 +| CCCL概念 | 我们的对应 | +|---------|-----------| +| compute_capability (SM80/90/100) | 请求类型 (tool_call/reasoning/basic) | +| input_size (1/2/4/8 bytes) | 请求复杂度 (simple/multimodal/multi-turn) | +| flagged/unflagged | has_tools/no_tools | +| keep_rejects/discard | enable_thinking/disable_thinking | +| threads_per_block | max_tokens cap | +| items_per_thread | default_max_tokens计算 | +| delay_constructor | token budget 分配策略 | +| benchmark注释 (4个加速比) | Sub168日志实测数据 | + +### Sub168 benchmark数据（=我们的tuning表） +| 请求类型 | 时间 | token数 | TPS | +|---------|------|---------|-----| +| d01 basic | 8.49s | 139 | 16.4 | +| d03 tool_call | 2.12s | ~34 | ~16 | +| d04 reasoning | 17.78s | 1192 | 67 | +| d07 reasoning+content | 61.11s | 4451 | 72.8 | +| replay avg | - | - | 11.86 |