dylanyunlon
|
8002900af0
|
[ENGINE] CCCL kernel_scan.cuh dual-algorithm dispatch: v1 (online norm) vs v2 (deferred norm)
Source: cccl_upstream/cub/cub/device/dispatch/kernels/kernel_scan.cuh
Target: vllm/attention/ops/prefix_prefill.py
CCCL kernel_scan.cuh implements compile-time algorithm selection:
- lookback: AgentScan with delay_constructor_t (safe default)
- lookahead: warpspeed pipeline stages (SM90+, deferred reduction)
Applied to prefix_prefill Triton kernels:
- _fwd_kernel (v1) = lookback: online softmax norm per block
- _fwd_kernel_flash_attn_v2 = lookahead: deferred normalization
Saves (ctx_len / BLOCK_N) divisions per query row.
Before: v2 kernel NEVER called — dead code since initial commit.
After: v2 dispatched for standard Qwen3.6 path (no alibi, no sliding
window, power-of-2 head_dim, no FP8).
BI-V100: v2 saves 64 fdiv/row at ctx_len=4096, BLOCK_N=64.
|
2026-08-07 06:37:03 +00:00 |
|
muh-bot
|
ab81329cb4
|
feat(engine): CCCL system design integration into prefill + decode hot paths
Source input for this commit:
- CCCL bench/adjacent_difference/subtract_left.cu (randomly selected)
→ Learned: %RANGE% parameter search + policy_selector_t override pattern
- CCCL bench/reduce/sum.cu + base.cuh
→ Learned: scale_mem_bound adapts (threads, items, vec) to hardware
→ 3 search dims: ipt 7:24, tpb 128:1024, ipv 1:2
- CCCL bench/scan/exclusive/sum.cu
→ Learned: 7 search dims including delay_ns, L2_write_latency
→ This is why nobody wins by guessing — NVIDIA searches 7D space
- CCCL thrust/examples/summary_statistics.cu
→ Welford parallel merge = paged_attention_v2 partition merge pattern
- Base engine: vllm/worker/cache_engine.py (already has CCCL layout/slot)
- Base engine: vllm/attention/ops/paged_attn.py (V1/V2 dispatch)
- Base engine: vllm/attention/ops/prefix_prefill.py (Triton prefill)
Changes:
prefix_prefill.py:
- Replaced hardcoded BLOCK=64/NUM_WARPS=4 with CCCL-informed
SMEM-aware policy selection
- Documents the actual SMEM model: BLOCK_N * Lk * elem_bytes * 2
- For BI-V100: derives BLOCK from smem_limit dynamically
- NUM_WARPS follows CCCL pattern: fewer warps when SM count is low
- Search space documented: BLOCK ∈ {16,32,64}, NUM_WARPS ∈ {2,4,8}
paged_attn.py:
- Enriched _PARTITION_SIZE documentation with CCCL scan benchmark
7-dimensional parameter space reference
- Added scale_mem_bound analysis for future float16 vs float32
partition size differentiation
- Connected GridEvenShare dispatch to scan delay parameters
NOT changed (correctly):
- _PARTITION_SIZE value stays 512 (precompiled .so constraint)
- V1/V2 threshold logic stays max_num_partitions == 1
- These require .so recompilation to change
|
2026-08-07 02:42:08 +00:00 |
|
dylanyunlon
|
8a38c04b4c
|
[vllm] 3 个运行时 bug 修复: SMEM 32KB→48KB, NUM_WARPS 8→4, v2 归一化
基于完整读入 CCCL agent_reduce.cuh (412行) + vllm 运行时代码分析。
这些改动影响实际 kernel 执行，不是 tuning 参数。
1. _custom_ops.py: get_max_shared_memory 32KB → 49152 (48KB)
BI-V100 实际有 48KB SMEM (via ixsmi 确认)。
32KB 限制了 vllm/utils.py:get_max_shared_memory_bytes() 的返回值，
可能影响 Triton 编译器 SMEM budget 和 ixformer 内部 tile size 选择。
2. prefix_prefill.py: NUM_WARPS 8→4 for non-SM80 devices
BLOCK=64 时只有 64 行 query 要处理。8 warps = 256 threads，
64/256 = 0.25 rows/thread，大部分 thread 空闲浪费 register。
4 warps = 128 threads，64/128 = 0.5 rows/thread，更好的利用率。
同时用 if/else 结构替代三元表达式，为未来 BI-V100 特化留位置。
3. prefix_prefill.py: _fwd_kernel_flash_attn_v2 归一化 bug 修复
v2 kernel 的 acc_scale = alpha (不除 l_i_new)，
所以 acc 是未归一化的 softmax 加权和。
最后的 acc /= l_i[:, None] 被注释掉了 → 输出错误。
对比 v1 kernel: 用 p_scale=beta/l_i_new, acc_scale=l_i/l_i_new*alpha
在循环内做在线归一化，所以不需要最后除。
v2 的设计是 defer normalization → 最后必须除。
当前是 dead code (use_v1=True)，但修复后可以安全启用 v2 路径。
|
2026-08-04 12:26:24 +00:00 |
|
dylanyunlon
|
ef6abf3dc7
|
[DEPLOY] Complete submission: baseline + all optimizations
Adds ALL files needed for Dockerfile build:
- qwen3_6_scripts/ (baseline patches + our optimizations)
- vllm/ (full vllm package)
- paged_attention_v2_pytorch.py (V2 with single-bmm optimization)
- Dockerfile + computility-run.yaml
Our optimizations vs baseline:
1. paged_attn.py: pre-gathered context KV (eliminates 194 gather calls),
Triton try/fallback, V2 heuristic, threshold 32K→64K
2. paged_attention_v2_pytorch.py: fills NotImplementedError,
single-bmm Phase 1 (195 launches → 3)
3. patch_enable_triton.py: HAS_TRITON=True with safety fallback
4. patch_triton_tuning.py: BLOCK=64, NUM_WARPS=4 for BI-V100
5. computility-run.yaml: gpu-memory-utilization 0.9→0.95,
max-num-batched-tokens 8192→16384
This repo can now be submitted to dev.modelhub.org.cn as-is.
|
2026-07-30 16:06:20 +00:00 |
|