Key findings from reading dispatch_scan.cuh:
1. Lookahead scan requires PTX ISA >= 860 (NVIDIA SM100+), completely
unavailable on BI-V100. Our lookback-only strategy is correct.
2. Lookback scan passes 0 dynamic SMEM — SMEM is all static via
__shared__. Different from lookahead which uses dynamic stages.
3. Scan launches exactly num_tiles blocks (not sm_count * subscription),
one CTA per tile. For 100K tokens: ~12 tiles all fit in one wave
on 16 SMs, explaining why no_delay (dcid=0) is optimal.
4. Lookahead's num_stages auto-tuning is irrelevant for BI-V100 but
reveals NVIDIA's pipeline depth selection strategy.
Read dispatch_reduce.cuh, kernel_reduce.cuh, agent_reduce.cuh,
tuning_reduce.cuh, and util_arch.cuh from cccl_upstream.
Key findings:
1. Reduce tile data is in REGISTERS, not SMEM. Our test_smem_safety
model (tile = threads * items * type_size) checks scale_mem_bound's
register-pressure cap, not actual SMEM usage. Real SMEM ≈ threads *
sizeof(AccumT), which is 2-8 KB, not 32-49 KB.
2. scale_mem_bound vs scale_reg_bound serve different purposes:
mem_bound allows items to 2x expand (for small types), reg_bound
does not. Both use 48KB as register-spill prevention, not SMEM.
3. Our float64 tuning (threads=384) may be too conservative. CCCL
SM100 uses threads=640 for float64 — this doesn't overflow SMEM
because SMEM is only used for BlockReduce communication.
4. paged_attn.py line 99 hardcodes use_v1=True, completely disabling
V2 partitioned attention. For 100K token sequences this is suboptimal.
5. _PARTITION_SIZE=512 is hardcoded, should be tunable via muh.
vllm 0.6.3 KeyError on qwen3_5_moe model type.
Model is hybrid linear+full attention MoE with 256 experts (top-8).
enginex-vllm-bi100-qwen36-main.zip in repo likely contains the fix.
Architecture document: docs/paged_attention_kernel_architecture.md
Defines every module from CCCL algorithm patterns before code.
Three-level decomposition from CCCL:
Level 1 (warp_reduce_shfl): shfl.down butterfly for per-thread QK scores
Level 2 (block_reduce_warp_reductions): warp partials → SMEM → block aggregate
Level 3 (agent_scan decoupled lookback): cross-partition combine
Compound type (from summary_statistics.cu):
attention_partial = (max_score, exp_sum, weighted_v[256])
combine(a, b) = online softmax rescaling (same math as Flash Attention)
Key design change: Grid on num_kv_heads, not num_heads.
Before: grid = (1, 24, 200) = 4800 blocks, KV loaded 6x redundantly
After: grid = (1, 4, 200) = 800 blocks, KV loaded once per kv_head
Each block computes GQA_RATIO=6 query heads with shared KV loads.
Reduces KV cache bandwidth by 6x (the GQA ratio).
SMEM budget verified:
K tile [32, 256] fp16 = 16KB
V tile [32, 256] fp16 = 16KB
Total = 32KB ≤ 48KB ✓
Phase 1 kernel: _partition_attn_kernel
Processes query heads sequentially within the GQA group
to minimize register pressure (6 × 256 = 1536 registers
too many if all loaded simultaneously).
Phase 2 kernel: _reduce_partitions_kernel
Also gridded on kv_heads, reduces all partitions for
GQA_RATIO heads per block.
This replaces the previous Triton V2 which was gridded on num_heads
and had no GQA awareness at the kernel level.
