From reading cccl_upstream/cub/cub/grid/grid_even_share.cuh:
Key finding: For Qwen3.6 attention score reduction (100K seq_len),
with tile_items=12288 (512 threads × 24 items), only 9 CTAs are
needed. All fit in one wave on 16 SMs.
This means reduce tuning (items/threads) matters less than the V1/V2
dispatch choice in paged_attn.py. V1 uses a single CTA iterating
sequentially over all KV blocks, completely bypassing GridEvenShare's
parallel distribution. V2 would enable partition-based parallelism.
Also documents: RAKE (scan) vs STRIP_MINE (reduce) strategies,
'big shares' load balancing, and the SingleTile fast path for
short sequences.
From reading cccl_upstream/cub/cub/device/dispatch/tuning/tuning_transform.cuh:
1. BI-V100 can only use prefetch and vectorized algorithms.
ldgsts (SM80+ cp.async) and ublkcp (SM90+ bulk copy) are NVIDIA-only.
2. bytes_in_flight only affects the PREFETCH path. For vllm's
contiguous fp16 element-wise ops (RMSNorm/SiLU/RoPE), the
VECTORIZED path is selected, where items_per_thread is fixed
at compile time, not derived from bytes_in_flight.
3. CCCL's cc_to_min_bytes_in_flight: B200=64KB, H100=48KB, A100=16KB,
V100=12KB. Our 64KB matches B200 level (56 GB/s/SM ≈ B200).
4. Bench result alg=1 confirms vectorized path is used on BI-V100.
The vectorized default {256, 8, 4} matches the benchmark winner.
Source: cccl_upstream/cub/cub/device/dispatch/tuning/tuning_transform.cuh
Two findings from reading CCCL source code:
1. single_pass_scan_operators.cuh: delay() has GridThreshold=500 gate.
BI-V100 scan launches ~12 blocks (100K elements / tile_size).
12 < 500, so ALL delay policies collapse to __threadfence_block().
Conclusion: delay_ns, delay_l2w, delay_algorithm are IRRELEVANT
on BI-V100. Only threads/items/load/scan algorithms matter.
2. summary_statistics.cu compound reduce pattern maps directly to
paged_attention V2's cross-partition reduce. Updated muh_kernel_map.py
with the structural mapping and the V2 dispatch bug (use_v1=True
hardcoded in paged_attn.py line 99).
Source: cccl_upstream/cub/cub/agent/single_pass_scan_operators.cuh
cccl_upstream/thrust/examples/summary_statistics.cu
Critical finding: scan and reduce have fundamentally different SMEM
models. Scan uses BlockLoad/BlockStore with WARP_TRANSPOSE which
puts tile data through SMEM (threads*items*type_size bytes). Reduce
keeps tile data in registers and only uses SMEM for BlockReduce
communication (~threads*4 bytes).
This means:
- Our SMEM constraint is CORRECT for scan (tuning_scan.cuh values
are properly bounded)
- Our SMEM constraint is WRONG for reduce (tuning_reduce.cuh could
use larger items_per_thread, especially for small types)
- The same check_smem() function should NOT be used for both algorithms
Source: cccl_upstream/cub/cub/agent/agent_scan.cuh _TempStorage union
Key findings from reading dispatch_scan.cuh:
1. Lookahead scan requires PTX ISA >= 860 (NVIDIA SM100+), completely
unavailable on BI-V100. Our lookback-only strategy is correct.
2. Lookback scan passes 0 dynamic SMEM — SMEM is all static via
__shared__. Different from lookahead which uses dynamic stages.
3. Scan launches exactly num_tiles blocks (not sm_count * subscription),
one CTA per tile. For 100K tokens: ~12 tiles all fit in one wave
on 16 SMs, explaining why no_delay (dcid=0) is optimal.
4. Lookahead's num_stages auto-tuning is irrelevant for BI-V100 but
reveals NVIDIA's pipeline depth selection strategy.
Read dispatch_reduce.cuh, kernel_reduce.cuh, agent_reduce.cuh,
tuning_reduce.cuh, and util_arch.cuh from cccl_upstream.
Key findings:
1. Reduce tile data is in REGISTERS, not SMEM. Our test_smem_safety
model (tile = threads * items * type_size) checks scale_mem_bound's
register-pressure cap, not actual SMEM usage. Real SMEM ≈ threads *
sizeof(AccumT), which is 2-8 KB, not 32-49 KB.
2. scale_mem_bound vs scale_reg_bound serve different purposes:
mem_bound allows items to 2x expand (for small types), reg_bound
does not. Both use 48KB as register-spill prevention, not SMEM.
3. Our float64 tuning (threads=384) may be too conservative. CCCL
SM100 uses threads=640 for float64 — this doesn't overflow SMEM
because SMEM is only used for BlockReduce communication.
4. paged_attn.py line 99 hardcodes use_v1=True, completely disabling
V2 partitioned attention. For 100K token sequences this is suboptimal.
5. _PARTITION_SIZE=512 is hardcoded, should be tunable via muh.
