1a4e100583
Source: cccl_upstream/cub/cub/agent/agent_reduce.cuh
agent_reduce.cuh has two data load paths:
1. Vectorized (ConsumeFullTile<CanVectorize=true>): loads float4/int4
when aligned, contiguous, trivially_relocatable, sizeof≤8
2. Scalar (ConsumeFullTile<CanVectorize=false>): striped access via
CacheModifiedInputIterator
PyTorch equivalent: .contiguous() enables vectorized GPU memory access.
Applied to decode KV gather:
- Added del statements for intermediate tensors (k_gathered, v_gathered)
to free GPU memory immediately — critical for 16-SM BI-V100 with tight
memory budget at seq_len=100K
- Documented the memory access pattern matching agent_reduce's approach
Also from dispatch_reduce.cuh GridEvenShare:
- Adaptive tile sizing for prefix attention context phase
- tile_sz computed from score tensor memory budget per sequence
- Decode (q_len=1) gets larger tiles, prefill gets smaller ones