Source: cccl_upstream/cub/cub/device/dispatch/dispatch_transform.cuh
(CacheAsyncConfiguration + spread_out_items_per_thread)
CCCL dispatch_transform.cuh insight: element-wise transforms have
deterministic output shapes. Cache output tensors to avoid cudaMalloc.
Quote from CCCL: 'This computation MUST NOT depend on runtime state
... since the result will be cached.'
Applied to:
1. GeluAndMul.forward_cuda — output tensor cached during decode
2. RMSNorm.forward_cuda — output tensor cached during decode
(64 layers × 2 norms/layer = 128 cudaMalloc eliminated per step)
SiluAndMul already had this pattern from previous commit.
BI-V100 has no async memory allocator — synchronous cudaMalloc blocks
the entire SM pipeline. Eliminating 128+ allocations per decode step
directly improves Output TPS (83% competition weight).
Source: cccl_upstream/cub/cub/device/dispatch/dispatch_transform.cuh
Target: vllm/model_executor/layers/activation.py
CCCL system design applied:
- dispatch_transform.cuh CacheAsyncConfiguration: cache occupancy/config
results across calls to avoid recomputation
- Applied: cache output tensor when shape/dtype/device unchanged
- BI-V100 has no async allocator → cudaMalloc is synchronous → caching
avoids blocking the stream on every decode step
- spread_out_items_per_thread: dynamic tile adjustment for occupancy
→ we only cache for stable decode shapes, not variable prefill
