Source: cccl_upstream/cub/cub/device/dispatch/dispatch_transform.cuh
(CacheAsyncConfiguration + spread_out_items_per_thread)
CCCL dispatch_transform.cuh insight: element-wise transforms have
deterministic output shapes. Cache output tensors to avoid cudaMalloc.
Quote from CCCL: 'This computation MUST NOT depend on runtime state
... since the result will be cached.'
Applied to:
1. GeluAndMul.forward_cuda — output tensor cached during decode
2. RMSNorm.forward_cuda — output tensor cached during decode
(64 layers × 2 norms/layer = 128 cudaMalloc eliminated per step)
SiluAndMul already had this pattern from previous commit.
BI-V100 has no async memory allocator — synchronous cudaMalloc blocks
the entire SM pipeline. Eliminating 128+ allocations per decode step
directly improves Output TPS (83% competition weight).
4ca0115af7