CCCL saxpy.cu demonstrates the principle: fused operations should minimize
wasted work. The saxpy_fast (single transform) vs saxpy_slow (two transforms)
comparison shows that eliminating unnecessary memory round-trips is the
primary optimization lever for element-wise ops.
Applied to MoE: during decode, M=8 (max-num-seqs) × topk=8 = 64 tokens.
Old heuristic: numel≤64 → BLOCK_SIZE_M=32 → 2 tiles of 32, no waste.
But for smaller batches (M=1,2,4 × topk=8 = 8,16,32 tokens):
BLOCK_SIZE_M=32 → tile padding: 24/16/0 rows wasted per tile
BLOCK_SIZE_M=16 → tile padding: 8/0/0 rows wasted per tile
New heuristic adds a finer-grained tier:
numel ≤ 16 → BLOCK_SIZE_M = 16 (zero waste for ≤2 seqs)
numel ≤ 64 → BLOCK_SIZE_M = 32 (was: same, no change)
numel ≤ 1024 → BLOCK_SIZE_M = 64 (was: same, no change)
else → BLOCK_SIZE_M = 256 (was: same, no change)
ixformer only reads BLOCK_SIZE_M from the config dict. The 16→32 threshold
matters for low-batch decode on BI-V100 where 16 SMs benefit from more
tiles with less padding over fewer tiles with more padding.
Source: cccl_upstream/thrust/examples/saxpy.cu (fusion + waste minimization)
6bf73bdacb