6bf73bdacb
CCCL saxpy.cu demonstrates the principle: fused operations should minimize wasted work. The saxpy_fast (single transform) vs saxpy_slow (two transforms) comparison shows that eliminating unnecessary memory round-trips is the primary optimization lever for element-wise ops. Applied to MoE: during decode, M=8 (max-num-seqs) × topk=8 = 64 tokens. Old heuristic: numel≤64 → BLOCK_SIZE_M=32 → 2 tiles of 32, no waste. But for smaller batches (M=1,2,4 × topk=8 = 8,16,32 tokens): BLOCK_SIZE_M=32 → tile padding: 24/16/0 rows wasted per tile BLOCK_SIZE_M=16 → tile padding: 8/0/0 rows wasted per tile New heuristic adds a finer-grained tier: numel ≤ 16 → BLOCK_SIZE_M = 16 (zero waste for ≤2 seqs) numel ≤ 64 → BLOCK_SIZE_M = 32 (was: same, no change) numel ≤ 1024 → BLOCK_SIZE_M = 64 (was: same, no change) else → BLOCK_SIZE_M = 256 (was: same, no change) ixformer only reads BLOCK_SIZE_M from the config dict. The 16→32 threshold matters for low-batch decode on BI-V100 where 16 SMs benefit from more tiles with less padding over fewer tiles with more padding. Source: cccl_upstream/thrust/examples/saxpy.cu (fusion + waste minimization)