Critical finding from reading fused_moe.py end-to-end:
The real decode bottleneck is NOT tuning parameters. It's:
1. 640+ ixformer kernel launches per decode step (64 MoE layers ×
~10 ops each). At target 395 TPS = 253K launches/second.
2. 192 torch.empty calls per step (3 intermediate caches × 64 layers).
3. Python-level dispatch overhead for each of these calls.
The BLOCK_SIZE_M heuristic is already reasonable (16 for decode).
The fused_moe Triton kernel is dead code — ixformer's C++ kernel
is called instead.
Actionable optimization: pre-allocate intermediate caches outside the
layer loop to eliminate 192 CUDA mallocs per decode step.
Source: vllm/model_executor/layers/fused_moe/fused_moe.py
vllm/_custom_ops.py (ixf_F.vllm_invoke_fused_moe_kernel)
cccl_upstream/cub/cub/device/dispatch/tuning/tuning_batch_memcpy.cuh
4e16133c7a