torch profiler confirmed: torch.mm launches Gemm_tcu_bi_kernel::gemm_h_h_tcu_25
which is BI-V100 TCU (Tensor Compute Unit) hardware-accelerated GEMM.
0.58ms per call vs our custom kernel 7.7ms — TCU is 13x faster.
Python for-loop overhead measured: 0.892 ms/expert = 7.1 ms for 8 experts.
This C++ dispatch eliminates that overhead while using the same TCU kernel.
Three entry points:
- moe_decode: full MoE forward (FC1 + SiLU*mul + FC2) for decode
- moe_prefill: group-by-expert MoE forward for prefill
- moe_expert_gemm_tcu: raw GEMM loop for benchmarking
