CCCL source read: cub/device/dispatch/dispatch_reduce_by_key.cuh
- DeviceReduceByKey sorts input by key, pads to tile boundary, then
one fused kernel processes all key-value segments in parallel.
- This is architecturally identical to base engine's fused_moe.py:
moe_align_block_size (sort+pad) → invoke_fused_moe_kernel (one launch).
Discovery: _custom_ops.py (line 776-806) confirms ixformer HAS native MoE:
- ixf_F.vllm_moe_topk_softmax
- ixf_F.vllm_moe_align_block_size
- ixf_F.vllm_invoke_fused_moe_kernel (takes only BLOCK_SIZE_M config)
Previous code assumed 'ixformer lacks MoE kernels' and used _pure_pytorch_experts
(Python for-loop over 256 experts). This may have been wrong or outdated.
Change: MoeSparseBlock.forward now tries self.experts (FusedMoE native) first.
If the native kernel fails on BI-V100, it catches the exception, logs a warning,
and permanently falls back to _pure_pytorch_experts for that instance.
Impact if native works: one fused CUDA kernel vs 256× F.linear calls = massive
decode speedup. Impact if native fails: same behavior as before (fallback).
Qwen3.6-35B-A3B has 256 experts × top_k=8. The baseline prefill MoE:
for eid in unique_eids: # up to 256 iterations
tokens = hidden_states[tok_ids] # SCATTERED gather
F.linear(tokens, w13[eid])
Problem: hidden_states[tok_ids] creates a non-contiguous gather for each expert.
With 16384 tokens × 256 experts, this is 256 scattered gathers per layer.
Optimization (CCCL segmented-sort pattern):
1. Flatten all token-expert pairs: (T×K,) assignments
2. Sort by expert ID: tokens for same expert become CONTIGUOUS
3. Each F.linear gets contiguous input → much better memory access
4. Activation (silu × up) computed in ONE fused op across all pairs
5. index_add_ scatter-back is one kernel call
Memory access improvement:
Before: 256 × hidden_states[random_indices] → scattered HBM reads
After: sorted_tokens[start:end] → sequential HBM reads per expert
The expert loop still exists (can't batch variable-size GEMMs with F.linear),
but each iteration reads contiguous memory instead of scattered indices.
