muh_dispatch.py:
- Fix missing os/sys imports (was crashing on import)
- Fix SM count 50→16 (confirmed via ixsmi, matches hardware.cuh)
- Fix C++ struct name lookup to match actual tuning_reduce.cuh names:
bi100_plus_float32_o4, bi100_plus_float64_o4, bi100_plus_accum2_o4
(was: bi100_float32_plus_o4 — wrong name, would always fall through to default)
Dockerfile:
- Add COPY for prefix_prefill.py and muh_dispatch.py
- Deploy CCCL-tuned prefix_prefill.py into vllm attention ops
(BLOCK=64, NUM_WARPS=4 for BI-V100 SM=16)
- Deploy muh_dispatch.py into vllm package for type-dispatched kernel configs
- These files were written but never deployed — dead code until now
Impact: prefix_prefill.py deployment means the CCCL-derived block sizes
actually take effect at runtime. Previously the base image's original
prefix_prefill.py (BLOCK=128 for cc>=80, or 64 for cc<80) was used,
which is correct for BI-V100 but our version adds explicit SM=16
documentation and the path for future tuning.
computility-run.yaml:
max-num-seqs 1→256: benchmark sweeps [128,256] concurrent seqs,
current config processes 1 while 127 queue. KV cache budget:
256 seqs × 2048 tokens × 80KB/token = 41.9GB < 45GB available.
max-num-batched-tokens 8192→32768: support 256 concurrent prefills.
gpu-memory-utilization 0.9→0.95: provide KV cache headroom.
Dockerfile:
Deploy paged_attention_v2_triton.py to vllm package path so
try-triton-first logic in _custom_ops.py can find it. Falls back
to PyTorch V2 automatically if Triton V2 fails (SMEM/runtime).
muh/tuning/common.cuh:
scale_mem_bound max_smem now a parameter (default 48KB). Allows
policy_selectors to pass hw.max_shared_memory_per_block if actual
SMEM differs from CCCL 48KB assumption.
muh/tuning/tuning_transform.cuh:
bytes_in_flight 16KB→32KB. Old derivation used 900/50=18 GB/s/SM
(wrong, SM=16 confirmed). Actual per-SM BW = 56 GB/s.
32KB is estimate pending benchmark sweep.
SM count 50→16 corrections across all affected files.
Two optimizations that target the real bottlenecks:
1. patch_enable_triton.py: Enable Triton Flash Attention for prefill
- Sets HAS_TRITON = True (was hardcoded False)
- Adds try/except wrapper in forward_prefix: tries Triton kernel first,
permanently falls back to PyTorch if it hangs or errors
- Combined with patch_triton_tuning.py (BLOCK=64, NUM_WARPS=4),
this keeps SMEM at 32KB ≤ 48KB limit
- If Triton works: 10-50x prefill speedup (GPU-parallel Flash Attention
vs Python for-loop)
- If Triton still hangs: auto-fallback, no worse than baseline
2. patch_vectorized_decode.py: Raise _PYTORCH_DECODE_THRESHOLD 32768 → 65536
- Compiled ixf_F.paged_attention_v1 is ~100x faster than Python fallback
- Baseline conservatively falls back at 32K, may work fine at 64K
- If v1 crashes at higher seq_lens, threshold can be lowered back
Why these matter (competition scoring):
Token吞吐加权值 = Output TPS × 16.796 + Input TPS × 2.799 + Cache TPS × 0.56
Prefill (Input TPS, 14% weight): _forward_prefix_pytorch is a Python
for-loop doing matmul+softmax per tile. Triton kernel does this in a
single GPU launch with Flash Attention online softmax.
Decode (Output TPS, 83% weight): Every seq_len between 32K-65K that
stays on compiled v1 instead of falling to Python saves ~100x per token.
Deploy order in Dockerfile:
1. patch_ops.sh (baseline functional patches)
2. patch_triton_tuning.py (BLOCK=64, NUM_WARPS=4)
3. patch_enable_triton.py (HAS_TRITON=True + try/fallback)
4. patch_vectorized_decode.py (threshold 32K → 64K)
The single biggest performance bottleneck in the baseline:
paged_attention_v2 = raise NotImplementedError()
paged_attn.py: use_v1 = True (hardcoded to avoid calling V2)
V1 limitation: processes entire KV sequence in one kernel launch.
For seq_len=100K, this is a single massive attention computation.
V2: splits into PARTITION_SIZE=512 chunks, runs them in parallel,
then reduces with log-sum-exp. 195 parallel partitions vs 1.
Implementation (paged_attention_v2_pytorch.py):
Phase 1: Per-partition attention
- For each (seq, head, partition): compute QK^T, softmax, weighted V sum
- Store partial: tmp_output, exp_sums, max_logits (per partition)
Phase 2: Cross-partition reduction (log-sum-exp)
- global_max = max(max_logits across partitions)
- rescale = exp(partition_max - global_max) × partition_exp_sum
- output = Σ (rescale / total_sum) × partition_output
This is the same algorithm as vllm's paged_attention_v2_kernel.cu:
- The reduction pattern is identical to CCCL's block_reduce_warp_reductions
(combine partial statistics from independent segments)
- The online softmax tiling is the same as Flash Attention's partitioning
Integration:
- patch_paged_attention_v2.py patches _custom_ops.py and paged_attn.py
- Removes use_v1=True hardcode → V2 used for seq_len > 8192
- Dockerfile adds the patch step
This is a PyTorch implementation (no CUDA compilation needed).
Next step: if /usr/local/corex/ has ixcc or nvcc-compatible compiler,
replace with compiled CUDA kernel for further speedup.
After reading the full baseline (enginex-vllm-bi100-qwen36-main.zip):
KEY DISCOVERY: The competition optimization surface is Python/Triton,
not C++ CUDA. There is no csrc/ directory. All CUDA kernels are
precompiled in vllm._C and ixformer .so files. The muh C++ headers
have no injection point in this competition framework.
What CAN be optimized:
1. Triton kernel parameters (prefix_prefill.py):
- BLOCK: stays at 64 (correct — BLOCK_N=128 overflows 48KB SMEM
at head_dim=128: 128×128×2×2=64KB > 48KB)
- NUM_WARPS: 8 → 4 (derived from occupancy analysis:
at 8 warps + 32KB SMEM/block, only 1 block fits per SM;
at 4 warps, potentially 2 blocks per SM = 2× occupancy;
BI-V100 is bandwidth-limited (900GB/s), so more blocks
hiding bandwidth latency matters more than more warps
hiding instruction latency)
2. computility-run.yaml:
- max-num-batched-tokens: 8192 → 16384 (larger prefill chunks
reduce kernel launch overhead; with max-num-seqs=1, SMEM
pressure is determined by BLOCK, not batch token count)
- gpu-memory-utilization: 0.9 → 0.95 (model uses ~17.5GB/GPU,
KV cache for 100K tokens ≈ 1.38GB, plenty of headroom)
3. Added Dockerfile with patch_triton_tuning.py step.
4. Analysis document in optimizations/prefix_prefill_patch.py
with full SMEM/register/occupancy derivation.
