bench_triton_prefill.py:
- Split --block into --block (BLOCK_M) and --block-n (BLOCK_N)
- Each (M, N, warps) combo triggers Triton JIT recompilation
- Enables finding asymmetric optima like M=64,N=32 that save SMEM
triton_flash_attention.py:
- Re-add 3 BI-V100 autotune configs (64x32, 32x64, 64x64 with warps=4)
- These were wrongly reverted in 8c1955d -- autotune is zero-risk
run_on_bi100.sh:
- Updated to use asymmetric block search
Unlike bench_bi100.py which called torch.sum() without injecting params:
- Directly invokes prefix_prefill._fwd_kernel Triton JIT kernel
- Each (BLOCK, NUM_WARPS) constexpr pair triggers Triton recompilation
into a different kernel binary — same mechanism as CCCL #define TUNE_*
- Combos that exceed SMEM fail at compile time (caught, reported as COMPILE FAIL)
- Measures actual kernel execution time per compiled variant
- Outputs speedup vs baseline (BLOCK=64, WARPS=4) in CCCL format
Search space: BLOCK=[16,32,64,128] × WARPS=[1,2,4,8] = 16 variants
Problem sizes: ctx_len=[128,512,2048,8192] (Qwen3.6 typical workloads)
Test tensors match Qwen3.6: head_dim=128, num_heads=64, num_kv_heads=8 (GQA)
Requires GPU — will error immediately if no CUDA device available.
Reports GPU properties (SM count, SMEM, VRAM) to confirm BI-V100 hardware.
Translates NVIDIA CCCL benchmark infrastructure to Iluvatar hardware:
- Extracts ALL %RANGE% parameter spaces from 95 CUB benchmark .cu files
- SMEM constraint pruning: eliminates 25-63% of invalid combos
- 6 hot-path algorithms with validated space sizes:
reduce=1044 scan=5.4M(pruned) topk=1698 transform=25920 for=566
- CCCL-compatible output format
- --prune-only works without GPU
- --update-schema writes best results back to muh/schema/*.yaml
- --smem-limit flag for 32KB vs 48KB investigation
From 1KB/55 lines (46× compression vs CCCL 70KB) to 203 lines:
- Add 11 type specialization structs (key=1,2,4,8 × accum=1,2,4,8)
- SM=16 tile maximization: k4_a4 hot path 100% SMEM (256*24*8=49152)
- k8_a8 also at 100% SMEM (192*16*16=49152)
- Delay halved for L2=6MB across all branches
- CCCL-matching ReduceByKeyPolicy struct with ReduceByKeyAlgorithm enum
- Dynamic SMEM fallback for unknown pair sizes
computility-run.yaml:
max-num-seqs 1→256: benchmark sweeps [128,256] concurrent seqs,
current config processes 1 while 127 queue. KV cache budget:
256 seqs × 2048 tokens × 80KB/token = 41.9GB < 45GB available.
max-num-batched-tokens 8192→32768: support 256 concurrent prefills.
gpu-memory-utilization 0.9→0.95: provide KV cache headroom.
Dockerfile:
Deploy paged_attention_v2_triton.py to vllm package path so
try-triton-first logic in _custom_ops.py can find it. Falls back
to PyTorch V2 automatically if Triton V2 fails (SMEM/runtime).
muh/tuning/common.cuh:
scale_mem_bound max_smem now a parameter (default 48KB). Allows
policy_selectors to pass hw.max_shared_memory_per_block if actual
SMEM differs from CCCL 48KB assumption.
muh/tuning/tuning_transform.cuh:
bytes_in_flight 16KB→32KB. Old derivation used 900/50=18 GB/s/SM
(wrong, SM=16 confirmed). Actual per-SM BW = 56 GB/s.
32KB is estimate pending benchmark sweep.
SM count 50→16 corrections across all affected files.
Previous version used base topk policy's bits (11 for key>=2B),
causing SMEM overflow: 512*4*key_size + 2048*4*batches > 49152.
Fix: force bits=8 (same as radix_sort decision for BI-V100).
SMEM: 512*4*key_size + 256*4*batches = manageable.
Also adds while-loop SMEM check on max_batches.
Detected by test_smem_safety.py: 3 overflows at key_size=2,4,8.
Registers all 26 CUB algorithms with metadata:
- 6 'injection' mode: have VLLM_INJECTION_POINTS (reduce/scan/topk/transform/batch_memcpy/for)
- 20 'library' mode: used via CCCL device API, no direct #define injection
- struct_mode: 'named' (bi100_* structs) vs 'inline' (policy_selector returns)
Also adds coverage reporting to generate_patches().
The previous version had a `portioned_smem_per_warp` field that doesn't
exist in CCCL. The actual CCCL RadixSortOnesweepPolicy has:
threads, items, store_algorithm, rank_algorithm, scan_algorithm,
rank_private_partitions, radix_bits
Also adds proper SMEM calculation:
total = max(keys_tile, values_tile, rank_smem) + offsets
with 2KB headroom for kernel stack/locals.
rank_private_partitions set to 1 to minimize SMEM pressure.
Defensive guard: if SMEM cap computes max_threads_by_smem < 32
(or rounds to 0), floor at 32 (one warp). Prevents launching
0 threads which is undefined behavior.
scale_mem_bound now returns {items, threads} (items-first) to match
CCCL's scaling_result struct. All 7 call sites in this file updated.
Previously: auto [t, i] bound threads→t, items→i
Now: auto [i, t] binds items→i, threads→t
The ReducePassPolicy{t, i, ...} constructors remain correct because
they take (threads, items, ...) — t is threads, i is items in both cases.
The old code worked by accident (two reversals canceling out).
1. Return order: {threads, items} → {items, threads} matching CCCL scaling_result
2. Upper clamp: nominal*1 → nominal*2 (CCCL allows small types to double items)
3. Add threads SMEM cap: min(nominal, round_up(48KB/(ts*items), 32))
Verified against all 8 test vectors from CCCL catch2_test_util_arch.cu.
The old code was only safe because current bi100_* structs don't hit the
edge cases — but any future CCCL code copy would silently produce wrong
values.
Audit results:
- 20/20 files had IDENTICAL if-branch and fallback (dead code)
- 787 lines total, 5% coverage of 15116 lines in CCCL originals
- No type specializations, no offset_size branches, no benchmark data
- 0 of 20 algorithms appear on vllm's Qwen3.6 inference hot path
The 6 headers that remain (reduce, topk, scan, transform, batch_memcpy, for)
are the only algorithms that execute during vllm decode/prefill/cache operations.
These 6 have real type specializations and CCCL SM100 reference values.
CCCL has 26 algorithms because it's a general-purpose library.
muh targets one workload: Qwen3.6-35B-A3B on 4× BI-V100.
Covering algorithms that don't execute is worse than not covering them —
it creates the illusion of completeness.
Problems fixed:
1. gen_patch.py was reading .muh YAML (all nulls) instead of C++ headers.
Now it parses bi100_* structs directly from tuning_*.cuh via regex,
extracts constexpr values, and maps them to vllm injection points.
Verified: 11 patches generated from 6 algorithms.
2. C++ headers had no build system or tests.
Added CMakeLists.txt (header-only library target) and compile_test.cpp.
Verified: g++ -std=c++17 compiles all headers, 17/17 runtime checks pass.
Also added cuda_compile_test.cu for when nvcc is available.
3. baseline.muh had a tuning section full of nulls duplicating C++ values.
Stripped to vllm launch config only. Tuning values live exclusively
in muh/include/muh/tuning/tuning_*.cuh bi100_* structs.
4. Fixed constexpr goto in tuning_scan.cuh (C++17 doesn't allow goto in
constexpr; replaced with early-return + default: break pattern).
Data flow is now:
tuning_*.cuh (bi100_* constexpr) ──→ gen_patch.py ──→ vllm patches
baseline.muh (launch config) ──→ gen_yaml.py ──→ computility-run.yaml
compile_test.cpp ──→ g++/nvcc ──→ verify values are real
