bench_triton_prefill.py:
- Split --block into --block (BLOCK_M) and --block-n (BLOCK_N)
- Each (M, N, warps) combo triggers Triton JIT recompilation
- Enables finding asymmetric optima like M=64,N=32 that save SMEM
triton_flash_attention.py:
- Re-add 3 BI-V100 autotune configs (64x32, 32x64, 64x64 with warps=4)
- These were wrongly reverted in 8c1955d -- autotune is zero-risk
run_on_bi100.sh:
- Updated to use asymmetric block search
Unlike bench_bi100.py which called torch.sum() without injecting params:
- Directly invokes prefix_prefill._fwd_kernel Triton JIT kernel
- Each (BLOCK, NUM_WARPS) constexpr pair triggers Triton recompilation
into a different kernel binary — same mechanism as CCCL #define TUNE_*
- Combos that exceed SMEM fail at compile time (caught, reported as COMPILE FAIL)
- Measures actual kernel execution time per compiled variant
- Outputs speedup vs baseline (BLOCK=64, WARPS=4) in CCCL format
Search space: BLOCK=[16,32,64,128] × WARPS=[1,2,4,8] = 16 variants
Problem sizes: ctx_len=[128,512,2048,8192] (Qwen3.6 typical workloads)
Test tensors match Qwen3.6: head_dim=128, num_heads=64, num_kv_heads=8 (GQA)
Requires GPU — will error immediately if no CUDA device available.
Reports GPU properties (SM count, SMEM, VRAM) to confirm BI-V100 hardware.
Translates NVIDIA CCCL benchmark infrastructure to Iluvatar hardware:
- Extracts ALL %RANGE% parameter spaces from 95 CUB benchmark .cu files
- SMEM constraint pruning: eliminates 25-63% of invalid combos
- 6 hot-path algorithms with validated space sizes:
reduce=1044 scan=5.4M(pruned) topk=1698 transform=25920 for=566
- CCCL-compatible output format
- --prune-only works without GPU
- --update-schema writes best results back to muh/schema/*.yaml
- --smem-limit flag for 32KB vs 48KB investigation
muh_dispatch.py:
- Fix missing os/sys imports (was crashing on import)
- Fix SM count 50→16 (confirmed via ixsmi, matches hardware.cuh)
- Fix C++ struct name lookup to match actual tuning_reduce.cuh names:
bi100_plus_float32_o4, bi100_plus_float64_o4, bi100_plus_accum2_o4
(was: bi100_float32_plus_o4 — wrong name, would always fall through to default)
Dockerfile:
- Add COPY for prefix_prefill.py and muh_dispatch.py
- Deploy CCCL-tuned prefix_prefill.py into vllm attention ops
(BLOCK=64, NUM_WARPS=4 for BI-V100 SM=16)
- Deploy muh_dispatch.py into vllm package for type-dispatched kernel configs
- These files were written but never deployed — dead code until now
Impact: prefix_prefill.py deployment means the CCCL-derived block sizes
actually take effect at runtime. Previously the base image's original
prefix_prefill.py (BLOCK=128 for cc>=80, or 64 for cc<80) was used,
which is correct for BI-V100 but our version adds explicit SM=16
documentation and the path for future tuning.
From 1KB/55 lines (46× compression vs CCCL 70KB) to 203 lines:
- Add 11 type specialization structs (key=1,2,4,8 × accum=1,2,4,8)
- SM=16 tile maximization: k4_a4 hot path 100% SMEM (256*24*8=49152)
- k8_a8 also at 100% SMEM (192*16*16=49152)
- Delay halved for L2=6MB across all branches
- CCCL-matching ReduceByKeyPolicy struct with ReduceByKeyAlgorithm enum
- Dynamic SMEM fallback for unknown pair sizes
256 concurrent seqs risks OOM: worst case with long prompts in queue
can exhaust KV cache + activation memory. 32K batched-tokens prefill
activation ≈ 20GB competes with KV cache. 0.95 mem-util leaves only
5% headroom for spikes.
Conservative start: max-num-seqs=8 (8× improvement over baseline=1).
8 seqs × 2048 avg context × 80KB/token = 1.3GB KV cache, safe.
gpu-memory-utilization and max-num-batched-tokens restored to proven
baseline values.
Optimal max-num-seqs needs real-hardware sweep: 4→8→16→32→64→128.
The value where Output TPS plateaus (KV cache saturated) is the
answer. Can't determine this without Phanthy Cloud access.
computility-run.yaml:
max-num-seqs 1→256: benchmark sweeps [128,256] concurrent seqs,
current config processes 1 while 127 queue. KV cache budget:
256 seqs × 2048 tokens × 80KB/token = 41.9GB < 45GB available.
max-num-batched-tokens 8192→32768: support 256 concurrent prefills.
gpu-memory-utilization 0.9→0.95: provide KV cache headroom.
Dockerfile:
Deploy paged_attention_v2_triton.py to vllm package path so
try-triton-first logic in _custom_ops.py can find it. Falls back
to PyTorch V2 automatically if Triton V2 fails (SMEM/runtime).
muh/tuning/common.cuh:
scale_mem_bound max_smem now a parameter (default 48KB). Allows
policy_selectors to pass hw.max_shared_memory_per_block if actual
SMEM differs from CCCL 48KB assumption.
muh/tuning/tuning_transform.cuh:
bytes_in_flight 16KB→32KB. Old derivation used 900/50=18 GB/s/SM
(wrong, SM=16 confirmed). Actual per-SM BW = 56 GB/s.
32KB is estimate pending benchmark sweep.
SM count 50→16 corrections across all affected files.
CRITICAL FINDINGS from enginex-vllm-bi100-qwen36-main.zip analysis:
1. No .cu files — all CUDA kernels pre-compiled in ixf_F (ixformer.functions)
2. paged_attention_v2 is NotImplementedError, use_v1=True hardcoded
3. Real tuning surface: BLOCK/NUM_WARPS in Triton, BLOCK_SIZE_M/N/K in MoE
4. _custom_ops reports SMEM=32KB (not 48KB!) — needs hardware verification
5. muh strategy shifts from C++ injection to Python parameter optimization
6. CCCL methodology still applies but targets Triton kernels not CUB dispatch
588-line vllm model implementation based on qwen3_moe.py.
Bootstrap strategy: treat ALL layers as full attention (ignoring
linear_attention optimization). Correct but suboptimal.
Key adaptations:
- _get_text_config(): unwrap composite config -> text_config
- Shared expert support (shared_expert_intermediate_size)
- Skip linear attention weights (conv1d, delta_net, gated_delta)
- Skip vision encoder and MTP weights
- QK norm (Qwen3 style)
- Partial rotary embedding (rope_pct=0.25)
Includes deploy.sh and run_baseline.sh for server deployment.
vllm 0.6.3 KeyError on qwen3_5_moe model type.
Model is hybrid linear+full attention MoE with 256 experts (top-8).
enginex-vllm-bi100-qwen36-main.zip in repo likely contains the fix.
Previous version used base topk policy's bits (11 for key>=2B),
causing SMEM overflow: 512*4*key_size + 2048*4*batches > 49152.
Fix: force bits=8 (same as radix_sort decision for BI-V100).
SMEM: 512*4*key_size + 256*4*batches = manageable.
Also adds while-loop SMEM check on max_batches.
Detected by test_smem_safety.py: 3 overflows at key_size=2,4,8.
Registers all 26 CUB algorithms with metadata:
- 6 'injection' mode: have VLLM_INJECTION_POINTS (reduce/scan/topk/transform/batch_memcpy/for)
- 20 'library' mode: used via CCCL device API, no direct #define injection
- struct_mode: 'named' (bi100_* structs) vs 'inline' (policy_selector returns)
Also adds coverage reporting to generate_patches().
The previous version had a `portioned_smem_per_warp` field that doesn't
exist in CCCL. The actual CCCL RadixSortOnesweepPolicy has:
threads, items, store_algorithm, rank_algorithm, scan_algorithm,
rank_private_partitions, radix_bits
Also adds proper SMEM calculation:
total = max(keys_tile, values_tile, rank_smem) + offsets
with 2KB headroom for kernel stack/locals.
rank_private_partitions set to 1 to minimize SMEM pressure.
Replaces hand-written reduce_threads=512, reduce_items=16 with
_read_reduce_config(accum_size) that reads from tuning_reduce.cuh
via gen_patch.extract_bi100_structs().
Architecture change:
OLD: hand-write values in Python + verify_against_headers() asserts equal
NEW: _read_reduce_config() reads from C++ header (single source of truth)
Falls back to compiled-in defaults only when headers not on disk
(deployed container), with RuntimeWarning.
No hand-written tuning values remain in the normal code path.
verify_against_headers() removed — there is nothing to verify
when there is only one copy of the truth.
Defensive guard: if SMEM cap computes max_threads_by_smem < 32
(or rounds to 0), floor at 32 (one warp). Prevents launching
0 threads which is undefined behavior.
Adds verification that hand-written values in muh_dispatch.py
(reduce_threads=512, reduce_items=16, etc.) match the C++ headers
(bi100_float32_plus_o4 in tuning_reduce.cuh).
Previously: muh_dispatch.py had hand-coded values with no link to
the C++ source of truth. gen_patch.py reads from C++ headers,
but muh_dispatch.py was a separate copy that could diverge.
Now: verify_against_headers() calls gen_patch.extract_bi100_structs()
and compares. Self-test prints mismatches if any exist.
