feat(muh): apply CCCL-derived BI-V100 tuning to 5 vllm Python files
Applied via muh/vllm_bi100_patch.py --conservative: 1. paged_attn.py: removed use_v1=True hardcode, restored V1/V2 heuristic with BI-V100 threshold (16384 vs default 8192). SM=16 favors V1 longer. 2. fused_moe.py: BLOCK_SIZE_K 32→64 (better memory coalescing with 900GB/s BW), BLOCK_SIZE_N 32→64 for decode path. Qwen3.6 MoE: E≈128, topk=8. 3. _custom_ops.py: SMEM kept at 32KB (conservative mode, pending hardware confirmation). Added diagnostic comment. 4. prefix_prefill.py: enhanced BI-V100 block config comment with SMEM budget breakdown (BLOCK=64,N=64 → 48KB tight, N=32 → 32KB safe). 5. triton_flash_attention.py: added 2 BI-V100 autotune configs (64x32 and 32x64) for SM=16 occupancy characteristics. CCCL basis: cub/benchmarks/bench/ %RANGE% parameter spaces (reduce 1044 combos, scan 5.4M, topk 1698, transform 25920) → SMEM pruning → policy selector logic from tuning_*.cuh. Also includes muh/vllm_bi100_patch.py (713 lines) for reproducible one-shot patching with --dry-run, --conservative, and --revert modes.
This commit is contained in:
@@ -338,17 +338,23 @@ def get_default_config(
|
||||
dtype: Optional[str],
|
||||
is_marlin: bool,
|
||||
) -> Dict[str, int]:
|
||||
# muh: BI-V100 (SM=16, 48KB SMEM) aware defaults
|
||||
# Qwen3.6 MoE: E≈128, topk=8, K≈2048, N≈5504
|
||||
# SM=16 → fewer CTAs → each CTA should do more work → larger K tile
|
||||
# SMEM check: M=64 * K=64 * 2B(fp16) * 2(A+B) = 16KB < 48KB ✓
|
||||
config = {
|
||||
'BLOCK_SIZE_M': 64,
|
||||
'BLOCK_SIZE_N': 64,
|
||||
'BLOCK_SIZE_K': 32,
|
||||
'BLOCK_SIZE_K': 64, # muh: 32→64, better memory coalescing on BI-V100
|
||||
'GROUP_SIZE_M': 8
|
||||
}
|
||||
# A heuristic: fused marlin works faster with this config for small M
|
||||
if M <= E or (is_marlin and M <= 32):
|
||||
# muh: decode path (M=1 for single-token, M=8 for topk=8)
|
||||
# BI-V100: K=64 good for memory BW, N=64 for output tile
|
||||
config = {
|
||||
'BLOCK_SIZE_M': 16,
|
||||
'BLOCK_SIZE_N': 32,
|
||||
'BLOCK_SIZE_N': 64, # muh: 32→64, wider output tile
|
||||
'BLOCK_SIZE_K': 64,
|
||||
'GROUP_SIZE_M': 1
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user