dylanyunlon
|
8a38c04b4c
|
[vllm] 3 个运行时 bug 修复: SMEM 32KB→48KB, NUM_WARPS 8→4, v2 归一化
基于完整读入 CCCL agent_reduce.cuh (412行) + vllm 运行时代码分析。
这些改动影响实际 kernel 执行，不是 tuning 参数。
1. _custom_ops.py: get_max_shared_memory 32KB → 49152 (48KB)
BI-V100 实际有 48KB SMEM (via ixsmi 确认)。
32KB 限制了 vllm/utils.py:get_max_shared_memory_bytes() 的返回值，
可能影响 Triton 编译器 SMEM budget 和 ixformer 内部 tile size 选择。
2. prefix_prefill.py: NUM_WARPS 8→4 for non-SM80 devices
BLOCK=64 时只有 64 行 query 要处理。8 warps = 256 threads，
64/256 = 0.25 rows/thread，大部分 thread 空闲浪费 register。
4 warps = 128 threads，64/128 = 0.5 rows/thread，更好的利用率。
同时用 if/else 结构替代三元表达式，为未来 BI-V100 特化留位置。
3. prefix_prefill.py: _fwd_kernel_flash_attn_v2 归一化 bug 修复
v2 kernel 的 acc_scale = alpha (不除 l_i_new)，
所以 acc 是未归一化的 softmax 加权和。
最后的 acc /= l_i[:, None] 被注释掉了 → 输出错误。
对比 v1 kernel: 用 p_scale=beta/l_i_new, acc_scale=l_i/l_i_new*alpha
在循环内做在线归一化，所以不需要最后除。
v2 的设计是 defer normalization → 最后必须除。
当前是 dead code (use_v1=True)，但修复后可以安全启用 v2 路径。
|
2026-08-04 12:26:24 +00:00 |
|
dylanyunlon
|
8c1955dc92
|
fix: revert invalid patches, add honest tuning surface assessment
REVERTED (invalid):
- paged_attn.py: restored use_v1=True hardcode. V2 is NotImplementedError
on BI-V100, removing the guard would cause runtime crash.
- fused_moe.py: BLOCK_SIZE_N/K changes reverted. ixformer only reads
BLOCK_SIZE_M from config dict, ignores N/K/GROUP_SIZE_M entirely
(confirmed: _custom_ops.py:774 only passes config['BLOCK_SIZE_M']).
- _custom_ops.py: SMEM change reverted pending hardware confirmation.
- triton_flash_attention.py: autotune configs reverted (will re-add properly).
- prefix_prefill.py: comment enhancement reverted (was harmless but noisy).
ADDED:
- TUNING_SURFACE_TRUTH.md: honest assessment of what's actually tunable
on BI-V100 with ixformer. Documents that bench_bi100.py benchmark
functions are invalid (point params not injected into kernels).
Actual tuning surface is 5 parameters, not dozens:
1. BLOCK_SIZE_M (fused_moe, passes to ixformer)
2. use_v1 threshold (hardcoded True, V2 unimplemented)
3. BLOCK/NUM_WARPS (prefix_prefill Triton JIT)
4. SMEM declaration (affects Triton compiler)
5. autotune config set (triton_flash_attention)
|
2026-08-03 10:34:28 +00:00 |
|
dylanyunlon
|
dc9ac0a757
|
feat(muh): apply CCCL-derived BI-V100 tuning to 5 vllm Python files
Applied via muh/vllm_bi100_patch.py --conservative:
1. paged_attn.py: removed use_v1=True hardcode, restored V1/V2 heuristic
with BI-V100 threshold (16384 vs default 8192). SM=16 favors V1 longer.
2. fused_moe.py: BLOCK_SIZE_K 32→64 (better memory coalescing with 900GB/s
BW), BLOCK_SIZE_N 32→64 for decode path. Qwen3.6 MoE: E≈128, topk=8.
3. _custom_ops.py: SMEM kept at 32KB (conservative mode, pending hardware
confirmation). Added diagnostic comment.
4. prefix_prefill.py: enhanced BI-V100 block config comment with SMEM
budget breakdown (BLOCK=64,N=64 → 48KB tight, N=32 → 32KB safe).
5. triton_flash_attention.py: added 2 BI-V100 autotune configs
(64x32 and 32x64) for SM=16 occupancy characteristics.
CCCL basis: cub/benchmarks/bench/ %RANGE% parameter spaces (reduce 1044
combos, scan 5.4M, topk 1698, transform 25920) → SMEM pruning → policy
selector logic from tuning_*.cuh.
Also includes muh/vllm_bi100_patch.py (713 lines) for reproducible
one-shot patching with --dry-run, --conservative, and --revert modes.
|
2026-08-03 10:27:10 +00:00 |
|
dylanyunlon
|
ef6abf3dc7
|
[DEPLOY] Complete submission: baseline + all optimizations
Adds ALL files needed for Dockerfile build:
- qwen3_6_scripts/ (baseline patches + our optimizations)
- vllm/ (full vllm package)
- paged_attention_v2_pytorch.py (V2 with single-bmm optimization)
- Dockerfile + computility-run.yaml
Our optimizations vs baseline:
1. paged_attn.py: pre-gathered context KV (eliminates 194 gather calls),
Triton try/fallback, V2 heuristic, threshold 32K→64K
2. paged_attention_v2_pytorch.py: fills NotImplementedError,
single-bmm Phase 1 (195 launches → 3)
3. patch_enable_triton.py: HAS_TRITON=True with safety fallback
4. patch_triton_tuning.py: BLOCK=64, NUM_WARPS=4 for BI-V100
5. computility-run.yaml: gpu-memory-utilization 0.9→0.95,
max-num-batched-tokens 8192→16384
This repo can now be submitted to dev.modelhub.org.cn as-is.
|
2026-07-30 16:06:20 +00:00 |
|