muh-bot
ab81329cb4
feat(engine): CCCL system design integration into prefill + decode hot paths
...
Source input for this commit:
- CCCL bench/adjacent_difference/subtract_left.cu (randomly selected)
→ Learned: %RANGE% parameter search + policy_selector_t override pattern
- CCCL bench/reduce/sum.cu + base.cuh
→ Learned: scale_mem_bound adapts (threads, items, vec) to hardware
→ 3 search dims: ipt 7:24, tpb 128:1024, ipv 1:2
- CCCL bench/scan/exclusive/sum.cu
→ Learned: 7 search dims including delay_ns, L2_write_latency
→ This is why nobody wins by guessing — NVIDIA searches 7D space
- CCCL thrust/examples/summary_statistics.cu
→ Welford parallel merge = paged_attention_v2 partition merge pattern
- Base engine: vllm/worker/cache_engine.py (already has CCCL layout/slot)
- Base engine: vllm/attention/ops/paged_attn.py (V1/V2 dispatch)
- Base engine: vllm/attention/ops/prefix_prefill.py (Triton prefill)
Changes:
prefix_prefill.py:
- Replaced hardcoded BLOCK=64/NUM_WARPS=4 with CCCL-informed
SMEM-aware policy selection
- Documents the actual SMEM model: BLOCK_N * Lk * elem_bytes * 2
- For BI-V100: derives BLOCK from smem_limit dynamically
- NUM_WARPS follows CCCL pattern: fewer warps when SM count is low
- Search space documented: BLOCK ∈ {16,32,64}, NUM_WARPS ∈ {2,4,8}
paged_attn.py:
- Enriched _PARTITION_SIZE documentation with CCCL scan benchmark
7-dimensional parameter space reference
- Added scale_mem_bound analysis for future float16 vs float32
partition size differentiation
- Connected GridEvenShare dispatch to scan delay parameters
NOT changed (correctly):
- _PARTITION_SIZE value stays 512 (precompiled .so constraint)
- V1/V2 threshold logic stays max_num_partitions == 1
- These require .so recompilation to change
2026-08-07 02:42:08 +00:00
Claude
dd59ec95c2
[ENGINE] prefix_caching_block: CCCL DeviceCopy::Batched 3-phase swap_in/swap_out
...
Source: cccl_upstream/cub/test/catch2_test_device_copy_env.cu
Target: vllm/core/block/prefix_caching_block.py
CCCL system design applied:
- DeviceCopy::Batched separates index_to_ptr (offset collection),
get_size (range sizing), and kernel launch (execution) into 3 phases
- Applied to swap_in: Phase 1 classify, Phase 2 batch-allocate,
Phase 3 batch-assign block_ids
- Applied to swap_out: Phase 1 collect, Phase 2 batch-free
- Prevents evictor state corruption from interleaved alloc+assign
Also applied to paged_attn.py:
- V1/V2 dispatch: CCCL dispatch_reduce.cuh tile-capacity decision
replaces hardcoded max_seq_len<=8192
- Added BI-V100 GridEvenShare constants from grid_even_share.cuh
2026-08-06 04:12:19 +00:00
muh-pipeline
b6538fd10e
[BASE] vllm/attention/ops/paged_attn.py: fix num_kv_heads type annotation
...
Discovered by tracing call chain after reading CCCL catch2_test_block_reduce.cu
(randomly selected). The test covers multi-dim block configs (BlockDimX/Y/Z)
which maps to GQA group dimensions in attention.
Call chain trace:
xformers.py:__init__() builds self.head_mapping = tensor [num_heads]
xformers.py:forward() → PagedAttention.forward_decode(head_mapping=tensor)
paged_attn.py:forward_decode(num_kv_heads: int) ← WRONG TYPE ANNOTATION
_custom_ops.py:paged_attention_v1(head_mapping=tensor) ← expects tensor
The parameter is head_mapping tensor for V1 (ixformer precompiled),
but int num_kv_heads for V2 (our PyTorch implementation).
Fixed annotation to remove misleading int type hint.
CCCL source read: cub/test/catch2_test_block_reduce.cu (252 lines, full)
Base file modified: vllm/attention/ops/paged_attn.py
2026-08-06 02:28:12 +00:00
muh-engine
29f119c094
[ENGINE] vllm/attention/ops/paged_attn.py: CCCL block_reduce_raking V1/V2 dispatch
...
FIXED BASE FILE (not root custom file):
vllm/attention/ops/paged_attn.py — the actual vllm paged attention
Two changes from reading cub/block/specializations/block_reduce_raking.cuh:
1. V1/V2 dispatch restored (was hardcoded use_v1=True on line 119)
CCCL block_reduce_raking has WARP_SYNCHRONOUS conditional fast path:
when RAKING_THREADS == BLOCK_THREADS, skip SMEM and go to warp shuffle.
This is CONDITIONAL — not hardcoded. Our equivalent:
V1 (single-pass) is the WARP_SYNCHRONOUS fast path for short seqs.
V2 (partitioned reduce) is the raking path for long seqs.
For max_num_seqs=1: num_seqs*num_heads=24 < 512, so V2 triggers
when max_seq_len > 8192.
2. V2 temp tensor caching (agent_merge_sort union _TempStorage pattern)
Cache tmp_output/exp_sums/max_logits by shape key across decode steps.
For max_num_seqs=1, shapes are stable → zero CUDA malloc after warmup.
CCCL files: cub/block/specializations/block_reduce_raking.cuh,
cub/agent/agent_merge_sort.cuh
2026-08-06 01:18:39 +00:00
dylanyunlon
8c1955dc92
fix: revert invalid patches, add honest tuning surface assessment
...
REVERTED (invalid):
- paged_attn.py: restored use_v1=True hardcode. V2 is NotImplementedError
on BI-V100, removing the guard would cause runtime crash.
- fused_moe.py: BLOCK_SIZE_N/K changes reverted. ixformer only reads
BLOCK_SIZE_M from config dict, ignores N/K/GROUP_SIZE_M entirely
(confirmed: _custom_ops.py:774 only passes config['BLOCK_SIZE_M']).
- _custom_ops.py: SMEM change reverted pending hardware confirmation.
- triton_flash_attention.py: autotune configs reverted (will re-add properly).
- prefix_prefill.py: comment enhancement reverted (was harmless but noisy).
ADDED:
- TUNING_SURFACE_TRUTH.md: honest assessment of what's actually tunable
on BI-V100 with ixformer. Documents that bench_bi100.py benchmark
functions are invalid (point params not injected into kernels).
Actual tuning surface is 5 parameters, not dozens:
1. BLOCK_SIZE_M (fused_moe, passes to ixformer)
2. use_v1 threshold (hardcoded True, V2 unimplemented)
3. BLOCK/NUM_WARPS (prefix_prefill Triton JIT)
4. SMEM declaration (affects Triton compiler)
5. autotune config set (triton_flash_attention)
2026-08-03 10:34:28 +00:00
dylanyunlon
dc9ac0a757
feat(muh): apply CCCL-derived BI-V100 tuning to 5 vllm Python files
...
Applied via muh/vllm_bi100_patch.py --conservative:
1. paged_attn.py: removed use_v1=True hardcode, restored V1/V2 heuristic
with BI-V100 threshold (16384 vs default 8192). SM=16 favors V1 longer.
2. fused_moe.py: BLOCK_SIZE_K 32→64 (better memory coalescing with 900GB/s
BW), BLOCK_SIZE_N 32→64 for decode path. Qwen3.6 MoE: E≈128, topk=8.
3. _custom_ops.py: SMEM kept at 32KB (conservative mode, pending hardware
confirmation). Added diagnostic comment.
4. prefix_prefill.py: enhanced BI-V100 block config comment with SMEM
budget breakdown (BLOCK=64,N=64 → 48KB tight, N=32 → 32KB safe).
5. triton_flash_attention.py: added 2 BI-V100 autotune configs
(64x32 and 32x64) for SM=16 occupancy characteristics.
CCCL basis: cub/benchmarks/bench/ %RANGE% parameter spaces (reduce 1044
combos, scan 5.4M, topk 1698, transform 25920) → SMEM pruning → policy
selector logic from tuning_*.cuh.
Also includes muh/vllm_bi100_patch.py (713 lines) for reproducible
one-shot patching with --dry-run, --conservative, and --revert modes.
2026-08-03 10:27:10 +00:00
dylanyunlon
ef6abf3dc7
[DEPLOY] Complete submission: baseline + all optimizations
...
Adds ALL files needed for Dockerfile build:
- qwen3_6_scripts/ (baseline patches + our optimizations)
- vllm/ (full vllm package)
- paged_attention_v2_pytorch.py (V2 with single-bmm optimization)
- Dockerfile + computility-run.yaml
Our optimizations vs baseline:
1. paged_attn.py: pre-gathered context KV (eliminates 194 gather calls),
Triton try/fallback, V2 heuristic, threshold 32K→64K
2. paged_attention_v2_pytorch.py: fills NotImplementedError,
single-bmm Phase 1 (195 launches → 3)
3. patch_enable_triton.py: HAS_TRITON=True with safety fallback
4. patch_triton_tuning.py: BLOCK=64, NUM_WARPS=4 for BI-V100
5. computility-run.yaml: gpu-memory-utilization 0.9→0.95,
max-num-batched-tokens 8192→16384
This repo can now be submitted to dev.modelhub.org.cn as-is.
2026-07-30 16:06:20 +00:00
