Two changes based on CCCL source reading:
1. PARTITION_SIZE 512→1024 in paged_attention_v2_pytorch.py
From dispatch_scan.cuh: grid_size = num_tiles = ceil(N / tile_size).
Optimal tile_size balances parallelism vs overhead:
- BI-V100: 16 SMs, max ~32 concurrent CTAs
- Need num_partitions >= 32 to fill one wave
- 100K tokens / 1024 = 98 partitions (3 waves) ✓
- 100K tokens / 512 = 195 partitions (6 waves) — twice the Phase 2 cost
Note: only affects V2 (PyTorch path). V1 (ixformer) has its own partition size.
2. Fix V2 import path in _custom_ops.py
paged_attention_v2_pytorch.py is in repo root, not vllm package.
Added sys.path manipulation to find it at runtime.
Also read: cccl_upstream/thrust/examples/expand.cu (variable-length
replication pattern — maps to GQA expansion, but our broadcast approach
is already more efficient than physical replication).
Source: cccl_upstream/cub/cub/device/dispatch/dispatch_scan.cuh lines 350-380
cccl_upstream/thrust/examples/expand.cu
44e4f6f947