Random CCCL pick: cub/cub/block/block_load_to_shared.cuh (340 lines, full read)
CCCL's BlockLoadToShared reveals three-tier hardware dispatch:
SM90+: cp.async.bulk (TMA) — one instruction copies entire tile
SM80+: cp.async.cg — 16B aligned async copy, bypasses L1
SM70-: manual gmem→reg→smem fallback (vec_load_t 16B chunks)
BI-V100 (non-NVIDIA) takes the fallback path. This explains why all
competitors are stuck at 1560 max (vs 8000 target) — no async copy
hardware acceleration.
Applied CCCL pre-allocation pattern to _run_sdpa_fallback:
- k_pos = torch.arange(q_len) computed once per sequence (was correct
already but now documented why via CCCL mbarrier_init-before-loop)
- Added note about CommitToken pattern for mask caching
Also confirmed: _Q_CHUNK=256 is reasonable for BI-V100 given
256 × 256 × 4B = 256KB attention matrix fits in available memory.
Base file modified: qwen3_6_scripts/xformers.py (deployed via patch_ops.sh)
8056641f08