Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
- patch_ops.sh: build corex_gdn_chunk_recurrent.so alongside moe_index_combine
- qwen3_5.py: import corex_gdn_chunk_recurrent, use C++ version for prefill
chunks instead of Python _torch_chunk_gated_delta_rule
- C++ version from xllm upstream avoids Python loop overhead and has proper
fp32 accumulation (key for NaN prevention on BI-V100)
- Falls back to Python version if .so not available
Verified on real BI-V100:
flash_attn_func works with head_dim=256 (diff < 0.004, no NaN)
flash_attn_varlen_func works for variable-length batching
seq=1024: 1.7x faster than PyTorch matmul
The profiling-stage _run_sdpa_fallback now tries flash_attn_varlen_func
first, falls back to Python Q-tiling only on exception.
This addresses the 10-50x attention slowdown identified in the analysis:
Python Q-tiling: O(L^2) per-tile matmul in Python loop
flash_attn: fused kernel, O(L) memory, hardware-optimized
flash_attn_func WORKS with head_dim=256 on BI-V100!
This is the path to 10-50x attention speedup.
Tests: correctness vs ref, GQA, long seq, varlen, paged decode, perf.
The 10-50x slowdown is from bypassing ixformer SDPA and using Python
matmul fallback. Test if ixformer actually crashes on head_dim=256
or if the bypass was premature.
