Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
- patch_ops.sh: build corex_gdn_chunk_recurrent.so alongside moe_index_combine
- qwen3_5.py: import corex_gdn_chunk_recurrent, use C++ version for prefill
chunks instead of Python _torch_chunk_gated_delta_rule
- C++ version from xllm upstream avoids Python loop overhead and has proper
fp32 accumulation (key for NaN prevention on BI-V100)
- Falls back to Python version if .so not available
Verified on real BI-V100:
flash_attn_func works with head_dim=256 (diff < 0.004, no NaN)
flash_attn_varlen_func works for variable-length batching
seq=1024: 1.7x faster than PyTorch matmul
The profiling-stage _run_sdpa_fallback now tries flash_attn_varlen_func
first, falls back to Python Q-tiling only on exception.
This addresses the 10-50x attention slowdown identified in the analysis:
Python Q-tiling: O(L^2) per-tile matmul in Python loop
flash_attn: fused kernel, O(L) memory, hardware-optimized
flash_attn_func WORKS with head_dim=256 on BI-V100!
This is the path to 10-50x attention speedup.
Tests: correctness vs ref, GQA, long seq, varlen, paged decode, perf.
The 10-50x slowdown is from bypassing ixformer SDPA and using Python
matmul fallback. Test if ixformer actually crashes on head_dim=256
or if the bypass was premature.
Extracted torch_chunk_gated_delta_rule and torch_recurrent_gated_delta_rule
from xllm_latest/core/layers/npu_torch/qwen3_gated_delta_net_base.cpp.
Pure PyTorch C++ — no NPU/ACL deps, no custom CUDA kernels.
Same algorithm as our Python _torch_chunk_gated_delta_rule but
avoids Python interpreter overhead in the chunk loop.
Verify on real BI-V100: python3 verify_gdn_cpp.py
Verified on real BI-V100:
moe_compute_index: 11.48x speedup (0.035ms vs 0.397ms)
moe_combine_result: 2.66x speedup (0.022ms vs 0.059ms)
Integration:
- qwen3_5.py: import corex_moe_index_combine, use in prefill path
with _USE_COREX_MOE_INDEX_COMBINE flag (env BI100_MOE_COREX_INDEX_COMBINE)
Falls back to PyTorch argsort+bincount if .so unavailable
- patch_ops.sh: compile corex_moe_index_combine.cu during docker build
verify_topk_softmax.py results:
IDs match: True (0/32 mismatches)
Max weight diff: 0.00000003
Speedup: 2.54x vs PyTorch (0.025ms vs 0.064ms)
Previous disable was based on speculation, not measurement.
project_7 docker build succeeds on competition platform.
Diff was: 5 ENV lines + .dockerignore whitelist.
ENV lines may override base image paths or trigger patch_ops.sh failures.
.dockerignore whitelist may exclude files the build needs.
Now Dockerfile is byte-identical to project_7.
Sub 655 analysis: 634/881 connection errors (server crash during replay).
Root cause: max-model-len=256000 + gpu-memory-utilization=0.95 + max-num-seqs=2
caused OOM on long-context requests (128K+ tokens).
Changes:
- max-model-len: 256000 → 131072 (enough for replay, prevents OOM)
- gpu-memory-utilization: 0.95 → 0.90 (safety margin)
- max-num-seqs: 2 → 1 (avoid concurrent long-context OOM)
- max-num-batched-tokens: 4096 → 8192 (match proven config)
- BI100_MOE_COREX_TOPK_SOFTMAX=0 (CUB kernel causes garbled output on
BI-V100; PyTorch topk+softmax path is correct and fast enough)
Expected impact: server stays alive through entire replay+opencompass run.
Sub 655 successful requests had output_tps_avg=11.5 — the TPS is fine,
we just need the server to not crash.
2ac877ce removed ENV lines to match project_7, but our base image needs them.
Without PATH, python3 not found at /usr/local/corex/bin.
Without PYTHONPATH, cannot import vllm/torch from corex dist-packages.
Without LD_LIBRARY_PATH, .so libraries not found at link time.
Restores the proven Dockerfile from 07e8681e (sub 655 scoring commit).
