flash_attn_varlen_func allocates O(n²) temp memory for 4096 dummy tokens
during profile_run, causing OOM at gpu_memory_utilization=0.80.
BI100_IN_STARTUP_PROFILE=1 env var is already set by
patch_worker_startup_profile_guard.py during the synthetic forward pass.
Real inference requests still use flash_attn_varlen (much faster).
verify_submission.sh confirmed: CCCL preload compiles and loads on BI-V100.
Was removed in 8d2f30f0 due to crash concern, but the .so is built by
patch_ops.sh during docker build so it will always exist at runtime.
These 14 lines were added after a3c45d3b (last confirmed working build).
The cccl_preload build and corex extension compile steps may cause
docker build failure on the competition platform even with || fallback.
Reverting to the exact patch_ops.sh from a3c45d3b.
.cu and .sh source files remain in the repo for future use.
commit 4c365b8c added 1106 CCCL device-level headers (294K lines) to
qwen3_6_scripts/cccl_preload/include/. These are NOT used by the allocator
preload (which only needs cub/util_allocator.cuh + 288 transitive deps)
and cannot compile on corex CUDA 10.2 anyway.
The extra headers doubled docker context from 15MB to 31MB, likely
causing platform build timeout or size limit failure.
Restoring to the original 288-header set that is proven to compile and run.
Profiling OOM: flash_attn_varlen_func allocates large temp buffers during
profile_run with 8192 dummy tokens. patch_worker_profile_override.py already
has skip logic when num_gpu_blocks_override is set.
4000 blocks × 16 tokens/block = 64K token KV capacity.
With max-model-len=80000 and prefix caching, this is sufficient.
Sub 168 reference: ran 262K context on 0.95 util without override because
base image profiling doesn't use flash_attn_varlen (uses native xformers).
The CCCL preload allocator build frequently fails (CUB headers conflict
with CoreX CUDA 10.2). When the .so doesn't exist, LD_PRELOAD causes
the Python process to abort immediately, resulting in Connection Refused
for all 881 replay requests (0 score).
This was the root cause of sub 655's failure: service never started.
CCCL CachingDeviceAllocator reserves some memory for its bin cache.
With 0.90 utilization + max-model-len=80000, profiling stage OOMs.
0.85 leaves ~1.6GB headroom per GPU for profiling + allocator cache.
Two problems from real BI-V100 build:
1. 'CUDA versions below 12 are not supported'
→ Add CCCL_IGNORE_DEPRECATED_CUDA_BELOW_12 (official suppress macro)
2. corex thrust/complex.h conflicts with CCCL thrust headers
→ Split into two compilation units:
- cccl_moe_sort_scatter.cu: CCCL headers only, C API, no torch
- cccl_moe_sort_scatter_pybind.cpp: torch headers only, no CCCL
Same pattern as proven cccl_allocator_preload.cu
3. Variadic device functions rejected by corex clang:
→ is_referenceable.h: __test(...) → __test(long)
→ invoke.h: __any(...) → template __any(_T)
→ conjunction.h: __and_helper(...) → __and_helper(long)
SFINAE still works: int overload wins, long is fallback.
CCCL latest requires CUDA 12+, corex is 10.2. Device-level CUB headers
(DeviceRadixSort etc) pull in thrust/detail/type_traits.h which conflicts
with corex's thrust/complex.h namespace.
Rewrite to use block-level CUB BlockScan only (same pattern as the proven
corex_moe_index_combine.cu): histogram + prefix_sum + scatter.
No extra_include_paths needed — uses corex's built-in cub/block/block_scan.cuh.
CoreX CUDACachingAllocator does not support expandable_segments.
Setting it causes ASSERT failure at startup:
'expandable_segments:True not supported on corex CUDACachingAllocator'
Replace with max_split_size_mb:512 only.
CUB CachingDeviceAllocator::DeviceAllocate calls cudaMalloc internally
on cache miss. Without a guard, our intercepted cudaMalloc recurses
into DeviceAllocate → cudaMalloc → DeviceAllocate → segfault.
thread_local g_in_allocator flag detects reentrant calls and forwards
them directly to the real cudaMalloc/cudaFree via dlsym(RTLD_NEXT).
CUB_DISABLE_NAMESPACE_MAGIC requires CUB_WRAPPED_NAMESPACE.
CUB_WRAPPED_NAMESPACE=cccl_preload wraps cub into cccl_preload::cub.
Source must use cccl_preload::cub::CachingDeviceAllocator.
CUB_WRAPPED_NAMESPACE=cccl_preload wraps cub into cccl_preload::cub
but cccl_allocator_preload.cu uses bare cub:: — compilation fails.
_CCCL_COMPILER_GCC=1 conflicts with CCCL auto-detection (redefined warning).
Drop both. CUB_DISABLE_NAMESPACE_MAGIC alone is sufficient.
Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
- patch_ops.sh: build corex_gdn_chunk_recurrent.so alongside moe_index_combine
- qwen3_5.py: import corex_gdn_chunk_recurrent, use C++ version for prefill
chunks instead of Python _torch_chunk_gated_delta_rule
- C++ version from xllm upstream avoids Python loop overhead and has proper
fp32 accumulation (key for NaN prevention on BI-V100)
- Falls back to Python version if .so not available
