Profiling OOM: flash_attn_varlen_func allocates large temp buffers during
profile_run with 8192 dummy tokens. patch_worker_profile_override.py already
has skip logic when num_gpu_blocks_override is set.
4000 blocks × 16 tokens/block = 64K token KV capacity.
With max-model-len=80000 and prefix caching, this is sufficient.
Sub 168 reference: ran 262K context on 0.95 util without override because
base image profiling doesn't use flash_attn_varlen (uses native xformers).
The CCCL preload allocator build frequently fails (CUB headers conflict
with CoreX CUDA 10.2). When the .so doesn't exist, LD_PRELOAD causes
the Python process to abort immediately, resulting in Connection Refused
for all 881 replay requests (0 score).
This was the root cause of sub 655's failure: service never started.
CCCL CachingDeviceAllocator reserves some memory for its bin cache.
With 0.90 utilization + max-model-len=80000, profiling stage OOMs.
0.85 leaves ~1.6GB headroom per GPU for profiling + allocator cache.
CoreX CUDACachingAllocator does not support expandable_segments.
Setting it causes ASSERT failure at startup:
'expandable_segments:True not supported on corex CUDACachingAllocator'
Replace with max_split_size_mb:512 only.
Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
Route C: replace PyTorch's cudaMalloc/cudaFree with CCCL CUB's
CachingDeviceAllocator via LD_PRELOAD. Eliminates driver-level allocation
overhead by reusing freed GPU memory from a bin-based cache.
Based on cccl_upstream/cub/cub/util_allocator.cuh (901 lines).
Self-contained .so with no CCCL header dependencies at compile time.
Files:
- cccl_preload_allocator.cu: the allocator (405 lines)
- build_cccl_preload_allocator.sh: build script (corex clang++ or g++ fallback)
- test_cccl_preload.sh: smoke test suite for BI-V100
- patch_ops.sh: build during docker build
- computility-run.yaml: LD_PRELOAD env var for runtime
Config via env:
CCCL_ALLOC_BIN_GROWTH=8, MIN_BIN=3, MAX_BIN=13, MAX_CACHED_MB=4096
Test on real machine:
cd qwen3_6_scripts && bash test_cccl_preload.sh
verify_topk_softmax.py results:
IDs match: True (0/32 mismatches)
Max weight diff: 0.00000003
Speedup: 2.54x vs PyTorch (0.025ms vs 0.064ms)
Previous disable was based on speculation, not measurement.
Sub 655 analysis: 634/881 connection errors (server crash during replay).
Root cause: max-model-len=256000 + gpu-memory-utilization=0.95 + max-num-seqs=2
caused OOM on long-context requests (128K+ tokens).
Changes:
- max-model-len: 256000 → 131072 (enough for replay, prevents OOM)
- gpu-memory-utilization: 0.95 → 0.90 (safety margin)
- max-num-seqs: 2 → 1 (avoid concurrent long-context OOM)
- max-num-batched-tokens: 4096 → 8192 (match proven config)
- BI100_MOE_COREX_TOPK_SOFTMAX=0 (CUB kernel causes garbled output on
BI-V100; PyTorch topk+softmax path is correct and fast enough)
Expected impact: server stays alive through entire replay+opencompass run.
Sub 655 successful requests had output_tps_avg=11.5 — the TPS is fine,
we just need the server to not crash.
Engine crash: ValueError: You set image=0 (or defaulted to 1) in
--limit-mm-per-prompt, but found 1 items in the same prompt.
This kills the entire vLLM engine (AsyncEngineDeadError), making all
subsequent requests return 503. Competition sends image requests in
functional tests (d08/d09 multimodal).
Fix: --limit-mm-per-prompt image=5 allows up to 5 images per prompt.
Root cause from latest docker build log:
ValueError: You set image=0 in --limit-mm-per-prompt, but found 1 items
→ Engine background task crashes → AsyncEngineDeadError → all subsequent 503
Fixes:
1. computility-run.yaml: add --limit-mm-per-prompt image=1
Prevents multimodal ValueError from killing the engine process.
2. patch_ops.sh: DON'T overwrite base image's corex_gdn.py/corex_moe.py
Comp 168 log proves base image's corex modules work with libcorex_gdn.so.
Our overwrite broke CoreXGDN.__init__ (unexpected kwarg 'num_v_heads').
Only deploy ours if base has NO corex modules at all.
Also deploy corex_fa2.py if base lacks it.
3. qwen3_5.py: try multiple CoreXGDN init signatures
Base image CoreXGDN may accept different kwargs than ours.
Try kwargs form first, fall back to positional.
4. corex_gdn.py: accept both calling conventions in __init__
Future-proof for when we DO need to deploy ours.
5. Copied upstream_ref headers: ilu_layer_fused_moe.h, ilu_layer_attention.h
Last 2 missing ILU files from xllm. All 14/14 now present.
Docker log proves two fatal issues:
1. max_model_len=100000 > KV cache capacity 88112 → ValueError crash
'max seq len (100000) is larger than maximum number of tokens
that can be stored in KV cache (88112)'
Fix: set max_model_len=80000 (safe margin below 88112)
2. NaN in GatedDeltaNet layers 34,36,37,38 (frac=1.0000)
Root cause: g.cumsum() → g.exp() overflow to inf → inf*0 = NaN
Fix: clamp all g values to [-80,80] before exp() calls
(max safe float32 exp input ~88, use 80 for margin)
Applied to: cumsum result, k_cumdecay, attn_inter, last_state update
3. CoreX modules confirmed NOT in base image:
'CoreX GDN module not found'
'CoreX MoE module not found'
→ pure PyTorch is the only path, must be numerically stable
Both base engine yaml and Sub168 use max-seq-len-to-capture=32768.
We were missing it.
Also fixed patch_ops.sh ending comments that claimed files were NOT
deployed when they actually ARE deployed.
CCCL buddy_allocator.cu teaches: control memory block fragmentation
at the allocator level. Sub168 OOM trace shows 'max_split_size_mb'
suggestion. Adding PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512
prevents PyTorch memory fragmentation that caused Sub168's final OOM.
OMP_NUM_THREADS=1 matches Sub168 docker log:
'Reducing Torch parallelism from 64 threads to 1'
CCCL device_reduce policy_selector pattern: hardware-adaptive params
through environment, not code changes. computility-run.yaml env vars
are the serving-safe equivalent of CCCL policy_selector.
THE OTHER CLAUDE'S COMMIT (5b8d7c6) IS WRONG. IT DEPLOYS ALL CUSTOM FILES.
Docker log evidence proves this is the root cause of ALL our failures:
Sub168 (07-23, PASS all d-tests):
corex_gdn.py:56 'Loaded fused CoreX GDN decode operator'
corex_moe.py:339 'Using CoreX fused MoE prefill: expert-grouped-wmma'
model_runner.py:1074 (BASE IMAGE native)
weights: 17.3529 GB
NaN: 0 times
Our Sub508 (08-07, 41.2%):
NO corex_gdn loading
model_runner.py:1119 (OUR CUSTOM — wrong)
weights: 16.2303 GB (1.1GB MISSING)
NaN: 16 times, FusedMoE fail: 19 times
Our custom qwen3_5.py REPLACES the base image's CoreX-accelerated model
with pure-PyTorch code that:
- Produces 99.98% NaN in every GatedDeltaNet layer
- Falls back to Python MoE loop (base image uses WMMA hardware)
- Loses 1.1GB of weights (broken load_weights function)
THIS COMMIT: deploy ONLY serving layer, keep base image model intact.
computility-run.yaml: exact Sub168 params (256K, 0.95, seqs=2, chunked).
1. computility-run.yaml: restore Sub168's proven params:
- max-model-len=256000 (not 100000)
- gpu-memory-utilization=0.95 (not 0.90)
- max-num-seqs=2 (not 1)
- max-num-batched-tokens=4096 (restored)
- enable-chunked-prefill (restored)
These params worked for Sub168. Now that pip install is removed,
they should work for us too.
2. patch_ops.sh: disable patch_numerical_stability.py
If corex_gdn loads (which it should without pip install breaking deps),
Python GatedDeltaNet fallback never runs, so numerical patches are
unnecessary. Running regex replacements on qwen3_5.py risks breaking
corex import conditions.
From competitor docker log analysis:
- max_model_len: 100000 → 256000 (competitor proven, 19259 GPU blocks)
- gpu_memory_utilization: 0.90 → 0.95 (competitor proven)
- max_num_batched_tokens: None → 4096 (competitor proven)
- enable_chunked_prefill: off → on (competitor proven, critical for 256K context)
- max_num_seqs stays at 2 (matches competitor)
Competitor Sub168 scored 60194 with these exact params before OOM at replay tail.
Our code has OOM-surviving advantages they lack (n>1 clamp, max_completion_tokens).
Docker log evidence: competitor's vLLM started with 19259 GPU blocks at 0.95 util,
ran for ~1h18m before OOM in layernorm.py x.float() at 31.72 GiB GPU.
Sub168 (competitor) passes t2_n_2 with n=2 at 1.50s even with
max_num_seqs likely >1. Our max_num_seqs=1 made n=2 crash.
Changes:
- computility-run.yaml: max-num-seqs 1→2 (200GB total VRAM sufficient)
- protocol.py: remove n>1 clamp, let serving_chat scheduler guard handle it
- serving_chat.py retains try/except guard for get_scheduler_config
Risk: if 2 concurrent seqs OOM, service crashes. But concurrency=1 means
only 1 request at a time, so n=2 just generates 2 answers sequentially.
CCCL input: tuning_topk.cuh (bits_per_pass=11 for float32, threads=512),
tuning_transform.cuh (cc_to_min_bytes_in_flight: B200=64KB, A100=16KB,
BI-V100 should use 48-64KB based on per-SM BW=56GB/s)
