build_unified_bridge.sh:
- set -euo → set -eo (avoid unbound var failures)
- Drop -ltorch_cuda -lc10_cuda (unavailable at Docker build time)
- Add -Wl,--unresolved-symbols=ignore-in-shared-libs
ixformer::infer symbols resolved at runtime via RTLD_GLOBAL preload
Dockerfile Step 6:
- Wrap in (... || echo non-fatal) so Docker build continues if bridge fails
ix_unified.py:
- 3-phase preload: lib*.so → _ixformer_torch*.so → remaining .so
- All loaded with ctypes.RTLD_GLOBAL so symbols visible to bridge
- Added /workspace and /home/dylan search paths
Verified on real machine: bridge compiles (272K), undefined symbols expected
until ixformer .so preloaded at runtime by ix_unified.py
Working commit (26e6cb40) uses:
bash ./patch_ops.sh 2>&1 | tee ... ; echo exit code
Current was:
bash ./patch_ops.sh (strict, any failure kills Docker build)
patch_ops.sh has set -euo pipefail internally, and some patches may
legitimately skip/fail on different base images. The tee+echo pattern
lets the build complete while logging any issues.
Root cause: patch_ops.sh deploys to VLLM_ROOT (found by importlib, typically
/usr/local/lib/python3.10/site-packages/vllm/) but runtime PYTHONPATH loads
/usr/local/corex/lib/python3/dist-packages/vllm/ first. The base image's
paged_attn.py calls context_attention_fwd (Triton kernel) which is undefined
on BI-V100 → NameError → AsyncEngineDeadError → all requests 503.
Fix: discover VLLM2 path and mirror ALL patched files (paged_attn.py,
qwen3_5.py, serving layer, corex .so, block overrides) to both installs.
Same pattern as Sub 520's working patch_ops.sh (db8e677b line 124-133).
Error: 'Given groups=1, weight [1,1,4], expected input [1,128,4099] to have 1 channels but got 128'
Root cause: kh_pad is (kd, N+pad) = (128, 4099), but weight was (1, 1, 4) with groups=1.
Conv1d requires in_channels == input_channels/groups, so 1 != 128/1.
Fix: expand weight to (kd, 1, conv_kernel_size) and use groups=kd for depthwise conv.
This matches the pattern in qwen3_5.py:212 (_causal_conv1d_fwd) which uses groups=channels.
This was the cause of 'evaluation failed' — GDN crash on first request killed the engine.
Root cause of Sub 520 output_tps=2.6 (vs Sub 168 output_tps=11.9):
- patch_xformers_sdpa_seq.py replaces ixformer flash attention with
pure PyTorch O(L^2) matmul+softmax serial implementation
- 32 full attention layers x every token = 4.6x slower
Sub 168 (base image) proof:
- output_tps_avg=11.9, output_tps_p50=13.0, output_tps_p90=18.1
- XFormers backend used WITHOUT any patches
- ixformer flash_attn works correctly on BI-V100
This commit: skip xformers patches in patch_ops.sh
Expected: output_tps should recover to ~11.9 (Sub 168 level)
Tested on real machine (cc-b2042074, BI-V100, IX-ML 3.2.3):
_moe_C.topk_softmax() → SUCCESS, correct output
Two fixes proven on hardware:
1. cuda_compat.h: WARP_SIZE=64 (BI-V100 warp is 64, not 32)
2. topk_softmax_kernels.cu: cub/block/block_reduce.cuh instead of cub/cub.cuh
(cub.cuh pulls radix_sort which has WARP_SIZE conflict)
Key finding: ixformer SDK on this base image does NOT have topk_softmax.
The ixformer::infer namespace from xllm's ixformer.h is for newer SDK.
We MUST compile our own _moe_C kernel — which now works.
Build flags (clang 16, ivcore10):
CUDA: -O3 -cl-fast-relaxed-math (NOT --use_fast_math)
C++: -O2 -std=c++17
Dockerfile simplified: 3 steps (was 6)
_custom_ops.py: _moe_C as Priority 0, in-place vllm API
3 changes that close the MoE performance gap:
1. _custom_ops.py: Add ix_bridge as Priority 0 for topk_softmax
- Before: tries our .cu kernel (fails) → PyTorch fallback (1-3 TPS)
- After: tries ix_bridge → ixformer::infer::topk_softmax() → FAST
- Call chain: _custom_ops.topk_softmax() → ix_bridge.topk_softmax()
→ ix_moe_bridge.so → ixformer::infer::topk_softmax()
2. Dockerfile: Add ix_moe_bridge.cpp precompile step
- This was the missing link: code existed but was never compiled
- Uses torch.utils.cpp_extension.load() to link against libixformer.so
3. upstream_ref sync from GitHub (cloned, not rewritten):
- xLLM-AI/xllm: ILU kernels + CUDA MoE + GDN fp32 state mgmt
- Deep-Spark/vllm: latest MoE kernel sources
ixformer's vllm_copy_cache (functions/vllm.py:249) iterates block_mapping
with .items() expecting a dict {src: [dst_list]}. But vllm 0.6.3 passes
a Tensor of shape [N,2]. Convert before calling.
Error: 'Tensor' object has no attribute 'items'
at ixformer/functions/vllm.py:249 in vllm_copy_cache
ixformer.functions exposes vllm_copy_cache and vllm_swap_blocks,
NOT copy_blocks/swap_blocks. Wrong function names crash engine
when prefix cache starts copying KV blocks (~9 min into eval).
Error was: AttributeError: module 'ixformer.functions' has no attribute 'copy_blocks'
at _custom_ops.py:1145 in copy_blocks
Root cause: base image paged_attn.py imports Triton context_attention_fwd
which does not exist on BI-V100 (no Triton). Our paged_attn.py replaces
it with PyTorch fallback but was NEVER deployed — missing from patch_ops.sh.
SYSTEM_DESIGN.md step 9 lists it, patch_ops.sh didn't have it.
Also deploys prefix_prefill.py as safety net.
Error was: paged_attn.py:203 NameError: name 'context_attention_fwd' is not defined
→ AsyncEngineDeadError → all requests 503
Three fixes for the three bugs in latest docker log:
1. corex_gdn.py REWRITTEN — interface now matches qwen3_5.py:
OLD: CoreXGDN(num_heads, head_dim, layer_idx, chunk_size, eps)
NEW: CoreXGDN(num_v_heads, num_k_heads, head_k_dim, head_v_dim, conv_kernel_size, layer_idx)
OLD forward: (q, k, v, gate, beta, conv_state, temporal_state, attn_metadata)
NEW forward: (hidden_states, attn_metadata, conv_state, temporal_state,
in_proj_qkv, in_proj_z, in_proj_b, in_proj_a,
conv1d_weight, A_log, dt_bias, norm, out_proj)
Fixes: 'CoreXGDN.__init__() got unexpected keyword argument num_v_heads'
2. serving_chat.py — engine death protection for multimodal:
When model has no multimodal_config, return 400 instead of passing image data
to engine (which causes permanent AsyncEngineDeadError).
Fixes: 'ValueError: You set image=0 but found 1 items'
3. patch_ops.sh — ALWAYS deploy our modules (base image has bugs):
- qwen3_5.py: ALWAYS deploy (base has NaN)
- corex_gdn/moe/fa2.py: ALWAYS deploy (base interface mismatch)
- corex_fa2.py was MISSING from base → now deployed
Reverted the NO-FALLBACK rewrite of corex_gdn.py and qwen3_5.py.
Policy: do NOT rewrite modules that already exist in base image or
upstream_ref. If an interface doesn't match, fix the interface call
site — don't rewrite the entire module in pure PyTorch.
Base image has corex_gdn.py, corex_moe.py, corex_fa2.py with C++
backends. The right approach is to match their __init__ signatures,
not replace them with slower Python reimplementations.
Engine crash: ValueError: You set image=0 (or defaulted to 1) in
--limit-mm-per-prompt, but found 1 items in the same prompt.
This kills the entire vLLM engine (AsyncEngineDeadError), making all
subsequent requests return 503. Competition sends image requests in
functional tests (d08/d09 multimodal).
Fix: --limit-mm-per-prompt image=5 allows up to 5 images per prompt.
Root cause from latest docker build log:
ValueError: You set image=0 in --limit-mm-per-prompt, but found 1 items
→ Engine background task crashes → AsyncEngineDeadError → all subsequent 503
Fixes:
1. computility-run.yaml: add --limit-mm-per-prompt image=1
Prevents multimodal ValueError from killing the engine process.
2. patch_ops.sh: DON'T overwrite base image's corex_gdn.py/corex_moe.py
Comp 168 log proves base image's corex modules work with libcorex_gdn.so.
Our overwrite broke CoreXGDN.__init__ (unexpected kwarg 'num_v_heads').
Only deploy ours if base has NO corex modules at all.
Also deploy corex_fa2.py if base lacks it.
3. qwen3_5.py: try multiple CoreXGDN init signatures
Base image CoreXGDN may accept different kwargs than ours.
Try kwargs form first, fall back to positional.
4. corex_gdn.py: accept both calling conventions in __init__
Future-proof for when we DO need to deploy ours.
5. Copied upstream_ref headers: ilu_layer_fused_moe.h, ilu_layer_attention.h
Last 2 missing ILU files from xllm. All 14/14 now present.
Base image qwen3_5.py (81706 bytes, 1777 lines) produces NaN frac=0.5000:
CoreXGDN.__init__() got unexpected keyword argument 'num_v_heads'
→ all GDN layers fallback to base PyTorch GDN → NaN
Our qwen3_5.py has the xllm-aligned GDN fix (cumsum + difference form).
verify_single_card.py confirmed ZERO NaN on real BI-V100.
Remove conditional deploy — always overwrite base qwen3_5.py.
Revert to the exact Dockerfile structure that built successfully on
the competition platform. Uses '; echo' pattern (not '&&') and
'| tee' for logging, matching the proven c2807549 submission.
Docker build fails if base image lacks gcc (needed for ex_registry.c)
and ninja (needed for torch.utils.cpp_extension). Install both in a
dedicated RUN layer before build.sh and patch_ops.sh.
