ROOT CAUSE FOUND from competitor sub168 docker log comparison:
Sub168 (competitor, works):
- corex_gdn.py:56] Loaded fused CoreX GDN decode operator ✓
- corex_moe.py:339] Using CoreX fused MoE prefill operator ✓
- corex_fa2.py:333] Using CoreX FA2 packed prefill ✓
- NO NaN warnings, NO MoE fallback
- max_model_len=256000, gpu_mem=0.95, max_num_seqs=2 (yaml params work)
Sub509 (ours, broken):
- NaN in prefill GatedDeltaNet layer 0 (frac=0.9998) ✗
- FusedMoE native kernel failed, falling back to PyTorch ✗
- NO corex_gdn/corex_moe/corex_fa2 loading logs at all
- max_model_len=100000, gpu_mem=0.9, max_num_seqs=1 (yaml params ignored)
The pip install transformers==4.55.3 in patch_ops.sh was the likely cause:
it changed dependencies that broke corex kernel loading paths.
Without corex_gdn, GatedDeltaNet falls back to Python → NaN.
Without corex_moe, MoE falls back to PyTorch → 10x slower.
Fix: Remove pip install, use base image's transformers version.
Only register qwen3_5 config files without upgrading the package.
3342d18bcc