3342d18bcc4dc9a22c61db55d043a6019de13777
ROOT CAUSE FOUND from competitor sub168 docker log comparison: Sub168 (competitor, works): - corex_gdn.py:56] Loaded fused CoreX GDN decode operator ✓ - corex_moe.py:339] Using CoreX fused MoE prefill operator ✓ - corex_fa2.py:333] Using CoreX FA2 packed prefill ✓ - NO NaN warnings, NO MoE fallback - max_model_len=256000, gpu_mem=0.95, max_num_seqs=2 (yaml params work) Sub509 (ours, broken): - NaN in prefill GatedDeltaNet layer 0 (frac=0.9998) ✗ - FusedMoE native kernel failed, falling back to PyTorch ✗ - NO corex_gdn/corex_moe/corex_fa2 loading logs at all - max_model_len=100000, gpu_mem=0.9, max_num_seqs=1 (yaml params ignored) The pip install transformers==4.55.3 in patch_ops.sh was the likely cause: it changed dependencies that broke corex kernel loading paths. Without corex_gdn, GatedDeltaNet falls back to Python → NaN. Without corex_moe, MoE falls back to PyTorch → 10x slower. Fix: Remove pip install, use base image's transformers version. Only register qwen3_5 config files without upgrading the package.
project_6
Description
Languages
C++
41.8%
Cuda
31.6%
Python
22.2%
C
2.1%
CMake
1.1%
Other
1.1%