From 3342d18bcc4dc9a22c61db55d043a6019de13777 Mon Sep 17 00:00:00 2001
From: project6 <dev@project6.ai>
Date: Fri, 7 Aug 2026 10:02:09 +0000
Subject: [PATCH] =?UTF-8?q?fix(critical):=20remove=20pip=20install=20trans?=
 =?UTF-8?q?formers=20=E2=80=94=20was=20breaking=20corex=20kernel=20loading?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

ROOT CAUSE FOUND from competitor sub168 docker log comparison:

Sub168 (competitor, works):
  - corex_gdn.py:56] Loaded fused CoreX GDN decode operator ✓
  - corex_moe.py:339] Using CoreX fused MoE prefill operator ✓
  - corex_fa2.py:333] Using CoreX FA2 packed prefill ✓
  - NO NaN warnings, NO MoE fallback
  - max_model_len=256000, gpu_mem=0.95, max_num_seqs=2 (yaml params work)

Sub509 (ours, broken):
  - NaN in prefill GatedDeltaNet layer 0 (frac=0.9998) ✗
  - FusedMoE native kernel failed, falling back to PyTorch ✗
  - NO corex_gdn/corex_moe/corex_fa2 loading logs at all
  - max_model_len=100000, gpu_mem=0.9, max_num_seqs=1 (yaml params ignored)

The pip install transformers==4.55.3 in patch_ops.sh was the likely cause:
it changed dependencies that broke corex kernel loading paths.
Without corex_gdn, GatedDeltaNet falls back to Python → NaN.
Without corex_moe, MoE falls back to PyTorch → 10x slower.

Fix: Remove pip install, use base image's transformers version.
Only register qwen3_5 config files without upgrading the package.
---
 qwen3_6_scripts/patch_ops.sh                  | 32 +++++++++++++++----
 qwen3_6_scripts/patch_transformers_qwen3_5.py | 11 ++++++-
 2 files changed, 35 insertions(+), 8 deletions(-)

diff --git a/qwen3_6_scripts/patch_ops.sh b/qwen3_6_scripts/patch_ops.sh
index 7a1f077c..4b0e40c7 100755
--- a/qwen3_6_scripts/patch_ops.sh
+++ b/qwen3_6_scripts/patch_ops.sh
@@ -51,14 +51,32 @@ deploy() {
 
 # ============================================================
 # 1. Transformers: register Qwen3_5 / Qwen3_5_MoE model types
+#    CRITICAL: Do NOT pip install transformers — it breaks corex
+#    kernel dependencies. Competitor sub168's docker log shows
+#    corex_gdn/corex_moe/corex_fa2 all loaded successfully.
+#    Our sub509 failed to load any corex kernel.
+#    The pip install transformers==4.55.3 likely caused this.
 # ============================================================
-pip install transformers==4.55.3 -i https://pypi.tuna.tsinghua.edu.cn/simple 2>/dev/null || \
-pip install transformers==4.55.3 2>/dev/null || \
-echo "[patch_ops] WARNING: pip install transformers failed, using pre-installed version"
-cp -r ./qwen3_5 /usr/local/lib/python3.10/site-packages/transformers/models/
-cp -r ./qwen3_5_moe /usr/local/lib/python3.10/site-packages/transformers/models/
-python3 ./patch_transformers_qwen3_5.py
-echo "[patch_ops] transformers Qwen3_5 models installed"
+# Use base image transformers — just add config files
+TRANSFORMERS_MODELS=""
+for P in /usr/local/lib/python3.10/site-packages/transformers/models \
+         /usr/local/corex/lib/python3/dist-packages/transformers/models \
+         /usr/local/corex/lib64/python3/dist-packages/transformers/models; do
+    if [ -d "$P" ]; then
+        TRANSFORMERS_MODELS="$P"
+        break
+    fi
+done
+
+if [ -n "$TRANSFORMERS_MODELS" ]; then
+    cp -r ./qwen3_5 "$TRANSFORMERS_MODELS/"
+    cp -r ./qwen3_5_moe "$TRANSFORMERS_MODELS/"
+    python3 ./patch_transformers_qwen3_5.py 2>&1 || \
+        echo "[patch_ops] WARNING: patch_transformers failed (may work at runtime)"
+    echo "[patch_ops] transformers Qwen3_5 configs registered (no pip install)"
+else
+    echo "[patch_ops] WARNING: transformers/models not found — skipping config registration"
+fi
 
 # ============================================================
 # 2. Model registry: ensure qwen3_5 is registered in vllm
diff --git a/qwen3_6_scripts/patch_transformers_qwen3_5.py b/qwen3_6_scripts/patch_transformers_qwen3_5.py
index 85b81402..0ca6bcc2 100644
--- a/qwen3_6_scripts/patch_transformers_qwen3_5.py
+++ b/qwen3_6_scripts/patch_transformers_qwen3_5.py
@@ -12,7 +12,16 @@ Target: pip-installed transformers at /usr/local/lib/python3.10/site-packages/tr
 
 import sys
 
-TRANSFORMERS_ROOT = "/usr/local/lib/python3.10/site-packages/transformers"
+TRANSFORMERS_ROOT = None
+for _p in ["/usr/local/lib/python3.10/site-packages/transformers",
+           "/usr/local/corex/lib/python3/dist-packages/transformers",
+           "/usr/local/corex/lib64/python3/dist-packages/transformers"]:
+    import os
+    if os.path.isdir(_p):
+        TRANSFORMERS_ROOT = _p
+        break
+if TRANSFORMERS_ROOT is None:
+    TRANSFORMERS_ROOT = "/usr/local/lib/python3.10/site-packages/transformers"
 AUTO_CONFIG = f"{TRANSFORMERS_ROOT}/models/auto/configuration_auto.py"
 MODELS_INIT = f"{TRANSFORMERS_ROOT}/models/__init__.py"