fix(critical): remove pip install transformers — was breaking corex kernel loading

ROOT CAUSE FOUND from competitor sub168 docker log comparison:

Sub168 (competitor, works):
  - corex_gdn.py:56] Loaded fused CoreX GDN decode operator ✓
  - corex_moe.py:339] Using CoreX fused MoE prefill operator ✓
  - corex_fa2.py:333] Using CoreX FA2 packed prefill ✓
  - NO NaN warnings, NO MoE fallback
  - max_model_len=256000, gpu_mem=0.95, max_num_seqs=2 (yaml params work)

Sub509 (ours, broken):
  - NaN in prefill GatedDeltaNet layer 0 (frac=0.9998) ✗
  - FusedMoE native kernel failed, falling back to PyTorch ✗
  - NO corex_gdn/corex_moe/corex_fa2 loading logs at all
  - max_model_len=100000, gpu_mem=0.9, max_num_seqs=1 (yaml params ignored)

The pip install transformers==4.55.3 in patch_ops.sh was the likely cause:
it changed dependencies that broke corex kernel loading paths.
Without corex_gdn, GatedDeltaNet falls back to Python → NaN.
Without corex_moe, MoE falls back to PyTorch → 10x slower.

Fix: Remove pip install, use base image's transformers version.
Only register qwen3_5 config files without upgrading the package.
This commit is contained in:
project6
2026-08-07 10:02:09 +00:00
parent 4e04674283
commit 3342d18bcc
2 changed files with 35 additions and 8 deletions

View File

@@ -51,14 +51,32 @@ deploy() {
# ============================================================
# 1. Transformers: register Qwen3_5 / Qwen3_5_MoE model types
# CRITICAL: Do NOT pip install transformers — it breaks corex
# kernel dependencies. Competitor sub168's docker log shows
# corex_gdn/corex_moe/corex_fa2 all loaded successfully.
# Our sub509 failed to load any corex kernel.
# The pip install transformers==4.55.3 likely caused this.
# ============================================================
pip install transformers==4.55.3 -i https://pypi.tuna.tsinghua.edu.cn/simple 2>/dev/null || \
pip install transformers==4.55.3 2>/dev/null || \
echo "[patch_ops] WARNING: pip install transformers failed, using pre-installed version"
cp -r ./qwen3_5 /usr/local/lib/python3.10/site-packages/transformers/models/
cp -r ./qwen3_5_moe /usr/local/lib/python3.10/site-packages/transformers/models/
python3 ./patch_transformers_qwen3_5.py
echo "[patch_ops] transformers Qwen3_5 models installed"
# Use base image transformers — just add config files
TRANSFORMERS_MODELS=""
for P in /usr/local/lib/python3.10/site-packages/transformers/models \
/usr/local/corex/lib/python3/dist-packages/transformers/models \
/usr/local/corex/lib64/python3/dist-packages/transformers/models; do
if [ -d "$P" ]; then
TRANSFORMERS_MODELS="$P"
break
fi
done
if [ -n "$TRANSFORMERS_MODELS" ]; then
cp -r ./qwen3_5 "$TRANSFORMERS_MODELS/"
cp -r ./qwen3_5_moe "$TRANSFORMERS_MODELS/"
python3 ./patch_transformers_qwen3_5.py 2>&1 || \
echo "[patch_ops] WARNING: patch_transformers failed (may work at runtime)"
echo "[patch_ops] transformers Qwen3_5 configs registered (no pip install)"
else
echo "[patch_ops] WARNING: transformers/models not found — skipping config registration"
fi
# ============================================================
# 2. Model registry: ensure qwen3_5 is registered in vllm

View File

@@ -12,7 +12,16 @@ Target: pip-installed transformers at /usr/local/lib/python3.10/site-packages/tr
import sys
TRANSFORMERS_ROOT = "/usr/local/lib/python3.10/site-packages/transformers"
TRANSFORMERS_ROOT = None
for _p in ["/usr/local/lib/python3.10/site-packages/transformers",
"/usr/local/corex/lib/python3/dist-packages/transformers",
"/usr/local/corex/lib64/python3/dist-packages/transformers"]:
import os
if os.path.isdir(_p):
TRANSFORMERS_ROOT = _p
break
if TRANSFORMERS_ROOT is None:
TRANSFORMERS_ROOT = "/usr/local/lib/python3.10/site-packages/transformers"
AUTO_CONFIG = f"{TRANSFORMERS_ROOT}/models/auto/configuration_auto.py"
MODELS_INIT = f"{TRANSFORMERS_ROOT}/models/__init__.py"