From a20e8614a470cc65e0814f1b061e7c61f522c682 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 7 Aug 2026 09:21:43 +0000 Subject: [PATCH] =?UTF-8?q?fix(critical):=20stop=20replacing=20base=20imag?= =?UTF-8?q?e=20compute=20files=20=E2=80=94=20use=20corex=20native=20kernel?= =?UTF-8?q?s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ROOT CAUSE OF ALL FAILURES: patch_ops.sh was replacing qwen3_5.py, _custom_ops.py, model_runner.py, xformers.py, paged_attn.py, prefix_prefill.py, logits_processor.py, sampler.py, arg_utils.py — killing base image's CoreX fused kernels. Evidence from competitor sub168 docker logs (d03 PASS in 2.12s): - 'Using fused CoreX GDN decode operator' (DeltaNet) - 'Using CoreX fused MoE prefill operator: tokens=4096, kernel=expert-grouped-wmma' - 'Using CoreX FA2 packed prefill: B=2 Hq=4 Hkv=1 D=256' - ZERO NaN warnings - Model weights: 17.35GB (full) Our sub509 (d03 FAIL in 49s): - 'NaN in prefill GatedDeltaNet layer 0 (frac=0.9998)' — 99.98% NaN! - 'FusedMoE native kernel failed, falling back to pure PyTorch' - No CoreX FA2 - Model weights: 16.23GB (incomplete — 1.1GB missing) CCCL design principle (dispatch_reduce_deterministic.cuh, transform.cu): Let the framework's policy_selector choose optimal kernel config per hardware — never hand-replace the dispatch layer. Now patch_ops.sh ONLY patches serving layer: - protocol.py, serving_chat.py, api_server.py, chat_utils.py, cli_args.py - qwen3coder_tool_parser.py (tool call XML parsing) - reasoning/ (think tag parsing) - registry.py (register Qwen3_5 model type) - transformers models (qwen3_5 config) Base image compute files PRESERVED: qwen3_5.py, _custom_ops.py, model_runner.py, xformers.py, paged_attn.py, prefix_prefill.py, logits_processor.py, sampler.py, arg_utils.py, sequence.py, scheduler.py --- qwen3_6_scripts/patch_ops.sh | 128 ++++++++++++++--------------------- 1 file changed, 50 insertions(+), 78 deletions(-) diff --git a/qwen3_6_scripts/patch_ops.sh b/qwen3_6_scripts/patch_ops.sh index e52e21c0..51e30902 100755 --- a/qwen3_6_scripts/patch_ops.sh +++ b/qwen3_6_scripts/patch_ops.sh @@ -2,23 +2,28 @@ set -eo pipefail # BI-V100 engine patches for Qwen3.6-35B-A3B (Qwen3_5 architecture) # -# All modifications are FULL FILE REPLACEMENTS — no AST patch scripts. -# Each file was read in full from the base image vllm source, modified -# with the necessary fixes, and placed here as a complete copy. +# STRATEGY: Only patch serving/protocol layer. NEVER replace core compute +# files (qwen3_5.py model, _custom_ops.py, model_runner.py, xformers.py, +# paged_attn.py, prefix_prefill.py, logits_processor.py, sampler.py). # -# Base image: git.modelhub.org.cn:9443/enginex-iluvatar/bi100-3.2.3-x86-ubuntu20.04-py3.10-poc-llm-infer:v1.2.3 -# vllm install path: /usr/local/corex/lib/python3/dist-packages/vllm/ +# The base image has optimized CoreX kernels: +# - corex_gdn.py — fused GatedDeltaNet (decode + prefill) +# - corex_moe.py — fused MoE (expert-grouped-wmma) +# - corex_fa2.py — FlashAttention2 (packed prefill + paged chunked) +# Replacing model files breaks these kernel paths and causes: +# - DeltaNet NaN (99.98% of activations) → model output garbage +# - MoE fallback to pure PyTorch → 10x slower +# - FA2 → XFormers fallback → slower attention +# +# Reference: competitor sub168 uses base image qwen3_5.py + these CoreX +# kernels and achieves d03_tool_call in 2.12s (vs our sub509's 49s FAIL). -# CRITICAL: cd into this script's directory so all ./relative paths work -# regardless of WORKDIR in Dockerfile or caller's cwd. cd "$(dirname "$0")" echo "[patch_ops] working directory: $(pwd)" VLLM=/usr/local/corex/lib/python3/dist-packages/vllm VLLM64=/usr/local/corex/lib64/python3/dist-packages/vllm -# Deploy to ALL existing vllm paths — Python may load from either one -# depending on PYTHONPATH ordering and namespace package resolution. TARGETS=() if [ -d "$VLLM" ]; then TARGETS+=("$VLLM") @@ -34,10 +39,9 @@ fi echo "[patch_ops] vllm paths found: ${TARGETS[*]}" -# Helper: copy file to all target vllm roots deploy() { local src="$1" - local rel_dst="$2" # relative path within vllm, e.g. "attention/ops/paged_attn.py" + local rel_dst="$2" for V in "${TARGETS[@]}"; do local dst="$V/$rel_dst" mkdir -p "$(dirname "$dst")" @@ -45,50 +49,9 @@ deploy() { done } -# --- _custom_ops.py: SMEM 48KB fix + hardware ops bindings ------------------- -# Base image returns 32KB (32768) for get_max_shared_memory_per_block, but -# BI-V100 actually has 48KB (49152) confirmed via ixsmi. This limits Triton -# tile sizes and ixformer internal allocations if not corrected. -# CCCL GridEvenShare test (catch2_test_grid_even_share.cu) validates that -# work distribution depends on correct hardware parameters — wrong SMEM -# means wrong tile_size means wrong grid_size. -# FULL FILE REPLACEMENT. -deploy ./_custom_ops.py "_custom_ops.py" -echo "[patch_ops] _custom_ops.py → / (SMEM 32KB→48KB fix)" - -# --- paged_attn.py: pure-PyTorch attention fallback -------------------------- -deploy ./paged_attn.py "attention/ops/paged_attn.py" -echo "[patch_ops] paged_attn.py → attention/ops/" - -# --- prefix_prefill.py: Triton-free prefix attention ------------------------- -deploy ./prefix_prefill.py "attention/ops/prefix_prefill.py" -echo "[patch_ops] prefix_prefill.py → attention/ops/" - -# --- model_runner.py: prefix_cache_hit fix ----------------------------------- -deploy ./model_runner.py "worker/model_runner.py" -echo "[patch_ops] model_runner.py → worker/" - -# --- xformers.py: head_dim>128 fallback + Q-tiling -------------------------- -deploy ./xformers.py "attention/backends/xformers.py" -echo "[patch_ops] xformers.py → attention/backends/" - -# --- arg_utils.py: disable auto chunked-prefill for 32K+ -------------------- -deploy ./arg_utils.py "engine/arg_utils.py" -echo "[patch_ops] arg_utils.py → engine/" - -# --- logits_processor.py: seq_groups=None guard ------------------------------ -deploy ./logits_processor.py "model_executor/layers/logits_processor.py" -echo "[patch_ops] logits_processor.py → model_executor/layers/" - -# --- sampler.py: CCCL-ported top-k fast path for sampling -------------------- -deploy ./sampler.py "model_executor/layers/sampler.py" -echo "[patch_ops] sampler.py → model_executor/layers/" - -# --- transformers: Qwen3_5 tokenizer / model files -------------------------- -# NOTE: patch_transformers_qwen3_5.py is the ONLY remaining patch script. -# It modifies pip-installed transformers' configuration_auto.py and __init__.py -# to register qwen3_5/qwen3_5_moe. These files come from pip (version-specific) -# so we can't pre-copy them — the patch script inserts lines after known anchors. +# ============================================================ +# 1. Transformers: register Qwen3_5 / Qwen3_5_MoE model types +# ============================================================ pip install transformers==4.55.3 -i https://pypi.tuna.tsinghua.edu.cn/simple 2>/dev/null || \ pip install transformers==4.55.3 2>/dev/null || \ echo "[patch_ops] WARNING: pip install transformers failed, using pre-installed version" @@ -97,37 +60,24 @@ cp -r ./qwen3_5_moe /usr/local/lib/python3.10/site-packages/transformers/models/ python3 ./patch_transformers_qwen3_5.py echo "[patch_ops] transformers Qwen3_5 models installed" -# --- vllm model: Qwen3.6 (Qwen3_5 arch) ------------------------------------ -for V in "${TARGETS[@]}"; do - cp ./mamba_cache.py "$V/model_executor/models/" -done -deploy ./qwen3_5.py "model_executor/models/qwen3_5.py" +# ============================================================ +# 2. Model registry: ensure qwen3_5 is registered in vllm +# ============================================================ deploy ./registry.py "model_executor/models/registry.py" -echo "[patch_ops] qwen3_5.py + registry.py deployed" +echo "[patch_ops] registry.py deployed" -# --- paged_attention_v2_pytorch.py: PyTorch V2 attention fallback ------------ -for V in "${TARGETS[@]}"; do - cp ./paged_attention_v2_pytorch.py "$V/paged_attention_v2_pytorch.py" -done -cp ./paged_attention_v2_pytorch.py /workspace/paged_attention_v2_pytorch.py -echo "[patch_ops] paged_attention_v2_pytorch.py → all paths + /workspace/" +# ============================================================ +# 3. Serving layer patches (protocol, chat, tool parsing, reasoning) +# ============================================================ -# --- sequence.py: fix completion_tokens inflation ---------------------------- -deploy ./sequence.py "sequence.py" -echo "[patch_ops] sequence.py → /" - -# --- scheduler.py: record num_cached_tokens --------------------------------- -deploy ./scheduler.py "core/scheduler.py" -echo "[patch_ops] scheduler.py → core/" - -# --- tool parser: Qwen3 XML tool call format -------------------------------- +# --- Tool parser: Qwen3 XML tool call format --- for V in "${TARGETS[@]}"; do cp ./qwen3coder_tool_parser.py "$V/entrypoints/openai/tool_parsers/" cp ./tool_parsers_init.py "$V/entrypoints/openai/tool_parsers/__init__.py" done echo "[patch_ops] qwen3_coder tool parser deployed" -# --- reasoning parser: Qwen3 ... split ----------------------- +# --- Reasoning parser + serving files --- for V in "${TARGETS[@]}"; do cp -r ./reasoning "$V/" cp ./protocol.py "$V/entrypoints/openai/protocol.py" @@ -138,4 +88,26 @@ for V in "${TARGETS[@]}"; do done echo "[patch_ops] reasoning parser + serving files installed" -echo "[patch_ops] DONE — all patches applied via full file replacement" +# ============================================================ +# 4. DO NOT PATCH sequence.py or scheduler.py +# 168 (reference competitor) did not patch these. +# Our custom versions may conflict with base image internals. +# Token counting fixes are minor; NaN-free output is critical. +# ============================================================ + +# ============================================================ +# 5. DO NOT PATCH these files — base image has optimized versions: +# - qwen3_5.py (model) — has corex_gdn/corex_moe/corex_fa2 integration +# - _custom_ops.py — base image ixformer bindings +# - model_runner.py — base image worker +# - xformers.py — base image attention backend +# - paged_attn.py — base image paged attention +# - prefix_prefill.py — base image prefix prefill +# - logits_processor.py — base image logits +# - sampler.py — base image sampler +# - arg_utils.py — base image arg parsing +# - paged_attention_v2_pytorch.py — not needed with native kernels +# ============================================================ + +echo "[patch_ops] DONE — serving-layer-only patches applied" +echo "[patch_ops] Core compute files preserved from base image (corex_gdn + corex_moe + corex_fa2)"