fix(critical): stop replacing base image compute files — use corex native kernels
ROOT CAUSE OF ALL FAILURES: patch_ops.sh was replacing qwen3_5.py, _custom_ops.py, model_runner.py, xformers.py, paged_attn.py, prefix_prefill.py, logits_processor.py, sampler.py, arg_utils.py — killing base image's CoreX fused kernels. Evidence from competitor sub168 docker logs (d03 PASS in 2.12s): - 'Using fused CoreX GDN decode operator' (DeltaNet) - 'Using CoreX fused MoE prefill operator: tokens=4096, kernel=expert-grouped-wmma' - 'Using CoreX FA2 packed prefill: B=2 Hq=4 Hkv=1 D=256' - ZERO NaN warnings - Model weights: 17.35GB (full) Our sub509 (d03 FAIL in 49s): - 'NaN in prefill GatedDeltaNet layer 0 (frac=0.9998)' — 99.98% NaN! - 'FusedMoE native kernel failed, falling back to pure PyTorch' - No CoreX FA2 - Model weights: 16.23GB (incomplete — 1.1GB missing) CCCL design principle (dispatch_reduce_deterministic.cuh, transform.cu): Let the framework's policy_selector choose optimal kernel config per hardware — never hand-replace the dispatch layer. Now patch_ops.sh ONLY patches serving layer: - protocol.py, serving_chat.py, api_server.py, chat_utils.py, cli_args.py - qwen3coder_tool_parser.py (tool call XML parsing) - reasoning/ (think tag parsing) - registry.py (register Qwen3_5 model type) - transformers models (qwen3_5 config) Base image compute files PRESERVED: qwen3_5.py, _custom_ops.py, model_runner.py, xformers.py, paged_attn.py, prefix_prefill.py, logits_processor.py, sampler.py, arg_utils.py, sequence.py, scheduler.py
This commit is contained in:
Claude
@@ -2,23 +2,28 @@
|
||||
set -eo pipefail
|
||||
# BI-V100 engine patches for Qwen3.6-35B-A3B (Qwen3_5 architecture)
|
||||
#
|
||||
# All modifications are FULL FILE REPLACEMENTS — no AST patch scripts.
|
||||
# Each file was read in full from the base image vllm source, modified
|
||||
# with the necessary fixes, and placed here as a complete copy.
|
||||
# STRATEGY: Only patch serving/protocol layer. NEVER replace core compute
|
||||
# files (qwen3_5.py model, _custom_ops.py, model_runner.py, xformers.py,
|
||||
# paged_attn.py, prefix_prefill.py, logits_processor.py, sampler.py).
|
||||
#
|
||||
# Base image: git.modelhub.org.cn:9443/enginex-iluvatar/bi100-3.2.3-x86-ubuntu20.04-py3.10-poc-llm-infer:v1.2.3
|
||||
# vllm install path: /usr/local/corex/lib/python3/dist-packages/vllm/
|
||||
# The base image has optimized CoreX kernels:
|
||||
# - corex_gdn.py — fused GatedDeltaNet (decode + prefill)
|
||||
# - corex_moe.py — fused MoE (expert-grouped-wmma)
|
||||
# - corex_fa2.py — FlashAttention2 (packed prefill + paged chunked)
|
||||
# Replacing model files breaks these kernel paths and causes:
|
||||
# - DeltaNet NaN (99.98% of activations) → model output garbage
|
||||
# - MoE fallback to pure PyTorch → 10x slower
|
||||
# - FA2 → XFormers fallback → slower attention
|
||||
#
|
||||
# Reference: competitor sub168 uses base image qwen3_5.py + these CoreX
|
||||
# kernels and achieves d03_tool_call in 2.12s (vs our sub509's 49s FAIL).
|
||||
|
||||
# CRITICAL: cd into this script's directory so all ./relative paths work
|
||||
# regardless of WORKDIR in Dockerfile or caller's cwd.
|
||||
cd "$(dirname "$0")"
|
||||
echo "[patch_ops] working directory: $(pwd)"
|
||||
|
||||
VLLM=/usr/local/corex/lib/python3/dist-packages/vllm
|
||||
VLLM64=/usr/local/corex/lib64/python3/dist-packages/vllm
|
||||
|
||||
# Deploy to ALL existing vllm paths — Python may load from either one
|
||||
# depending on PYTHONPATH ordering and namespace package resolution.
|
||||
TARGETS=()
|
||||
if [ -d "$VLLM" ]; then
|
||||
TARGETS+=("$VLLM")
|
||||
@@ -34,10 +39,9 @@ fi
|
||||
|
||||
echo "[patch_ops] vllm paths found: ${TARGETS[*]}"
|
||||
|
||||
# Helper: copy file to all target vllm roots
|
||||
deploy() {
|
||||
local src="$1"
|
||||
local rel_dst="$2" # relative path within vllm, e.g. "attention/ops/paged_attn.py"
|
||||
local rel_dst="$2"
|
||||
for V in "${TARGETS[@]}"; do
|
||||
local dst="$V/$rel_dst"
|
||||
mkdir -p "$(dirname "$dst")"
|
||||
@@ -45,50 +49,9 @@ deploy() {
|
||||
done
|
||||
}
|
||||
|
||||
# --- _custom_ops.py: SMEM 48KB fix + hardware ops bindings -------------------
|
||||
# Base image returns 32KB (32768) for get_max_shared_memory_per_block, but
|
||||
# BI-V100 actually has 48KB (49152) confirmed via ixsmi. This limits Triton
|
||||
# tile sizes and ixformer internal allocations if not corrected.
|
||||
# CCCL GridEvenShare test (catch2_test_grid_even_share.cu) validates that
|
||||
# work distribution depends on correct hardware parameters — wrong SMEM
|
||||
# means wrong tile_size means wrong grid_size.
|
||||
# FULL FILE REPLACEMENT.
|
||||
deploy ./_custom_ops.py "_custom_ops.py"
|
||||
echo "[patch_ops] _custom_ops.py → / (SMEM 32KB→48KB fix)"
|
||||
|
||||
# --- paged_attn.py: pure-PyTorch attention fallback --------------------------
|
||||
deploy ./paged_attn.py "attention/ops/paged_attn.py"
|
||||
echo "[patch_ops] paged_attn.py → attention/ops/"
|
||||
|
||||
# --- prefix_prefill.py: Triton-free prefix attention -------------------------
|
||||
deploy ./prefix_prefill.py "attention/ops/prefix_prefill.py"
|
||||
echo "[patch_ops] prefix_prefill.py → attention/ops/"
|
||||
|
||||
# --- model_runner.py: prefix_cache_hit fix -----------------------------------
|
||||
deploy ./model_runner.py "worker/model_runner.py"
|
||||
echo "[patch_ops] model_runner.py → worker/"
|
||||
|
||||
# --- xformers.py: head_dim>128 fallback + Q-tiling --------------------------
|
||||
deploy ./xformers.py "attention/backends/xformers.py"
|
||||
echo "[patch_ops] xformers.py → attention/backends/"
|
||||
|
||||
# --- arg_utils.py: disable auto chunked-prefill for 32K+ --------------------
|
||||
deploy ./arg_utils.py "engine/arg_utils.py"
|
||||
echo "[patch_ops] arg_utils.py → engine/"
|
||||
|
||||
# --- logits_processor.py: seq_groups=None guard ------------------------------
|
||||
deploy ./logits_processor.py "model_executor/layers/logits_processor.py"
|
||||
echo "[patch_ops] logits_processor.py → model_executor/layers/"
|
||||
|
||||
# --- sampler.py: CCCL-ported top-k fast path for sampling --------------------
|
||||
deploy ./sampler.py "model_executor/layers/sampler.py"
|
||||
echo "[patch_ops] sampler.py → model_executor/layers/"
|
||||
|
||||
# --- transformers: Qwen3_5 tokenizer / model files --------------------------
|
||||
# NOTE: patch_transformers_qwen3_5.py is the ONLY remaining patch script.
|
||||
# It modifies pip-installed transformers' configuration_auto.py and __init__.py
|
||||
# to register qwen3_5/qwen3_5_moe. These files come from pip (version-specific)
|
||||
# so we can't pre-copy them — the patch script inserts lines after known anchors.
|
||||
# ============================================================
|
||||
# 1. Transformers: register Qwen3_5 / Qwen3_5_MoE model types
|
||||
# ============================================================
|
||||
pip install transformers==4.55.3 -i https://pypi.tuna.tsinghua.edu.cn/simple 2>/dev/null || \
|
||||
pip install transformers==4.55.3 2>/dev/null || \
|
||||
echo "[patch_ops] WARNING: pip install transformers failed, using pre-installed version"
|
||||
@@ -97,37 +60,24 @@ cp -r ./qwen3_5_moe /usr/local/lib/python3.10/site-packages/transformers/models/
|
||||
python3 ./patch_transformers_qwen3_5.py
|
||||
echo "[patch_ops] transformers Qwen3_5 models installed"
|
||||
|
||||
# --- vllm model: Qwen3.6 (Qwen3_5 arch) ------------------------------------
|
||||
for V in "${TARGETS[@]}"; do
|
||||
cp ./mamba_cache.py "$V/model_executor/models/"
|
||||
done
|
||||
deploy ./qwen3_5.py "model_executor/models/qwen3_5.py"
|
||||
# ============================================================
|
||||
# 2. Model registry: ensure qwen3_5 is registered in vllm
|
||||
# ============================================================
|
||||
deploy ./registry.py "model_executor/models/registry.py"
|
||||
echo "[patch_ops] qwen3_5.py + registry.py deployed"
|
||||
echo "[patch_ops] registry.py deployed"
|
||||
|
||||
# --- paged_attention_v2_pytorch.py: PyTorch V2 attention fallback ------------
|
||||
for V in "${TARGETS[@]}"; do
|
||||
cp ./paged_attention_v2_pytorch.py "$V/paged_attention_v2_pytorch.py"
|
||||
done
|
||||
cp ./paged_attention_v2_pytorch.py /workspace/paged_attention_v2_pytorch.py
|
||||
echo "[patch_ops] paged_attention_v2_pytorch.py → all paths + /workspace/"
|
||||
# ============================================================
|
||||
# 3. Serving layer patches (protocol, chat, tool parsing, reasoning)
|
||||
# ============================================================
|
||||
|
||||
# --- sequence.py: fix completion_tokens inflation ----------------------------
|
||||
deploy ./sequence.py "sequence.py"
|
||||
echo "[patch_ops] sequence.py → /"
|
||||
|
||||
# --- scheduler.py: record num_cached_tokens ---------------------------------
|
||||
deploy ./scheduler.py "core/scheduler.py"
|
||||
echo "[patch_ops] scheduler.py → core/"
|
||||
|
||||
# --- tool parser: Qwen3 XML tool call format --------------------------------
|
||||
# --- Tool parser: Qwen3 XML tool call format ---
|
||||
for V in "${TARGETS[@]}"; do
|
||||
cp ./qwen3coder_tool_parser.py "$V/entrypoints/openai/tool_parsers/"
|
||||
cp ./tool_parsers_init.py "$V/entrypoints/openai/tool_parsers/__init__.py"
|
||||
done
|
||||
echo "[patch_ops] qwen3_coder tool parser deployed"
|
||||
|
||||
# --- reasoning parser: Qwen3 <think>...</think> split -----------------------
|
||||
# --- Reasoning parser + serving files ---
|
||||
for V in "${TARGETS[@]}"; do
|
||||
cp -r ./reasoning "$V/"
|
||||
cp ./protocol.py "$V/entrypoints/openai/protocol.py"
|
||||
@@ -138,4 +88,26 @@ for V in "${TARGETS[@]}"; do
|
||||
done
|
||||
echo "[patch_ops] reasoning parser + serving files installed"
|
||||
|
||||
echo "[patch_ops] DONE — all patches applied via full file replacement"
|
||||
# ============================================================
|
||||
# 4. DO NOT PATCH sequence.py or scheduler.py
|
||||
# 168 (reference competitor) did not patch these.
|
||||
# Our custom versions may conflict with base image internals.
|
||||
# Token counting fixes are minor; NaN-free output is critical.
|
||||
# ============================================================
|
||||
|
||||
# ============================================================
|
||||
# 5. DO NOT PATCH these files — base image has optimized versions:
|
||||
# - qwen3_5.py (model) — has corex_gdn/corex_moe/corex_fa2 integration
|
||||
# - _custom_ops.py — base image ixformer bindings
|
||||
# - model_runner.py — base image worker
|
||||
# - xformers.py — base image attention backend
|
||||
# - paged_attn.py — base image paged attention
|
||||
# - prefix_prefill.py — base image prefix prefill
|
||||
# - logits_processor.py — base image logits
|
||||
# - sampler.py — base image sampler
|
||||
# - arg_utils.py — base image arg parsing
|
||||
# - paged_attention_v2_pytorch.py — not needed with native kernels
|
||||
# ============================================================
|
||||
|
||||
echo "[patch_ops] DONE — serving-layer-only patches applied"
|
||||
echo "[patch_ops] Core compute files preserved from base image (corex_gdn + corex_moe + corex_fa2)"
|
||||
|
||||
Reference in New Issue
Block a user