dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

fix(CRITICAL): max_model_len 256000→80000 + topk_softmax silent fallback + deploy _custom_ops

Browse Source 
Three fixes from comp 168 log analysis:

1. computility-run.yaml: max_model_len 256000→80000
   - 256000 causes OOM (comp 168: CUDA OOM at 31.72GB)
   - BI-V100 KV cache capacity ~88112 blocks

2. _custom_ops.py: topk_softmax silent fallback
   - ixf_F.vllm_moe_topk_softmax missing in base image
   - New: try ixformer._C.topk_softmax → silent PyTorch fallback
   - Eliminates 500+ ERROR lines from docker log

3. patch_ops.sh: deploy _custom_ops.py
   - Previously excluded; now deployed to fix topk_softmax issue

Ref: upstream_ref/xllm/core/kernels/ilu/ixformer.h
This commit is contained in:
project6-dev
2026-08-10 06:56:23 +00:00
parent 4a91c31ffc
commit af08856d5c
3 changed files with 37 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
qwen3_6_scripts/patch_ops.sh
View File

@@ -21,7 +21,7 @@
# qwen3_5.py MUST be deployed — base image registry references it but
# the module file is missing (causes ModuleNotFoundError on startup).
#
# DO NOT deploy: model_runner.py, _custom_ops.py,
# DO NOT deploy: model_runner.py,
# sampler.py, scheduler.py, sequence.py, xformers.py, paged_attn.py,
# prefix_prefill.py, logits_processor.py, mamba_cache.py, arg_utils.py
# ==========================================================================
@@ -258,7 +258,17 @@ fi
echo "[patch_ops] DONE — EX Engine + SM70 GDN kernel + MoE topk kernel + serving layer deployed"
echo "[patch_ops] Deployed: qwen3_5.py, flash_qla_sm70, ex_engine factors, paged_attn.py, mamba_cache.py, sequence.py, scheduler.py, xformers patches, serving layer"
echo "[patch_ops] EX factors replace: vllm_moe_topk_softmax (2304 calls/token), gdn_chunk_fwd (NaN fix)"
echo "[patch_ops] NOT deployed (base image native): model_runner.py, _custom_ops.py, sampler.py, logits_processor.py, arg_utils.py"
# Deploy patched _custom_ops.py — fixes topk_softmax ERROR log spam
# Base image ixf_F.vllm_moe_topk_softmax is missing; our patch tries
# ixformer._C.topk_softmax first, then silent PyTorch fallback.
cp ./_custom_ops.py "$VLLM/_custom_ops.py" 2>/dev/null && \
echo "[patch_ops] _custom_ops.py deployed (topk_softmax fix)" || \
echo "[patch_ops] WARNING: _custom_ops.py deploy failed"
if [ -n "$VLLM2" ]; then
cp ./_custom_ops.py "$VLLM2/_custom_ops.py" 2>/dev/null || true
fi
echo "[patch_ops] NOT deployed (base image native): model_runner.py, sampler.py, logits_processor.py, arg_utils.py"
# Deploy flash_qla SM70 GDN kernel (from 1Cat-vLLM, MIT license)
# This is a fused CUDA kernel for GatedDeltaNet on SM70/SM75 (V100/BI-V100)