diag: comp 168 完整分析 + verify_deployment.sh
关键发现: - 07-23: base corex chain 完好(GDN+MoE+FA2全fused), 但OOM(max_model_len=100000) - 08-07: 我们覆盖qwen3_5.py打断corex chain → GDN NaN + MoE Python fallback - 修复: patch_ops.sh v2 不覆盖base, max_model_len=80000 - 新增: verify_deployment.sh 真机验证, COMP168_DIAGNOSIS.md
This commit is contained in:
project6-dev
165
COMP168_DIAGNOSIS.md
Normal file
165
COMP168_DIAGNOSIS.md
Normal file
@@ -0,0 +1,165 @@
|
||||
# comp 168 Docker 诊断 → .so 开发清单
|
||||
|
||||
> 基于 `2d5232c5d6bc` (comp 168 docker log, 3786 行)
|
||||
> 当前 HEAD: `b25fc53e` (414 commits)
|
||||
|
||||
## 一、comp 168 日志三大致命问题
|
||||
|
||||
| # | 错误 | 出现次数 | 根因 | 状态 |
|
||||
|---|------|----------|------|------|
|
||||
| 1 | `GDN NaN frac=0.9998` | 16次(layer 0-4) | 我们的 GDN prefill 实现产生 NaN → replace with zeros → 模型质量归零 | **P0 未修** |
|
||||
| 2 | `vllm_moe_topk_softmax not found` | 39次 | `ixformer.functions` 没有 Python binding → fallback to Python for 循环 | **P0 需 .so** |
|
||||
| 3 | `CUDA OOM 32 MiB` | 17次 | `max_model_len=100000` 超过 KV cache 容量 → engine 死亡 | ✅ 已修为 80000 |
|
||||
|
||||
## 二、真机探测确认的事实
|
||||
|
||||
从你贴的真机 probe 输出：
|
||||
|
||||
```
|
||||
ixformer.functions 有:
|
||||
✓ silu_and_mul, rms_norm, fused_add_rms_norm, rotary_embedding
|
||||
✓ flash_attn_*, vllm_single_query_cached_kv_attention_v2
|
||||
✓ vllm_cache_ops_reshape_and_cache, vllm_swap_blocks, vllm_copy_cache
|
||||
✗ vllm_moe_topk_softmax (不存在!)
|
||||
✗ moe_compute_token_index_api (不存在!)
|
||||
✗ moe_w16a16_group_gemm (不存在!)
|
||||
|
||||
libixformer.so 中:
|
||||
✓ 上述函数全部存在 (C++ 符号, xllm 的 ixformer.h 声明了它们)
|
||||
但 Python binding (_C.so) 没有暴露
|
||||
```
|
||||
|
||||
**结论**: MoE 7 步 pipeline 中的 topk_softmax / gen_idx / expand / group_gemm / combine 全部需要通过 `ix_moe_bridge.so` 桥接。
|
||||
|
||||
## 三、需要开发/修复的 .so 清单
|
||||
|
||||
### SO-1: `ix_moe_bridge.so` (MoE 7步 pipeline) — ✅ 代码已有，需真机编译验证
|
||||
|
||||
**源码**: `ex_engine/csrc/ix_moe_bridge.cpp` (258行)
|
||||
**编译**: `ex_engine/precompile_ix_bridge.py` → `torch.utils.cpp_extension.load(-lixformer)`
|
||||
**状态**: 代码写好了，Dockerfile 有 build step，但从未在真机验证过编译成功
|
||||
|
||||
真机验证命令:
|
||||
```bash
|
||||
cd /workspace/ex_engine
|
||||
python3 precompile_ix_bridge.py
|
||||
ls -la build/ix_moe_bridge*.so
|
||||
python3 -c "import torch; from torch.utils.cpp_extension import load; m=load('test', sources=['csrc/ix_moe_bridge.cpp'], extra_ldflags=['-L/usr/local/corex/lib64/python3/dist-packages/ixformer', '-lixformer']); print(dir(m))"
|
||||
```
|
||||
|
||||
### SO-2: GDN prefill 修复 — **P0 最高优先级**
|
||||
|
||||
**现状**: 我们的 `_torch_chunk_gated_delta_rule` 在 fp16 下产生 99.98% NaN
|
||||
**参考**: `upstream_ref/xllm/core/layers/npu_torch/qwen3_gated_delta_net_base.cpp` (576行)
|
||||
|
||||
关键差异:
|
||||
- xllm 用 `fp32` accumulation: `decay_mask = ... .exp().float()`
|
||||
- xllm 用 `torch::matmul` 而不是自定义 chunk kernel
|
||||
- xllm 的 recurrent state 管理有精确的 `clamp(-20, 20)` 限制
|
||||
|
||||
**解决方案**: 不写新 .so，而是从 xllm 搬运 GDN 的 PyTorch 实现（C++ torch ops, 全 fp32 accumulation），替换我们的 chunk kernel。
|
||||
|
||||
### SO-3: `_custom_ops.py` patch — ✅ 已有 fallback 逻辑
|
||||
|
||||
base image 的 `_custom_ops.py` 调用 `ixf_F.vllm_moe_topk_softmax` 时会报错。
|
||||
但 comp 168 的 base 镜像绕过了 `_custom_ops`，直接走 `corex_moe.py` 的 7 步 pipeline。
|
||||
|
||||
**如果 base 有 corex_moe.py**: 不需要 patch
|
||||
**如果 base 没有 corex_moe.py**: 我们的版本 + ix_moe_bridge.so 补位
|
||||
|
||||
## 四、upstream 已有、不需要重写的代码
|
||||
|
||||
| upstream 文件 | 行数 | 我们的对应文件 | 搬运状态 |
|
||||
|--------------|------|---------------|---------|
|
||||
| `xllm/core/kernels/ilu/ixformer.h` | 147 | `ex_engine/csrc/ilu/ixformer.h` | ✅ 已搬 |
|
||||
| `xllm/core/kernels/ilu/fused_moe.cpp` | 99 | `ex_engine/csrc/ilu_kernel_fused_moe.cpp` | ✅ 已搬 |
|
||||
| `xllm/core/layers/ilu/fused_moe.cpp` | 797 | `ex_engine/csrc/ilu_layer_fused_moe.cpp` | ✅ 已搬 |
|
||||
| `xllm/core/kernels/ilu/attention.cpp` | 162 | `ex_engine/csrc/ilu_kernel_attention.cpp` | ✅ 已搬 |
|
||||
| `xllm/core/layers/ilu/attention.cpp` | 189 | `ex_engine/csrc/ilu_layer_attention.cpp` | ✅ 已搬 |
|
||||
| `xllm/core/kernels/ilu/norm.cpp` | 50 | `ex_engine/csrc/ilu_kernel_norm.cpp` | ✅ 已搬 |
|
||||
| `xllm/core/kernels/ilu/activation.cpp` | 32 | `ex_engine/csrc/ilu_kernel_activation.cpp` | ✅ 已搬 |
|
||||
| `xllm/core/kernels/ilu/rope.cpp` | 31 | `ex_engine/csrc/ilu_kernel_rope.cpp` | ✅ 已搬 |
|
||||
| `xllm/core/kernels/ilu/group_gemm.cpp` | 39 | `ex_engine/csrc/ilu_kernel_group_gemm.cpp` | ✅ 已搬 |
|
||||
| `xllm/core/kernels/ilu/matmul.cpp` | 73 | `ex_engine/csrc/ilu_kernel_matmul.cpp` | ✅ 已搬 |
|
||||
| `xllm/core/layers/npu_torch/qwen3_gated_delta_net_base.cpp` | 576 | `ex_engine/csrc/qwen3_gated_delta_net_base.cpp` | ✅ 已搬 |
|
||||
| `ds_vllm/csrc/moe/topk_softmax_kernels.cu` | 874 | `ex_engine/csrc/moe_v055/topk_softmax_kernels.cu` | ✅ 已搬 |
|
||||
| `xllm/core/kernels/cuda/moe/moe_topk_softmax_kernels.cuh` | ~400 | `ex_engine/csrc/moe/moe_topk_softmax_kernels.cuh` | ✅ 已搬 |
|
||||
|
||||
## 五、真机验证 checklist
|
||||
|
||||
在真机上按顺序执行:
|
||||
|
||||
```bash
|
||||
# 1. 验证 ix_moe_bridge.so 编译
|
||||
cd /workspace/ex_engine && python3 precompile_ix_bridge.py
|
||||
ls build/ix_moe_bridge*.so # 必须存在
|
||||
|
||||
# 2. 验证符号解析
|
||||
python3 -c "
|
||||
import torch
|
||||
import importlib.util
|
||||
spec = importlib.util.spec_from_file_location('ix', 'build/ix_moe_bridge.cpython-310-x86_64-linux-gnu.so')
|
||||
m = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(m)
|
||||
print([x for x in dir(m) if not x.startswith('_')])
|
||||
# 应输出: ['topk_softmax', 'moe_gen_idx', 'moe_expand_input', 'moe_group_gemm',
|
||||
# 'silu_and_mul', 'moe_combine_result', 'paged_attention', 'rms_norm',
|
||||
# 'fused_add_rms_norm', 'linear', 'reshape_and_cache', 'rotary_embedding']
|
||||
"
|
||||
|
||||
# 3. 验证 topk_softmax 功能
|
||||
python3 -c "
|
||||
import torch
|
||||
# ... load ix_moe_bridge ...
|
||||
gating = torch.randn(4, 64, device='cuda', dtype=torch.float32)
|
||||
tw = torch.empty(4, 8, device='cuda', dtype=torch.float32)
|
||||
ti = torch.empty(4, 8, device='cuda', dtype=torch.int32)
|
||||
tei = torch.empty(4, 8, device='cuda', dtype=torch.int32)
|
||||
m.topk_softmax(tw, ti, tei, gating)
|
||||
print('topk_weights:', tw)
|
||||
print('topk_ids:', ti)
|
||||
"
|
||||
|
||||
# 4. 验证 GDN 不再 NaN
|
||||
# (需要先修复 GDN prefill 代码)
|
||||
|
||||
# 5. 启动服务验证
|
||||
python3 -m vllm.entrypoints.openai.api_server --model /model ...
|
||||
```
|
||||
|
||||
## 六、最关键发现：07-23 的 base image 自带完整 corex_* chain
|
||||
|
||||
**07-23 日志证据** (dockerrizhi.txt):
|
||||
```
|
||||
corex_gdn.py:56 → Loaded fused CoreX GDN decode operator from /usr/local/corex/lib64/libcorex_gdn.so ✅
|
||||
corex_gdn.py:228 → Using fused CoreX GDN prefill operator ✅
|
||||
corex_moe.py:339 → Using CoreX fused MoE prefill operator: tokens=4096, kernel=expert-grouped-wmma ✅
|
||||
corex_fa2.py:333 → Using CoreX FA2 packed prefill: B=2 Hq=4 Hkv=1 D=256 ✅
|
||||
corex_fa2.py:507 → Using CoreX paged FA2 chunked prefill ✅
|
||||
```
|
||||
|
||||
**08-07 日志**: 零条 corex_* 加载记录。取而代之的是 `qwen3_5.py:445 NaN in prefill` + `_custom_ops.py:58 topk_softmax not found`。
|
||||
|
||||
**根因**: 08-07 提交部署了我们自己的 `qwen3_5.py`，覆盖了 base image 自带的版本，打断了 `corex_gdn.py` / `corex_moe.py` / `corex_fa2.py` 的调用链。
|
||||
|
||||
**当前状态**: `patch_ops.sh v2` 已经有条件跳过逻辑（`_QW_SIZE > 1000 → KEEPING IT`），但需要确保下次提交时不再触发 qwen3_5.py 覆盖。
|
||||
|
||||
**结论**: 如果 base image 有工作的 corex_* chain，我们只需要:
|
||||
1. 不覆盖 qwen3_5.py
|
||||
2. 只部署 serving 层（protocol/serving_chat/api_server/tool_parser）
|
||||
3. `max_model_len=80000`（已修）
|
||||
4. `ix_moe_bridge.so` 作为备用（如果 base 的 _custom_ops 有路径碰到 topk_softmax）
|
||||
|
||||
## 七、代码量评估
|
||||
|
||||
| 组件 | 文件数 | 总行数 | 状态 |
|
||||
|------|--------|--------|------|
|
||||
| ex_engine/csrc (C++) | 39 | ~8000 | 全部已有，需真机编译 |
|
||||
| ex_engine/python (Python) | 7 | ~1200 | 全部已有，dispatch chain 完整 |
|
||||
| qwen3_6_scripts (serving) | 20+ | ~6000 | 全部已有，patch_ops.sh 管部署 |
|
||||
| upstream_ref (xllm reference) | 500+ | ~100K | 参考用，关键文件已搬到 ex_engine |
|
||||
|
||||
**结论**: 代码量是够的。问题不是代码不够，而是:
|
||||
1. GDN NaN 没修（需要用 xllm 的 fp32 accumulation 逻辑替换）
|
||||
2. ix_moe_bridge.so 从未在真机编译成功
|
||||
3. 没有 "不允许 fallback" 的硬要求落实到代码里
|
||||
184
verify_deployment.sh
Executable file
184
verify_deployment.sh
Executable file
@@ -0,0 +1,184 @@
|
||||
#!/bin/bash
|
||||
# verify_deployment.sh — Run on real BI-V100 machine to verify deployment correctness
|
||||
#
|
||||
# USAGE: bash verify_deployment.sh
|
||||
#
|
||||
# This script verifies:
|
||||
# 1. Base image qwen3_5.py is intact (NOT overwritten)
|
||||
# 2. Base corex_gdn.py / corex_moe.py / corex_fa2.py exist
|
||||
# 3. libcorex_gdn.so / libixformer.so exist
|
||||
# 4. ix_moe_bridge.so compiles successfully
|
||||
# 5. Serving layer patches are deployed
|
||||
# 6. computility-run.yaml has correct max_model_len
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
PASS=0
|
||||
FAIL=0
|
||||
WARN=0
|
||||
|
||||
check_pass() { echo " ✅ PASS: $1"; PASS=$((PASS+1)); }
|
||||
check_fail() { echo " ❌ FAIL: $1"; FAIL=$((FAIL+1)); }
|
||||
check_warn() { echo " ⚠️ WARN: $1"; WARN=$((WARN+1)); }
|
||||
|
||||
echo "============================================"
|
||||
echo " DEPLOYMENT VERIFICATION — $(date)"
|
||||
echo "============================================"
|
||||
|
||||
# --- Find vllm ---
|
||||
VLLM=""
|
||||
for P in /usr/local/corex/lib/python3/dist-packages/vllm \
|
||||
/usr/local/corex/lib64/python3/dist-packages/vllm; do
|
||||
[ -d "$P" ] && VLLM="$P" && break
|
||||
done
|
||||
if [ -z "$VLLM" ]; then
|
||||
check_fail "vllm not found in expected paths"
|
||||
exit 1
|
||||
fi
|
||||
echo " vllm path: $VLLM"
|
||||
|
||||
echo ""
|
||||
echo "=== 1. BASE IMAGE MODEL LAYER ==="
|
||||
_QW="$VLLM/model_executor/models/qwen3_5.py"
|
||||
if [ -f "$_QW" ]; then
|
||||
_SIZE=$(wc -c < "$_QW")
|
||||
_LINES=$(wc -l < "$_QW")
|
||||
if [ "$_SIZE" -gt 1000 ]; then
|
||||
check_pass "qwen3_5.py exists (${_SIZE} bytes, ${_LINES} lines) — base image intact"
|
||||
else
|
||||
check_fail "qwen3_5.py exists but tiny (${_SIZE} bytes) — may be overwritten"
|
||||
fi
|
||||
else
|
||||
check_fail "qwen3_5.py NOT FOUND — model can't load"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== 2. COREX MODULES (base image) ==="
|
||||
for m in corex_gdn.py corex_moe.py corex_fa2.py; do
|
||||
_F="$VLLM/model_executor/models/$m"
|
||||
if [ -f "$_F" ]; then
|
||||
check_pass "$m exists ($(wc -c < "$_F") bytes)"
|
||||
else
|
||||
check_warn "$m NOT in base — our version will be deployed"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "=== 3. NATIVE .so LIBRARIES ==="
|
||||
for so in /usr/local/corex/lib64/libcorex_gdn.so \
|
||||
/usr/local/corex/lib64/python3/dist-packages/ixformer/libixformer.so; do
|
||||
if [ -f "$so" ]; then
|
||||
check_pass "$(basename $so) exists at $so"
|
||||
else
|
||||
check_fail "$(basename $so) NOT FOUND at $so"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "=== 4. IXFORMER PYTHON BINDING ==="
|
||||
python3 -c "
|
||||
import ixformer.functions as ixf_F
|
||||
# Check critical functions
|
||||
critical = ['silu_and_mul', 'rms_norm', 'fused_add_rms_norm', 'flash_attn',
|
||||
'vllm_single_query_cached_kv_attention_v2']
|
||||
missing_critical = [f for f in critical if not hasattr(ixf_F, f)]
|
||||
if missing_critical:
|
||||
print(f'FAIL: missing critical ixformer functions: {missing_critical}')
|
||||
exit(1)
|
||||
else:
|
||||
print(f'PASS: all critical ixformer functions present')
|
||||
|
||||
# Check topk_softmax (expected to be missing from Python binding)
|
||||
if hasattr(ixf_F, 'vllm_moe_topk_softmax'):
|
||||
print('NOTE: vllm_moe_topk_softmax IS in ixf_F (unexpected but good)')
|
||||
else:
|
||||
print('NOTE: vllm_moe_topk_softmax NOT in ixf_F (expected — corex_moe.py bypasses this)')
|
||||
" 2>&1 && check_pass "ixformer Python binding OK" || check_fail "ixformer Python binding broken"
|
||||
|
||||
echo ""
|
||||
echo "=== 5. IX_MOE_BRIDGE COMPILE TEST ==="
|
||||
if [ -f /workspace/ex_engine/csrc/ix_moe_bridge.cpp ]; then
|
||||
python3 /workspace/ex_engine/precompile_ix_bridge.py 2>&1
|
||||
if ls /workspace/ex_engine/build/ix_moe_bridge*.so 1>/dev/null 2>&1; then
|
||||
check_pass "ix_moe_bridge.so compiled: $(ls /workspace/ex_engine/build/ix_moe_bridge*.so)"
|
||||
# Verify symbols
|
||||
python3 -c "
|
||||
import importlib.util, glob
|
||||
so = glob.glob('/workspace/ex_engine/build/ix_moe_bridge*.so')[0]
|
||||
spec = importlib.util.spec_from_file_location('ix', so)
|
||||
m = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(m)
|
||||
fns = [x for x in dir(m) if not x.startswith('_')]
|
||||
print(f'Functions: {fns}')
|
||||
required = ['topk_softmax', 'moe_gen_idx', 'moe_group_gemm', 'silu_and_mul', 'moe_combine_result']
|
||||
missing = [f for f in required if f not in fns]
|
||||
if missing:
|
||||
print(f'FAIL: missing functions: {missing}')
|
||||
exit(1)
|
||||
print('PASS: all MoE pipeline functions present')
|
||||
" 2>&1 && check_pass "ix_moe_bridge symbols verified" || check_warn "ix_moe_bridge symbol check failed"
|
||||
else
|
||||
check_warn "ix_moe_bridge.so compile failed (OK if base has corex_moe.py)"
|
||||
fi
|
||||
else
|
||||
check_warn "ix_moe_bridge.cpp not found at /workspace/ex_engine/csrc/"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== 6. SERVING LAYER ==="
|
||||
for f in protocol.py serving_chat.py api_server.py; do
|
||||
_F="$VLLM/entrypoints/openai/$f"
|
||||
if [ -f "$_F" ]; then
|
||||
check_pass "$f deployed"
|
||||
else
|
||||
check_fail "$f NOT deployed — serving broken"
|
||||
fi
|
||||
done
|
||||
|
||||
# Check tool parser
|
||||
_TP="$VLLM/entrypoints/openai/tool_parsers/qwen3coder_tool_parser.py"
|
||||
if [ -f "$_TP" ]; then
|
||||
check_pass "qwen3_coder tool parser deployed"
|
||||
else
|
||||
check_warn "qwen3_coder tool parser missing"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== 7. COMPUTILITY-RUN.YAML ==="
|
||||
_YAML="/workspace/computility-run.yaml"
|
||||
if [ -f "$_YAML" ]; then
|
||||
_MAX_LEN=$(grep -oP "max-model-len.*?'(\d+)'" "$_YAML" | grep -oP "\d+" | head -1)
|
||||
if [ "$_MAX_LEN" = "80000" ]; then
|
||||
check_pass "max_model_len=80000 (correct)"
|
||||
else
|
||||
check_fail "max_model_len=$_MAX_LEN (should be 80000)"
|
||||
fi
|
||||
else
|
||||
check_fail "computility-run.yaml not found"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== 8. FUNCTIONAL SMOKE TEST ==="
|
||||
python3 -c "
|
||||
import torch
|
||||
print(f'CUDA available: {torch.cuda.is_available()}')
|
||||
print(f'GPU count: {torch.cuda.device_count()}')
|
||||
for i in range(torch.cuda.device_count()):
|
||||
print(f' GPU {i}: {torch.cuda.get_device_name(i)}, {torch.cuda.get_device_properties(i).total_mem / 1e9:.1f} GB')
|
||||
print(f'torch version: {torch.__version__}')
|
||||
import vllm
|
||||
print(f'vllm version: {vllm.__version__}')
|
||||
" 2>&1 && check_pass "CUDA + vllm import OK" || check_fail "CUDA or vllm import failed"
|
||||
|
||||
echo ""
|
||||
echo "============================================"
|
||||
echo " RESULTS: $PASS passed, $FAIL failed, $WARN warnings"
|
||||
echo "============================================"
|
||||
|
||||
if [ $FAIL -gt 0 ]; then
|
||||
echo " ⛔ DEPLOYMENT HAS FAILURES — DO NOT SUBMIT"
|
||||
exit 1
|
||||
else
|
||||
echo " ✅ DEPLOYMENT READY FOR SUBMISSION"
|
||||
exit 0
|
||||
fi
|
||||
Reference in New Issue
Block a user