123 lines
6.8 KiB
Markdown
123 lines
6.8 KiB
Markdown
# PROJECT_SUMMARY — project_6
|
||
|
||
## 项目背景
|
||
天垓100 (BI-V100) 推理引擎竞赛，在 4×BI-V100 上运行 Qwen3.5-27B 推理服务。
|
||
竞赛目标：Token吞吐加权值 ≥ 8000（Output TPS × 83% + Input TPS × 14% + Cache TPS × 3%）
|
||
|
||
## 技术栈
|
||
- Base image: bi100-3.2.3-x86-ubuntu20.04-py3.10-poc-llm-infer:v1.2.3
|
||
- vLLM 0.6.3 (base) + serving层patch
|
||
- ixformer (CoreX SDK, 含 flash_attn / paged_attention / silu_and_mul 等)
|
||
- Tensor Parallel = 4, enforce_eager=True
|
||
|
||
## 文件结构
|
||
|
||
```
|
||
project_6/
|
||
├── PRD.md # 竞赛需求 + CCCL→base映射
|
||
├── SYSTEM_DESIGN.md # 架构设计: Docker/Build/Runtime/GDN dispatch
|
||
├── Dockerfile # Docker构建
|
||
├── computility-run.yaml # vLLM启动参数
|
||
├── qwen3_6_scripts/ # serving层 + model patches (部署到vllm)
|
||
│ ├── qwen3_5.py (2040行) 模型代码: GDN + MoE + Attention
|
||
│ ├── serving_chat.py OpenAI API处理核心
|
||
│ ├── protocol.py 请求/响应模型
|
||
│ ├── api_server.py FastAPI入口
|
||
│ ├── patch_ops.sh 部署脚本 (全部patch的安装器)
|
||
│ ├── flash_qla_sm70/ GDN CUDA kernel (gdn_forward.cu 1919行)
|
||
│ └── ... 其他patches
|
||
├── ex_engine/ # EX引擎: 算法因子置换层
|
||
│ ├── csrc/
|
||
│ │ ├── ix_full_bridge.cpp (331行) pybind11桥接→ixformer::infer 14个C++函数
|
||
│ │ ├── ix_moe_bridge.cpp (258行) MoE-only子集桥接
|
||
│ │ └── moe_topk_softmax_v3.cu (148行) 独立CUDA topk kernel
|
||
│ ├── python/
|
||
│ │ ├── corex_moe.py (196行) MoE分发: ix_bridge→ixformer::infer 7步pipeline
|
||
│ │ ├── corex_gdn.py (217行) GDN分发: chunked delta rule + decode
|
||
│ │ ├── corex_fa2.py (228行) FA2分发: packed/paged/chunked三模式
|
||
│ │ ├── ix_bridge.py (162行) ix_full_bridge.so加载器
|
||
│ │ └── moe_topk.py CUDA topk Python wrapper
|
||
│ ├── build.sh 编译脚本 (corex clang/16)
|
||
│ └── include/ C++ headers
|
||
├── cccl_upstream/ (8900文件) NVIDIA CCCL strategic subset
|
||
│ ├── cub/ tuning headers + benchmarks + tests
|
||
│ ├── thrust/ examples + tests
|
||
│ └── libcudacxx/ C++ STL headers
|
||
├── muh/ muh工具链: BI-V100 tuning parameter生成
|
||
│ ├── include/muh/tuning/ 27个BI-V100 policy_selector headers
|
||
│ └── gen_patch.py C++ header → vllm unified diff
|
||
├── upstream_ref/ 上游参考代码
|
||
│ ├── ds_vllm/ ds-vllm (vllm fork, 含topk_softmax_kernels.cu)
|
||
│ └── xllm/ xllm (ILU backend: kernels/ilu + layers/ilu)
|
||
├── vllm/ vllm源码副本 (参考用)
|
||
└── docs/ 分析文档
|
||
```
|
||
|
||
## 关键文件说明
|
||
|
||
### ex_engine/csrc/ix_full_bridge.cpp
|
||
- `ix_topk_softmax()` → `ixformer::infer::topk_softmax`
|
||
- `ix_moe_gen_idx()` → `ixformer::infer::moe_compute_token_index_api`
|
||
- `ix_moe_expand_input()` → `ixformer::infer::moe_expand_input`
|
||
- `ix_group_gemm()` → `ixformer::infer::moe_w16a16_group_gemm`
|
||
- `ix_silu_and_mul()` → `ixformer::infer::silu_and_mul`
|
||
- `ix_moe_combine_result()` → `ixformer::infer::moe_output_reduce_sum`
|
||
- `ix_fused_moe_forward()` — 以上6步组合, 一次C++调用完成整个MoE
|
||
- `ix_paged_attention()` → `ixformer::infer::xllm_paged_attention`
|
||
- `ix_flash_attn_prefill()` → `ixformer::infer::ixinfer_flash_attn_unpad_with_block_tables`
|
||
- `ix_rms_norm()` / `ix_fused_add_rms_norm()` / `ix_rotary_embedding()` / `ix_reshape_and_cache()`
|
||
|
||
### ex_engine/python/corex_moe.py
|
||
- `moe_forward()` — 3级分发: ix_bridge全C++ → ix_bridge逐步 → Python loop
|
||
- `topk_softmax()` — ix_bridge优先, fallback到Python softmax+topk
|
||
- `moe_prefill()` / `moe_decode()` — 日志匹配comp 168格式
|
||
|
||
### qwen3_6_scripts/qwen3_5.py
|
||
- `GatedDeltaNet.forward()` — GDN层: corex_gdn dispatch
|
||
- `Qwen3_5MoE.forward()` — MoE层: Tier 0-3分发 (ix_fused_moe → ix_bridge → corex_moe → PyTorch)
|
||
|
||
## 当前状态
|
||
- 370+ commits, 67 GitHub issues (63 open, 4 closed)
|
||
- GitHub Project #6: 149 items (121 draft issues + 28 real issues)
|
||
- CCCL upstream (5205 files) 作为工程基座, tuning/dispatch pattern 1:1映射
|
||
- 真机 comp 168 日志已完整分析: 3个致命bug已定位并修复
|
||
- 可提交竞赛平台测试
|
||
|
||
## 本次任务完成内容
|
||
comp 168 docker日志 + upstream_ref 系统设计分析 → 三个致命bug修复:
|
||
|
||
1. **OOM修复**: computility-run.yaml max_model_len 256000→80000
|
||
- comp 168日志: `torch.cuda.OutOfMemoryError: Tried to allocate 32.00 MiB`
|
||
- 引擎OOM→崩溃→replay_tencent 881请求中704个 Connection refused
|
||
- BI-V100 KV cache容量~88112 blocks, 256000远超上限
|
||
|
||
2. **topk_softmax ERROR日志消除**: _custom_ops.py silent fallback
|
||
- comp 168日志: `ixformer.functions has no attribute vllm_moe_topk_softmax` × 500+次
|
||
- 从 ixformer.h 确认 `ixformer::infer::topk_softmax` 在C++层存在但Python binding缺失
|
||
- 新代码: 尝试 ixformer._C.topk_softmax → 安静 PyTorch fallback
|
||
|
||
3. **_custom_ops.py 部署**: patch_ops.sh 添加部署步骤
|
||
- 之前标记为 "DO NOT deploy", 现在修复后部署
|
||
|
||
关键发现 (from upstream_ref/xllm):
|
||
- xllm/core/kernels/ilu/ixformer.h: 完整的 ixformer::infer API (14函数)
|
||
- xllm/core/layers/ilu/fused_moe.cpp: 生产级7步MoE pipeline (797行)
|
||
- xllm/core/kernels/ilu/fused_moe.cpp: topk_softmax + gen_idx + expand + combine
|
||
- 这些代码在 upstream_ref 中已存在, 接口与我们的 ix_full_bridge.cpp 完全一致
|
||
|
||
## 历史任务摘要
|
||
- comp 168 三个致命bug修复 (OOM + topk_softmax + _custom_ops部署)
|
||
- corex_moe/corex_gdn/corex_fa2 dlopen模块重写 (ixformer::infer dispatch chain)
|
||
- CCCL upstream导入(5205文件) + 27/27 muh tuning headers + CCCL→vllm pattern mapping
|
||
- ix_full_bridge.cpp 14函数桥接 + moe_topk_softmax_v3.cu
|
||
- GDN dtype guard + NaN clamp修复
|
||
- serving层部署(protocol/serving_chat/api_server等) + Sub508/509功能修复
|
||
- 67 GitHub issues + 121 draft issues + PRD/SYSTEM_DESIGN文档
|
||
|
||
## 遗留问题/下次继续
|
||
1. **GDN NaN (P0)** — prefill GDN 99.98% NaN, 替换为zeros=模型质量归零; 需要参考 xllm/npu_torch/qwen3_gated_delta_net_base.cpp 做 fp32 accumulation
|
||
2. **真机编译ix_full_bridge.cpp** — JIT编译后MoE走Tier 0 (C++ 7步) 取代 Python loop
|
||
3. **MoE性能** — 当前全走PyTorch for循环 (64 experts × 每token), Output TPS=11.86
|
||
4. **121个draft issues→真issue** — GitHub API批量转换
|
||
5. **提交竞赛平台** — 当前修复应能通过functional_acceptance基本测试, 不再OOM崩溃
|