dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
734 Commits 1 Branch 0 Tags
f439f67f393287c577a82fa9cba3e75b28f59d88
Commit Graph

734 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
root
f439f67f39 baseline4 trigger rebuild 2026-08-17 17:16:11 +00:00
Claude
f287382f99 fix: update SHA256SUMS for rebuilt ix_full_bridge.so 
ix_full_bridge.so was rebuilt in be5e23f3 (without MoE ixformer::infer deps)
but SHA256SUMS was not updated. install_prebuilt_corex.sh sha256sum --check
fails → set -eo pipefail kills patch_ops.sh → cli_args.py/api_server.py
never deployed → vllm rejects --reasoning-parser qwen3 at startup.
 2026-08-17 17:12:41 +00:00
root
ff18454eeb trigger rebuild 2026-08-17 16:52:37 +00:00
root
be5e23f335 fix: rebuild ix_full_bridge.so without MoE ixformer::infer deps 2026-08-17 16:25:16 +00:00
root
5c97e3dcb8 trigger rebuild 2026-08-17 14:39:56 +00:00
root
5b96a91156 [fix] baseline4 不是 CUDA 不可用，是 .so 在非主进程里被 dlopen 时段错误。需要用进程级别的判断 2026-08-17 14:17:03 +00:00
root
053dc036b8 [fix] prebuilt 的 ix_full_bridge.so 在子进程（没有 GPU 初始化）里被 dlopen 2026-08-17 14:01:36 +00:00
root
9c46f5a04e [fix] baseline4 copy_blocks 崩溃和 ex_engine import 路径两个接线没接对 2026-08-17 13:19:26 +00:00
root
1af45de371 [fix] baseline4 docker build move ex_engine into qwen3_6_scripts, remove COPY ex_engine from Dockerfile 2026-08-17 11:51:01 +00:00
root
c655c1d29e fix: replace symlinks with real files for Docker build 2026-08-17 10:52:34 +00:00
root
657fef6766 trigger rebuild 2026-08-17 10:37:29 +00:00
root
0e21445220 Merge branch 'main' of https://dev.modelhub.org.cn/dylanyunlong/project_6 into main 2026-08-17 10:11:44 +00:00
root
36909bf964 [fix] prebuilt 2026-08-17 10:11:24 +00:00
root
f10cccf9df [fix] baseline4 :ix_moe_bridge.so 。 v2 bridge 真机编译通过，13函数全导出 2026-08-17 09:18:32 +00:00
Claude
7d6884d15a tool: verify_bridge.sh — 验证prebuilt .so是v1还是v2，缺MoE则自动重编 
5步验证流程:
  Step 1: 定位 prebuilt ix_full_bridge.so
  Step 2: nm -D 检查导出符号 (v1=5函数 vs v2=13函数)
  Step 3: 检查编译源文件+工具链+依赖库
  Step 4: 缺MoE函数则调用 build_moe_bridge.sh 重编
  Step 5: Python import 验证最终 .so 的函数列表
 2026-08-17 08:49:14 +00:00
root
b81a071f84 merge modelhub: bridge_v2 + patch_vllm_ops fix 2026-08-17 08:34:14 +00:00
root
b342eb6b98 [fix] baseline4 fix prebuilt ix_full_bridge.so 大概率是从 v2 编译的 2026-08-17 08:26:54 +00:00
root
c044d04509 merge modelhub: dockerignore fix + build_moe_bridge 2026-08-17 07:54:23 +00:00
root
ba127fe66b delay baseline4 docker build 2026-08-17 07:39:52 +00:00
root
8211a45464 platform test baseline4 2026-08-17 07:22:14 +00:00
root
f7070b2075 platform test baseline4 2026-08-17 07:19:44 +00:00
dev
be4d661191 revert: undo 2 premature pushes (ee516bd2, 9ea0a1d4) — code needs review first 2026-08-17 07:04:50 +00:00
dev
9ea0a1d4f4 fix: resolve all 8 deployment pipeline breaks 
Breaks found and fixed:

1. Dockerfile: COPY 5 individual files → COPY entire ex_engine/
2. patch_ops.sh EX_ENGINE_DIR: /workspace/ex_engine not found → added fallback
3. patch_ops.sh deploy: ix_ops.py to ex_engine/ (flat) → ex_engine/python/ (correct package)
4. ix_startup_patch.py: import from vllm.ex_engine.patch_vllm_ops → vllm.ex_engine.python.patch_vllm_ops
5. ix_moe_bridge.so: only deployed to ex_engine/ → also copy to model_executor/models/ and vllm root
6. ex_engine/__init__.py: missing re-exports → add imports so 'from vllm.ex_engine import x' works
7. gemm_grouped.so: compiled but never imported → add import + flag + prefill GEMM path in qwen3_5.py
8. build_moe_bridge.sh Python heredoc: SCRIPT_DIR not exported + wrong nested path → export + search both layouts

Also added:
- CUTLASS batched GEMM compile step (corex_batched_gemm.so for decode)
- Full ex_engine/python/*.py deployment (was deploying only 2 of 19 files)
- EX_ENGINE_INFRA_AUDIT.md documenting all findings
 2026-08-17 07:01:41 +00:00
dev
ee516bd206 fix: connect ex_engine to Docker build pipeline 
Path breaks fixed:
1. Dockerfile: COPY entire ex_engine/ instead of 5 individual files
2. patch_ops.sh: EX_ENGINE_DIR fallback to /workspace/ex_engine
3. build_moe_bridge.sh: search csrc/ under both SCRIPT_DIR and SCRIPT_DIR/ex_engine

New build steps added to patch_ops.sh:
- Build gemm_grouped.so (CUTLASS Cu10 grouped GEMM, verified on device)
- Build corex_batched_gemm.so (CUTLASS batched GEMM for decode)
- Build ix_moe_bridge.so (7-step fused MoE pipeline)
- Deploy all ex_engine/python/*.py to VLLM_ROOT/ex_engine/python/

Previously broken: ix_ops.py, patch_vllm_ops.py, ix_startup_patch.py
were never deployed because EX_ENGINE_DIR resolved to nonexistent path.
xllm_activation/norm/rope/cache.so were copied but had no Python caller.
 2026-08-17 06:56:11 +00:00
Claude
461addf428 Revert "feat: 10-file algorithm factor system — full 10-layer AST call chain" 
This reverts commit 3581dd5435.
 2026-08-17 05:32:26 +00:00
project6-dev
79562c342d Revert "feat: build_all.sh — compile 3 .so from 10 algorithm factor files" 
This reverts commit 401d33ca6b.
 2026-08-17 05:31:17 +00:00
Claude
3581dd5435 feat: 10-file algorithm factor system — full 10-layer AST call chain 
Layer 1: hw_config.h (245 lines) — BI-V100 hardware descriptor + tuning tables
Layer 2: moe_pipeline.py (461 lines) — MoE 7-step pipeline orchestrator
Layer 3: attn_dispatch.py (270 lines) — Attention prefill/decode dispatch
Layer 4: ilu_ops_api.h (182 lines) — Dispatch signature contract
Layer 5: kernel_moe_ops.cpp (155 lines) — MoE kernel-level ops wrappers
Layer 6: kernel_elem_ops.cpp (210 lines) — Element-wise kernel wrappers
Layer 7: ixformer_infer.h (246 lines) — ixformer::infer namespace contract
Layer 8: factor_topk_softmax.cu (456 lines) — MoE routing CUDA kernel
Layer 9: factor_moe_compute_index.cu (174 lines) — Token index CUDA kernel
Layer 10: factor_moe_combine.cu (154 lines) — Weighted combine CUDA kernel

Total: 2553 lines across 10 files (h/cpp/cu/py)
Upstream reference: 2787 lines across corresponding 10 xllm AST layers

Each file follows the read-read-read-write pattern from upstream xllm,
ds_vllm, and fla repos. No hand-written inference code — all kernel
logic is cat-migrated from the upstream references.
 2026-08-17 05:30:53 +00:00
project6-dev
401d33ca6b feat: build_all.sh — compile 3 .so from 10 algorithm factor files 
AST chain (aligned to xllm ILU layer):
  _ixformer_torch.so (base) → ix_full_bridge.so (7 ops) → Python
  moe_ops_impl.cu (5 CUDA kernels) → ix_moe_bridge.so (7-step MoE) → Python
  gemm_grouped.cu (CUTLASS Cu10) → gemm_grouped.so (verified 1.97x) → Python
  cuinfer_gemm_wrapper.cu → cuinferCustomGemm C API → Python

Symbol sources verified from cat_files/symbol_dumps
 2026-08-17 05:30:21 +00:00
root
512f384a49 Merge branch 'main' of https://dev.modelhub.org.cn/dylanyunlong/project_6 into main 2026-08-17 04:19:31 +00:00
root
cdcf115037 feat: CUTLASS Cu10 grouped GEMM — real device verified 
BI-V100 real device results:
  moe_group_gemm: err=0.000015 PASS
  moe_decode_cutlass: NaN=False PASS
  cutlass grouped: 4.77ms vs torch.mm loop: 9.38ms → 1.97x speedup

Fix: gemm_grouped.cu ldb=K (not N) for ColumnMajor B view
Link: -lcuinfer from /usr/local/corex-3.2.3/lib64/libcuinfer.so.7
 2026-08-17 04:18:29 +00:00
root
cb926707af Merge branch 'main' of https://github.com/dylanyunlon/project_6 2026-08-17 02:21:09 +00:00
root
beaa8dbb65 Merge branch 'main' of https://dev.modelhub.org.cn/dylanyunlong/project_6 2026-08-17 02:18:46 +00:00
root
03be5f2b15 [feat] group gemm 2026-08-17 02:16:58 +00:00
Claude
330669b309 Revert "fix: ix_full_bridge_v2.cpp — align namespace+signatures to real nm -D symbol dump" 
This reverts commit 5c03156978.
 2026-08-17 02:08:03 +00:00
Claude
5c03156978 fix: ix_full_bridge_v2.cpp — align namespace+signatures to real nm -D symbol dump 
Non-MoE functions: ixformer::infer → ixformer_torch_ext (real namespace)
  silu_and_mul_forward, rms_norm_forward, fused_add_rms_norm_forward,
  ixformer_linear, ixformer_linear_ex, vllm_rotary_embedding_neox,
  vllm_cache_ops_reshape_and_cache, vllm_single_query_cached_kv_attention

MoE functions: keep ixformer::infer (provided by moe_ops_impl.cu)
  topk_softmax, moe_compute_token_index_api, moe_expand_input,
  moe_w16a16_group_gemm, moe_output_reduce_sum

Removed: flash_attn_prefill, xllm_paged_attention (not in any .so)
Fixed: c10::optional vs std::optional, parameter counts, arg order
 2026-08-17 02:04:55 +00:00
project_6
34a8fbf27e revert: undo 2 premature pushes (c54923a1, 49034d1d) — code needs review first 2026-08-16 17:48:17 +00:00
project_6
49034d1d09 feat: 10-file MoE bridge pipeline — compile, dispatch, patch, test 
The complete chain to replace 180 Python fallback calls/token with C++:

BUILD:
  1. moe_ops_impl.cu (489L) — 5 MoE functions in ixformer::infer namespace
     - topk_softmax: dynamic num_experts (128 for Qwen3.5), shared-mem
     - moe_compute_token_index: histogram + prefix_sum + scatter
     - moe_expand_input: gather kernel
     - moe_w16a16_group_gemm: per-expert cuinferCustomGemm loop
     - moe_output_reduce_sum: weighted combine
  2. ix_full_bridge_v2.cpp (461L) — pybind11 bridge, 14+1 functions
  3. build_moe_bridge.sh — torch.utils.cpp_extension compile, link cuinfer+ixformer

DISPATCH:
  4. moe_dispatch.py — 3-tier fallback (fused → individual → PyTorch)
  5. patch_moe_hot_path.py — monkey-patch Qwen3_5MoE.forward()

CONFIG:
  6. computility-run.yaml — max_num_seqs 1→2 (match sub168 baseline)
  7. patch_ops.sh — add build + deploy steps for MoE bridge

VERIFY:
  8. probe_moe_symbols.sh — nm -D .so to confirm 5 MoE symbols present
  9. test_moe_bridge.py — random-tensor integration test (no weights needed)

DEPLOY:
 10. Dockerfile — COPY ex_engine sources for in-container compilation
 2026-08-16 17:46:53 +00:00
project_6
c54923a17e feat: implement 5 missing MoE ops — topk_softmax + token_index + expand + group_gemm + combine 
Symbol dump from real device confirms: libixformer.so has 0 MoE symbols.
topk_softmax, moe_compute_token_index_api, moe_expand_input,
moe_w16a16_group_gemm, moe_output_reduce_sum — all missing.

Non-MoE symbols (silu_and_mul, rms_norm, flash_attn, reshape_and_cache,
rotary_embedding) are present and working.

Implementation strategy — use available primitives:
- topk_softmax: pure CUDA kernel (64-expert, shared-mem argmax)
- moe_compute_token_index: histogram + prefix_sum + scatter (3 kernels)
- moe_expand_input: gather kernel
- moe_w16a16_group_gemm: per-expert loop calling cuinferCustomGemm
  (confirmed in libcuinfer.so symbol dump: cuinferCustomGemm exists)
- moe_output_reduce_sum: weighted combine kernel

All in ixformer::infer namespace so ix_full_bridge_v2.cpp links directly.
Compile: nvcc moe_ops_impl.cu + ix_full_bridge_v2.cpp → single .so
 2026-08-16 17:35:08 +00:00
root
3712c06861 data: full symbol dumps 2026-08-16 17:17:26 +00:00
root
dec268d252 data: full symbol dumps 2026-08-16 17:15:11 +00:00
project_6
415ca12afc fix: group_gemm format "TN" + Layer 3 ops_api dispatch from xllm upstream 
AST chain alignment with upstream_ref/xllm/xllm/core/kernels/ilu/:

Layer 5: ixformer::infer (binary .so on device)
Layer 4: xllm_kernels/ilu/*.cpp -> calls ixformer::infer (0-diff with upstream)
Layer 3: xllm_kernels/ops_api.h+cpp + param.h (NEW from upstream 2719 lines)
         kernels/kernels.h aggregation header (NEW)
Layer 2: xllm_layers/ilu/*.cpp (0-diff with upstream)
Layer 1: ix_full_bridge_v2.cpp pybind11 bridge (FIXED)

Critical fixes in ix_full_bridge_v2.cpp:
- group_gemm format "default" -> "TN" (match upstream ilu/group_gemm.cpp)
- fused_moe_forward: pass 3D weights directly instead of .view({-1,...})
- group_gemm output_n: use tokens_per_experts.sum() per upstream convention
 2026-08-16 16:09:15 +00:00
Claude
5172f94b1f Revert "feat: 3-tier ixformer flash prefill dispatch + OpenCompass max_tokens clamp + n>1 fanout + index sanitizer" 
This reverts commit cdec569977.
 2026-08-16 15:42:36 +00:00
Claude
7a7ddf38db Revert "test: deploy_and_verify.sh — pull+patch+probe ixformer backends+clamp test" 
This reverts commit 587e18309b.
 2026-08-16 15:42:36 +00:00
Claude
587e18309b test: deploy_and_verify.sh — pull+patch+probe ixformer backends+clamp test 2026-08-16 15:40:47 +00:00
Claude
cdec569977 feat: 3-tier ixformer flash prefill dispatch + OpenCompass max_tokens clamp + n>1 fanout + index sanitizer 
paged_attn.py (+218 lines):
  - Tier 0: ixformer flash_attn_varlen_func (cu_seqlens packed)
  - Tier 0.5: ixformer flash_attn_func (non-varlen, batch layout)
  - Tier 1: CoreXFA2 3-mode dispatch (packed/paged/chunked)
  - Tier 2 fallback: existing Python Q-tiling (unchanged)
  - Import chain: flash_attn_func + CoreXFA2Class + varlen

serving_chat.py (+23 lines):
  - max_tokens clamp: fixes OpenCompass 0 score (5 benchmarks all 400)
  - n>1 fanout: remove temperature==0 restriction for t2_n_2 FAIL

api_server.py (+33 lines):
  - HTTP middleware: strip index from messages before pydantic validation
  - Fixes ValidatorIterator 0.index Extra inputs are not permitted x6
 2026-08-16 15:37:23 +00:00
root
522e8376b6 data: sub 694 (683分) 日志 + resolve merge 2026-08-16 15:28:08 +00:00
Claude
4189f44d27 test: dump ALL symbols from ALL ixformer/cuinfer .so — no grep filter, find what we missed 2026-08-16 04:58:47 +00:00
Claude
8eabbac857 test: probe_model_shapes.sh — 真机验证模型config和MoE权重shape 2026-08-15 15:05:26 +00:00
Claude
b7149f810a fix: decode MoE路径对齐base — F.linear+bmm替换pre-transpose+bmm 
base qwen3_5.py的decode路径（已验证可跑通竞赛）:
  F.linear(hidden, w13_sel.reshape(-1,H)) → view → act → bmm(w2_sel, act)

我们之前的路径（未验证，probe显示更慢）:
  pre-transpose(w13全量) → w13_t[eids] → bmm(x_expand, w13_t_sel) → act → bmm(act, w2_t_sel)

probe真机数据: loop matmul 19ms < torch.bmm 24ms
说明F.linear路径在BI-V100单token场景下更优

保持的corex加速:
  ✓ corex_moe_topk_softmax (topk+softmax fused)
  ✓ corex_moe_weight_gather (gather fused)
  ✓ corex_moe_exact_reduce (weighted sum fused)
  ✓ corex_moe_index_combine (prefill token routing fused)
 2026-08-15 14:56:40 +00:00
Claude
ea2c15f699 fix: decode MoE路径对齐base — F.linear+bmm替换pre-transpose+bmm 
base qwen3_5.py的decode路径（已验证可跑通竞赛）:
  F.linear(hidden, w13_sel.reshape(-1,H)) → view → act → bmm(w2_sel, act)

我们之前的路径（未验证，probe显示更慢）:
  pre-transpose(w13全量) → w13_t[eids] → bmm(x_expand, w13_t_sel) → act → bmm(act, w2_t_sel)

probe真机数据: loop matmul 19ms < torch.bmm 24ms
说明F.linear路径在BI-V100单token场景下更优

保持的corex加速:
  ✓ corex_moe_topk_softmax (topk+softmax fused)
  ✓ corex_moe_weight_gather (gather fused)
  ✓ corex_moe_exact_reduce (weighted sum fused)
  ✓ corex_moe_index_combine (prefill token routing fused)
 2026-08-15 14:55:05 +00:00