dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
707 Commits 1 Branch 0 Tags
f7070b20759e1f9e1cae6cdc9c1dce36ab575c9f
Commit Graph

707 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
root
f7070b2075 platform test baseline4 2026-08-17 07:19:44 +00:00
root
512f384a49 Merge branch 'main' of https://dev.modelhub.org.cn/dylanyunlong/project_6 into main 2026-08-17 04:19:31 +00:00
root
cdcf115037 feat: CUTLASS Cu10 grouped GEMM — real device verified 
BI-V100 real device results:
  moe_group_gemm: err=0.000015 PASS
  moe_decode_cutlass: NaN=False PASS
  cutlass grouped: 4.77ms vs torch.mm loop: 9.38ms → 1.97x speedup

Fix: gemm_grouped.cu ldb=K (not N) for ColumnMajor B view
Link: -lcuinfer from /usr/local/corex-3.2.3/lib64/libcuinfer.so.7
 2026-08-17 04:18:29 +00:00
root
cb926707af Merge branch 'main' of https://github.com/dylanyunlon/project_6 2026-08-17 02:21:09 +00:00
root
beaa8dbb65 Merge branch 'main' of https://dev.modelhub.org.cn/dylanyunlong/project_6 2026-08-17 02:18:46 +00:00
root
03be5f2b15 [feat] group gemm 2026-08-17 02:16:58 +00:00
Claude
330669b309 Revert "fix: ix_full_bridge_v2.cpp — align namespace+signatures to real nm -D symbol dump" 
This reverts commit 5c03156978.
 2026-08-17 02:08:03 +00:00
Claude
5c03156978 fix: ix_full_bridge_v2.cpp — align namespace+signatures to real nm -D symbol dump 
Non-MoE functions: ixformer::infer → ixformer_torch_ext (real namespace)
  silu_and_mul_forward, rms_norm_forward, fused_add_rms_norm_forward,
  ixformer_linear, ixformer_linear_ex, vllm_rotary_embedding_neox,
  vllm_cache_ops_reshape_and_cache, vllm_single_query_cached_kv_attention

MoE functions: keep ixformer::infer (provided by moe_ops_impl.cu)
  topk_softmax, moe_compute_token_index_api, moe_expand_input,
  moe_w16a16_group_gemm, moe_output_reduce_sum

Removed: flash_attn_prefill, xllm_paged_attention (not in any .so)
Fixed: c10::optional vs std::optional, parameter counts, arg order
 2026-08-17 02:04:55 +00:00
project_6
34a8fbf27e revert: undo 2 premature pushes (c54923a1, 49034d1d) — code needs review first 2026-08-16 17:48:17 +00:00
project_6
49034d1d09 feat: 10-file MoE bridge pipeline — compile, dispatch, patch, test 
The complete chain to replace 180 Python fallback calls/token with C++:

BUILD:
  1. moe_ops_impl.cu (489L) — 5 MoE functions in ixformer::infer namespace
     - topk_softmax: dynamic num_experts (128 for Qwen3.5), shared-mem
     - moe_compute_token_index: histogram + prefix_sum + scatter
     - moe_expand_input: gather kernel
     - moe_w16a16_group_gemm: per-expert cuinferCustomGemm loop
     - moe_output_reduce_sum: weighted combine
  2. ix_full_bridge_v2.cpp (461L) — pybind11 bridge, 14+1 functions
  3. build_moe_bridge.sh — torch.utils.cpp_extension compile, link cuinfer+ixformer

DISPATCH:
  4. moe_dispatch.py — 3-tier fallback (fused → individual → PyTorch)
  5. patch_moe_hot_path.py — monkey-patch Qwen3_5MoE.forward()

CONFIG:
  6. computility-run.yaml — max_num_seqs 1→2 (match sub168 baseline)
  7. patch_ops.sh — add build + deploy steps for MoE bridge

VERIFY:
  8. probe_moe_symbols.sh — nm -D .so to confirm 5 MoE symbols present
  9. test_moe_bridge.py — random-tensor integration test (no weights needed)

DEPLOY:
 10. Dockerfile — COPY ex_engine sources for in-container compilation
 2026-08-16 17:46:53 +00:00
project_6
c54923a17e feat: implement 5 missing MoE ops — topk_softmax + token_index + expand + group_gemm + combine 
Symbol dump from real device confirms: libixformer.so has 0 MoE symbols.
topk_softmax, moe_compute_token_index_api, moe_expand_input,
moe_w16a16_group_gemm, moe_output_reduce_sum — all missing.

Non-MoE symbols (silu_and_mul, rms_norm, flash_attn, reshape_and_cache,
rotary_embedding) are present and working.

Implementation strategy — use available primitives:
- topk_softmax: pure CUDA kernel (64-expert, shared-mem argmax)
- moe_compute_token_index: histogram + prefix_sum + scatter (3 kernels)
- moe_expand_input: gather kernel
- moe_w16a16_group_gemm: per-expert loop calling cuinferCustomGemm
  (confirmed in libcuinfer.so symbol dump: cuinferCustomGemm exists)
- moe_output_reduce_sum: weighted combine kernel

All in ixformer::infer namespace so ix_full_bridge_v2.cpp links directly.
Compile: nvcc moe_ops_impl.cu + ix_full_bridge_v2.cpp → single .so
 2026-08-16 17:35:08 +00:00
root
3712c06861 data: full symbol dumps 2026-08-16 17:17:26 +00:00
root
dec268d252 data: full symbol dumps 2026-08-16 17:15:11 +00:00
project_6
415ca12afc fix: group_gemm format "TN" + Layer 3 ops_api dispatch from xllm upstream 
AST chain alignment with upstream_ref/xllm/xllm/core/kernels/ilu/:

Layer 5: ixformer::infer (binary .so on device)
Layer 4: xllm_kernels/ilu/*.cpp -> calls ixformer::infer (0-diff with upstream)
Layer 3: xllm_kernels/ops_api.h+cpp + param.h (NEW from upstream 2719 lines)
         kernels/kernels.h aggregation header (NEW)
Layer 2: xllm_layers/ilu/*.cpp (0-diff with upstream)
Layer 1: ix_full_bridge_v2.cpp pybind11 bridge (FIXED)

Critical fixes in ix_full_bridge_v2.cpp:
- group_gemm format "default" -> "TN" (match upstream ilu/group_gemm.cpp)
- fused_moe_forward: pass 3D weights directly instead of .view({-1,...})
- group_gemm output_n: use tokens_per_experts.sum() per upstream convention
 2026-08-16 16:09:15 +00:00
Claude
5172f94b1f Revert "feat: 3-tier ixformer flash prefill dispatch + OpenCompass max_tokens clamp + n>1 fanout + index sanitizer" 
This reverts commit cdec569977.
 2026-08-16 15:42:36 +00:00
Claude
7a7ddf38db Revert "test: deploy_and_verify.sh — pull+patch+probe ixformer backends+clamp test" 
This reverts commit 587e18309b.
 2026-08-16 15:42:36 +00:00
Claude
587e18309b test: deploy_and_verify.sh — pull+patch+probe ixformer backends+clamp test 2026-08-16 15:40:47 +00:00
Claude
cdec569977 feat: 3-tier ixformer flash prefill dispatch + OpenCompass max_tokens clamp + n>1 fanout + index sanitizer 
paged_attn.py (+218 lines):
  - Tier 0: ixformer flash_attn_varlen_func (cu_seqlens packed)
  - Tier 0.5: ixformer flash_attn_func (non-varlen, batch layout)
  - Tier 1: CoreXFA2 3-mode dispatch (packed/paged/chunked)
  - Tier 2 fallback: existing Python Q-tiling (unchanged)
  - Import chain: flash_attn_func + CoreXFA2Class + varlen

serving_chat.py (+23 lines):
  - max_tokens clamp: fixes OpenCompass 0 score (5 benchmarks all 400)
  - n>1 fanout: remove temperature==0 restriction for t2_n_2 FAIL

api_server.py (+33 lines):
  - HTTP middleware: strip index from messages before pydantic validation
  - Fixes ValidatorIterator 0.index Extra inputs are not permitted x6
 2026-08-16 15:37:23 +00:00
root
522e8376b6 data: sub 694 (683分) 日志 + resolve merge 2026-08-16 15:28:08 +00:00
Claude
4189f44d27 test: dump ALL symbols from ALL ixformer/cuinfer .so — no grep filter, find what we missed 2026-08-16 04:58:47 +00:00
Claude
8eabbac857 test: probe_model_shapes.sh — 真机验证模型config和MoE权重shape 2026-08-15 15:05:26 +00:00
Claude
b7149f810a fix: decode MoE路径对齐base — F.linear+bmm替换pre-transpose+bmm 
base qwen3_5.py的decode路径（已验证可跑通竞赛）:
  F.linear(hidden, w13_sel.reshape(-1,H)) → view → act → bmm(w2_sel, act)

我们之前的路径（未验证，probe显示更慢）:
  pre-transpose(w13全量) → w13_t[eids] → bmm(x_expand, w13_t_sel) → act → bmm(act, w2_t_sel)

probe真机数据: loop matmul 19ms < torch.bmm 24ms
说明F.linear路径在BI-V100单token场景下更优

保持的corex加速:
  ✓ corex_moe_topk_softmax (topk+softmax fused)
  ✓ corex_moe_weight_gather (gather fused)
  ✓ corex_moe_exact_reduce (weighted sum fused)
  ✓ corex_moe_index_combine (prefill token routing fused)
 2026-08-15 14:56:40 +00:00
Claude
ea2c15f699 fix: decode MoE路径对齐base — F.linear+bmm替换pre-transpose+bmm 
base qwen3_5.py的decode路径（已验证可跑通竞赛）:
  F.linear(hidden, w13_sel.reshape(-1,H)) → view → act → bmm(w2_sel, act)

我们之前的路径（未验证，probe显示更慢）:
  pre-transpose(w13全量) → w13_t[eids] → bmm(x_expand, w13_t_sel) → act → bmm(act, w2_t_sel)

probe真机数据: loop matmul 19ms < torch.bmm 24ms
说明F.linear路径在BI-V100单token场景下更优

保持的corex加速:
  ✓ corex_moe_topk_softmax (topk+softmax fused)
  ✓ corex_moe_weight_gather (gather fused)
  ✓ corex_moe_exact_reduce (weighted sum fused)
  ✓ corex_moe_index_combine (prefill token routing fused)
 2026-08-15 14:55:05 +00:00
root
b187f52ced data: so import chain probe 2026-08-15 14:53:57 +00:00
root
ee62ea13ba data: so import chain probe 2026-08-15 14:50:29 +00:00
root
924e48b502 data: so import chain probe 2026-08-15 14:49:42 +00:00
Claude
1700c35bd7 test: probe_so_import_chain.sh — 验证.so部署路径+import链+flag值+shape匹配 2026-08-15 14:48:47 +00:00
Claude
c290278b35 test: probe_so_import_chain.sh — 验证.so部署路径+import链+flag值+shape匹配 2026-08-15 14:46:49 +00:00
root
a6cc233880 data: base MoE forward + corex_moe签名 2026-08-15 14:41:09 +00:00
root
784dea96c0 data: base MoE forward + corex_moe签名 2026-08-15 14:39:03 +00:00
Claude
adf05b6bfb test: probe_base_moe_forward.sh — cat base qwen3_5.py的完整MoE forward + 所有corex_moe_*.so签名 2026-08-15 14:37:48 +00:00
Claude
7a8545f7c2 test: push_probe_results.sh — 真机commit probe结果到modelhub 2026-08-15 14:37:48 +00:00
Claude
b00429d81c test: probe_base_moe_forward.sh — cat base qwen3_5.py的完整MoE forward + 所有corex_moe_*.so签名 2026-08-15 14:36:29 +00:00
root
6a4459d405 data: probe bridge output 2026-08-15 14:35:35 +00:00
root
437ad8aaa4 data: probe bridge output 2026-08-15 14:35:07 +00:00
Claude
6e22415a91 test: push_probe_results.sh — 真机commit probe结果到modelhub 2026-08-15 14:34:49 +00:00
Claude
de1212c271 test: probe_ix_unified_bridge.sh — cat base镜像的ix_unified_bridge + corex_*.so + _custom_ops.py完整接口 2026-08-15 14:30:53 +00:00
Claude
a823bdf9ea test: probe_real_machine.sh — cat ixformer/vllm/cublas真机数据 2026-08-15 14:28:10 +00:00
Claude
6415249693 data: port complete MoE + xllm layer call chains from upstream repos 
MoE call chain from ds_vllm (vllm-project/vllm latest):
  ex_engine/moe/ — 20 files, 8736 lines
  - modular_kernel.py (1630 lines) — base classes for modular MoE
  - experts/fused_batched_moe.py (972 lines) — NaiveBatchedExperts
  - prepare_finalize/batched.py (171 lines) — token grouping by expert
  - topk_weight_and_reduce.py (176 lines) — scatter-add finalize
  - fused_moe.py (1740 lines) — main fused_moe dispatch
  - config.py (1407 lines) — FusedMoEQuantConfig
  - activation.py, utils.py, layer.py, etc.

xllm layer code (jd-opensource/xllm):
  ex_engine/xllm_layers/ — 39 files, 5859 lines
  - ilu/fused_moe.cpp (797 lines) — production ixformer 7-step MoE pipeline
  - ilu/attention.cpp (189 lines) — paged_attention + flash_attn bridge
  - npu_torch/qwen3_gated_delta_net_base.cpp (576 lines) — GDN reference
  - common/rms_norm.cpp, rotary_embedding.cpp, activation.cpp, dense_mlp.cpp

xllm ILU kernels — synced 10 files to upstream (diffs from prior edits)

These are reference implementations, NOT hand-written.
Source repos: vllm-project/vllm, jd-opensource/xllm
 2026-08-15 14:26:24 +00:00
dylan
7aa5054574 feat: ILU kernel pipeline — ix_full_bridge_v2 build + deploy + 7-step MoE dispatch 
System design: algorithm factor replacement, not a connector.
All ops go through ixformer::infer C++ namespace (no Python fallback).

New files:
  build_ix_bridge.sh        — compile ix_full_bridge_v2.cpp on BI-V100
  build_xllm_ilu_kernels.sh — compile upstream xllm ILU wrappers
  deploy_ilu_pipeline.sh    — wire everything into patch_ops.sh
  ix_ops_dispatch.py        — runtime dispatcher (12 ops via C++ bridge)
  corex_fa2_dispatch.py     — 3-mode attention (prefill/v1/flash paged)
  fused_moe_ilu.py          — 7-step MoE pipeline (no expert for-loop)

Upstream sources used (not rewritten):
  xllm/core/kernels/ilu/*.cpp  (ILU kernel wrappers)
  xllm/core/kernels/ilu/ixformer.h (14 C++ function declarations)
  ds_vllm/csrc/libtorch_stable/*.cu (kernel references)

Call chain:
  patch_ops.sh → deploy_ilu_pipeline.sh → build_ix_bridge.sh
    → ix_full_bridge_v2.so → ixformer::infer::*
    → silu_and_mul, rms_norm, rotary_embedding, paged_attention,
      topk_softmax, group_gemm, expand_input, combine_result
 2026-08-15 14:15:47 +00:00
Claude
52e2ef31a8 feat: xllm_ops NO-FALLBACK kernel loader + 6 missing .so build targets + hot-path patcher 
Build infrastructure:
- build_xllm_kernels.sh: add 5 missing build targets (norm, rope, activation, cache, moe)
  Previously only built xllm_fused_qknorm_rope.so, now builds all 6 .so files

Kernel loader (xllm_ops.py):
- NO-FALLBACK architecture matching xllm/core/kernels/ilu/ dispatch chain
- Loads: xllm_norm.so, xllm_rope.so, xllm_activation.so, xllm_cache.so,
         xllm_moe.so, ix_full_bridge.so, xllm_fused_qknorm_rope.so
- check_all(strict=True) verifies ALL .so at startup

Hot-path patcher (patch_vllm_hot_path.py):
- Monkey-patches vllm._custom_ops to route through xllm .so
- Critical fix: topk_softmax patch prevents comp 168 cascade
- Patches: topk_softmax, rms_norm, silu_and_mul, rotary_embedding, reshape_and_cache

Source mapping: xllm/core/kernels/ilu/*.cpp -> our xllm_*.so files
 2026-08-15 14:13:13 +00:00
dylan
e873e5f27b fix: eliminate 8x CUDA sync in MoE decode — tolist() once instead of .item() per expert 2026-08-15 13:14:11 +00:00
dylan
23fe535985 fix: add ex_engine/__init__.py for Python package import 2026-08-15 13:08:24 +00:00
dylan
e18ece8f3a feat: port NaiveBatchedExperts from ds_vllm — view transpose + cublas transB 
Source: upstream_ref/ds_vllm/vllm/model_executor/layers/fused_moe/experts/fused_batched_moe.py
        upstream_ref/ds_vllm/vllm/model_executor/layers/fused_moe/activation.py

New files (ported from ds_vllm, adapted for BI-V100):
  ex_engine/moe/__init__.py
  ex_engine/moe/activation.py
    - MoEActivation enum + apply_moe_activation
    - torch.ops._C.silu_and_mul replaced with F.silu(gate)*up fallback
  ex_engine/moe/naive_batched_experts.py
    - naive_batched_moe_forward()
    - Decode: per-expert loop, w13[eid].transpose(0,1) is VIEW (zero copy)
    - @ operator → cublas passes transB=CUBLAS_OP_T internally
    - Prefill: group tokens by expert, batch @ per expert

Modified:
  qwen3_6_scripts/qwen3_5.py
    - Import naive_batched_moe_forward
    - Tier 0.5: after ix_fused_moe, before corex point-optimized loop
    - Uses existing topk routing (xllm/corex/pytorch)

Key difference from previous approach:
  - NO physical transpose (was 22ms overhead)
  - NO weight gather into contiguous buffer
  - View transpose is O(0), cublas handles transB
 2026-08-15 13:05:45 +00:00
dylan
6f1904aa8c perf: MoE decode — pre-transposed bmm replaces F.linear (6.9ms vs 8.0ms, 14%) 
Probe data (probe_moe_fused_breakdown.sh on BI-V100):
  F.linear loop 8 experts:     8.060 ms
  bmm pre-transposed full MoE: 6.918 ms  ← 14% faster
  transpose+contiguous runtime: 22.219 ms ← why CUTLASS was 27ms

Changes:
  - Lazy-cache w13_t (E,H,2I) and w2_t (E,I,H) on first decode call
  - FC1: torch.bmm(x_expand, w13_t_sel) replaces F.linear(x, w13_sel.reshape)
  - FC2: torch.bmm(act, w2_t_sel) replaces torch.bmm(w2_sel, act^T)
  - Zero runtime transpose cost after first call
 2026-08-15 12:52:55 +00:00
Claude
9f265894cc test: MoE breakdown — F.linear vs torch.mm vs torch.bmm vs bmm pre-transposed 2026-08-15 12:46:37 +00:00
Claude
e47b66e268 fix: module name in probe_moe_fused_breakdown.sh 2026-08-15 12:43:55 +00:00
Claude
b0af7d54ff test: breakdown moe_decode_fused timing by step — find the real bottleneck 2026-08-15 12:41:33 +00:00
dylan
0795e064b2 fix: corex_batched_gemm TCU OpClassTensorOp + Cu10 + float accum (merge) 2026-08-15 12:36:18 +00:00
dylan
3b2a0bc4d3 fix: corex_batched_gemm TCU OpClassTensorOp + Cu10 + float accum 2026-08-15 12:36:06 +00:00