dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
676 Commits 1 Branch 0 Tags
1700c35bd7d00a5001bbe226760d7e46bd56fcb6
Commit Graph

676 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Claude
1700c35bd7 test: probe_so_import_chain.sh — 验证.so部署路径+import链+flag值+shape匹配 2026-08-15 14:48:47 +00:00
root
784dea96c0 data: base MoE forward + corex_moe签名 2026-08-15 14:39:03 +00:00
Claude
adf05b6bfb test: probe_base_moe_forward.sh — cat base qwen3_5.py的完整MoE forward + 所有corex_moe_*.so签名 2026-08-15 14:37:48 +00:00
Claude
7a8545f7c2 test: push_probe_results.sh — 真机commit probe结果到modelhub 2026-08-15 14:37:48 +00:00
root
437ad8aaa4 data: probe bridge output 2026-08-15 14:35:07 +00:00
Claude
de1212c271 test: probe_ix_unified_bridge.sh — cat base镜像的ix_unified_bridge + corex_*.so + _custom_ops.py完整接口 2026-08-15 14:30:53 +00:00
Claude
a823bdf9ea test: probe_real_machine.sh — cat ixformer/vllm/cublas真机数据 2026-08-15 14:28:10 +00:00
Claude
6415249693 data: port complete MoE + xllm layer call chains from upstream repos 
MoE call chain from ds_vllm (vllm-project/vllm latest):
  ex_engine/moe/ — 20 files, 8736 lines
  - modular_kernel.py (1630 lines) — base classes for modular MoE
  - experts/fused_batched_moe.py (972 lines) — NaiveBatchedExperts
  - prepare_finalize/batched.py (171 lines) — token grouping by expert
  - topk_weight_and_reduce.py (176 lines) — scatter-add finalize
  - fused_moe.py (1740 lines) — main fused_moe dispatch
  - config.py (1407 lines) — FusedMoEQuantConfig
  - activation.py, utils.py, layer.py, etc.

xllm layer code (jd-opensource/xllm):
  ex_engine/xllm_layers/ — 39 files, 5859 lines
  - ilu/fused_moe.cpp (797 lines) — production ixformer 7-step MoE pipeline
  - ilu/attention.cpp (189 lines) — paged_attention + flash_attn bridge
  - npu_torch/qwen3_gated_delta_net_base.cpp (576 lines) — GDN reference
  - common/rms_norm.cpp, rotary_embedding.cpp, activation.cpp, dense_mlp.cpp

xllm ILU kernels — synced 10 files to upstream (diffs from prior edits)

These are reference implementations, NOT hand-written.
Source repos: vllm-project/vllm, jd-opensource/xllm
 2026-08-15 14:26:24 +00:00
dylan
7aa5054574 feat: ILU kernel pipeline — ix_full_bridge_v2 build + deploy + 7-step MoE dispatch 
System design: algorithm factor replacement, not a connector.
All ops go through ixformer::infer C++ namespace (no Python fallback).

New files:
  build_ix_bridge.sh        — compile ix_full_bridge_v2.cpp on BI-V100
  build_xllm_ilu_kernels.sh — compile upstream xllm ILU wrappers
  deploy_ilu_pipeline.sh    — wire everything into patch_ops.sh
  ix_ops_dispatch.py        — runtime dispatcher (12 ops via C++ bridge)
  corex_fa2_dispatch.py     — 3-mode attention (prefill/v1/flash paged)
  fused_moe_ilu.py          — 7-step MoE pipeline (no expert for-loop)

Upstream sources used (not rewritten):
  xllm/core/kernels/ilu/*.cpp  (ILU kernel wrappers)
  xllm/core/kernels/ilu/ixformer.h (14 C++ function declarations)
  ds_vllm/csrc/libtorch_stable/*.cu (kernel references)

Call chain:
  patch_ops.sh → deploy_ilu_pipeline.sh → build_ix_bridge.sh
    → ix_full_bridge_v2.so → ixformer::infer::*
    → silu_and_mul, rms_norm, rotary_embedding, paged_attention,
      topk_softmax, group_gemm, expand_input, combine_result
 2026-08-15 14:15:47 +00:00
Claude
52e2ef31a8 feat: xllm_ops NO-FALLBACK kernel loader + 6 missing .so build targets + hot-path patcher 
Build infrastructure:
- build_xllm_kernels.sh: add 5 missing build targets (norm, rope, activation, cache, moe)
  Previously only built xllm_fused_qknorm_rope.so, now builds all 6 .so files

Kernel loader (xllm_ops.py):
- NO-FALLBACK architecture matching xllm/core/kernels/ilu/ dispatch chain
- Loads: xllm_norm.so, xllm_rope.so, xllm_activation.so, xllm_cache.so,
         xllm_moe.so, ix_full_bridge.so, xllm_fused_qknorm_rope.so
- check_all(strict=True) verifies ALL .so at startup

Hot-path patcher (patch_vllm_hot_path.py):
- Monkey-patches vllm._custom_ops to route through xllm .so
- Critical fix: topk_softmax patch prevents comp 168 cascade
- Patches: topk_softmax, rms_norm, silu_and_mul, rotary_embedding, reshape_and_cache

Source mapping: xllm/core/kernels/ilu/*.cpp -> our xllm_*.so files
 2026-08-15 14:13:13 +00:00
dylan
e873e5f27b fix: eliminate 8x CUDA sync in MoE decode — tolist() once instead of .item() per expert 2026-08-15 13:14:11 +00:00
dylan
23fe535985 fix: add ex_engine/__init__.py for Python package import 2026-08-15 13:08:24 +00:00
dylan
e18ece8f3a feat: port NaiveBatchedExperts from ds_vllm — view transpose + cublas transB 
Source: upstream_ref/ds_vllm/vllm/model_executor/layers/fused_moe/experts/fused_batched_moe.py
        upstream_ref/ds_vllm/vllm/model_executor/layers/fused_moe/activation.py

New files (ported from ds_vllm, adapted for BI-V100):
  ex_engine/moe/__init__.py
  ex_engine/moe/activation.py
    - MoEActivation enum + apply_moe_activation
    - torch.ops._C.silu_and_mul replaced with F.silu(gate)*up fallback
  ex_engine/moe/naive_batched_experts.py
    - naive_batched_moe_forward()
    - Decode: per-expert loop, w13[eid].transpose(0,1) is VIEW (zero copy)
    - @ operator → cublas passes transB=CUBLAS_OP_T internally
    - Prefill: group tokens by expert, batch @ per expert

Modified:
  qwen3_6_scripts/qwen3_5.py
    - Import naive_batched_moe_forward
    - Tier 0.5: after ix_fused_moe, before corex point-optimized loop
    - Uses existing topk routing (xllm/corex/pytorch)

Key difference from previous approach:
  - NO physical transpose (was 22ms overhead)
  - NO weight gather into contiguous buffer
  - View transpose is O(0), cublas handles transB
 2026-08-15 13:05:45 +00:00
dylan
6f1904aa8c perf: MoE decode — pre-transposed bmm replaces F.linear (6.9ms vs 8.0ms, 14%) 
Probe data (probe_moe_fused_breakdown.sh on BI-V100):
  F.linear loop 8 experts:     8.060 ms
  bmm pre-transposed full MoE: 6.918 ms  ← 14% faster
  transpose+contiguous runtime: 22.219 ms ← why CUTLASS was 27ms

Changes:
  - Lazy-cache w13_t (E,H,2I) and w2_t (E,I,H) on first decode call
  - FC1: torch.bmm(x_expand, w13_t_sel) replaces F.linear(x, w13_sel.reshape)
  - FC2: torch.bmm(act, w2_t_sel) replaces torch.bmm(w2_sel, act^T)
  - Zero runtime transpose cost after first call
 2026-08-15 12:52:55 +00:00
Claude
9f265894cc test: MoE breakdown — F.linear vs torch.mm vs torch.bmm vs bmm pre-transposed 2026-08-15 12:46:37 +00:00
Claude
e47b66e268 fix: module name in probe_moe_fused_breakdown.sh 2026-08-15 12:43:55 +00:00
Claude
b0af7d54ff test: breakdown moe_decode_fused timing by step — find the real bottleneck 2026-08-15 12:41:33 +00:00
dylan
0795e064b2 fix: corex_batched_gemm TCU OpClassTensorOp + Cu10 + float accum (merge) 2026-08-15 12:36:18 +00:00
dylan
3b2a0bc4d3 fix: corex_batched_gemm TCU OpClassTensorOp + Cu10 + float accum 2026-08-15 12:36:06 +00:00
dylan
e2fc3f270f fix: corex_batched_gemm use TCU OpClassTensorOp + Cu10 + float accum 
Previous version used default SIMT path (25ms).
Fixed version matches moe_cutlass_batched.cu (2.462ms):
  - ElementAccumulator = float (was half_t)
  - OpClassTensorOp (was default OpClassSimt)
  - arch::Cu10 (was default Sm61)
  - RowMajor layout (was ColumnMajor)
  - torch::sigmoid(x)*x instead of torch::silu (not in corex torch)

Also fixed bind.cpp: removed col-major transposition logic,
kernel now RowMajor so A(M,K) @ B(K,N) = C(M,N) directly.
 2026-08-15 12:35:56 +00:00
Claude
d8d241bf9f fix: corex_batched_gemm_kernel — use OpClassTensorOp + arch::Cu10 + FP32 accumulator 
Root cause of 25ms (vs expected 2.5ms):
1. ElementAccumulator was half_t → now float (FP32 accumulation)
2. Missing OpClassTensorOp → was defaulting to OpClassSimt (CUDA cores only)
3. Missing arch::Cu10 → was defaulting to arch::Sm61

With these fixes it should use __ivcorex_matrix_mad_f32x4_f16x4 (TCU)
same as moe_cutlass_batched.cu which benchmarked at 2.462ms.
 2026-08-15 12:34:01 +00:00
dylan
3481f2903f fix(build): cutlass.h lives under tensorflow/include on this image 2026-08-15 12:03:12 +00:00
dylan
f41900c06b fix(build): auto-find cutlass/cutlass.h under COREX_ROOT 2026-08-15 12:02:23 +00:00
dylan
bfa18cd5b4 fix(build): use CoreX clang++ instead of nvcc — match working build scripts 2026-08-15 12:01:17 +00:00
dylan
04cc9b88af fix(build): add CUDA include path to g++ step in build_corex_batched_gemm.sh 2026-08-15 11:58:53 +00:00
dylan
ddcfbad431 feat: pybind wrapper for CUTLASS batched GEMM → MoE decode path 
Based on verified result (issue #68):
  CUTLASS Cu10 TensorOp batched: 2.462ms (8 experts, 1 launch)
  vs 8× torch.matmul: 4.6ms (8 launches)
  vs Python F.linear loop: 10.36ms

New files:
  ex_engine/xllm_kernels/cuda/bindings/corex_batched_gemm_bind.cpp
    pybind11 wrapper: batched_gemm_fp16() + moe_decode_fused()
  ex_engine/xllm_kernels/cuda/corex_batched_gemm_kernel.cu
    CUTLASS GemmBatched<half> kernel (from cat_files/batched_gemm.cu)
  qwen3_6_scripts/build_corex_batched_gemm.sh
    Build script for BI-V100 (ivcore10)

Modified:
  qwen3_6_scripts/qwen3_5.py
    import corex_batched_gemm + _USE_COREX_BATCHED_GEMM flag
    Tier 1.5 in MoE decode: after corex_direct_routed, before corex_gather

Build on device: bash qwen3_6_scripts/build_corex_batched_gemm.sh
Output: prebuilt/corex-3.2.3-ivcore10/corex_batched_gemm.so
 2026-08-15 11:54:26 +00:00
dylan
a875fa5d4c Revert "data: cat SGEMM files from 3 repos into cat_files/" 
This reverts commit 7cfa87b5ac.
 2026-08-15 11:48:21 +00:00
Claude
36676f2d1b data: complete SGEMM upstream from 3 repos (siboehm+wangzyon+edtallison) + xllm fused_qknorm_rope + xattention kernels 
SGEMM repos (upstream_ref/sgemm_cuda/, 41 files):
  siboehm/SGEMM_CUDA: kernel 1-12, runner, CMake, cuBLAS benchmark
  wangzyon/NVIDIA_SGEMM_PRACTICE: kernel 1-7 (Chinese comments), utils
  edtallison/sgemm-cuda: kernel 01-09 (learning notes), Makefile

xllm kernels (ex_engine/xllm_kernels/cuda/):
  fused_qknorm_rope.cu + bind — saves 128 kernel launches/fwd
  xattention/ — 6 files from upstream xllm
  headers: corex_compat_utils.h, topk_last_dim.cuh
  ilu/CMakeLists.txt

SO_BUILD_MANIFEST.md — complete .so inventory and call chain analysis
 2026-08-15 07:00:09 +00:00
dylan
7cfa87b5ac data: cat SGEMM files from 3 repos into cat_files/ 
siboehm/SGEMM_CUDA (19 files):
  siboehm_sgemm.cu, siboehm_runner.cu, siboehm_runner.cuh, siboehm_kernels.cuh
  siboehm_cuBLAS_sgemm.cu, siboehm_simplest_kernel.cu, siboehm_CMakeLists.txt
  siboehm_{1_naive..12_kernel_double_buffering}.cuh

wangzyon/NVIDIA_SGEMM_PRACTICE (12 files):
  wangzyon_sgemm.cu, wangzyon_utils.cu, wangzyon_utils.cuh, wangzyon_kernel.cuh
  wangzyon_CMakeLists.txt, wangzyon_kernel_{1..7}.cuh

edtallison/sgemm-cuda (19 files):
  edtallison_sgemm.cu, edtallison_runner.cu, edtallison_runner.cuh
  edtallison_kernels.cuh, edtallison_cuBLAS_sgemm.cu, edtallison_simplest_kernel.cu
  edtallison_CMakeLists.txt, edtallison_{01_naive..12_kernel_double_buffering}.cuh

cat_files/ total: 25 → 75 files
 2026-08-15 06:59:18 +00:00
dylan
284804ac53 data: cat 3 SGEMM repos — siboehm, wangzyon, edtallison (full clone, no --depth) 
Sources:
  siboehm/SGEMM_CUDA        → upstream_ref/sgemm_siboehm/     (25 files)
  wangzyon/NVIDIA_SGEMM_PRACTICE → upstream_ref/nvidia_sgemm_practice/ (23 files, filled gaps)
  edtallison/sgemm-cuda      → upstream_ref/sgemm_edtallison/  (41 files)

All files cat'd one by one from git clone (no --depth).
These are the 3 public SGEMM repos that can compile on CUDA 10.2 + CoreX ivcore10.

Key files for BI-V100 porting:
  kernel 10 (warp tiling) — already proven on device with WARPSIZE=64
  kernel 11/12 (double buffering) — next optimization target
  sgemm.cu + runner.cu — complete build+benchmark harness
  CMakeLists.txt — build system reference
 2026-08-15 06:58:07 +00:00
dylan
854fb93a8e test: add test_ex_engine_cuda.py — test all 22 prebuilt .so on BI-V100 
Tests xllm_norm, xllm_activation, xllm_rope, xllm_moe, ix_full_bridge,
and 11 corex_*.so modules with correctness checks against PyTorch reference.

Run: python3 test_ex_engine_cuda.py
 2026-08-15 06:28:34 +00:00
dylan
e8f0948fe1 feat: ix_ops integration layer — wire ix_full_bridge.so into vllm hot path 
Architecture (CCCL dispatch pattern):
  base_image ixformer::infer → ix_full_bridge.so → ix_ops.py → vllm patches

New files:
  ex_engine/python/ix_ops.py          — Python API for all 14 ixformer::infer ops
  ex_engine/python/patch_vllm_ops.py  — monkey-patch vllm GemmaRMSNorm, SiluAndMul
  ex_engine/deploy_ix_bridge.sh       — build-time deployment script

Modified:
  qwen3_6_scripts/patch_ops.sh        — integrated ix_bridge deployment + startup hook

Call chain: DecoderLayer.forward → GemmaRMSNorm → ix_ops.fused_add_rms_norm
            → ixformer::infer::residual_rms_norm (fused C++ kernel)
 2026-08-15 06:15:17 +00:00
root
109d29fa60 Merge remote-tracking branch 'modelhub/main' 2026-08-15 05:58:37 +00:00
Claude
045ea5df79 feat: Cu10 TensorOp batched HGEMM via Iluvatar CUTLASS framework 
From corex-samples batched_gemm.cu, changed:
  float → half_t, OpClassSimt → OpClassTensorOp, Sm61 → Cu10
Uses __ivcorex_matrix_mad_f32x4_f16x4 via mma_cu10.h
Default config: TB<128,128,32> Warp<32,32,32> Inst<16,16,16>

Standalone test: correctness + perf for MoE decode (8 × 1x4096@4096x11008)
 2026-08-15 05:44:15 +00:00
root
30f98c0674 data: Cu10 CUTLASS part 2 — tensorop example, arch.h, cutlass.h 2026-08-15 05:41:30 +00:00
Claude
f006ab1a01 test: cat tensorop GEMM example + arch.h + cutlass.h from corex-samples 2026-08-15 05:41:03 +00:00
root
b922d694dc data: Cu10 CUTLASS headers from corex-samples 2026-08-15 05:32:10 +00:00
Claude
1d36754efc fix: cat_cutlass_cu10.sh writes to cat_files/ directory instead of stdout 2026-08-15 05:31:33 +00:00
Claude
4abb4df215 test: cat Cu10 CUTLASS files — mma_cu10.h, iluvatar_mma.hpp, batched_gemm.cu, default_mma_core_cu10.h 2026-08-15 05:30:15 +00:00
Claude
6b9086c3a9 test: probe Cu10 CUTLASS fork — find mma_cu10.h, tensor op files, batched_gemm example 2026-08-15 05:27:43 +00:00
Claude
a465dd1d75 fix: use F.silu in test script for old corex torch 2026-08-15 05:22:31 +00:00
Claude
a12d070d82 fix: replace torch::silu with x*sigmoid(x) for old corex torch 2026-08-15 05:22:25 +00:00
Claude
c840c9159f feat: moe_tcu_dispatch.cpp — C++ MoE expert loop via torch::mm (TCU kernel) 
torch profiler confirmed: torch.mm launches Gemm_tcu_bi_kernel::gemm_h_h_tcu_25
which is BI-V100 TCU (Tensor Compute Unit) hardware-accelerated GEMM.
0.58ms per call vs our custom kernel 7.7ms — TCU is 13x faster.

Python for-loop overhead measured: 0.892 ms/expert = 7.1 ms for 8 experts.
This C++ dispatch eliminates that overhead while using the same TCU kernel.

Three entry points:
- moe_decode: full MoE forward (FC1 + SiLU*mul + FC2) for decode
- moe_prefill: group-by-expert MoE forward for prefill
- moe_expert_gemm_tcu: raw GEMM loop for benchmarking
 2026-08-15 05:20:05 +00:00
Claude
9514092980 test: probe torch.matmul backend + ixformer.matmul/linear + Python loop overhead 2026-08-15 05:17:13 +00:00
Claude
395b3e4042 test: clean rebuild + debug output for kernel 10 correctness 2026-08-15 05:14:18 +00:00
Claude
a8ca42b59c perf: hgemm_warptiling Config B — beats cublas on MoE-sized GEMM (0.7x) 
probe_k10_configs.sh results on BI-V100:
  256x4096 @ 4096x11008:
    cublas:   10.554 ms
    Config B:  7.649 ms (0.7x cublas — FASTER)
    Config A:  2308 ms  (old broken config)

Config B: BM128 BN128 BK16 WM64 WN64 WNITER2 TM8 TN4 NT128
Root cause of Config A slowness: WN=128 WNITER=4 caused
excessive register pressure and smem bank conflicts.
 2026-08-15 05:11:32 +00:00
Claude
21417319bc test: sweep 6 kernel 10 configs + cublas baseline — find best params for warp64 2026-08-15 05:08:11 +00:00
Claude
27bb8d28df test: probe kernel 10 perf with CUDA events — isolate bottleneck 2026-08-14 17:23:41 +00:00
Claude
2b12fe687e feat: hgemm_warptiling.cu — siboehm kernel 10 ported to WARPSIZE=64 FP16 
1:1 from upstream_ref/sgemm_cuda/10_kernel_warptiling.cuh.
3 changes: WARPSIZE 32→64, float→__half, FP32 accumulator.

Launch config (confirmed by probe_warp64.sh):
  NUM_THREADS=128, 2 warps of 64
  BM=128 BN=128 BK=16 WM=64 WN=128 WNITER=4 TM=4 TN=4
  WMITER=2, WSUBM=32, WSUBN=32, threads_per_warp=64 ✓
 2026-08-14 17:05:59 +00:00
Claude
11b8a98eea test: probe warp_size=64 behavior + kernel 10 warp tiling with WARPSIZE=64 on BI-V100 2026-08-14 17:02:55 +00:00