dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
792 Commits 1 Branch 0 Tags
c8e8473367cfcb8aac22a3ef5e5ee6c73e3bf497
Commit Graph

350 Commits

Author SHA1 Message Date
root
c8e8473367 test docker build 2026-08-19 03:09:44 +00:00
root
7d74ad4e8e [fix] baseline docker build from ex_engine.python.xllm_ops import _get error 2026-08-19 02:46:40 +00:00
root
cbbc3a6100 [fix] \ No newline at end of file 2026-08-19 02:07:29 +00:00
root
679f2c1e29 [fix] baseline4 _xllm_block_copy arg 2026-08-19 01:55:54 +00:00
root
5d2faa3c00 [fix] baseline4 ixf to xllm 2026-08-19 01:07:16 +00:00
root
e1870880c5 [fix ]build baseline4 2026-08-18 17:40:53 +00:00
root
54834c7644 [fix] baseline4 ops_sh fix 2026-08-18 17:26:02 +00:00
root
37a49ee655 [fix] bseline4 sh 添加换行符 2026-08-18 17:05:30 +00:00
root
06f4ce3bb9 [fix] baseline4 ixf_F.copy_blocks名字不匹配 2026-08-18 15:35:03 +00:00
root
8c206feb91 [fix] baseline4 paged_attn.py 的 seq_lens_tensor 包含 0x7FFF7FFF 2026-08-18 10:32:55 +00:00
root
614bad6232 [feat] baseline4 profiling 2026-08-18 10:21:15 +00:00
root
3c53587de2 [fix] baseline4 NaN 0X7FFF7FFF 2026-08-18 09:54:25 +00:00
root
7a02a5bf92 baseline4 NAN 0X7FFF7FFF 2026-08-18 08:37:30 +00:00
root
ab078989da [fix] reshape_paged_cache slot_mapping int64→int32 (C++ data_ptr<int>要求) 2026-08-18 07:41:38 +00:00
root
9e1df1482e [fix] xllm_ops 5处C++参数签名 2026-08-18 07:10:18 +00:00
root
c92135111e [fix] reshape_paged_cache参数顺序 2026-08-18 06:44:01 +00:00
root
b89bd3d72f [fix] ix_moe_bridge pybind名无ix_前缀: ix_paged_attention→paged_attention, ix_linear→linear 2026-08-18 06:14:15 +00:00
root
54a9b572af [fix] xllm_ops 7处函数名匹配.so真实导出符号 
- reshape_and_cache → reshape_paged_cache (xllm_cache.so)
- residual_rms_norm → fused_add_rms_norm (xllm_norm.so)
- topk_softmax → moe_fused_topk (xllm_moe.so)
- moe_compute_token_index → moe_compute_index (xllm_moe.so)
- ix_full_bridge → ix_moe_bridge (paged_attention, ix_linear)
- ix_linear 5参数 → 3参数 (input, weight, bias)
- check_all: ix_moe_bridge改为required, ix_full_bridge改为optional
 2026-08-18 06:11:03 +00:00
root
fccb78df09 Merge branch 'main' of https://dev.modelhub.org.cn/dylanyunlong/project_6 into main 2026-08-18 04:17:58 +00:00
root
8cc6a91b8a [fix] baseline4 use ix_moe_bridge.so instead of broken ix_full_bridge.so 2026-08-18 04:17:37 +00:00
root
13b12aac4c [fix] baseline4 rms_norm crash 2026-08-18 03:35:26 +00:00
root
714c41d17b trigger rebuild 2026-08-18 02:32:00 +00:00
root
96ef5f27d3 [fix] baseline4 debug info 2026-08-18 02:20:43 +00:00
root
ae27d0405b Merge branch 'main' of https://github.com/dylanyunlon/project_6 2026-08-17 18:11:57 +00:00
Claude
f7b1b2d119 fix: patched_reshape_and_cache signature 7→8 args (k_scale, v_scale) 
Original vllm ops.reshape_and_cache passes 8 args:
  key, value, key_cache, value_cache, slot_mapping, kv_cache_dtype, k_scale, v_scale
Our patch had 7 (single kv_scale) → TypeError on first request → engine dead.
 2026-08-17 18:10:52 +00:00
root
d42b0c1c04 Merge branch 'main' of https://dev.modelhub.org.cn/dylanyunlong/project_6 2026-08-17 17:45:24 +00:00
Claude
8d969822b4 fix: revert ix_full_bridge.so to d32822c5 original (160KB) 
be5e23f3 rebuilt this .so but introduced ABI mismatch:
  undefined symbol: c10::detail::torchInternalAssertFail
The original 160KB .so does not reference this symbol and works
with the Docker image's libtorch.
 2026-08-17 17:39:21 +00:00
root
f439f67f39 baseline4 trigger rebuild 2026-08-17 17:16:11 +00:00
Claude
f287382f99 fix: update SHA256SUMS for rebuilt ix_full_bridge.so 
ix_full_bridge.so was rebuilt in be5e23f3 (without MoE ixformer::infer deps)
but SHA256SUMS was not updated. install_prebuilt_corex.sh sha256sum --check
fails → set -eo pipefail kills patch_ops.sh → cli_args.py/api_server.py
never deployed → vllm rejects --reasoning-parser qwen3 at startup.
 2026-08-17 17:12:41 +00:00
root
ff18454eeb trigger rebuild 2026-08-17 16:52:37 +00:00
root
be5e23f335 fix: rebuild ix_full_bridge.so without MoE ixformer::infer deps 2026-08-17 16:25:16 +00:00
root
5c97e3dcb8 trigger rebuild 2026-08-17 14:39:56 +00:00
root
5b96a91156 [fix] baseline4 不是 CUDA 不可用，是 .so 在非主进程里被 dlopen 时段错误。需要用进程级别的判断 2026-08-17 14:17:03 +00:00
root
053dc036b8 [fix] prebuilt 的 ix_full_bridge.so 在子进程（没有 GPU 初始化）里被 dlopen 2026-08-17 14:01:36 +00:00
root
9c46f5a04e [fix] baseline4 copy_blocks 崩溃和 ex_engine import 路径两个接线没接对 2026-08-17 13:19:26 +00:00
root
1af45de371 [fix] baseline4 docker build move ex_engine into qwen3_6_scripts, remove COPY ex_engine from Dockerfile 2026-08-17 11:51:01 +00:00
root
0e21445220 Merge branch 'main' of https://dev.modelhub.org.cn/dylanyunlong/project_6 into main 2026-08-17 10:11:44 +00:00
root
36909bf964 [fix] prebuilt 2026-08-17 10:11:24 +00:00
root
f10cccf9df [fix] baseline4 :ix_moe_bridge.so 。 v2 bridge 真机编译通过，13函数全导出 2026-08-17 09:18:32 +00:00
root
8211a45464 platform test baseline4 2026-08-17 07:22:14 +00:00
dev
be4d661191 revert: undo 2 premature pushes (ee516bd2, 9ea0a1d4) — code needs review first 2026-08-17 07:04:50 +00:00
dev
9ea0a1d4f4 fix: resolve all 8 deployment pipeline breaks 
Breaks found and fixed:

1. Dockerfile: COPY 5 individual files → COPY entire ex_engine/
2. patch_ops.sh EX_ENGINE_DIR: /workspace/ex_engine not found → added fallback
3. patch_ops.sh deploy: ix_ops.py to ex_engine/ (flat) → ex_engine/python/ (correct package)
4. ix_startup_patch.py: import from vllm.ex_engine.patch_vllm_ops → vllm.ex_engine.python.patch_vllm_ops
5. ix_moe_bridge.so: only deployed to ex_engine/ → also copy to model_executor/models/ and vllm root
6. ex_engine/__init__.py: missing re-exports → add imports so 'from vllm.ex_engine import x' works
7. gemm_grouped.so: compiled but never imported → add import + flag + prefill GEMM path in qwen3_5.py
8. build_moe_bridge.sh Python heredoc: SCRIPT_DIR not exported + wrong nested path → export + search both layouts

Also added:
- CUTLASS batched GEMM compile step (corex_batched_gemm.so for decode)
- Full ex_engine/python/*.py deployment (was deploying only 2 of 19 files)
- EX_ENGINE_INFRA_AUDIT.md documenting all findings
 2026-08-17 07:01:41 +00:00
dev
ee516bd206 fix: connect ex_engine to Docker build pipeline 
Path breaks fixed:
1. Dockerfile: COPY entire ex_engine/ instead of 5 individual files
2. patch_ops.sh: EX_ENGINE_DIR fallback to /workspace/ex_engine
3. build_moe_bridge.sh: search csrc/ under both SCRIPT_DIR and SCRIPT_DIR/ex_engine

New build steps added to patch_ops.sh:
- Build gemm_grouped.so (CUTLASS Cu10 grouped GEMM, verified on device)
- Build corex_batched_gemm.so (CUTLASS batched GEMM for decode)
- Build ix_moe_bridge.so (7-step fused MoE pipeline)
- Deploy all ex_engine/python/*.py to VLLM_ROOT/ex_engine/python/

Previously broken: ix_ops.py, patch_vllm_ops.py, ix_startup_patch.py
were never deployed because EX_ENGINE_DIR resolved to nonexistent path.
xllm_activation/norm/rope/cache.so were copied but had no Python caller.
 2026-08-17 06:56:11 +00:00
root
beaa8dbb65 Merge branch 'main' of https://dev.modelhub.org.cn/dylanyunlong/project_6 2026-08-17 02:18:46 +00:00
root
03be5f2b15 [feat] group gemm 2026-08-17 02:16:58 +00:00
Claude
5172f94b1f Revert "feat: 3-tier ixformer flash prefill dispatch + OpenCompass max_tokens clamp + n>1 fanout + index sanitizer" 
This reverts commit cdec569977.
 2026-08-16 15:42:36 +00:00
Claude
cdec569977 feat: 3-tier ixformer flash prefill dispatch + OpenCompass max_tokens clamp + n>1 fanout + index sanitizer 
paged_attn.py (+218 lines):
  - Tier 0: ixformer flash_attn_varlen_func (cu_seqlens packed)
  - Tier 0.5: ixformer flash_attn_func (non-varlen, batch layout)
  - Tier 1: CoreXFA2 3-mode dispatch (packed/paged/chunked)
  - Tier 2 fallback: existing Python Q-tiling (unchanged)
  - Import chain: flash_attn_func + CoreXFA2Class + varlen

serving_chat.py (+23 lines):
  - max_tokens clamp: fixes OpenCompass 0 score (5 benchmarks all 400)
  - n>1 fanout: remove temperature==0 restriction for t2_n_2 FAIL

api_server.py (+33 lines):
  - HTTP middleware: strip index from messages before pydantic validation
  - Fixes ValidatorIterator 0.index Extra inputs are not permitted x6
 2026-08-16 15:37:23 +00:00
Claude
4189f44d27 test: dump ALL symbols from ALL ixformer/cuinfer .so — no grep filter, find what we missed 2026-08-16 04:58:47 +00:00
Claude
ea2c15f699 fix: decode MoE路径对齐base — F.linear+bmm替换pre-transpose+bmm 
base qwen3_5.py的decode路径（已验证可跑通竞赛）:
  F.linear(hidden, w13_sel.reshape(-1,H)) → view → act → bmm(w2_sel, act)

我们之前的路径（未验证，probe显示更慢）:
  pre-transpose(w13全量) → w13_t[eids] → bmm(x_expand, w13_t_sel) → act → bmm(act, w2_t_sel)

probe真机数据: loop matmul 19ms < torch.bmm 24ms
说明F.linear路径在BI-V100单token场景下更优

保持的corex加速:
  ✓ corex_moe_topk_softmax (topk+softmax fused)
  ✓ corex_moe_weight_gather (gather fused)
  ✓ corex_moe_exact_reduce (weighted sum fused)
  ✓ corex_moe_index_combine (prefill token routing fused)
 2026-08-15 14:55:05 +00:00
dylan
e18ece8f3a feat: port NaiveBatchedExperts from ds_vllm — view transpose + cublas transB 
Source: upstream_ref/ds_vllm/vllm/model_executor/layers/fused_moe/experts/fused_batched_moe.py
        upstream_ref/ds_vllm/vllm/model_executor/layers/fused_moe/activation.py

New files (ported from ds_vllm, adapted for BI-V100):
  ex_engine/moe/__init__.py
  ex_engine/moe/activation.py
    - MoEActivation enum + apply_moe_activation
    - torch.ops._C.silu_and_mul replaced with F.silu(gate)*up fallback
  ex_engine/moe/naive_batched_experts.py
    - naive_batched_moe_forward()
    - Decode: per-expert loop, w13[eid].transpose(0,1) is VIEW (zero copy)
    - @ operator → cublas passes transB=CUBLAS_OP_T internally
    - Prefill: group tokens by expert, batch @ per expert

Modified:
  qwen3_6_scripts/qwen3_5.py
    - Import naive_batched_moe_forward
    - Tier 0.5: after ix_fused_moe, before corex point-optimized loop
    - Uses existing topk routing (xllm/corex/pytorch)

Key difference from previous approach:
  - NO physical transpose (was 22ms overhead)
  - NO weight gather into contiguous buffer
  - View transpose is O(0), cublas handles transB
 2026-08-15 13:05:45 +00:00