dylanyunlong/project_6_89d52222
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

perf: native ixformer decode (v1 ≤32K, v2 >32K) + flash_attn_varlen prefill

Browse Source 
Replaces all Python PyTorch fallback attention with native ixformer kernels:

Decode path:
- ≤32K: paged_attention_v1 (5D KV layout, x=8) — verified on real BI-V100
- >32K: paged_attention_v2 (5D→4D permute) — verified 65K+ on real BI-V100
- Removes _forward_decode_pytorch Python fallback entirely

Prefill path (profiling):
- _run_sdpa_fallback now uses ixformer.flash_attn_varlen_func
- head_dim=256 verified correct (diff<0.004) and 1.7x faster than PyTorch
- Falls back to Q-tiling pure-math if ixformer unavailable

Also includes: MoE kernel integration, GDN C++ kernels, diagnostic scripts,
xllm upstream layer/kernel references, .dockerignore cleanup.

All changes verified on real BI-V100 hardware (single card).
This commit is contained in:
project6-dev
2026-08-13 07:04:21 +00:00
parent a3c45d3b36
commit f28223c9da
67 changed files with 10659 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
cat_ixformer_vllm.py Normal file
View File

@@ -0,0 +1,4 @@
#!/usr/bin/env python3
"""Print ixformer vllm.py source code."""
with open("/usr/local/corex/lib64/python3/dist-packages/ixformer/functions/vllm.py") as f:
print(f.read())