dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

feat(SM70): add 1Cat-vLLM FlashQLA fused GDN CUDA kernel for BI-V100

Browse Source 
Source: github.com/1CatAI/1Cat-vLLM (MIT license)
flash_qla/ops/gated_delta_rule/chunk/sm70/

Files added:
- csrc/gdn_forward.cu (1919 lines) — 4 CUDA kernels for SM70/SM75:
  gdn_forward, gdn_forward_vlk_varlen,
  gdn_decode_mixed_qkv_global_state, gdn_decode_mixed_qkv_ddtree_state
- fused_fwd.py — Python wrapper, JIT compiles via torch.utils.cpp_extension.load()
- naive_gdn.py — fla reference PyTorch implementation for fallback
- __init__.py — exports chunk_gated_delta_rule_fwd_sm70

Build: JIT compiled at runtime (TORCH_CUDA_ARCH_LIST=7.0;7.5 -O3)
Deploy: patch_ops.sh copies flash_qla_sm70/ to vllm models dir

qwen3_5.py updated to try import flash_qla_sm70 before PyTorch fallback
This commit is contained in:
Claude
2026-08-10 01:06:32 +00:00
parent 3d5f75fefd
commit 8cf73ad39c
6 changed files with 2621 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
qwen3_6_scripts/patch_ops.sh
View File

@@ -175,6 +175,22 @@ if [ -n "$VLLM2" ]; then
cp ./chat_utils.py "$VLLM2/entrypoints/chat_utils.py" 2>/dev/null || true
fi
echo "[patch_ops] DONE — all patches deployed"
echo "[patch_ops] Deployed: qwen3_5.py, paged_attn.py, mamba_cache.py, sequence.py, scheduler.py, xformers patches, tool/reasoning parsers, serving layer"
echo "[patch_ops] NOT deployed (using base image native): model_runner.py, _custom_ops.py, sampler.py, logits_processor.py, arg_utils.py"
echo "[patch_ops] DONE — SM70 GDN kernel + serving layer + engine patches deployed"
echo "[patch_ops] Deployed: qwen3_5.py, flash_qla_sm70 (SM70 GDN CUDA kernel), paged_attn.py, mamba_cache.py, sequence.py, scheduler.py, xformers patches, serving layer"
echo "[patch_ops] SM70 GDN kernel: JIT compiles on first forward pass (~2min), then cached"
echo "[patch_ops] NOT deployed (base image native): model_runner.py, _custom_ops.py, sampler.py, logits_processor.py, arg_utils.py"
# Deploy flash_qla SM70 GDN kernel (from 1Cat-vLLM, MIT license)
# This is a fused CUDA kernel for GatedDeltaNet on SM70/SM75 (V100/BI-V100)
# JIT compiled at runtime via torch.utils.cpp_extension.load()
FLASH_QLA_DST="$VLLM/model_executor/models/flash_qla_sm70"
if [ -d "./flash_qla_sm70" ]; then
rm -rf "$FLASH_QLA_DST" 2>/dev/null
cp -r ./flash_qla_sm70 "$FLASH_QLA_DST" 2>/dev/null && \
echo "[patch_ops] flash_qla_sm70 deployed to $FLASH_QLA_DST" || true
# Also deploy to VLLM2 if present
if [ -n "$VLLM2" ]; then
rm -rf "$VLLM2/model_executor/models/flash_qla_sm70" 2>/dev/null
cp -r ./flash_qla_sm70 "$VLLM2/model_executor/models/flash_qla_sm70" 2>/dev/null || true
fi
fi