dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

feat: deploy CCCL-tuned prefix_prefill + muh_dispatch + fix SM=16 count

Browse Source 
muh_dispatch.py:
- Fix missing os/sys imports (was crashing on import)
- Fix SM count 50→16 (confirmed via ixsmi, matches hardware.cuh)
- Fix C++ struct name lookup to match actual tuning_reduce.cuh names:
  bi100_plus_float32_o4, bi100_plus_float64_o4, bi100_plus_accum2_o4
  (was: bi100_float32_plus_o4 — wrong name, would always fall through to default)

Dockerfile:
- Add COPY for prefix_prefill.py and muh_dispatch.py
- Deploy CCCL-tuned prefix_prefill.py into vllm attention ops
  (BLOCK=64, NUM_WARPS=4 for BI-V100 SM=16)
- Deploy muh_dispatch.py into vllm package for type-dispatched kernel configs
- These files were written but never deployed — dead code until now

Impact: prefix_prefill.py deployment means the CCCL-derived block sizes
actually take effect at runtime. Previously the base image's original
prefix_prefill.py (BLOCK=128 for cc>=80, or 64 for cc<80) was used,
which is correct for BI-V100 but our version adds explicit SM=16
documentation and the path for future tuning.
This commit is contained in:
Claude
2026-08-03 08:30:16 +00:00
parent 8e9c22f6c1
commit 9f93d695a9
2 changed files with 25 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
Dockerfile
View File

@@ -3,10 +3,12 @@ FROM git.modelhub.org.cn:9443/enginex-iluvatar/bi100-3.2.3-x86-ubuntu20.04-py3.1
RUN mkdir /workspace
WORKDIR /workspace/
# Copy all scripts and the V2 module
# Copy all scripts, V2 kernels, CCCL-tuned prefill, and muh dispatch
COPY ./qwen3_6_scripts /workspace/qwen3_6_scripts
COPY ./paged_attention_v2_pytorch.py /workspace/paged_attention_v2_pytorch.py
COPY ./paged_attention_v2_triton.py /workspace/paged_attention_v2_triton.py
COPY ./prefix_prefill.py /workspace/prefix_prefill.py
COPY ./muh_dispatch.py /workspace/muh_dispatch.py
# Run baseline patches (model registration, xformers fallback, tool parser, etc.)
RUN cd ./qwen3_6_scripts && ./patch_ops.sh
@@ -23,12 +25,25 @@ RUN python3 /workspace/qwen3_6_scripts/patch_ixformer_native.py
# Triton V2 risk: SMEM=32KB zero margin at head_dim=256 BLOCK_N=32.
# If Triton V2 crashes, PyTorch V2 (batched bmm, no intermediate tensor
# savings but correct) takes over automatically via try/except.
# Deploy Triton V2 kernel into vllm package
RUN cp /workspace/paged_attention_v2_triton.py \
/usr/local/corex/lib/python3/dist-packages/vllm/paged_attention_v2_triton.py 2>/dev/null || \
cp /workspace/paged_attention_v2_triton.py \
/usr/local/corex/lib64/python3/dist-packages/vllm/paged_attention_v2_triton.py 2>/dev/null || true
RUN python3 /workspace/qwen3_6_scripts/patch_paged_attention_v2.py
# Deploy CCCL-tuned prefix_prefill.py (SM=16: BLOCK=64, NUM_WARPS=4)
RUN cp /workspace/prefix_prefill.py \
/usr/local/corex/lib/python3/dist-packages/vllm/attention/ops/prefix_prefill.py 2>/dev/null || \
cp /workspace/prefix_prefill.py \
/usr/local/corex/lib64/python3/dist-packages/vllm/attention/ops/prefix_prefill.py 2>/dev/null || true
# Deploy muh_dispatch.py (CCCL-style type dispatch for kernel configs)
RUN cp /workspace/muh_dispatch.py \
/usr/local/corex/lib/python3/dist-packages/vllm/muh_dispatch.py 2>/dev/null || \
cp /workspace/muh_dispatch.py \
/usr/local/corex/lib64/python3/dist-packages/vllm/muh_dispatch.py 2>/dev/null || true
# 2. Triton kernel tuning: BLOCK=64, NUM_WARPS=4
# SMEM: BLOCK_N=64 × head_dim=128 × 2B × 2(K+V) = 32KB ≤ 48KB
# Occupancy: 4 warps allows 2 blocks/SM vs 1 at 8 warps