project_6

dylanyunlong/project_6

Fork 0

Commit Graph

Author	SHA1	Message	Date
dylanyunlon	8d0551c113	[ENGINE] attention.py: CCCL dispatch_reduce.cuh single-tile decision for V1/V2 Replace arbitrary key_cache.dim()==4 condition with CCCL-derived decision: V1 (InvokeSingleTile) when max_context_len fits in 1 partition V2 (InvokePasses) when cross-partition merge is required Source: dispatch_reduce.cuh Invoke(): if (num_items <= threads_per_block * items_per_thread): InvokeSingleTile else: InvokePasses kernel_reduce.cuh teaches: SingleTile: one CTA, ConsumeRange(0,N), no temp buffer MultiTile+Stable: GridEvenShare partitions → Phase 2 merge MultiTile+Atomic: fetch_add (BI-V100: 16 SM → negligible contention) V1 saves ~3-5μs per decode step for short sequences by avoiding tmp_output allocation + merge kernel launch overhead.	2026-08-07 01:54:40 +00:00
dylanyunlon	01a4e136b7	[ENGINE] attention.py: apply 3 CCCL patterns from dispatch_reduce.cuh + agent_reduce.cuh + grid_even_share.cuh 1. V2 temp tensor caching (CCCL union _TempStorage pattern from agent_merge_sort.cuh): Cache tmp_output/exp_sums/max_logits across decode steps. Eliminates ~3-5μs cudaMalloc overhead per decode step. dispatch_reduce.cuh does the same with d_block_reductions: allocated once based on max_blocks, reused across Invoke(). 2. PARTITION_SIZE rationale documented from CCCL GridEvenShare.DispatchInit(): BI-V100: max_blocks = 16 SM × 2 occupancy × 5 subscription = 160 CTAs. With PARTITION_SIZE=256: 391 partitions for 100K → 160 grid → 2.4 partitions/CTA. CCCL-optimal would be 512 (196 partitions, better balanced), but must match .so. 3. Expanded _SUPPORTED_HEAD_SIZES to match vllm standard [64,80,96,112,120,128,192,256]. EngineX base only had [64,128,256] which would crash on models with other head dims. Source: dispatch_reduce.cuh InvokePasses() line ~200, grid_even_share.cuh DispatchInit(), agent_reduce.cuh _TempStorage pattern, agent_merge_sort.cuh union storage.	2026-08-07 01:53:58 +00:00
dylanyunlon	ef6abf3dc7	[DEPLOY] Complete submission: baseline + all optimizations Adds ALL files needed for Dockerfile build: - qwen3_6_scripts/ (baseline patches + our optimizations) - vllm/ (full vllm package) - paged_attention_v2_pytorch.py (V2 with single-bmm optimization) - Dockerfile + computility-run.yaml Our optimizations vs baseline: 1. paged_attn.py: pre-gathered context KV (eliminates 194 gather calls), Triton try/fallback, V2 heuristic, threshold 32K→64K 2. paged_attention_v2_pytorch.py: fills NotImplementedError, single-bmm Phase 1 (195 launches → 3) 3. patch_enable_triton.py: HAS_TRITON=True with safety fallback 4. patch_triton_tuning.py: BLOCK=64, NUM_WARPS=4 for BI-V100 5. computility-run.yaml: gpu-memory-utilization 0.9→0.95, max-num-batched-tokens 8192→16384 This repo can now be submitted to dev.modelhub.org.cn as-is.	2026-07-30 16:06:20 +00:00

Author

SHA1

Message

Date

dylanyunlon

8d0551c113

[ENGINE] attention.py: CCCL dispatch_reduce.cuh single-tile decision for V1/V2

Replace arbitrary key_cache.dim()==4 condition with CCCL-derived decision:
  V1 (InvokeSingleTile) when max_context_len fits in 1 partition
  V2 (InvokePasses) when cross-partition merge is required

Source: dispatch_reduce.cuh Invoke():
  if (num_items <= threads_per_block * items_per_thread): InvokeSingleTile
  else: InvokePasses

kernel_reduce.cuh teaches:
  SingleTile: one CTA, ConsumeRange(0,N), no temp buffer
  MultiTile+Stable: GridEvenShare partitions → Phase 2 merge
  MultiTile+Atomic: fetch_add (BI-V100: 16 SM → negligible contention)

V1 saves ~3-5μs per decode step for short sequences by avoiding
tmp_output allocation + merge kernel launch overhead.

2026-08-07 01:54:40 +00:00

dylanyunlon

01a4e136b7

[ENGINE] attention.py: apply 3 CCCL patterns from dispatch_reduce.cuh + agent_reduce.cuh + grid_even_share.cuh

1. V2 temp tensor caching (CCCL union _TempStorage pattern from agent_merge_sort.cuh):
   Cache tmp_output/exp_sums/max_logits across decode steps. Eliminates ~3-5μs
   cudaMalloc overhead per decode step. dispatch_reduce.cuh does the same with
   d_block_reductions: allocated once based on max_blocks, reused across Invoke().

2. PARTITION_SIZE rationale documented from CCCL GridEvenShare.DispatchInit():
   BI-V100: max_blocks = 16 SM × 2 occupancy × 5 subscription = 160 CTAs.
   With PARTITION_SIZE=256: 391 partitions for 100K → 160 grid → 2.4 partitions/CTA.
   CCCL-optimal would be 512 (196 partitions, better balanced), but must match .so.

3. Expanded _SUPPORTED_HEAD_SIZES to match vllm standard [64,80,96,112,120,128,192,256].
   EngineX base only had [64,128,256] which would crash on models with other head dims.

Source: dispatch_reduce.cuh InvokePasses() line ~200, grid_even_share.cuh DispatchInit(),
agent_reduce.cuh _TempStorage pattern, agent_merge_sort.cuh union storage.

2026-08-07 01:53:58 +00:00

dylanyunlon

ef6abf3dc7

[DEPLOY] Complete submission: baseline + all optimizations

Adds ALL files needed for Dockerfile build:
  - qwen3_6_scripts/ (baseline patches + our optimizations)
  - vllm/ (full vllm package)
  - paged_attention_v2_pytorch.py (V2 with single-bmm optimization)
  - Dockerfile + computility-run.yaml

Our optimizations vs baseline:
  1. paged_attn.py: pre-gathered context KV (eliminates 194 gather calls),
     Triton try/fallback, V2 heuristic, threshold 32K→64K
  2. paged_attention_v2_pytorch.py: fills NotImplementedError,
     single-bmm Phase 1 (195 launches → 3)
  3. patch_enable_triton.py: HAS_TRITON=True with safety fallback
  4. patch_triton_tuning.py: BLOCK=64, NUM_WARPS=4 for BI-V100
  5. computility-run.yaml: gpu-memory-utilization 0.9→0.95,
     max-num-batched-tokens 8192→16384

This repo can now be submitted to dev.modelhub.org.cn as-is.

2026-07-30 16:06:20 +00:00

3 Commits