project_6

dylanyunlong/project_6

Fork 0

Commit Graph

Author	SHA1	Message	Date
dylanyunlon	ca3697f4b0	[ENGINE] CCCL SmemResource pattern: pre-allocate staging buffers in V2 paged attention Eliminates per-decode-step torch.full/torch.zeros GPU allocations that cause OOM after thousands of generation steps (case_truncation max_tokens=8192). Three allocation sites replaced with staging buffer .fill_()/.zero_() reuse: - scores_padded: torch.full([H, padded_len], -inf) → _staging_scores slice - v_padded_kv: torch.zeros([kv_h, padded_len, d]) → _staging_v_kv slice - v_padded: torch.zeros([H, padded_len, d]) → _staging_v slice Pattern from CCCL cub/detail/warpspeed/resource/smem_resource.cuh: SmemResource pre-allocates stageCount buffers, nextStage() cycles through them. PyTorch translation: allocate once at function entry, slice per step.	2026-08-07 04:37:44 +00:00
dylanyunlon	5ba9c1e731	[CRITICAL/deploy] fix 3 deployment gaps found from docker crash log 1. paged_attention_v2_pytorch.py was missing from container - _custom_ops.py imports it but Dockerfile only COPYs qwen3_6_scripts/ - Now: copied into qwen3_6_scripts/ + patch_ops deploys to both $V/ and /workspace/ 2. prefix_prefill.py was not deployed by patch_ops.sh - xformers.py may try to import context_attention_fwd from it - Now: patch_ops copies it to $V/attention/ops/ 3. _custom_ops.py paged_attention_v2 import path hardened - Try 3 locations: vllm package, /workspace/, repo root - Prevents ImportError in container where file locations differ CCCL source read: cub/block/block_exchange.cuh (blocked↔striped data rearrangement) → identified missing file deployment as analogous to incorrect data layout mapping	2026-08-06 07:01:14 +00:00

Author

SHA1

Message

Date

dylanyunlon

ca3697f4b0

[ENGINE] CCCL SmemResource pattern: pre-allocate staging buffers in V2 paged attention

Eliminates per-decode-step torch.full/torch.zeros GPU allocations that cause
OOM after thousands of generation steps (case_truncation max_tokens=8192).

Three allocation sites replaced with staging buffer .fill_()/.zero_() reuse:
  - scores_padded: torch.full([H, padded_len], -inf) → _staging_scores slice
  - v_padded_kv: torch.zeros([kv_h, padded_len, d]) → _staging_v_kv slice
  - v_padded: torch.zeros([H, padded_len, d]) → _staging_v slice

Pattern from CCCL cub/detail/warpspeed/resource/smem_resource.cuh:
  SmemResource pre-allocates stageCount buffers, nextStage() cycles through them.
  PyTorch translation: allocate once at function entry, slice per step.

2026-08-07 04:37:44 +00:00

dylanyunlon

5ba9c1e731

[CRITICAL/deploy] fix 3 deployment gaps found from docker crash log

1. paged_attention_v2_pytorch.py was missing from container
   - _custom_ops.py imports it but Dockerfile only COPYs qwen3_6_scripts/
   - Now: copied into qwen3_6_scripts/ + patch_ops deploys to both $V/ and /workspace/

2. prefix_prefill.py was not deployed by patch_ops.sh
   - xformers.py may try to import context_attention_fwd from it
   - Now: patch_ops copies it to $V/attention/ops/

3. _custom_ops.py paged_attention_v2 import path hardened
   - Try 3 locations: vllm package, /workspace/, repo root
   - Prevents ImportError in container where file locations differ

CCCL source read: cub/block/block_exchange.cuh (blocked↔striped data rearrangement)
→ identified missing file deployment as analogous to incorrect data layout mapping

2026-08-06 07:01:14 +00:00

2 Commits