dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

fix(OOM): lower blocks cap 5000→3000 — flash_attn needs ~4GB temp buffer

Browse Source 
5000 blocks KV cache fills GPU memory, flash_attn_varlen_func OOMs
allocating temp attention buffer on first real request.
3000 blocks × 16 = 48K tokens capacity, leaves room for attention.
This commit is contained in:
project6-dev
2026-08-14 01:33:06 +00:00
parent c8a982c4e8
commit 716034bdd0
1 changed files with 5 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
qwen3_6_scripts/patch_block_major_worker_capacity.py
View File

@@ -21,11 +21,12 @@ CAPACITY_REPLACEMENT = """\
num_gpu_blocks = reserve_block_major_gpu_blocks(
num_gpu_blocks, cache_block_size)
# BI100: profiling with zero-tensor attention underestimates memory.
# Hardcap at 5000 blocks (80K tokens) to prevent runtime OOM.
if num_gpu_blocks > 5000:
# Hardcap at 3000 blocks (48K tokens) to prevent runtime OOM.
# Must leave ~4GB free for flash_attn_varlen_func temp buffers.
if num_gpu_blocks > 3000:
logger.warning(
"[BI100] capping num_gpu_blocks: %d -> 5000", num_gpu_blocks)
num_gpu_blocks = 5000
"[BI100] capping num_gpu_blocks: %d -> 3000", num_gpu_blocks)
num_gpu_blocks = 3000
num_gpu_blocks = max(num_gpu_blocks, 0)
num_cpu_blocks = max(num_cpu_blocks, 0)
"""