fix(OOM): cap GPU blocks at 5000 via BI100_MAX_GPU_BLOCKS env var

Profiling zeros-out attention → vllm overestimates free memory → 7942 blocks
→ first real request OOMs. Cap at 5000 (80K tokens / 16 block_size).

patch_block_major_worker_capacity.py reads BI100_MAX_GPU_BLOCKS from env,
caps num_gpu_blocks after reserve_block_major_gpu_blocks.
This commit is contained in:
project6-dev
2026-08-14 00:12:39 +00:00
parent c6aa1b9c62
commit 456380eed0
2 changed files with 10 additions and 0 deletions

View File

@@ -49,3 +49,5 @@ env:
value: '1'
- name: PYTORCH_CUDA_ALLOC_CONF
value: max_split_size_mb:512
- name: BI100_MAX_GPU_BLOCKS
value: '5000'

View File

@@ -20,6 +20,14 @@ CAPACITY_ANCHOR = """\
CAPACITY_REPLACEMENT = """\
num_gpu_blocks = reserve_block_major_gpu_blocks(
num_gpu_blocks, cache_block_size)
# BI100: cap GPU blocks — profiling with zero-tensor attention
# underestimates memory, causing runtime OOM if uncapped.
_bi100_max = int(os.environ.get("BI100_MAX_GPU_BLOCKS", "0"))
if _bi100_max > 0 and num_gpu_blocks > _bi100_max:
logger.warning(
"[BI100] capping num_gpu_blocks: %d -> %d (BI100_MAX_GPU_BLOCKS)",
num_gpu_blocks, _bi100_max)
num_gpu_blocks = _bi100_max
num_gpu_blocks = max(num_gpu_blocks, 0)
num_cpu_blocks = max(num_cpu_blocks, 0)
"""