fix(OOM): cap GPU blocks at 5000 via BI100_MAX_GPU_BLOCKS env var
Profiling zeros-out attention → vllm overestimates free memory → 7942 blocks → first real request OOMs. Cap at 5000 (80K tokens / 16 block_size). patch_block_major_worker_capacity.py reads BI100_MAX_GPU_BLOCKS from env, caps num_gpu_blocks after reserve_block_major_gpu_blocks.
This commit is contained in:
@@ -49,3 +49,5 @@ env:
|
||||
value: '1'
|
||||
- name: PYTORCH_CUDA_ALLOC_CONF
|
||||
value: max_split_size_mb:512
|
||||
- name: BI100_MAX_GPU_BLOCKS
|
||||
value: '5000'
|
||||
|
||||
Reference in New Issue
Block a user