fix(OOM): add --num-gpu-blocks-override=4000 to skip profiling
Profiling OOM: flash_attn_varlen_func allocates large temp buffers during profile_run with 8192 dummy tokens. patch_worker_profile_override.py already has skip logic when num_gpu_blocks_override is set. 4000 blocks × 16 tokens/block = 64K token KV capacity. With max-model-len=80000 and prefix caching, this is sufficient. Sub 168 reference: ran 262K context on 0.95 util without override because base image profiling doesn't use flash_attn_varlen (uses native xformers).
This commit is contained in:
@@ -32,6 +32,8 @@ command:
|
||||
- --enforce-eager
|
||||
- --dtype
|
||||
- half
|
||||
- --num-gpu-blocks-override
|
||||
- '4000'
|
||||
env:
|
||||
- name: VLLM_ENGINE_ITERATION_TIMEOUT_S
|
||||
value: 3600
|
||||
|
||||
Reference in New Issue
Block a user