fix(OOM): add --num-gpu-blocks-override=4000 to skip profiling
Profiling OOM: flash_attn_varlen_func allocates large temp buffers during profile_run with 8192 dummy tokens. patch_worker_profile_override.py already has skip logic when num_gpu_blocks_override is set. 4000 blocks × 16 tokens/block = 64K token KV capacity. With max-model-len=80000 and prefix caching, this is sufficient. Sub 168 reference: ran 262K context on 0.95 util without override because base image profiling doesn't use flash_attn_varlen (uses native xformers).
This commit is contained in:
@@ -32,6 +32,8 @@ command:
|
|||||||
- --enforce-eager
|
- --enforce-eager
|
||||||
- --dtype
|
- --dtype
|
||||||
- half
|
- half
|
||||||
|
- --num-gpu-blocks-override
|
||||||
|
- '4000'
|
||||||
env:
|
env:
|
||||||
- name: VLLM_ENGINE_ITERATION_TIMEOUT_S
|
- name: VLLM_ENGINE_ITERATION_TIMEOUT_S
|
||||||
value: 3600
|
value: 3600
|
||||||
|
|||||||
Reference in New Issue
Block a user