fix(OOM): max-num-batched-tokens 4096→256 — profiling Q-tiling also OOMs
flash_attn skip worked but Q-tiling fallback still OOMs at 4096 tokens. K tensor: [28_heads, 4096, 256] float32 = 112MB per layer slice. At 256 tokens: [28, 256, 256] = 7MB — safe for profiling. This only affects profiling dummy batch size. Real inference chunked prefill still processes up to max_model_len tokens.
This commit is contained in:
@@ -19,7 +19,7 @@ command:
|
||||
- --disable-log-requests
|
||||
- --disable-frontend-multiprocessing
|
||||
- --max-num-batched-tokens
|
||||
- '4096'
|
||||
- '256'
|
||||
- --enable-chunked-prefill
|
||||
- --max-seq-len-to-capture
|
||||
- '32768'
|
||||
|
||||
Reference in New Issue
Block a user