fix(OOM): max-num-batched-tokens 4096→256 — profiling Q-tiling also OOMs

flash_attn skip worked but Q-tiling fallback still OOMs at 4096 tokens.
K tensor: [28_heads, 4096, 256] float32 = 112MB per layer slice.
At 256 tokens: [28, 256, 256] = 7MB — safe for profiling.

This only affects profiling dummy batch size. Real inference chunked
prefill still processes up to max_model_len tokens.
This commit is contained in:
project6-dev
2026-08-13 16:13:41 +00:00
parent 2717bafc30
commit e31bd69779

View File

@@ -19,7 +19,7 @@ command:
- --disable-log-requests
- --disable-frontend-multiprocessing
- --max-num-batched-tokens
- '4096'
- '256'
- --enable-chunked-prefill
- --max-seq-len-to-capture
- '32768'