fix: dial back max-num-seqs 256→8, revert batched-tokens and mem-util

256 concurrent seqs risks OOM: worst case with long prompts in queue
can exhaust KV cache + activation memory. 32K batched-tokens prefill
activation ≈ 20GB competes with KV cache. 0.95 mem-util leaves only
5% headroom for spikes.

Conservative start: max-num-seqs=8 (8× improvement over baseline=1).
8 seqs × 2048 avg context × 80KB/token = 1.3GB KV cache, safe.
gpu-memory-utilization and max-num-batched-tokens restored to proven
baseline values.

Optimal max-num-seqs needs real-hardware sweep: 4→8→16→32→64→128.
The value where Output TPS plateaus (KV cache saturated) is the
answer. Can't determine this without Phanthy Cloud access.
This commit is contained in:
Claude
2026-08-03 06:51:33 +00:00
parent cdc01bbc6a
commit 0ba4cdb025

View File

@@ -10,16 +10,16 @@ command:
- --max-model-len
- '100000'
- --gpu-memory-utilization
- '0.95'
- '0.9'
- --trust-remote-code
- -tp
- '4'
- --max-num-seqs
- '256'
- '8'
- --disable-log-requests
- --disable-frontend-multiprocessing
- --max-num-batched-tokens
- '32768'
- '8192'
- --enable-chunked-prefill
- --max-seq-len-to-capture
- '32768'