[OPT] Raise max-seq-len-to-capture to 65536 for more CUDA graph coverage
Analysis: CUDA graph eliminates kernel launch overhead (~10-20% for decode). At 32768, sequences >32K skip graph capture. At 65536, most competition workload sequences get graph acceleration. Memory: CUDA graph capture allocates one copy of all intermediate tensors at the max captured batch size. With max-num-seqs=1, this is one sequence's worth of tensors — small relative to model weights. Combined with V2 enabled for seq>8192 and threshold raised to 65536, the decode path is now: seq <= 8192: V1 compiled kernel (fastest) 8192 < seq <= 65536: V2 pytorch (single-bmm, good) seq > 65536: PyTorch fallback (rare at competition workload)
This commit is contained in:
@@ -22,7 +22,7 @@ command:
|
||||
- '16384'
|
||||
- --enable-chunked-prefill
|
||||
- --max-seq-len-to-capture
|
||||
- '32768'
|
||||
- '65536'
|
||||
- --enable-auto-tool-choice
|
||||
- --tool-call-parser
|
||||
- qwen3_coder
|
||||
|
||||
Reference in New Issue
Block a user