[OPT] Raise max-seq-len-to-capture to 65536 for more CUDA graph coverage

Analysis:
  CUDA graph eliminates kernel launch overhead (~10-20% for decode).
  At 32768, sequences >32K skip graph capture.
  At 65536, most competition workload sequences get graph acceleration.

  Memory: CUDA graph capture allocates one copy of all intermediate tensors
  at the max captured batch size. With max-num-seqs=1, this is one sequence's
  worth of tensors — small relative to model weights.

Combined with V2 enabled for seq>8192 and threshold raised to 65536,
the decode path is now:
  seq <= 8192:  V1 compiled kernel (fastest)
  8192 < seq <= 65536: V2 pytorch (single-bmm, good)
  seq > 65536: PyTorch fallback (rare at competition workload)
This commit is contained in:
dylanyunlon
2026-07-30 16:14:57 +00:00
parent 0c60ed8784
commit 8951d74936

View File

@@ -22,7 +22,7 @@ command:
- '16384'
- --enable-chunked-prefill
- --max-seq-len-to-capture
- '32768'
- '65536'
- --enable-auto-tool-choice
- --tool-call-parser
- qwen3_coder