From 8951d74936b61af95c387d45af7d3aa4d088ff9d Mon Sep 17 00:00:00 2001 From: dylanyunlon Date: Thu, 30 Jul 2026 16:14:57 +0000 Subject: [PATCH] [OPT] Raise max-seq-len-to-capture to 65536 for more CUDA graph coverage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Analysis: CUDA graph eliminates kernel launch overhead (~10-20% for decode). At 32768, sequences >32K skip graph capture. At 65536, most competition workload sequences get graph acceleration. Memory: CUDA graph capture allocates one copy of all intermediate tensors at the max captured batch size. With max-num-seqs=1, this is one sequence's worth of tensors — small relative to model weights. Combined with V2 enabled for seq>8192 and threshold raised to 65536, the decode path is now: seq <= 8192: V1 compiled kernel (fastest) 8192 < seq <= 65536: V2 pytorch (single-bmm, good) seq > 65536: PyTorch fallback (rare at competition workload) --- computility-run.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/computility-run.yaml b/computility-run.yaml index 3fad7ecf..286cc10c 100644 --- a/computility-run.yaml +++ b/computility-run.yaml @@ -22,7 +22,7 @@ command: - '16384' - --enable-chunked-prefill - --max-seq-len-to-capture - - '32768' + - '65536' - --enable-auto-tool-choice - --tool-call-parser - qwen3_coder