From 0ba4cdb025411b1e45c2b1732fd30159e504b8f6 Mon Sep 17 00:00:00 2001
From: Claude <claude@anthropic.com>
Date: Mon, 3 Aug 2026 06:51:33 +0000
Subject: [PATCH] =?UTF-8?q?fix:=20dial=20back=20max-num-seqs=20256?=
 =?UTF-8?q?=E2=86=928,=20revert=20batched-tokens=20and=20mem-util?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

256 concurrent seqs risks OOM: worst case with long prompts in queue
can exhaust KV cache + activation memory. 32K batched-tokens prefill
activation ≈ 20GB competes with KV cache. 0.95 mem-util leaves only
5% headroom for spikes.

Conservative start: max-num-seqs=8 (8× improvement over baseline=1).
8 seqs × 2048 avg context × 80KB/token = 1.3GB KV cache, safe.
gpu-memory-utilization and max-num-batched-tokens restored to proven
baseline values.

Optimal max-num-seqs needs real-hardware sweep: 4→8→16→32→64→128.
The value where Output TPS plateaus (KV cache saturated) is the
answer. Can't determine this without Phanthy Cloud access.
---
 computility-run.yaml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/computility-run.yaml b/computility-run.yaml
index d283b113..4522b6e1 100644
--- a/computility-run.yaml
+++ b/computility-run.yaml
@@ -10,16 +10,16 @@ command:
     - --max-model-len
     - '100000'
     - --gpu-memory-utilization
-    - '0.95'
+    - '0.9'
     - --trust-remote-code
     - -tp
     - '4'
     - --max-num-seqs
-    - '256'
+    - '8'
     - --disable-log-requests
     - --disable-frontend-multiprocessing
     - --max-num-batched-tokens
-    - '32768'
+    - '8192'
     - --enable-chunked-prefill
     - --max-seq-len-to-capture
     - '32768'