From e31bd697794c67a6b34e43a08b0eb1bec8815790 Mon Sep 17 00:00:00 2001
From: project6-dev <dev@project6.ai>
Date: Thu, 13 Aug 2026 16:13:41 +0000
Subject: [PATCH] =?UTF-8?q?fix(OOM):=20max-num-batched-tokens=204096?=
 =?UTF-8?q?=E2=86=92256=20=E2=80=94=20profiling=20Q-tiling=20also=20OOMs?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

flash_attn skip worked but Q-tiling fallback still OOMs at 4096 tokens.
K tensor: [28_heads, 4096, 256] float32 = 112MB per layer slice.
At 256 tokens: [28, 256, 256] = 7MB — safe for profiling.

This only affects profiling dummy batch size. Real inference chunked
prefill still processes up to max_model_len tokens.
---
 computility-run.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/computility-run.yaml b/computility-run.yaml
index b009c3fb..9beaf135 100644
--- a/computility-run.yaml
+++ b/computility-run.yaml
@@ -19,7 +19,7 @@ command:
     - --disable-log-requests
     - --disable-frontend-multiprocessing
     - --max-num-batched-tokens
-    - '4096'
+    - '256'
     - --enable-chunked-prefill
     - --max-seq-len-to-capture
     - '32768'