dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

fix: remove --enable-chunked-prefill — conflicts with small max_num_batched_tokens

Browse Source 
chunked_prefill requires max_num_batched_tokens >= max_model_len/max_num_seqs
= 80000/2 = 40000. But we need small batched_tokens for profiling OOM.

Without chunked_prefill, max_num_batched_tokens=2048 is fine for profiling
and real inference processes full sequences in one pass.
This commit is contained in:
project6-dev
2026-08-13 16:34:44 +00:00
parent e31bd69779
commit 15ad56a454
1 changed files with 1 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
computility-run.yaml
View File

@@ -19,8 +19,7 @@ command:
- --disable-log-requests
- --disable-frontend-multiprocessing
- --max-num-batched-tokens
- '256'
- --enable-chunked-prefill
- '2048'
- --max-seq-len-to-capture
- '32768'
- --enable-auto-tool-choice