dylanyunlong/project_7
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

fix: cap default_max_tokens at 8192 — prevent OOM kill on unlimited generation

Browse Source 
t3_max_tokens_none test sends request without max_tokens.
With max_model_len=262144, default_max_tokens was ~260K tokens.
Model generates indefinitely at 12 TPS, container gets OOM killed after ~6min.
Cap at 8192 prevents memory exhaustion while still allowing long outputs.
This commit is contained in:
project6-dev
2026-08-12 04:06:21 +00:00
parent dbc1ce9a71
commit f1a408c75f
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
qwen3_6_scripts/serving_chat.py
View File

@@ -394,7 +394,7 @@ class OpenAIServingChat(OpenAIServing):
assert prompt_inputs is not None
sampling_params: Union[SamplingParams, BeamSearchParams]
default_max_tokens = self.max_model_len - len(
default_max_tokens = min(8192, self.max_model_len - len(
prompt_inputs["prompt_token_ids"])
if request.use_beam_search:
sampling_params = request.to_beam_search_params(