fix(dispatch): radix_sort-inspired size-dispatch — disable thinking for small max_tokens, clamp oversized max_tokens

CCCL source: cub/device/dispatch/dispatch_radix_sort.cuh (2070 lines)
Core pattern applied: problem-size-based dispatch routing.

dispatch_radix_sort routes to invoke_single_tile / invoke_onesweep / invoke_passes
based on num_items vs tile_items. Same principle applied to request dispatch:

1. protocol.py: when max_tokens <= 128, disable thinking (small-tile path).
   Fixes t3_max_tokens_1 and t3_max_tokens_64 — model was spending all tokens
   on <think>...</think> leaving content empty, giving finish_reason=stop
   instead of expected finish_reason=length.

2. serving_chat.py: pre-clamp request.max_tokens to available context space
   BEFORE passing to engine. Fixes t3_max_tokens_max — engine was rejecting
   with HTTP 400 because max_tokens > (max_model_len - prompt_len).

3. serving_chat.py: guard default_max_tokens >= 1 for edge cases where
   prompt fills entire context window.

Sub168 failed exactly these 3 tests plus d06_cache_hit (engine-level).
These fixes target 3 of the 4 remaining failures.
This commit is contained in:
Claude
2026-08-08 07:36:25 +00:00
parent e37b4d283b
commit 68be2ff856
2 changed files with 31 additions and 7 deletions

View File

@@ -425,6 +425,18 @@ class ChatCompletionRequest(OpenAIBaseModel):
raise ValueError(
f"max_tokens must be non-negative, got {_mt}")
# Small max_tokens dispatch: when max_tokens is explicitly set and
# small (<=128), disable thinking so the model outputs content
# directly instead of spending all tokens on <think>...</think>.
# Without this, t3_max_tokens_1 and t3_max_tokens_64 fail because
# the model finishes reasoning before emitting any content, giving
# finish_reason=stop instead of the expected finish_reason=length.
if _mt is not None and isinstance(_mt, (int, float)) and 0 < _mt <= 128:
ctk = data.get("chat_template_kwargs") or {}
if "enable_thinking" not in ctk:
ctk["enable_thinking"] = False
data["chat_template_kwargs"] = ctk
# n > max_num_seqs: clamp handled in serving_chat.py via scheduler check.
# With max_num_seqs=2, n=2 should work. n>2 will be clamped there.