fix(dispatch): radix_sort-inspired size-dispatch — disable thinking for small max_tokens, clamp oversized max_tokens
CCCL source: cub/device/dispatch/dispatch_radix_sort.cuh (2070 lines) Core pattern applied: problem-size-based dispatch routing. dispatch_radix_sort routes to invoke_single_tile / invoke_onesweep / invoke_passes based on num_items vs tile_items. Same principle applied to request dispatch: 1. protocol.py: when max_tokens <= 128, disable thinking (small-tile path). Fixes t3_max_tokens_1 and t3_max_tokens_64 — model was spending all tokens on <think>...</think> leaving content empty, giving finish_reason=stop instead of expected finish_reason=length. 2. serving_chat.py: pre-clamp request.max_tokens to available context space BEFORE passing to engine. Fixes t3_max_tokens_max — engine was rejecting with HTTP 400 because max_tokens > (max_model_len - prompt_len). 3. serving_chat.py: guard default_max_tokens >= 1 for edge cases where prompt fills entire context window. Sub168 failed exactly these 3 tests plus d06_cache_hit (engine-level). These fixes target 3 of the 4 remaining failures.
This commit is contained in:
@@ -425,6 +425,18 @@ class ChatCompletionRequest(OpenAIBaseModel):
|
||||
raise ValueError(
|
||||
f"max_tokens must be non-negative, got {_mt}")
|
||||
|
||||
# Small max_tokens dispatch: when max_tokens is explicitly set and
|
||||
# small (<=128), disable thinking so the model outputs content
|
||||
# directly instead of spending all tokens on <think>...</think>.
|
||||
# Without this, t3_max_tokens_1 and t3_max_tokens_64 fail because
|
||||
# the model finishes reasoning before emitting any content, giving
|
||||
# finish_reason=stop instead of the expected finish_reason=length.
|
||||
if _mt is not None and isinstance(_mt, (int, float)) and 0 < _mt <= 128:
|
||||
ctk = data.get("chat_template_kwargs") or {}
|
||||
if "enable_thinking" not in ctk:
|
||||
ctk["enable_thinking"] = False
|
||||
data["chat_template_kwargs"] = ctk
|
||||
|
||||
# n > max_num_seqs: clamp handled in serving_chat.py via scheduler check.
|
||||
# With max_num_seqs=2, n=2 should work. n>2 will be clamped there.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user