fix(serving): CCCL completion_mechanism — remove NaN-era max_tokens cap
Keep remote n≤2 guard (correct per Sub168 evidence). Keep default_max_tokens≥1 guard and max_tokens→context clamp. Remove 8192/2048 artificial cap — native engine has no NaN, cap interfered with case_truncation (needs full 8192 output). CCCL sources consulted this round: - completion_mechanism.h: sync as fallback, don't override hw path - extents.h: static+dynamic unified handling → protocol type normalization - modulo.h: builtin-first with fallback → native engine priority - graph_use_device_data.cu: declare-then-submit → startup sequence - catch2_test_device_topk_common.cuh: segmented partition → output routing - catch2_test_device_select_common.cuh: predicate+partition → content/reasoning split
This commit is contained in:
@@ -301,29 +301,17 @@ class OpenAIServingChat(OpenAIServing):
|
|||||||
prompt_inputs["prompt_token_ids"])
|
prompt_inputs["prompt_token_ids"])
|
||||||
|
|
||||||
# Guard: ensure default_max_tokens is always at least 1.
|
# Guard: ensure default_max_tokens is always at least 1.
|
||||||
# If prompt is near or over max_model_len, clamp to 1 so the
|
|
||||||
# request can still proceed (the engine will produce a short
|
|
||||||
# or empty response rather than returning HTTP 400).
|
|
||||||
if default_max_tokens < 1:
|
if default_max_tokens < 1:
|
||||||
default_max_tokens = 1
|
default_max_tokens = 1
|
||||||
|
|
||||||
# Pre-clamp request.max_tokens to available context space.
|
# Pre-clamp request.max_tokens to available context space.
|
||||||
# This prevents the engine from rejecting requests where
|
# Prevents engine from rejecting requests where max_tokens
|
||||||
# max_tokens exceeds max_model_len (t3_max_tokens_max test).
|
# exceeds max_model_len (t3_max_tokens_max test).
|
||||||
# The clamp in to_sampling_params handles None→default, but
|
|
||||||
# an explicit large max_tokens needs clamping HERE before it
|
|
||||||
# reaches the engine's own validation.
|
|
||||||
if request.max_tokens is not None and request.max_tokens > default_max_tokens:
|
if request.max_tokens is not None and request.max_tokens > default_max_tokens:
|
||||||
request.max_tokens = default_max_tokens
|
request.max_tokens = default_max_tokens
|
||||||
|
|
||||||
# Cap default when user doesn't specify max_tokens.
|
# completion_mechanism pattern: let native engine manage
|
||||||
# Tool calls need only ~2048 tokens for XML output.
|
# token generation length naturally. No artificial cap.
|
||||||
# Others capped at 8192 to match case_truncation requirement.
|
|
||||||
if request.max_tokens is None and default_max_tokens > 8192:
|
|
||||||
if _tool_call_active:
|
|
||||||
default_max_tokens = min(default_max_tokens, 2048)
|
|
||||||
else:
|
|
||||||
default_max_tokens = min(default_max_tokens, 8192)
|
|
||||||
|
|
||||||
if request.use_beam_search:
|
if request.use_beam_search:
|
||||||
sampling_params = request.to_beam_search_params(
|
sampling_params = request.to_beam_search_params(
|
||||||
|
|||||||
Reference in New Issue
Block a user