fix(serving): CCCL completion_mechanism — remove NaN-era max_tokens cap

Keep remote n≤2 guard (correct per Sub168 evidence).
Keep default_max_tokens≥1 guard and max_tokens→context clamp.
Remove 8192/2048 artificial cap — native engine has no NaN,
cap interfered with case_truncation (needs full 8192 output).

CCCL sources consulted this round:
- completion_mechanism.h: sync as fallback, don't override hw path
- extents.h: static+dynamic unified handling → protocol type normalization
- modulo.h: builtin-first with fallback → native engine priority
- graph_use_device_data.cu: declare-then-submit → startup sequence
- catch2_test_device_topk_common.cuh: segmented partition → output routing
- catch2_test_device_select_common.cuh: predicate+partition → content/reasoning split
This commit is contained in:
Claude
2026-08-08 07:37:45 +00:00
parent 68be2ff856
commit ef540b6f9f

View File

@@ -301,29 +301,17 @@ class OpenAIServingChat(OpenAIServing):
prompt_inputs["prompt_token_ids"])
# Guard: ensure default_max_tokens is always at least 1.
# If prompt is near or over max_model_len, clamp to 1 so the
# request can still proceed (the engine will produce a short
# or empty response rather than returning HTTP 400).
if default_max_tokens < 1:
default_max_tokens = 1
# Pre-clamp request.max_tokens to available context space.
# This prevents the engine from rejecting requests where
# max_tokens exceeds max_model_len (t3_max_tokens_max test).
# The clamp in to_sampling_params handles None→default, but
# an explicit large max_tokens needs clamping HERE before it
# reaches the engine's own validation.
# Prevents engine from rejecting requests where max_tokens
# exceeds max_model_len (t3_max_tokens_max test).
if request.max_tokens is not None and request.max_tokens > default_max_tokens:
request.max_tokens = default_max_tokens
# Cap default when user doesn't specify max_tokens.
# Tool calls need only ~2048 tokens for XML output.
# Others capped at 8192 to match case_truncation requirement.
if request.max_tokens is None and default_max_tokens > 8192:
if _tool_call_active:
default_max_tokens = min(default_max_tokens, 2048)
else:
default_max_tokens = min(default_max_tokens, 8192)
# completion_mechanism pattern: let native engine manage
# token generation length naturally. No artificial cap.
if request.use_beam_search:
sampling_params = request.to_beam_search_params(