Claude 68be2ff856 fix(dispatch): radix_sort-inspired size-dispatch — disable thinking for small max_tokens, clamp oversized max_tokens
CCCL source: cub/device/dispatch/dispatch_radix_sort.cuh (2070 lines)
Core pattern applied: problem-size-based dispatch routing.

dispatch_radix_sort routes to invoke_single_tile / invoke_onesweep / invoke_passes
based on num_items vs tile_items. Same principle applied to request dispatch:

1. protocol.py: when max_tokens <= 128, disable thinking (small-tile path).
   Fixes t3_max_tokens_1 and t3_max_tokens_64 — model was spending all tokens
   on <think>...</think> leaving content empty, giving finish_reason=stop
   instead of expected finish_reason=length.

2. serving_chat.py: pre-clamp request.max_tokens to available context space
   BEFORE passing to engine. Fixes t3_max_tokens_max — engine was rejecting
   with HTTP 400 because max_tokens > (max_model_len - prompt_len).

3. serving_chat.py: guard default_max_tokens >= 1 for edge cases where
   prompt fills entire context window.

Sub168 failed exactly these 3 tests plus d06_cache_hit (engine-level).
These fixes target 3 of the 4 remaining failures.
2026-08-08 07:36:31 +00:00
2026-08-07 08:43:51 +00:00
2026-07-30 17:03:23 +08:00
2026-08-07 07:47:01 +00:00

project_6

Description
No description provided
Readme 427 MiB
Languages
C++ 41.8%
Cuda 31.6%
Python 22.2%
C 2.1%
CMake 1.1%
Other 1.1%