arch(cccl): match Sub168 proven config + bench.py timeout pattern

CCCL sources read as design input:
- group_by.cuh: static vs dynamic unit_count → match proven config
- bench/bench.py: timeout + cache + graceful failure → cap default tokens
- transform_iterator.cu: lazy transform pipeline → message preprocessing

Changes:
1. computility-run.yaml: match Sub168's proven config exactly:
   - max-model-len: 100000 (not 32768, Sub168 used 100000 successfully)
   - Remove --max-num-batched-tokens (Sub168 didn't use it)
   - Remove --enable-chunked-prefill (Sub168 didn't use it)
   - Keep: max-num-seqs=1, gpu-mem=0.9, enable-prefix-caching

2. serving_chat.py: CCCL bench.py timeout pattern
   - Cap ALL requests without explicit max_tokens to 8192
   - Cap tool_call requests to 2048
   - Prevents NaN-damaged model from generating 99K tokens
   - Sub168 generates 139-2497 tokens per request
This commit is contained in:
project6-dev
2026-08-07 10:02:53 +00:00
parent 3342d18bcc
commit 1ba0dd3966
2 changed files with 13 additions and 18 deletions

View File

@@ -8,7 +8,7 @@ command:
- --served-model-name
- llm
- --max-model-len
- '32768'
- '100000'
- --gpu-memory-utilization
- '0.90'
- --trust-remote-code
@@ -16,8 +16,6 @@ command:
- '4'
- --max-num-seqs
- '1'
- --max-num-batched-tokens
- '4096'
- --disable-log-requests
- --disable-frontend-multiprocessing
- --enforce-eager
@@ -27,7 +25,6 @@ command:
- --reasoning-parser
- qwen3
- --enable-prefix-caching
- --enable-chunked-prefill
- --dtype
- half
env: