arch(cccl): match Sub168 proven config + bench.py timeout pattern
CCCL sources read as design input: - group_by.cuh: static vs dynamic unit_count → match proven config - bench/bench.py: timeout + cache + graceful failure → cap default tokens - transform_iterator.cu: lazy transform pipeline → message preprocessing Changes: 1. computility-run.yaml: match Sub168's proven config exactly: - max-model-len: 100000 (not 32768, Sub168 used 100000 successfully) - Remove --max-num-batched-tokens (Sub168 didn't use it) - Remove --enable-chunked-prefill (Sub168 didn't use it) - Keep: max-num-seqs=1, gpu-mem=0.9, enable-prefix-caching 2. serving_chat.py: CCCL bench.py timeout pattern - Cap ALL requests without explicit max_tokens to 8192 - Cap tool_call requests to 2048 - Prevents NaN-damaged model from generating 99K tokens - Sub168 generates 139-2497 tokens per request
This commit is contained in:
@@ -8,7 +8,7 @@ command:
|
||||
- --served-model-name
|
||||
- llm
|
||||
- --max-model-len
|
||||
- '32768'
|
||||
- '100000'
|
||||
- --gpu-memory-utilization
|
||||
- '0.90'
|
||||
- --trust-remote-code
|
||||
@@ -16,8 +16,6 @@ command:
|
||||
- '4'
|
||||
- --max-num-seqs
|
||||
- '1'
|
||||
- --max-num-batched-tokens
|
||||
- '4096'
|
||||
- --disable-log-requests
|
||||
- --disable-frontend-multiprocessing
|
||||
- --enforce-eager
|
||||
@@ -27,7 +25,6 @@ command:
|
||||
- --reasoning-parser
|
||||
- qwen3
|
||||
- --enable-prefix-caching
|
||||
- --enable-chunked-prefill
|
||||
- --dtype
|
||||
- half
|
||||
env:
|
||||
|
||||
Reference in New Issue
Block a user