[ENGINE] muh_cc_dispatch + analysis: max_num_seqs=1 from computility-run.yaml
CRITICAL FINDING from reading computility-run.yaml: --max-num-seqs 1 This means the competition ALWAYS runs single-sequence inference. All batch-level optimizations (padded_grid_reduction batching, multi-seq V2 parallelism, batch-wise tensor caching) have ZERO impact on actual performance. The real bottleneck is single-sequence KV cache access: - decode: 1 seq × all heads × all KV blocks - prefill: 1 seq × chunked (max_num_batched_tokens=8192) - MoE: 1 seq × top_k=8 experts × 64 layers Updated muh_cc_dispatch.py to record QWEN36_MAX_NUM_SEQS=1. CCCL insight from padded_grid_reduction.cu: the padded grid batching pattern is only beneficial when num_seqs > 1. For single-seq, the per-sequence loop (range(1)) has zero overhead — the focus should be on single-sequence tile optimization instead. CCCL files: thrust/examples/padded_grid_reduction.cu, cub/block/block_exchange.cuh
This commit is contained in:
@@ -297,6 +297,7 @@ def select_moe_config(
|
||||
QWEN36_HEAD_DIM = 256 # text_cfg.head_dim
|
||||
QWEN36_NUM_KV_HEADS = 4 # num_key_value_heads
|
||||
QWEN36_MAX_SEQ_LEN = 100000 # from computility-run.yaml
|
||||
QWEN36_MAX_NUM_SEQS = 1 # CRITICAL: computility-run.yaml --max-num-seqs 1
|
||||
QWEN36_NUM_EXPERTS = 256 # MoE experts
|
||||
QWEN36_TOP_K = 8 # MoE top-k
|
||||
QWEN36_HIDDEN = 3584 # hidden_size
|
||||
|
||||
Reference in New Issue
Block a user