[CI]Add Kimi k2 nightly test (#5682)
### What this PR does / why we need it?
The PR add performance and accuracy tests for **Kimi-K2-Instruct-W8A8**
and **Kimi-K2-Thinking** models to the Nightly test suite.
#### Test Configuration
**Kimi-K2-Instruct-W8A8**
- model: vllm-ascend/Kimi-K2-Instruct-W8A8
- Hardware: A3, 2 Nodes (32 NPUs total, 16 NPUs per node)
- Architecture: Unified Distributed Inference
- Parallelism: **DP4 + TP8 + EP** (Data Parallel 4, Tensor Parallel 8,
Expert Parallel enabled).
- Optimization: **torchair graph**, **no-prefix-caching**.
- Node 0: DP Rank 0-1, Local DP 2, Tensor Parallel 8.
- Node 1: DP Rank 2-3, Local DP 2, Tensor Parallel 8.
- Benchmarks:
- Performance: vllm-ascend/GSM8K-in3500-bs2800.
- Accuracy: vllm-ascend/gsm8k-lite.
**Kimi-K2-Thinking**
- Model: moonshotai/Kimi-K2-Thinking
- Hardware: A3, 1 Node (16 NPUs total)
- Architecture: Single Node Distributed Inference
- Parallelism: TP16 + EP (Tensor Parallel 16, Expert Parallel enabled).
- Optimization: **no-prefix-caching**
- Benchmarks:
- Performance: vllm-ascend/GSM8K-in3500-bs400.
- Accuracy: vllm-ascend/gsm8k-lite.
### Does this PR introduce _any_ user-facing change?
**Yes.** This PR enhances the ```AisbenchRunner``` to support dynamic
configuration of the ```trust_remote_code``` flag. This allows the
AISBench client to successfully load tokenizers for models that require
custom code execution (e.g., **Kimi-K2-Thinking and
Kimi-K2-Instruct-W8A8**).
**Changes:**
1. ```AisbenchRunner.__init__ ```Added the ability to capture the
```trust_remote_code``` parameter from the case configuration.
``` python
self.batch_size = aisbench_config["batch_size"]
self.request_rate = aisbench_config.get("request_rate", 0)
+ self.trust_remote_code = aisbench_config.get("trust_remote_code", False)
self.temperature = aisbench_config.get("temperature")
self.top_k = aisbench_config.get("top_k")
```
2. ```AisbenchRunner._init_request_conf``` Added regex substitution to
inject the parameter into the generated dynamic configuration file.
``` python
content = re.sub(r'batch_size.*', f'batch_size = {self.batch_size},',
content)
+ content = re.sub(r'trust_remote_code=.*',
+ f'trust_remote_code={self.trust_remote_code},',
+ content)
content = content.replace("top_k", "#top_k")
content = content.replace("seed", "#seed")
```
**Details:**
- New Config Key: Users can add ```"trust_remote_code": True``` to any
dictionary within the ```aisbench_cases``` list.
- Default Value: Defaults to ```False``` to maintain existing security
protocols for standard models.
- Impact: Resolves ```ValueError``` when benchmarking reasoning models
or models with custom tokenizers that previously failed during the
AISBench local initialization phase.
**User Example:**
Users can now enable custom code execution for specific models (like
Kimi-K2-Thinking) directly in their test suite:
```
# Now supported in test scripts:
aisbench_cases = [{
"case_type": "performance",
"request_conf": "vllm_api_stream_chat",
"trust_remote_code": True, # New user-facing parameter
...
}]
```
### How was this patch tested?
Actions:
- https://github.com/vllm-project/vllm-ascend/actions/runs/20849768433
Result as following:
- **Kimi-K2-Instruct-W8A8**(25m25s)
1. Accuracy test
```
dataset version metric mode vllm-api-general-chat
--------- --------- -------- ------ -----------------------
gsm8k 7cd45e accuracy gen 96.88
```
2. Perf test
```
╒══════════════════════════╤═════════╤════════════════╤════════════════╤═══════════════╤════════════════╤════════════════╤════════════════╤════════════════╤═════╕
│ Performance Parameters │ Stage │ Average │ Min │ Max │ Median │ P75 │ P90 │ P99 │ N │
╞══════════════════════════╪═════════╪════════════════╪════════════════╪═══════════════╪════════════════╪════════════════╪════════════════╪════════════════╪═════╡
│ E2EL │ total │ 34571.489 ms │ 28657.8054 ms │ 36294.1788 ms │ 34714.7329 ms │ 35247.2724 ms │ 35526.6758 ms │ 36146.4314 ms │ 512 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼───────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ TTFT │ total │ 2043.9136 ms │ 627.4718 ms │ 3532.3978 ms │ 1906.0194 ms │ 2307.7979 ms │ 2883.8528 ms │ 3283.7012 ms │ 512 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼───────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ TPOT │ total │ 127.5591 ms │ 106.4937 ms │ 137.107 ms │ 128.3135 ms │ 129.5704 ms │ 131.1332 ms │ 134.1087 ms │ 512 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼───────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ ITL │ total │ 126.5571 ms │ 0.0095 ms │ 1340.783 ms │ 104.1398 ms │ 110.1272 ms │ 119.6124 ms │ 950.2924 ms │ 512 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼───────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ InputTokens │ total │ 3516.6055 │ 3014.0 │ 3985.0 │ 3525.0 │ 3525.0 │ 3586.8 │ 3800.67 │ 512 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼───────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ OutputTokens │ total │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 512 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼───────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ OutputTokenThroughput │ total │ 7.4143 token/s │ 7.0535 token/s │ 8.933 token/s │ 7.3744 token/s │ 7.4118 token/s │ 7.5608 token/s │ 8.7051 token/s │ 512 │
╘══════════════════════════╧═════════╧════════════════╧════════════════╧═══════════════╧════════════════╧════════════════╧════════════════╧════════════════╧═════╛
╒══════════════════════════╤═════════╤═══════════════════╕
│ Common Metric │ Stage │ Value │
╞══════════════════════════╪═════════╪═══════════════════╡
│ Benchmark Duration │ total │ 279430.9375 ms │
├──────────────────────────┼─────────┼───────────────────┤
│ Total Requests │ total │ 512 │
├──────────────────────────┼─────────┼───────────────────┤
│ Failed Requests │ total │ 0 │
├──────────────────────────┼─────────┼───────────────────┤
│ Success Requests │ total │ 512 │
├──────────────────────────┼─────────┼───────────────────┤
│ Concurrency │ total │ 63.3452 │
├──────────────────────────┼─────────┼───────────────────┤
│ Max Concurrency │ total │ 64 │
├──────────────────────────┼─────────┼───────────────────┤
│ Request Throughput │ total │ 1.8323 req/s │
├──────────────────────────┼─────────┼───────────────────┤
│ Total Input Tokens │ total │ 1800502 │
├──────────────────────────┼─────────┼───────────────────┤
│ Prefill Token Throughput │ total │ 1720.5255 token/s │
├──────────────────────────┼─────────┼───────────────────┤
│ Total generated tokens │ total │ 131072 │
├──────────────────────────┼─────────┼───────────────────┤
│ Input Token Throughput │ total │ 6443.4598 token/s │
├──────────────────────────┼─────────┼───────────────────┤
│ Output Token Throughput │ total │ 469.0676 token/s │
├──────────────────────────┼─────────┼───────────────────┤
│ Total Token Throughput │ total │ 6912.5274 token/s │
╘══════════════════════════╧═════════╧═══════════════════╛
```
- **Kimi-K2-Thinking**(43m51s)
1. Accuracy test
```
dataset version metric mode vllm-api-general-chat
--------- --------- -------- ------ -----------------------
gsm8k 7cd45e accuracy gen 100.00
```
2. Perf test
```
╒══════════════════════════╤═════════╤════════════════╤════════════════╤════════════════╤════════════════╤════════════════╤════════════════╤════════════════╤═════╕
│ Performance Parameters │ Stage │ Average │ Min │ Max │ Median │ P75 │ P90 │ P99 │ N │
╞══════════════════════════╪═════════╪════════════════╪════════════════╪════════════════╪════════════════╪════════════════╪════════════════╪════════════════╪═════╡
│ E2EL │ total │ 172384.3573 ms │ 34456.5517 ms │ 205922.9407 ms │ 174844.2216 ms │ 202656.092 ms │ 204428.9502 ms │ 205468.6776 ms │ 400 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ TTFT │ total │ 138740.3228 ms │ 655.1066 ms │ 171777.3003 ms │ 141088.0561 ms │ 169237.5599 ms │ 170716.4954 ms │ 171393.1278 ms │ 400 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ TPOT │ total │ 131.9374 ms │ 90.6331 ms │ 135.4144 ms │ 132.405 ms │ 132.948 ms │ 133.7549 ms │ 135.2543 ms │ 400 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ ITL │ total │ 130.9028 ms │ 0.0099 ms │ 960.3683 ms │ 116.9623 ms │ 122.3127 ms │ 132.0522 ms │ 886.4662 ms │ 400 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ InputTokens │ total │ 3514.575 │ 3014.0 │ 3843.0 │ 3525.0 │ 3525.0 │ 3588.0 │ 3801.08 │ 400 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ OutputTokens │ total │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 400 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ OutputTokenThroughput │ total │ 1.6799 token/s │ 1.2432 token/s │ 7.4296 token/s │ 1.4642 token/s │ 1.4737 token/s │ 1.8754 token/s │ 7.125 token/s │ 400 │
╘══════════════════════════╧═════════╧════════════════╧════════════════╧════════════════╧════════════════╧════════════════╧════════════════╧════════════════╧═════╛
╒══════════════════════════╤═════════╤═══════════════════╕
│ Common Metric │ Stage │ Value │
╞══════════════════════════╪═════════╪═══════════════════╡
│ Benchmark Duration │ total │ 1166795.568 ms │
├──────────────────────────┼─────────┼───────────────────┤
│ Total Requests │ total │ 400 │
├──────────────────────────┼─────────┼───────────────────┤
│ Failed Requests │ total │ 0 │
├──────────────────────────┼─────────┼───────────────────┤
│ Success Requests │ total │ 400 │
├──────────────────────────┼─────────┼───────────────────┤
│ Concurrency │ total │ 59.0967 │
├──────────────────────────┼─────────┼───────────────────┤
│ Max Concurrency │ total │ 64 │
├──────────────────────────┼─────────┼───────────────────┤
│ Request Throughput │ total │ 0.3428 req/s │
├──────────────────────────┼─────────┼───────────────────┤
│ Total Input Tokens │ total │ 1405830 │
├──────────────────────────┼─────────┼───────────────────┤
│ Prefill Token Throughput │ total │ 25.332 token/s │
├──────────────────────────┼─────────┼───────────────────┤
│ Total generated tokens │ total │ 102400 │
├──────────────────────────┼─────────┼───────────────────┤
│ Input Token Throughput │ total │ 1204.864 token/s │
├──────────────────────────┼─────────┼───────────────────┤
│ Output Token Throughput │ total │ 87.7617 token/s │
├──────────────────────────┼─────────┼───────────────────┤
│ Total Token Throughput │ total │ 1292.6258 token/s │
╘══════════════════════════╧═════════╧═══════════════════╛
```
- vLLM version: v0.13.0
- vLLM main:
2f4e6548ef
---------
Signed-off-by: MrZ20 <2609716663@qq.com>
Signed-off-by: root <root@LAPTOP-VQKDDVMG.localdomain>
This commit is contained in:
SILONG ZENG
@@ -143,5 +143,3 @@ jobs:
|
||||
# ignore test_dispatch_ffn_combine until the test is fixed
|
||||
pytest -sv ${{ inputs.tests }} \
|
||||
--ignore=tests/e2e/nightly/single_node/ops/singlecard_ops/test_fused_moe.py
|
||||
|
||||
|
||||
|
||||
6
.github/workflows/nightly_test_a3.yaml
vendored
6
.github/workflows/nightly_test_a3.yaml
vendored
@@ -83,6 +83,9 @@ jobs:
|
||||
- name: multi-node-qwen-vl-disagg-pd
|
||||
config_file_path: Qwen3-VL-235B-disagg-pd.yaml
|
||||
size: 2
|
||||
- name: multi-node-kimi-k2-instruct-w8a8
|
||||
config_file_path: Kimi-K2-Instruct-W8A8.yaml
|
||||
size: 2
|
||||
uses: ./.github/workflows/_e2e_nightly_multi_node.yaml
|
||||
with:
|
||||
soc_version: a3
|
||||
@@ -144,6 +147,9 @@ jobs:
|
||||
- name: qwen3-next-w8a8
|
||||
os: linux-aarch64-a3-4
|
||||
tests: tests/e2e/nightly/single_node/models/test_qwen3_next_w8a8.py
|
||||
- name: kimi-k2-thinking
|
||||
os: linux-aarch64-a3-16
|
||||
tests: tests/e2e/nightly/single_node/models/test_kimi_k2_thinking.py
|
||||
# TODO: Replace deepseek3.2-exp with deepseek3.2 after nightly tests pass
|
||||
# - name: deepseek3_2-exp-w8a8
|
||||
# os: linux-aarch64-a3-16
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
test_name: "test Kimi-K2-Instruct-W8A8 2-nodes-dp4-tp8-torchair"
|
||||
model: "vllm-ascend/Kimi-K2-Instruct-W8A8"
|
||||
|
||||
num_nodes: 2
|
||||
npu_per_node: 16
|
||||
env_common:
|
||||
VLLM_USE_MODELSCOPE: true
|
||||
HCCL_BUFFSIZE: 1024
|
||||
SERVER_PORT: 8080
|
||||
OMP_PROC_BIND: false
|
||||
OMP_NUM_THREADS: 100
|
||||
NUMEXPR_MAX_THREADS: 128
|
||||
|
||||
deployment:
|
||||
-
|
||||
server_cmd: >
|
||||
vllm serve "vllm-ascend/Kimi-K2-Instruct-W8A8"
|
||||
--host 0.0.0.0
|
||||
--port $SERVER_PORT
|
||||
--data-parallel-size 4
|
||||
--data-parallel-size-local 2
|
||||
--data-parallel-start-rank 0
|
||||
--data-parallel-address $LOCAL_IP
|
||||
--data-parallel-rpc-port 13389
|
||||
--tensor-parallel-size 8
|
||||
--seed 1024
|
||||
--enable-expert-parallel
|
||||
--max-num-seqs 32
|
||||
--max-model-len 8192
|
||||
--max-num-batched-tokens 8192
|
||||
--quantization ascend
|
||||
--trust-remote-code
|
||||
--no-enable-prefix-caching
|
||||
--gpu-memory-utilization 0.9
|
||||
--additional-config '{"torchair_graph_config":{"enabled":true}}'
|
||||
|
||||
-
|
||||
server_cmd: >
|
||||
vllm serve "vllm-ascend/Kimi-K2-Instruct-W8A8"
|
||||
--headless
|
||||
--data-parallel-size 4
|
||||
--data-parallel-size-local 2
|
||||
--data-parallel-start-rank 2
|
||||
--data-parallel-address $MASTER_IP
|
||||
--data-parallel-rpc-port 13389
|
||||
--tensor-parallel-size 8
|
||||
--seed 1024
|
||||
--enable-expert-parallel
|
||||
--max-num-seqs 32
|
||||
--max-model-len 8192
|
||||
--max-num-batched-tokens 8192
|
||||
--quantization ascend
|
||||
--trust-remote-code
|
||||
--no-enable-prefix-caching
|
||||
--gpu-memory-utilization 0.9
|
||||
--additional-config '{"torchair_graph_config":{"enabled":true}}'
|
||||
|
||||
benchmarks:
|
||||
perf:
|
||||
case_type: performance
|
||||
dataset_path: vllm-ascend/GSM8K-in3500-bs2800
|
||||
request_conf: vllm_api_stream_chat
|
||||
dataset_conf: gsm8k/gsm8k_gen_0_shot_cot_str_perf
|
||||
num_prompts: 512
|
||||
max_out_len: 256
|
||||
batch_size: 64
|
||||
trust_remote_code: True
|
||||
request_rate: 11.2
|
||||
baseline: 1
|
||||
threshold: 0.97
|
||||
acc:
|
||||
case_type: accuracy
|
||||
dataset_path: vllm-ascend/gsm8k-lite
|
||||
request_conf: vllm_api_general_chat
|
||||
dataset_conf: gsm8k/gsm8k_gen_0_shot_cot_chat_prompt
|
||||
max_out_len: 7680
|
||||
batch_size: 64
|
||||
baseline: 95
|
||||
threshold: 5
|
||||
110
tests/e2e/nightly/single_node/models/test_kimi_k2_thinking.py
Normal file
110
tests/e2e/nightly/single_node/models/test_kimi_k2_thinking.py
Normal file
@@ -0,0 +1,110 @@
|
||||
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
|
||||
# Copyright 2023 The vLLM team.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# This file is a part of the vllm-ascend project.
|
||||
#
|
||||
from typing import Any
|
||||
|
||||
import openai
|
||||
import pytest
|
||||
from vllm.utils.network_utils import get_open_port
|
||||
|
||||
from tests.e2e.conftest import RemoteOpenAIServer
|
||||
from tools.aisbench import run_aisbench_cases
|
||||
|
||||
MODELS = [
|
||||
"moonshotai/Kimi-K2-Thinking",
|
||||
]
|
||||
|
||||
TENSOR_PARALLELS = [16]
|
||||
|
||||
prompts = [
|
||||
"San Francisco is a",
|
||||
]
|
||||
|
||||
api_keyword_args = {
|
||||
"max_tokens": 10,
|
||||
}
|
||||
|
||||
aisbench_cases = [{
|
||||
"case_type": "accuracy",
|
||||
"dataset_path": "vllm-ascend/gsm8k-lite",
|
||||
"request_conf": "vllm_api_general_chat",
|
||||
"dataset_conf": "gsm8k/gsm8k_gen_0_shot_cot_chat_prompt",
|
||||
"max_out_len": 4096,
|
||||
"batch_size": 32,
|
||||
"baseline": 95,
|
||||
"threshold": 5
|
||||
}, {
|
||||
"case_type": "performance",
|
||||
"dataset_path": "vllm-ascend/GSM8K-in3500-bs400",
|
||||
"request_conf": "vllm_api_stream_chat",
|
||||
"dataset_conf": "gsm8k/gsm8k_gen_0_shot_cot_str_perf",
|
||||
"num_prompts": 512,
|
||||
"max_out_len": 256,
|
||||
"batch_size": 64,
|
||||
"trust_remote_code": True,
|
||||
"request_rate": 11.2,
|
||||
"baseline": 1,
|
||||
"threshold": 0.97
|
||||
}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("model", MODELS)
|
||||
@pytest.mark.parametrize("tp_size", TENSOR_PARALLELS)
|
||||
async def test_models(model: str, tp_size: int) -> None:
|
||||
port = get_open_port()
|
||||
env_dict = {
|
||||
"HCCL_BUFFSIZE": "1024",
|
||||
"TASK_QUEUE_ENABLE": "1",
|
||||
"OMP_PROC_BIND": "false",
|
||||
"HCCL_OP_EXPANSION_MODE": "AIV",
|
||||
"PYTORCH_NPU_ALLOC_CONF": "expandable_segments:True"
|
||||
}
|
||||
server_args = [
|
||||
"--tensor-parallel-size",
|
||||
str(tp_size),
|
||||
"--port",
|
||||
str(port),
|
||||
"--max-model-len",
|
||||
"8192",
|
||||
"--max-num-batched-tokens",
|
||||
"8192",
|
||||
"--max-num-seqs",
|
||||
"12",
|
||||
"--gpu-memory-utilization",
|
||||
"0.9",
|
||||
"--trust-remote-code",
|
||||
"--enable-expert-parallel",
|
||||
"--no-enable-prefix-caching",
|
||||
]
|
||||
request_keyword_args: dict[str, Any] = {
|
||||
**api_keyword_args,
|
||||
}
|
||||
with RemoteOpenAIServer(model,
|
||||
server_args,
|
||||
server_port=port,
|
||||
env_dict=env_dict,
|
||||
auto_port=False) as server:
|
||||
client = server.get_async_client()
|
||||
batch = await client.completions.create(
|
||||
model=model,
|
||||
prompt=prompts,
|
||||
**request_keyword_args,
|
||||
)
|
||||
choices: list[openai.types.CompletionChoice] = batch.choices
|
||||
assert choices[0].text, "empty response"
|
||||
# aisbench test
|
||||
run_aisbench_cases(model, port, aisbench_cases)
|
||||
@@ -92,6 +92,8 @@ class AisbenchRunner:
|
||||
self.max_out_len = aisbench_config["max_out_len"]
|
||||
self.batch_size = aisbench_config["batch_size"]
|
||||
self.request_rate = aisbench_config.get("request_rate", 0)
|
||||
self.trust_remote_code = aisbench_config.get("trust_remote_code",
|
||||
False)
|
||||
self.temperature = aisbench_config.get("temperature")
|
||||
self.top_k = aisbench_config.get("top_k")
|
||||
self.top_p = aisbench_config.get("top_p")
|
||||
@@ -145,6 +147,9 @@ class AisbenchRunner:
|
||||
f'max_out_len = {self.max_out_len},', content)
|
||||
content = re.sub(r'batch_size.*', f'batch_size = {self.batch_size},',
|
||||
content)
|
||||
content = re.sub(r'trust_remote_code=.*',
|
||||
f'trust_remote_code={self.trust_remote_code},',
|
||||
content)
|
||||
content = content.replace("top_k", "#top_k")
|
||||
content = content.replace("seed", "#seed")
|
||||
content = content.replace("repetition_penalty", "#repetition_penalty")
|
||||
|
||||
Reference in New Issue
Block a user