init v0.23.0

Signed-off-by: Sun Ruoxi <sunruoxi@4paradigm.com>
2026-08-27 15:11:51 +08:00
parent b582a8e7d1
commit 7f8a1b1f7a
2849 changed files with 712887 additions and 22001 deletions
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -1,8 +1,13 @@
-# Introduction
+# vLLM Ascend Benchmarks
+
+## Introduction
+
 This document outlines the benchmarking methodology for vllm-ascend, aimed at evaluating the performance under a variety of workloads. The primary goal is to help developers assess whether their pull requests improve or degrade vllm-ascend's performance.

-# Overview
+## Overview
+
 **Benchmarking Coverage**: We measure latency, throughput, and fixed-QPS serving on the Atlas800I A2 (see [quick_start](../docs/source/quick_start.md) to learn more supported devices list), with different models(coming soon).
+
 - Latency tests
    - Input length: 32 tokens.
    - Output length: 128 tokens.
@@ -24,10 +29,12 @@ This document outlines the benchmarking methodology for vllm-ascend, aimed at ev
    - Models: Qwen2.5-VL-7B-Instruct, Qwen2.5-7B-Instruct, Qwen3-8B.
    - Evaluation metrics: throughput, TTFT (time to the first token, with mean, median and p99), ITL (inter-token latency, with mean, median and p99).

-**Benchmarking Duration**: about 800 senond for single model.
+**Benchmarking Duration**: about 800 seconds for single model.
+
+## Quick Use
+
+### Prerequisites

-# Quick Use
-## Prerequisites
 Before running the benchmarks, ensure the following:

 - vllm and vllm-ascend are installed and properly set up in an NPU environment, as these scripts are specifically designed for NPU devices.
@@ -39,9 +46,9 @@ Before running the benchmarks, ensure the following:
  ```
  
 - For performance benchmark, it is recommended to set the [load-format](https://github.com/vllm-project/vllm-ascend/blob/5897dc5bbe321ca90c26225d0d70bff24061d04b/benchmarks/tests/latency-tests.json#L7) as `dummy`, It will construct random weights based on the passed model without downloading the weights from internet, which can greatly reduce the benchmark time.
- If you want to run benchmark customized, feel free to add your own models and parameters in the [JSON](https://github.com/vllm-project/vllm-ascend/tree/main/benchmarks/tests), let's take `Qwen2.5-VL-7B-Instruct`as an example:
+- If you want to run a customized benchmark, feel free to add your own models and parameters in the [JSON](https://github.com/vllm-project/vllm-ascend/tree/main/benchmarks/tests), let's take `Qwen2.5-VL-7B-Instruct`as an example:

-  ```shell
+  ```json
  [
  {
    "test_name": "serving_qwen2_5vl_7B_tp1",
@@ -75,45 +82,46 @@ Before running the benchmarks, ensure the following:
  
 this Json will be structured and parsed into server parameters and client parameters by the benchmark script. This configuration defines a test case named `serving_qwen2_5vl_7B_tp1`, designed to evaluate the performance of the `Qwen/Qwen2.5-VL-7B-Instruct` model under different request rates. The test includes both server and client parameters, for more parameters details, see vllm benchmark [cli](https://github.com/vllm-project/vllm/tree/main/vllm/benchmarks).

-  - **Test Overview**
-     - Test Name: serving_qwen2_5vl_7B_tp1
+- **Test Overview**
+    - Test Name: serving_qwen2_5vl_7B_tp1

-     - Queries Per Second (QPS): The test is run at four different QPS levels: 1, 4, 16, and inf (infinite load, typically used for stress testing).
+    - Queries Per Second (QPS): The test is run at four different QPS levels: 1, 4, 16, and inf (infinite load, typically used for stress testing).

-  - Server Parameters
-     - Model: Qwen/Qwen2.5-VL-7B-Instruct
+- Server Parameters
+    - Model: Qwen/Qwen2.5-VL-7B-Instruct

-     - Tensor Parallelism: 1 (no model parallelism is used; the model runs on a single device or node)
+    - Tensor Parallelism: 1 (no model parallelism is used; the model runs on a single device or node)

-     - Swap Space: 16 GB (used to handle memory overflow by swapping to disk)
+    - Swap Space: 16 GB (used to handle memory overflow by swapping to disk)

-     - disable_log_stats: disables logging of performance statistics.
+    - disable_log_stats: disables logging of performance statistics.

-     - disable_log_requests: disables logging of individual requests.
+    - disable_log_requests: disables logging of individual requests.

-     - Trust Remote Code: enabled (allows execution of model-specific custom code)
+    - Trust Remote Code: enabled (allows execution of model-specific custom code)

-     - Max Model Length: 16,384 tokens (maximum context length supported by the model)
+    - Max Model Length: 16,384 tokens (maximum context length supported by the model)

-  - Client Parameters
+- Client Parameters

-     - Model: Qwen/Qwen2.5-VL-7B-Instruct (same as the server)
+    - Model: Qwen/Qwen2.5-VL-7B-Instruct (same as the server)

-     - Backend: openai-chat (suggests the client uses the OpenAI-compatible chat API format)
+    - Backend: openai-chat (suggests the client uses the OpenAI-compatible chat API format)

-     - Dataset Source: Hugging Face (hf)
+    - Dataset Source: Hugging Face (hf)

-     - Dataset Split: train
+    - Dataset Split: train

-     - Endpoint: /v1/chat/completions (the REST API endpoint to which chat requests are sent)
+    - Endpoint: /v1/chat/completions (the REST API endpoint to which chat requests are sent)

-     - Dataset Path: lmarena-ai/vision-arena-bench-v0.1 (the benchmark dataset used for evaluation, hosted on Hugging Face)
+    - Dataset Path: lmarena-ai/vision-arena-bench-v0.1 (the benchmark dataset used for evaluation, hosted on Hugging Face)

-     - Number of Prompts: 200 (the total number of prompts used during the test)
+    - Number of Prompts: 200 (the total number of prompts used during the test)

-## Run benchmarks
+### Run benchmarks
+
+#### Use benchmark script

-### Use benchmark script
 The provided scripts automatically execute performance tests for serving, throughput, and latency. To start the benchmarking process, run command in the vllm-ascend root directory:

 ```shell
@@ -124,21 +132,22 @@ Once the script completes, you can find the results in the benchmarks/results fo

 ```shell
 .
-|-- serving_qwen2_5_7B_tp1_qps_1.json
-|-- serving_qwen2_5_7B_tp1_qps_16.json
-|-- serving_qwen2_5_7B_tp1_qps_4.json
-|-- serving_qwen2_5_7B_tp1_qps_inf.json
-|-- latency_qwen2_5_7B_tp1.json
-|-- throughput_qwen2_5_7B_tp1.json
+|-- serving_qwen2_5_7Bvl_tp1_qps_1.json
+|-- serving_qwen2_5_7Bvl_tp1_qps_16.json
+|-- serving_qwen2_5_7Bvl_tp1_qps_4.json
+|-- serving_qwen2_5_7Bvl_tp1_qps_inf.json
+|-- throughput_qwen2_5_7Bvl_tp1.json
 ```

 These files contain detailed benchmarking results for further analysis.

-### Use benchmark cli
+#### Use benchmark cli

 For more flexible and customized use, benchmark cli is also provided to run online/offline benchmarks
-Similarly, let’s take `Qwen2.5-VL-7B-Instruct` benchmark as an example:
-#### Online serving
+Similarly, let's take `Qwen2.5-VL-7B-Instruct` benchmark as an example:
+
+##### Online serving
+
 1. Launch the server:

    ```shell
@@ -156,7 +165,8 @@ Similarly, let’s take `Qwen2.5-VL-7B-Instruct` benchmark as an example:
    --request-rate 16
    ```

-#### Offline
+##### Offline
+
 - **Throughput**

  ```shell
--- a/benchmarks/ops/ben_vocabparallelembedding.py
+++ b/benchmarks/ops/ben_vocabparallelembedding.py
@@ -1,5 +1,3 @@
-from typing import Tuple
-
 import numpy as np
 import pytest
 import torch
@@ -47,20 +45,12 @@ def get_masked_input_and_mask_ref(
    num_org_vocab_padding: int,
    added_vocab_start_index: int,
    added_vocab_end_index: int,
-) -> Tuple[torch.Tensor, torch.Tensor]:
+) -> tuple[torch.Tensor, torch.Tensor]:
    """Reference implementation for verification"""
    org_vocab_mask = (input_ >= org_vocab_start_index) & (input_ < org_vocab_end_index)
-    added_vocab_mask = (input_ >= added_vocab_start_index) & (
-        input_ < added_vocab_end_index
-    )
-    added_offset = (
-        added_vocab_start_index
-        - (org_vocab_end_index - org_vocab_start_index)
-        - num_org_vocab_padding
-    )
-    valid_offset = (org_vocab_start_index * org_vocab_mask) + (
-        added_offset * added_vocab_mask
-    )
+    added_vocab_mask = (input_ >= added_vocab_start_index) & (input_ < added_vocab_end_index)
+    added_offset = added_vocab_start_index - (org_vocab_end_index - org_vocab_start_index) - num_org_vocab_padding
+    valid_offset = (org_vocab_start_index * org_vocab_mask) + (added_offset * added_vocab_mask)
    vocab_mask = org_vocab_mask | added_vocab_mask
    masked_input = vocab_mask * (input_ - valid_offset)
    return masked_input, ~vocab_mask
@@ -78,7 +68,7 @@ SEEDS = [0]
@pytest.mark.parametrize("seed", SEEDS)
@torch.inference_mode()
 def test_get_masked_input_and_mask(
-    shape: Tuple[int, ...],
+    shape: tuple[int, ...],
    dtype: torch.dtype,
    device: str,
    seed: int,
--- a/benchmarks/scripts/convert_json_to_markdown.py
+++ b/benchmarks/scripts/convert_json_to_markdown.py
@@ -59,9 +59,7 @@ def results_to_json(latency, throughput, serving):


 if __name__ == "__main__":
-    parser = argparse.ArgumentParser(
-        description="Process the results of the benchmark tests."
-    )
+    parser = argparse.ArgumentParser(description="Process the results of the benchmark tests.")
    parser.add_argument(
        "--results_folder",
        type=str,
@@ -80,12 +78,8 @@ if __name__ == "__main__":
        default="./perf_result_template.md",
        help="The template file for the markdown report.",
    )
-    parser.add_argument(
-        "--tag", default="main", help="Tag to be used for release message."
-    )
-    parser.add_argument(
-        "--commit_id", default="", help="Commit ID to be used for release message."
-    )
+    parser.add_argument("--tag", default="main", help="Tag to be used for release message.")
+    parser.add_argument("--commit_id", default="", help="Commit ID to be used for release message.")

    args = parser.parse_args()
    results_folder = (CUR_PATH / args.results_folder).resolve()
@@ -116,9 +110,7 @@ if __name__ == "__main__":
            # get different percentiles
            for perc in [10, 25, 50, 75, 90, 99]:
                # Multiply 1000 to convert the time unit from s to ms
-                raw_result.update(
-                    {f"P{perc}": 1000 * raw_result["percentiles"][str(perc)]}
-                )
+                raw_result.update({f"P{perc}": 1000 * raw_result["percentiles"][str(perc)]})
            raw_result["avg_latency"] = raw_result["avg_latency"] * 1000

            # add the result to raw_result
@@ -142,38 +134,24 @@ if __name__ == "__main__":
    serving_results = pd.DataFrame.from_dict(serving_results)
    throughput_results = pd.DataFrame.from_dict(throughput_results)

-    raw_results_json = results_to_json(
-        latency_results, throughput_results, serving_results
-    )
+    raw_results_json = results_to_json(latency_results, throughput_results, serving_results)

    # remapping the key, for visualization purpose
    if not latency_results.empty:
-        latency_results = latency_results[list(latency_column_mapping.keys())].rename(
-            columns=latency_column_mapping
-        )
+        latency_results = latency_results[list(latency_column_mapping.keys())].rename(columns=latency_column_mapping)
    if not serving_results.empty:
-        serving_results = serving_results[list(serving_column_mapping.keys())].rename(
-            columns=serving_column_mapping
-        )
+        serving_results = serving_results[list(serving_column_mapping.keys())].rename(columns=serving_column_mapping)
    if not throughput_results.empty:
-        throughput_results = throughput_results[
-            list(throughput_results_column_mapping.keys())
-        ].rename(columns=throughput_results_column_mapping)
+        throughput_results = throughput_results[list(throughput_results_column_mapping.keys())].rename(
+            columns=throughput_results_column_mapping
+        )

-    processed_results_json = results_to_json(
-        latency_results, throughput_results, serving_results
-    )
+    processed_results_json = results_to_json(latency_results, throughput_results, serving_results)

    # get markdown tables
-    latency_md_table = tabulate(
-        latency_results, headers="keys", tablefmt="pipe", showindex=False
-    )
-    serving_md_table = tabulate(
-        serving_results, headers="keys", tablefmt="pipe", showindex=False
-    )
-    throughput_md_table = tabulate(
-        throughput_results, headers="keys", tablefmt="pipe", showindex=False
-    )
+    latency_md_table = tabulate(latency_results, headers="keys", tablefmt="pipe", showindex=False)
+    serving_md_table = tabulate(serving_results, headers="keys", tablefmt="pipe", showindex=False)
+    throughput_md_table = tabulate(throughput_results, headers="keys", tablefmt="pipe", showindex=False)

    # document the result
    print(output_folder)
--- a/benchmarks/scripts/perf_result_template.md
+++ b/benchmarks/scripts/perf_result_template.md
@@ -10,6 +10,7 @@
 {serving_tests_markdown_table}

 ## Offline tests
+
 ### Latency tests

 - Input length: 32 tokens.
--- a/benchmarks/tests/serving-tests.json
+++ b/benchmarks/tests/serving-tests.json
@@ -18,7 +18,7 @@
    },
    "client_parameters": {
      "model": "Qwen/Qwen2.5-VL-7B-Instruct",
-      "endpoint_type": "openai-chat",
+      "backend": "openai-chat",
      "dataset_name": "hf",
      "hf_split": "train",
      "endpoint": "/v1/chat/completions",
@@ -45,7 +45,7 @@
    },
    "client_parameters": {
      "model": "Qwen/Qwen3-8B",
-      "endpoint_type": "vllm",
+      "backend": "vllm",
      "dataset_name": "sharegpt",
      "dataset_path": "/github/home/.cache/datasets/ShareGPT_V3_unfiltered_cleaned_split.json",
      "num_prompts": 200
@@ -69,7 +69,7 @@
    },
    "client_parameters": {
      "model": "Qwen/Qwen2.5-7B-Instruct",
-      "endpoint_type": "vllm",
+      "backend": "vllm",
      "dataset_name": "sharegpt",
      "dataset_path": "/github/home/.cache/datasets/ShareGPT_V3_unfiltered_cleaned_split.json",
      "num_prompts": 200