xc-llm-ascend

Go to file

drslark 384d84c7ef [Bugfix] Avoided a bug of drafter when dp and sp are enabled (#6226 )

### What this PR does / why we need it?

Avoided a bug of drafter when `dp` and `sp` are enabled.

Specifically, disable `sp` when drafter is dense.

### Does this PR introduce _any_ user-facing change?

N/A

### How was this patch tested?

An aisbench test:

```shell
python3 aisbench_test.py --input_len 3500 --output_len 1000 --data_num 100 --concurrency 320 --request_rate 8
```

The result is okay.

```text
[2026-01-24 22:38:20,256] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Calculate global interval offsets time: 0.5922 s
01/24 22:38:20 - AISBench - INFO - Process 0 using precomputed sleep offsets with 100 requests
Process-0 pid:220279: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [09:40<00:00,  5.81s/it]
Pid:      220279 | Post:        100 | Received:    100 | Failed:        0 | Post Time:12.51s | Receive Time:580.92s: 
Encoding output text...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:01<00:00, 93.75it/s]
01/24 22:48:02 - AISBench - INFO - Start converting origin data to detailed data ...
01/24 22:48:02 - AISBench - INFO - Finish converting origin data to detailed data█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:01<00:00, 95.08it/s]
01/24 22:48:02 - AISBench - INFO - Added 'Actual RPS: After Excluding Anomalies' to group 'Time - RPS: ' in legend explanation table
01/24 22:48:02 - AISBench - INFO - Successfully merged chart into position (1, 1)
01/24 22:48:02 - AISBench - INFO - RPS distribution charts saved to outputs/default/20260124_223809/performances/vllm-api-stream-chat/gsm8kdataset_rps_distribution_plot_with_actual_rps.html
01/24 22:48:02 - AISBench - INFO - Updated chart with actual RPS saved to outputs/default/20260124_223809/performances/vllm-api-stream-chat/gsm8kdataset_rps_distribution_plot_with_actual_rps.html
[2026-01-24 22:48:02,557] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_perf_inferencer] [INFO] Start extracting pref datas ...
[2026-01-24 22:48:02,558] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_perf_inferencer] [INFO] Finish extracting pref datas!
[2026-01-24 22:48:02,558] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_perf_inferencer] [INFO] Dumping detail perf data ...
Dumping data to h5: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 75.31it/s]
[2026-01-24 22:48:02,588] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_perf_inferencer] [INFO] Dump detail perf data cost: 0.02995561994612217(s)
[2026-01-24 22:48:02,588] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_perf_inferencer] [INFO] Performance task finished, results saved in outputs/default/20260124_223809/performances/vllm-api-stream-chat
01/24 22:48:02 - AISBench - INFO - time elapsed: 586.32s
Running tasks: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [09:55<00:00, 595.91s/it]
01/24 22:48:05 - AISBench - INFO - Performance evaluation tasks completed.
01/24 22:48:05 - AISBench - INFO - Loading detail perf data of model='vllm-api-stream-chat' dataset='gsm8kdataset' ...
01/24 22:48:05 - AISBench - INFO - Starting request timeline processing...
01/24 22:48:05 - AISBench - INFO - Data preprocessing completed in 0.0004s
01/24 22:48:05 - AISBench - INFO - Generating timeline traces for 100 requests...
01/24 22:48:05 - AISBench - INFO - Generated timeline trace chunks in 0.0441s
01/24 22:48:05 - AISBench - INFO - Generating concurrency traces...
01/24 22:48:05 - AISBench - INFO - Generated concurrency trace chunks in 0.0011s
01/24 22:48:05 - AISBench - INFO - Creating figure layout...
01/24 22:48:05 - AISBench - INFO - Figure layout created in 0.0504s
01/24 22:48:05 - AISBench - INFO - Writing to outputs/default/20260124_223809/performances/vllm-api-stream-chat/gsm8kdataset_plot.html...
01/24 22:48:05 - AISBench - INFO - HTML written in 0.0181s
01/24 22:48:05 - AISBench - INFO - Completed! Total execution time: 0.1148s
01/24 22:48:05 - AISBench - INFO - The gsm8kdataset_plot has been saved in outputs/default/20260124_223809/performances/vllm-api-stream-chat/gsm8kdataset_plot.html
01/24 22:48:05 - AISBench - INFO - Converting perf results of stage ...
01/24 22:48:05 - AISBench - INFO - Finish Converting!
01/24 22:48:05 - AISBench - INFO - Start calculating metrics ...
01/24 22:48:05 - AISBench - INFO - Start calculating common metrics ...
01/24 22:48:05 - AISBench - INFO - Start calculating add units ...
01/24 22:48:05 - AISBench - INFO - Finish calculating perf data!
01/24 22:48:05 - AISBench - INFO - Summarizing performance results...
01/24 22:48:05 - AISBench - INFO - Performance Results of task: vllm-api-stream-chat/gsm8kdataset: 
╒══════════════════════════╤═════════╤════════════════╤════════════════╤════════════════╤════════════════╤════════════════╤════════════════╤════════════════╤═════╕
│ Performance Parameters   │ Stage   │ Average        │ Min            │ Max            │ Median         │ P75            │ P90            │ P99            │  N  │
╞══════════════════════════╪═════════╪════════════════╪════════════════╪════════════════╪════════════════╪════════════════╪════════════════╪════════════════╪═════╡
│ E2EL                     │ total   │ 300806.1781 ms │ 189326.0489 ms │ 568345.5121 ms │ 380629.6785 ms │ 384208.3527 ms │ 385363.7709 ms │ 566871.7684 ms │ 100 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ TTFT                     │ total   │ 107441.2231 ms │ 343.8054 ms    │ 378132.3979 ms │ 188817.4877 ms │ 190985.8451 ms │ 192547.6847 ms │ 378008.356 ms  │ 100 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ TPOT                     │ total   │ 193.5585 ms    │ 185.1008 ms    │ 197.262 ms     │ 193.8146 ms    │ 195.0803 ms    │ 196.0323 ms    │ 196.9688 ms    │ 100 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ ITL                      │ total   │ 194.2067 ms    │ 0.0108 ms      │ 2782.7124 ms   │ 184.9998 ms    │ 194.2631 ms    │ 221.2895 ms    │ 304.363 ms     │ 100 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ InputTokens              │ total   │ 3506.86        │ 3431.0         │ 3508.0         │ 3508.0         │ 3508.0         │ 3508.0         │ 3508.0         │ 100 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ OutputTokens             │ total   │ 1000.0         │ 1000.0         │ 1000.0         │ 1000.0         │ 1000.0         │ 1000.0         │ 1000.0         │ 100 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────┤
│ OutputTokenThroughput    │ total   │ 3.7745 token/s │ 1.7595 token/s │ 5.2819 token/s │ 2.6272 token/s │ 5.1028 token/s │ 5.1502 token/s │ 5.2754 token/s │ 100 │
╘══════════════════════════╧═════════╧════════════════╧════════════════╧════════════════╧════════════════╧════════════════╧════════════════╧════════════════╧═════╛
╒══════════════════════════╤═════════╤══════════════════╕
│ Common Metric            │ Stage   │ Value            │
╞══════════════════════════╪═════════╪══════════════════╡
│ Benchmark Duration       │ total   │ 580456.2704 ms   │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Requests           │ total   │ 100              │
├──────────────────────────┼─────────┼──────────────────┤
│ Failed Requests          │ total   │ 0                │
├──────────────────────────┼─────────┼──────────────────┤
│ Success Requests         │ total   │ 100              │
├──────────────────────────┼─────────┼──────────────────┤
│ Concurrency              │ total   │ 51.8224          │
├──────────────────────────┼─────────┼──────────────────┤
│ Max Concurrency          │ total   │ 320              │
├──────────────────────────┼─────────┼──────────────────┤
│ Request Throughput       │ total   │ 0.1723 req/s     │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Input Tokens       │ total   │ 350686           │
├──────────────────────────┼─────────┼──────────────────┤
│ Prefill Token Throughput │ total   │ 32.6398 token/s  │
├──────────────────────────┼─────────┼──────────────────┤
│ Total generated tokens   │ total   │ 100000           │
├──────────────────────────┼─────────┼──────────────────┤
│ Input Token Throughput   │ total   │ 604.1558 token/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Output Token Throughput  │ total   │ 172.2783 token/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Token Throughput   │ total   │ 776.434 token/s  │
╘══════════════════════════╧═════════╧══════════════════╛
01/24 22:48:05 - AISBench - INFO - Performance Result files locate in outputs/default/20260124_223809/performances/vllm-api-stream-chat.
```
- vLLM version: v0.14.0
- vLLM main:
d68209402d

Signed-off-by: drslark <slarksblood@qq.com>

2026-01-25 17:45:29 +08:00

.gemini

Configure Gemini (#2298 )

2025-08-11 22:21:29 +08:00

.github

[310P]: refactoring for 310p kvcache and some ops class (#6117 )

2026-01-24 20:34:29 +08:00

benchmarks

[Lint]Style: reformat markdown files via markdownlint (#5884 )

2026-01-15 09:06:01 +08:00

cmake

[Misc] Cleanup useless file and code (#5877 )

2026-01-15 20:32:47 +08:00

csrc

[BugFix]bug fix for dispatch_ffn_combine (#6156 )

2026-01-23 21:14:18 +08:00

docs

[Fusion] change fusion env variable (#6201 )

2026-01-24 22:49:33 +08:00

examples

[Lint]Style: reformat markdown files via markdownlint (#5884 )

2026-01-15 09:06:01 +08:00

tests

Revert "[Refactor] Unify full-graph parameter update logic (#6041 )" (#6227 )

2026-01-25 15:25:38 +08:00

tools

[Doc] Add release note for 0.13.0rc2 (#6207 )

2026-01-24 12:51:47 +08:00

vllm_ascend

[Bugfix] Avoided a bug of drafter when dp and sp are enabled (#6226 )

2026-01-25 17:45:29 +08:00

.gitignore

[Misc] Add cann custom ops to .gitignore (#4670 )

2025-12-03 18:29:10 +08:00

.gitmodules

add dispatch_gmm_combine kernel (#3532 )

2025-12-04 23:00:59 +08:00

.markdownlint.yaml

[Lint]Style: reformat markdown files via markdownlint (#5884 )

2026-01-15 09:06:01 +08:00

.pre-commit-config.yaml

[CI] optimize lint term (#5986 )

2026-01-22 15:46:59 +08:00

.readthedocs.yaml

[Doc] Add sphinx build for vllm-ascend (#55 )

2025-02-13 18:44:17 +08:00

CMakeLists.txt

[Main] Upgrade PTA to 2.9.0 (#6112 )

2026-01-22 17:59:06 +08:00

CODE_OF_CONDUCT.md

[1/2/N] Enable pymarkdown and python __init__ for lint system (#2011 )

2025-07-25 22:16:10 +08:00

codecov.yml

ut: add ci guard for ut coverage (#2317 )

2025-08-12 08:05:01 +08:00

collect_env.py

[Lint]Style: Convert root, benchmarks, tools and docs to ruff format (#5843 )

2026-01-13 15:29:34 +08:00

CONTRIBUTING.md

[Doc] Update doc url link (#5781 )

2026-01-12 11:21:31 +08:00

DCO

[Core] Init vllm-ascend (#3 )

2025-02-05 10:53:12 +08:00

Dockerfile

[Misc] Revert "[Misc] Bump mooncake version to v0.3.8.post1 (#6110 )" (#6164 )

2026-01-23 09:53:32 +08:00

Dockerfile.310p

[Image] fix 310p image build (#6228 )

2026-01-25 16:07:13 +08:00

Dockerfile.310p.openEuler

[Image] fix 310p image build (#6228 )

2026-01-25 16:07:13 +08:00

Dockerfile.a3

[Misc] Revert "[Misc] Bump mooncake version to v0.3.8.post1 (#6110 )" (#6164 )

2026-01-23 09:53:32 +08:00

Dockerfile.a3.openEuler

[Misc] Revert "[Misc] Bump mooncake version to v0.3.8.post1 (#6110 )" (#6164 )

2026-01-23 09:53:32 +08:00

Dockerfile.openEuler

[Misc] Revert "[Misc] Bump mooncake version to v0.3.8.post1 (#6110 )" (#6164 )

2026-01-23 09:53:32 +08:00

format.sh

[Doc] Update doc url link (#5781 )

2026-01-12 11:21:31 +08:00

LICENSE

Initial commit

2025-01-29 02:44:13 -08:00

mypy.ini

[Lint]Style: Convert vllm-ascend/ to ruff format(Batch #5 ) (#5996 )

2026-01-24 22:45:38 +08:00

packages.txt

[CI/UT][PD Disaggreate] Initialize PD Disaggreate UT (#889 )

2025-05-29 10:17:12 +08:00

pyproject.toml

[Lint]Style: Convert vllm-ascend/ to ruff format(Batch #5 ) (#5996 )

2026-01-24 22:45:38 +08:00

README.md

[Doc] Improved English grammar and integrated the DeepWiki badge for Ask AI (#6216 )

2026-01-24 20:11:18 +08:00

README.zh.md

[Main] Upgrade PTA to 2.9.0 (#6112 )

2026-01-22 17:59:06 +08:00

requirements-dev.txt

[1/N][Feat] Xlite Qwen3 MoE Support (#5951 )

2026-01-21 09:26:03 +08:00

requirements-lint.txt

[Test] Remove VLLM_USE_V1 in example and tests (#1733 )

2025-07-15 12:49:57 +08:00

requirements.txt

[Bugfix] Fix structured outputs errors: TypeError: apply_token_bitmask_inplace_cpu() (#6151 )

2026-01-23 09:52:55 +08:00

setup.py

[Lint]Style: Convert root, benchmarks, tools and docs to ruff format (#5843 )

2026-01-13 15:29:34 +08:00

typos.toml

[OPS] add bmm_transpose ops (#3990 )

2025-12-01 09:09:51 +08:00

README.md

vLLM Ascend Plugin

English | 中文

Latest News 🔥

[2025/12] We released the new official version v0.11.0! Please follow the official guide to start using vLLM Ascend Plugin on Ascend.
[2025/09] We released the new official version v0.9.1! Please follow the official guide to start deploying large-scale Expert Parallelism (EP) on Ascend.
[2025/08] We hosted the vLLM Beijing Meetup with vLLM and Tencent! Please find the meetup slides here.
[2025/06] User stories page is now live! It kicks off with LLaMA-Factory/verl/TRL/GPUStack to demonstrate how vLLM Ascend‌ assists Ascend users in enhancing their experience across fine-tuning, evaluation, reinforcement learning (RL), and deployment scenarios.
[2025/06] Contributors page is now live! All contributions deserve to be recorded, thanks for all contributors.
[2025/05] We've released the first official version v0.7.3! We collaborated with the vLLM community to publish a blog post sharing our practice: Introducing vLLM Hardware Plugin, Best Practice from Ascend NPU.
[2025/03] We hosted the vLLM Beijing Meetup with vLLM team! Please find the meetup slides here.
[2025/02] vLLM community officially created vllm-project/vllm-ascend repo for running vLLM seamlessly on the Ascend NPU.
[2024/12] We are working with the vLLM community to support [RFC]: Hardware pluggable.

Overview

vLLM Ascend (vllm-ascend) is a community maintained hardware plugin for running vLLM seamlessly on the Ascend NPU.

It is the recommended approach for supporting the Ascend backend within the vLLM community. It adheres to the principles outlined in the [RFC]: Hardware pluggable, providing a hardware-pluggable interface that decouples the integration of the Ascend NPU with vLLM.

By using vLLM Ascend plugin, popular open-source models, including Transformer-like, Mixture-of-Experts (MoE), Embedding, Multi-modal LLMs can run seamlessly on the Ascend NPU.

Prerequisites

Hardware: Atlas 800I A2 Inference series, Atlas A2 Training series, Atlas 800I A3 Inference series, Atlas A3 Training series, Atlas 300I Duo (Experimental)
OS: Linux
Software:
- Python >= 3.10, < 3.12
- CANN == 8.5.0 (Ascend HDK version refers to here)
- PyTorch == 2.9.0, torch-npu == 2.9.0
- vLLM (the same version as vllm-ascend)

Getting Started

Please use the following recommended versions to get started quickly:

Version	Release type	Doc
v0.13.0rc1	Latest release candidate	See QuickStart and Installation for more details
v0.11.0	Latest stable version	See QuickStart and Installation for more details

Contributing

See CONTRIBUTING for more details, which is a step-by-step guide to help you set up the development environment, build and test.

We welcome and value any contributions and collaborations:

Please let us know if you encounter a bug by filing an issue
Please use User forum for usage questions and help.

Branch

vllm-ascend has a main branch and a dev branch.

main: main branch, corresponds to the vLLM main branch, and is continuously monitored for quality through Ascend CI.
releases/vX.Y.Z: development branch, created alongside new releases of vLLM. For example, releases/v0.13.0 is the dev branch for vLLM v0.13.0 version.

Below are the maintained branches:

Branch	Status	Note
main	Maintained	CI commitment for vLLM main branch and vLLM v0.13.0 tag
v0.7.1-dev	Unmaintained	Only doc fixes are allowed
v0.7.3-dev	Maintained	CI commitment for vLLM 0.7.3 version, only bug fixes are allowed, and no new release tags anymore.
v0.9.1-dev	Maintained	CI commitment for vLLM 0.9.1 version
v0.11.0-dev	Maintained	CI commitment for vLLM 0.11.0 version
releases/v0.13.0	Maintained	CI commitment for vLLM 0.13.0 version
rfc/feature-name	Maintained	Feature branches for collaboration

Please refer to Versioning policy for more details.

Weekly Meeting

vLLM Ascend Weekly Meeting: https://tinyurl.com/vllm-ascend-meeting
Wednesday, 15:00 - 16:00 (UTC+8, Convert to your timezone)

License

Apache License 2.0, as found in the LICENSE file.

Languages

C++ 51.8%

Python 45.8%

CMake 1.1%

Shell 0.7%

C 0.5%

Other 0.1%

README.md Unescape Escape

vLLM Ascend Plugin

Overview

Prerequisites

Getting Started

Contributing

Branch

Weekly Meeting

License

README.md