xc-llm-ascend

Go to file

sdmyzlp 7bdc606677 Support multistream of shared experts in FusedMoE (#997 )

Contains on #1111 for completeness.

<!--  Thanks for sending a pull request!

BEFORE SUBMITTING, PLEASE READ
https://docs.vllm.ai/en/latest/contributing/overview.html

-->
### What this PR does / why we need it?
Implement multi-stream parallelism for MoE layers with shared experts,
where computation of shared experts will be overlapped with expert token
dispatch and combine. Also, when multi-stream is enabled, weights of
shared experts will be force to replicate across all cards, regardless
of any tensor parallelism configurations, to avoid AllReduce operations.

With the expected overlaping being:
```
| shared gate_up | shared act |              | shared down |
|    dispatch    | routed gate_up, act, down |   combine   |
```

<!--
- Please clarify what changes you are proposing. The purpose of this
section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster
reviews in your PR.

- Please clarify why the changes are needed. For instance, the use case
and bug description.

- Fixes #
-->

### Does this PR introduce _any_ user-facing change?
No.

<!--
Note that it means *any* user-facing change including all aspects such
as API, interface or other behavior changes.
Documentation-only updates are not considered user-facing changes.
-->

### How was this patch tested?
Tested on 1x16 910 node, with tailored 2 layer DSKv2.
<!--
CI passed with new added/existing test.
If it was tested in a way different from regular unit tests, please
clarify how you tested step by step, ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future.
If tests were not added, please describe why they were not added and/or
why it was difficult to add.
-->

---------

Signed-off-by: sdmyzlp <lrwei2@petalmail.com>

2025-06-11 09:18:38 +08:00

.github

Support multistream of shared experts in FusedMoE (#997 )

2025-06-11 09:18:38 +08:00

benchmarks

[CI] Make accuarcy CI and report work (#1078 )

2025-06-10 14:35:44 +08:00

cmake

[core] Support custom ascendc kernels in vllm-ascend (#233 )

2025-04-03 14:52:34 +08:00

csrc

[Performance]: Custom AscendC Kernel of Multi-Step Prepare Input (#814 )

2025-05-20 09:31:30 +08:00

docs

Support multistream of shared experts in FusedMoE (#997 )

2025-06-11 09:18:38 +08:00

examples

[perf]: support dual-batch overlap(dbo) for deepseek (#941 )

2025-06-07 16:46:58 +08:00

tests

Support multistream of shared experts in FusedMoE (#997 )

2025-06-11 09:18:38 +08:00

tools

add workflow to build and release wheel (#775 )

2025-05-26 14:18:26 +08:00

vllm_ascend

Support multistream of shared experts in FusedMoE (#997 )

2025-06-11 09:18:38 +08:00

.gitignore

[Misc] version control by setuptools_scm (#21 )

2025-02-10 09:36:09 +08:00

.readthedocs.yaml

[Doc] Add sphinx build for vllm-ascend (#55 )

2025-02-13 18:44:17 +08:00

CMakeLists.txt

[Performance]: Custom AscendC Kernel of Multi-Step Prepare Input (#814 )

2025-05-20 09:31:30 +08:00

CODE_OF_CONDUCT.md

[Core] Init vllm-ascend (#3 )

2025-02-05 10:53:12 +08:00

collect_env.py

[CI]Add model basic accuracy test(Qwen2.5-0.5B-Instruct) (#460 )

2025-04-17 14:59:56 +08:00

DCO

[Core] Init vllm-ascend (#3 )

2025-02-05 10:53:12 +08:00

Dockerfile

[CI] upgrade to vllm 0.9.0 (#959 )

2025-05-28 21:18:41 +08:00

Dockerfile.openEuler

[CI] upgrade to vllm 0.9.0 (#959 )

2025-05-28 21:18:41 +08:00

format.sh

[CI] Refactor CI (#952 )

2025-05-28 06:31:35 +08:00

LICENSE

Initial commit

2025-01-29 02:44:13 -08:00

mypy.ini

Support multistream of shared experts in FusedMoE (#997 )

2025-06-11 09:18:38 +08:00

packages.txt

[CI/UT][PD Disaggreate] Initialize PD Disaggreate UT (#889 )

2025-05-29 10:17:12 +08:00

pyproject.toml

[Build] Move numba/quart to requirments and update DS baseline and sync graph typo fix (#1121 )

2025-06-08 22:33:37 +08:00

pytest.ini

[CI/UT] Ignore vllm/tests/test_vllm_port.py (#887 )

2025-05-16 18:52:59 +08:00

README.md

[doc] add 0.7.3.post1 release note (#1008 )

2025-05-29 17:38:34 +08:00

README.zh.md

Upgrade CANN version to 8.1.rc1 (#747 )

2025-05-06 05:44:18 +08:00

requirements-dev.txt

[Build] Move numba/quart to requirments and update DS baseline and sync graph typo fix (#1121 )

2025-06-08 22:33:37 +08:00

requirements-lint.txt

[CI] Fix mypy CI (#443 )

2025-04-01 09:25:33 +08:00

requirements.txt

[Build] Move numba/quart to requirments and update DS baseline and sync graph typo fix (#1121 )

2025-06-08 22:33:37 +08:00

setup.py

[Bug fix] fix a typo in setup.py (#762 )

2025-05-06 17:01:26 +08:00

README.md

vLLM Ascend Plugin

English | 中文

Latest News 🔥

[2025/03] We hosted the vLLM Beijing Meetup with vLLM team! Please find the meetup slides here.
[2025/02] vLLM community officially created vllm-project/vllm-ascend repo for running vLLM seamlessly on the Ascend NPU.
[2024/12] We are working with the vLLM community to support [RFC]: Hardware pluggable.

Overview

vLLM Ascend (vllm-ascend) is a community maintained hardware plugin for running vLLM seamlessly on the Ascend NPU.

It is the recommended approach for supporting the Ascend backend within the vLLM community. It adheres to the principles outlined in the [RFC]: Hardware pluggable, providing a hardware-pluggable interface that decouples the integration of the Ascend NPU with vLLM.

By using vLLM Ascend plugin, popular open-source models, including Transformer-like, Mixture-of-Expert, Embedding, Multi-modal LLMs can run seamlessly on the Ascend NPU.

Prerequisites

Hardware: Atlas 800I A2 Inference series, Atlas A2 Training series
OS: Linux
Software:
- Python >= 3.9, < 3.12
- CANN >= 8.1.RC1
- PyTorch >= 2.5.1, torch-npu >= 2.5.1
- vLLM (the same version as vllm-ascend)

Getting Started

Please refer to QuickStart and Installation for more details.

Contributing

See CONTRIBUTING for more details, which is a step-by-step guide to help you set up development environment, build and test.

We welcome and value any contributions and collaborations:

Please let us know if you encounter a bug by filing an issue
Please use User forum for usage questions and help.

Branch

vllm-ascend has main branch and dev branch.

main: main branch，corresponds to the vLLM main branch, and is continuously monitored for quality through Ascend CI.
vX.Y.Z-dev: development branch, created with part of new releases of vLLM. For example, v0.7.3-dev is the dev branch for vLLM v0.7.3 version.

Below is maintained branches:

Branch	Status	Note
main	Maintained	CI commitment for vLLM main branch and vLLM 0.9.x branch
v0.7.1-dev	Unmaintained	Only doc fixed is allowed
v0.7.3-dev	Maintained	CI commitment for vLLM 0.7.3 version

Please refer to Versioning policy for more details.

Weekly Meeting

vLLM Ascend Weekly Meeting: https://tinyurl.com/vllm-ascend-meeting
Wednesday, 15:00 - 16:00 (UTC+8, Convert to your timezone)

License

Apache License 2.0, as found in the LICENSE file.

Languages

Python 66.8%

C++ 31.8%

Shell 1%

CMake 0.2%

Dockerfile 0.1%

Other 0.1%

README.md Unescape Escape

vLLM Ascend Plugin

Overview

Prerequisites

Getting Started

Contributing

Branch

Weekly Meeting

License

README.md