<!-- Thanks for sending a pull request!
BEFORE SUBMITTING, PLEASE READ
https://docs.vllm.ai/en/latest/contributing/overview.html
-->
### What this PR does / why we need it?
Currently, the implementation for MLA V1 pads q, k, v to `head_dim` 256
to conform to early MLA kernel. But the new MLA kernel supports
`head_dim` that can't be devided by 128. Therefore we can remove those
unnecessary paddings to boost the performance
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
<!--
CI passed with new added/existing test.
If it was tested in a way different from regular unit tests, please
clarify how you tested step by step, ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future.
If tests were not added, please describe why they were not added and/or
why it was difficult to add.
-->
Signed-off-by: angazenn <zengyanjia@huawei.com>
Co-authored-by: angazenn <zengyanjia@huawei.com>
### What this PR does / why we need it?
Update attention nz and mla nz modules to improve TPOP 6ms performance
Convert W_UV and W_UK_T to NPU format in mla_v1.py
Convert layer.weight to NPU format in w8a8.py
Signed-off-by: ttanzhiqiang <389825161@qq.com>
Implement save kv cache logic for v1 disaggregated prefill in ascend
scheduler
This PR adds support for saving kv cache in the ascend scheduler, which
is part of the v1 disaggregated prefill design. The load functionality
is not yet implemented.
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
### What this PR does / why we need it?
Revert the default value of enable_chunked_prefill to 'False' in
additional_scheduler_config. In engine v1, enable_chunked_prefill is
forcibly set to True in VllmConfig, which causes it to be perceived as
True in check_and_update_config(). As a result, when the v0 scheduler is
enabled, the chunked prefill feature remains active, leading to the
failure of the v0 scheduler and causing it to fall back to the native v1
scheduling logic.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
CI passed with new added/existing test.
Signed-off-by: rjg-lyh <1318825571@qq.com>
### What this PR does / why we need it?
Fix the bug of #703, where vllm wrong raised the ERROR : Failed to
import vllm_ascend_C:No module named 'vllm_ascend.vllm_ascend_C'. The
format for reporting import vllm_ascend_C failure is unified by warning
("Failed to import vllm_ascend_C:%s", e).
### Does this PR introduce _any_ user-facing change?
No
---------
Signed-off-by: yangpuPKU <604425840@qq.com>
### What this PR does / why we need it?
Update feature support for lora
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
preview
Signed-off-by: paulyu <paulyu0307@gmail.com>
Co-authored-by: paulyu <paulyu0307@gmail.com>
### What this PR does / why we need it?
Add V1Engine LoRA support.
Add LoRA e2e test on single card and multiple cards.
### Does this PR introduce _any_ user-facing change?
support lora for V1
### How was this patch tested?
CI passed with new added test
---------
Signed-off-by: jesse <szxfml@gmail.com>
Signed-off-by: paulyu <paulyu0307@gmail.com>
Signed-off-by: paulyu12 <507435917@qq.com>
Co-authored-by: jesse <szxfml@gmail.com>
Co-authored-by: paulyu <paulyu0307@gmail.com>
### What this PR does / why we need it?
Fix the bugs when run deepseek model in engine v1.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
CI passed with new added/existing test.
---------
Signed-off-by: rjg-lyh <1318825571@qq.com>
<!-- Thanks for sending a pull request!
BEFORE SUBMITTING, PLEASE READ
https://docs.vllm.ai/en/latest/contributing/overview.html
-->
### What this PR does / why we need it?
<!--
- Please clarify what changes you are proposing. The purpose of this
section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster
reviews in your PR.
- Please clarify why the changes are needed. For instance, the use case
and bug description.
- Fixes #
-->
Set div_mode to False to use the ACLNN kernel, which is crucial when
using ACL Graph.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such
as API, interface or other behavior changes.
Documentation-only updates are not considered user-facing changes.
-->
### How was this patch tested?
<!--
CI passed with new added/existing test.
If it was tested in a way different from regular unit tests, please
clarify how you tested step by step, ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future.
If tests were not added, please describe why they were not added and/or
why it was difficult to add.
-->
Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com>
1. update modelslim version to fix deepseek related issues
2. add note for "--quantization ascend"
Signed-off-by: 22dimensions <waitingwind@foxmail.com>
### What this PR does / why we need it?
- According to https://github.com/vllm-project/vllm-ascend/issues/807,
we pull request for customer ascendc kernel of multi-step.
- also a bug we found in multi_step_runner.py is fixed when we use
multi-step on V0 Engine.
### Does this PR introduce _any_ user-facing change?
no user-facing change
### How was this patch tested?
we add Unit Test file and offline inference file to test the custom
ascendc kernel. See test/ops/test_multi_step.py and
examples/offline_multi_step.py
---------
Signed-off-by: wan_danfeng <wonderful199082@126.com>
For online serving, "ascend" quantization method is not a choice
natively, so we need to add "ascend" quantization method to quantization
methods list and the user can enable quantization using "vllm serve
--quantization ascend" command.
---------
Signed-off-by: 22dimensions <waitingwind@foxmail.com>
1. replace deepseek-v2-lite model with more pratical model QwQ 32B
2. fix some incorrect commands
3. replase modelslim version with a more formal tag
Signed-off-by: 22dimensions <waitingwind@foxmail.com>
The [vllm
commit](67da5720d4)
changed the input and rotary position embedding for qwen 2.5 vl which
break CI. This PR fix the CI failure for qwen2.5 vl in quick
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
### What this PR does / why we need it?
this PR fix CI failure broken by vllm.
1. add moe_config for fused_moe
2. adjust the change for kv cache group from vllm. currently vllm-ascend
doesn't support this feature. this is just a quick fix for backward
compatibility
fix: #872
---------
Signed-off-by: MengqingCao <cmq0113@163.com>
<!-- Thanks for sending a pull request!
BEFORE SUBMITTING, PLEASE READ
https://docs.vllm.ai/en/latest/contributing/overview.html
-->
### What this PR does / why we need it?
1. This PR introduces native `all_to_all` communication operator to fix
`allgather` bugs when dp_size > 1. Besides, it adds a naive
implementation of force-load-balance when doing profile runs.
2. The operator `npu_dequant_swiglu_quant` only supports input
hidden_states with dtype `torch.int32`. This tensor occupies space of
`global_bs * seq_len * topk * hidden_size`, which might be very large as
`ep_size` grows. Therefore we need to disable this operator and use
original `swiglu` && `quantize`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
By performing offline inference:
---------
Signed-off-by: angazenn <zengyanjia@huawei.com>
Co-authored-by: angazenn <zengyanjia@huawei.com>
make sure pytorch infer_schema check is patched before some case which
using fused moe ops:
1. model register
2. quantization loading
3. fused moe ut
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
### What this PR does / why we need it?
Add basic CI for PD disaggregation, and enable it when schedule and
label with `module:pd`
- Updated `.github/actionlint.yaml` to add a new self-hosted runner
configuration: `linux-arm64-npu-static-8`.
- Introduced a new GitHub Actions workflow
`.github/workflows/vllm_ascend_test_pd.yaml` for PD disaggregation
testing:
- Scheduled to run daily at 23:00 UTC and triggered by pull request
label `module:pd`.
- Added steps for baisci installation and other steps will add in
followup PR
Related: https://github.com/vllm-project/vllm-ascend/issues/841
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- CI passed
- No trigger by default
<img width="847" alt="image"
src="https://github.com/user-attachments/assets/23aa128f-526d-447f-91c8-8ebf6be8400f"
/>
- Trigger only if we tag with pd
<img width="930" alt="image"
src="https://github.com/user-attachments/assets/aef1caca-2029-48e8-a6e6-860136adcd37"
/>
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
### What this PR does / why we need it?
Add quickstart doctest CI
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
- CI passed
- Run `/vllm-ascend/tests/e2e/run_doctests.sh`
Related: https://github.com/vllm-project/vllm-ascend/issues/725
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
Patch torch.library.infer_schema for torch 2.5 backward compatibility
- Introduced a new module `patch_utils` under
`vllm_ascend/patch/worker/patch_common/`.
- Added a function `ascend_direct_register_custom_op` to handle custom
operator registration with backward compatibility for PyTorch < 2.7
(such as torch 2.5.1).
- Implemented type conversion logic for annotations to ensure
compatibility across different PyTorch versions.
- Registered the function `ascend_direct_register_custom_op` to
`utils.direct_register_custom_op`.
- Updated `__init__.py` to include `patch_utils` as the first import.
- Ensured `patch_utils` is available for use in other patch files and
skipped isort checks for `patch_utils` import.
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
### What this PR does / why we need it?
moe support for llama4 and mllama4 in vllm-ascend
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
start sever:
python -m vllm.entrypoints.openai.api_server --model
/data/nfs/benchmark/tokenizer/Llama-4-Scout-17B-16E-Instruct \
--max-num-seqs=256 \
--max-model-len=8192 \
--tensor-parallel-size=8 \
--block-size=128 \
--dtype bfloat16 \
--host=0.0.0.0 \
--port=8000 \
--gpu-memory-utilization=0.9 \
--trust-remote-code
client:
python online_server.py --model-path
/data/nfs/benchmark/tokenizer/Llama-4-Scout-17B-16E-Instruct
--image-path /data/nfs/w60040464/cherry_blossom.jpg --docker-ip
7.242.108.253 --served-port 8000 --text "what is the content of this
image?"
result:
{'id': 'chatcmpl-2b709a5d2e1a4017991ec4ba8248686a', 'object':
'chat.completion', 'created': 1747056823, 'model':
'/data/nfs/benchmark/tokenizer/Llama-4-Scout-17B-16E-Instruct',
'choices': [{'index': 0, 'message': {'role': 'assistant',
'reasoning_content': None, 'content': 'The image depicts a tower, likely
Tokyo Skytree, framed by branches of a cherry blossom tree. The tower is
white and has a distinctive shape, with a large sphere at the top and a
long, thin spire extending from it. The branches of the cherry blossom
tree are in the foreground, with pink flowers blooming on them. The
background is a clear blue sky.\n\n**Key Features:**\n\n* **Tower:**
White, spherical shape at the top, long thin spire\n', 'tool_calls':
[]}, 'logprobs': None, 'finish_reason': 'length', 'stop_reason': None}],
'usage': {'prompt_tokens': 2340, 'total_tokens': 2440,
'completion_tokens': 100, 'prompt_tokens_details': None},
'prompt_logprobs': None}
Signed-off-by: chenxu <chenxu68@huawei.com>
Co-authored-by: chenxu <chenxu68@huawei.com>
Co-authored-by: evian <eviantai@u.nus.edu>
### What this PR does / why we need it?
Fix the method of importing environment variables in DeepSeek model to
support successful compilation via aclgraph.
Signed-off-by: rjg-lyh <1318825571@qq.com>
1. Fix format check error to make format.sh work
2. Add codespell check CI
3. Add the missing required package for vllm-ascend.
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
This PR fixes two bugs in AscendScheduler:
1. When running with high concurrency, the length of running queue may
exceed the limit of max_num_seqs
2. When some requests are prempted and recomputing is activated, the
logic of computing new tokens is wrong.
Signed-off-by: whx-sjtu <2952154980@qq.com>
### What this PR does / why we need it?
Add padding for ACL Graph and refactor graph batch size adjustments to
utils.py
---------
Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com>
### What this PR does / why we need it?
1. update cann version to 8.1.0 for multimodal
2. fix huggingface-hub version to adapt to qwen3
3. change Qwen3-8B to Qwen-8B-Base,
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
### What this PR does / why we need it?
- Add qwen2.5-7b test
- Optimize the documentation to be more developer-friendly
Signed-off-by: xuedinge233 <damow890@gmail.com>
Co-authored-by: xuedinge233 <damow890@gmail.com>
### What this PR does / why we need it?
#### 1. fix spec ut in vllm-ascend main and vllm main
As https://github.com/vllm-project/vllm-ascend/pull/694 and
https://github.com/vllm-project/vllm-ascend/pull/749 verify, Now,
vllm-ascend main and vllm 0.8.5, spec UT is happy, but vllm-ascend main
and vllm main, CI is fail.
I found the reason is a triton bug
https://github.com/triton-lang/triton/issues/2266, but i I didn't figure
it out that why the bug did not effect vllm-ascend main and vllm 0.8.5,
maybe the usage of triton have changed when vllm 0.8.5 to latest main
As the bug describe, I changed the minimum block_size in UT from 8 to
16, and the modification is verified locally to be effective.
#### 2. modify some case skip form.
I modified some commented out cases to skipif form, which is more
standardized.
### Does this PR introduce _any_ user-facing change?
None
### How was this patch tested?
CI
Signed-off-by: mengwei805 <mengwei25@huawei.com>
### What this PR does / why we need it?
This PR aims to fix nightly ci
[broken](https://github.com/vllm-project/vllm-ascend/actions/runs/14848150987)
We have a workflow containing multiple triggers:
- push events (to the default branch)
- pull requests (against the default branch)
- scheduled events
Our paths-filter action works great for the first two use-cases,
detecting the context and base to compare against. However, it fails for
scheduled events giving the error `This action requires 'base' input to
be configured or 'repository.default_branch' to be set in the event
payload.`
For the scheduling trigger event, we choose to skip this filter
because we don't need its results:
```
- name: Check for changes in Speculative Decode
if: github.event_name != 'schedule'
```
Signed-off-by: wangli <wangli858794774@gmail.com>
### What this PR does / why we need it?
- Move Release Compatibility Matrix to top
- Remove v0.7.x rc info because v0.7.3 final release alread published
- Rename vllm-ascend to vLLM Ascend
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Preview
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
### What this PR does / why we need it?
Support the features of prefix cache and chunked prefill in v0/v1.
---------
Signed-off-by: rjg-lyh <1318825571@qq.com>
### What this PR does / why we need it?
In the w8a8 quantization code of `fused_experts`, the output of almost
every operator is assigned a new variable name. If we want to save NPU
memory, we manually `del` these variables to end their lifecycle, which
fills the code with `del` statements and looks inelegant.
Therefore, I plan to names the output of most operators as
`hidden_states`, thereby ending the lifecycle of the previous
`hidden_states`.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Signed-off-by: ApsarasX <apsarax@outlook.com>
This change ensures proper functionality for longer sequences by
correctly invoking the _set_cos_sin_cache method with self as the first
argument.
For example, with DeepSeek R1, if this change isn't made, the program
will crash when the input sequence exceeds 4096.
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
### What this PR does / why we need it?
add notes for OOM in faqs.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
CI passed
---------
Signed-off-by: zzzzwwjj <1183291235@qq.com>
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
Co-authored-by: Yikun Jiang <yikunkero@gmail.com>
<!-- Thanks for sending a pull request!
BEFORE SUBMITTING, PLEASE READ
https://docs.vllm.ai/en/latest/contributing/overview.html
-->
### What this PR does / why we need it?
This PR add new function of : aclgraph_batch_size can dynamic adjust to
different model; before this PR, the aclgraph_batch_sizes given from
vllm to vllm-ascend always too large, and that may result in ERROR while
running on different, with the information: "The resources are
insufficient".
Now, with this PR, the code can dynamic adjust aclgraph_batch_sizes
depend on the model hidden_layer_nums and parallel config, for example:
a. for Qwen2.5-7B, the aclgraph_batch_size length is 33 total;
b. for Qwen2.5-72B, the aclgraph_batch_size length is 11 total;
Signed-off-by: chris668899 <15105191595@126.com>
<!-- Thanks for sending a pull request!
BEFORE SUBMITTING, PLEASE READ
https://docs.vllm.ai/en/latest/contributing/overview.html
-->
### What this PR does / why we need it?
<!--
- Please clarify what changes you are proposing. The purpose of this
section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster
reviews in your PR.
- Please clarify why the changes are needed. For instance, the use case
and bug description.
- Fixes #
-->
Fix output tensor shape in vanilla_chunked_prefill function.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such
as API, interface or other behavior changes.
Documentation-only updates are not considered user-facing changes.
-->
None.
### How was this patch tested?
<!--
CI passed with new added/existing test.
If it was tested in a way different from regular unit tests, please
clarify how you tested step by step, ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future.
If tests were not added, please describe why they were not added and/or
why it was difficult to add.
-->
Run offline inference on DeepSeek models.
---------
Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com>
### What this PR does / why we need it?
Add 0.8.5rc1 release note and bump vllm version to v0.8.5.post1
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
CI passed
---------
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
### What this PR does / why we need it?
The root cause of the bug is that numerical computations involving NaNs
cannot eliminate them. We addressed it by using `masked_fill_` to
eliminate NaNs while avoiding memory-wasting `torch.where` approach.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
This patch was tested with vllm v0.8.5 and vllm-ascend master. I run
deepseek_v3 model with offline inference scripts
(examples/dp_offline/run_dp.sh & data_parallel.py).
Signed-off-by: linfeng-yuan <1102311262@qq.com>
### What this PR does / why we need it?
Fix function name typo, make `mask_fill_` to `masked_fill_`
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
CI passed
Signed-off-by: ApsarasX <apsarax@outlook.com>
### What this PR does / why we need it?
- Revert "Re-patch TritonPlaceholder on main to make CI happy (#753)"
because upstream main CI already merged:
https://github.com/vllm-project/vllm/pull/17446
- Keep 0.8.5.post1 compatible
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
CI passed
---------
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
