Shanshan Shen
8a91e6e59c
[Misc][V0 Deprecation] Remove V0 Related Custom Ops ( #1871 )
...
### What this PR does / why we need it?
This PR is a part of
https://github.com/vllm-project/vllm-ascend/issues/1620 .
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/ca4eb82bcba97f4fb0a377287ff4d36e19c3d33e
---------
Signed-off-by: shen-shanshan <467638484@qq.com >
2025-07-18 23:06:03 +08:00
li chaoran
3e39d7234c
[CI] Switching to infra cache server to reduce network pressure ( #1792 )
...
### What this PR does / why we need it?
This PR introduce the infra cache server to speed up apt/pip package
installation
### Does this PR introduce _any_ user-facing change?
None
### How was this patch tested?
Tested locally, with this config, the network bandwith reduce from 100%
to 5% usage when a new PR was submitted.
<img width="807" height="334" alt="image"
src="https://github.com/user-attachments/assets/16f03bce-4531-4c71-ab6e-8308dc2c022c "
/>
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/8dfb45ca3379b3a789ec529af4bf725daa07f10d
---------
Signed-off-by: mywaaagh_admin <pkwarcraft@gmail.com >
2025-07-18 18:39:25 +08:00
Shanshan Shen
d08ff304cd
[Misc][V0 Deprecation] Remove V0 Attention ( #1835 )
...
### What this PR does / why we need it?
This PR is a part of
https://github.com/vllm-project/vllm-ascend/issues/1620 .
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/8dfb45ca3379b3a789ec529af4bf725daa07f10d
Signed-off-by: shen-shanshan <467638484@qq.com >
2025-07-18 14:10:13 +08:00
xudongLi-cmss
33ef5dc813
add unit test for func wrapper ( #1863 )
...
### What this PR does / why we need it?
test func wrapper file
### Does this PR introduce _any_ user-facing change?
N/A
### How was this patch tested?
CI passed with new added test.
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/8dfb45ca3379b3a789ec529af4bf725daa07f10d
Signed-off-by: lixudong <lixudong@cmss.chinamobile.com >
2025-07-18 11:05:17 +08:00
Li Wang
f9dfde02fd
[Bugfix] Fix broken CI ( #1848 )
...
### What this PR does / why we need it?
- Fix broken commit by
[#20927 ](https://github.com/vllm-project/vllm/pull/20927 )
- Fix broken commit by
[#20466 ](https://github.com/vllm-project/vllm/pull/20466 )
- TODO: more fully adapt to the upstream reconstruction, let's first
make CI happy
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/11dfdf21bf3ab057986a7f0053704dfcfee27a4e
---------
Signed-off-by: wangli <wangli858794774@gmail.com >
2025-07-17 20:10:12 +08:00
Zhu Yi Lin
538dd357e6
Add graph mode and improve on multi_npu_moge.md ( #1849 )
...
### What this PR does / why we need it?
Add graph mode and improve on multi_npu_moge.md
### Does this PR introduce _any_ user-facing change?
yes
### How was this patch tested?
CI passed with new existing test.
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/5a7fb3ab9ee4bd8174a5100595326f6f40024cdc
Signed-off-by: GDzhu01 <809721801@qq.com >
2025-07-17 17:53:37 +08:00
Shanshan Shen
aeb5aa8b88
[Misc][V0 Deprecation] Add __main__ guard to all offline examples ( #1837 )
...
### What this PR does / why we need it?
Add `__main__` guard to all offline examples.
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/76b494444fd864ffc53a623420668d1865c804b9
---------
Signed-off-by: shen-shanshan <467638484@qq.com >
2025-07-17 14:13:30 +08:00
Shanshan Shen
19e37cd379
[Misc] Add fusion_result.json to .gitignore ( #1836 )
...
### What this PR does / why we need it?
Add `fusion_result.json` to `.gitignore`.
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/72ad2735823e23b4e1cc79b7c73c3a5f3c093ab0
---------
Signed-off-by: shen-shanshan <467638484@qq.com >
2025-07-17 11:54:49 +08:00
Icey
875a920d4a
[Platform] Add support for Altlas A3 series ( #1794 )
...
### What this PR does / why we need it?
Add support for Ascend A3 and remove latest tag
### Does this PR introduce _any_ user-facing change?
User can run vLLM on Altlas A3 series
### How was this patch tested?
CI passed with:
- remove latest tag test:
https://github.com/wxsIcey/wxs-vllm-ascend/actions/runs/16267635040/job/45926924765
- E2E image build for A3
- CI test on A3 with e2e test and longterm test
- Unit test missing because need a real A3 hardware to have a test
Closes: https://github.com/vllm-project/vllm-ascend/issues/1696
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/d0dc4cfca48c2734da18ec42d6bba1346cbfc400
---------
Signed-off-by: Icey <1790571317@qq.com >
2025-07-17 11:13:02 +08:00
wangxiyuan
ef99fe1c54
[Test] Clean up duplicate test for ascend scheduler ( #1819 )
...
There are some duplicate tests for ascend scheduler. This PR remove them
to make the test clear.
After this PR. the singlecard e2e cost time is reduced from 47min to
46min.
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/1eb2b9c10205b68658dede9dac73390706ef2e05
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-16 17:57:48 +08:00
Shanshan Shen
c66b0827a7
[Misc][V0 Deprecation] Remove Pooling Model Runner ( #1824 )
...
### What this PR does / why we need it?
Remove pooling model runner.
This PR is a part of
https://github.com/vllm-project/vllm-ascend/issues/1620 .
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/d31a64712489d7c079fe48515c7ddd8a60bc0e71
Signed-off-by: shen-shanshan <467638484@qq.com >
2025-07-16 17:48:21 +08:00
Yikun Jiang
ba7e934b21
Remove redundant empty lines in commit msg ( #1814 )
...
### What this PR does / why we need it?
Remove redundant empty lines in commit msg
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
test locally: https://github.com/Yikun/vllm-ascend/pull/48
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/d0dc4cfca48c2734da18ec42d6bba1346cbfc400
Signed-off-by: Yikun Jiang <yikunkero@gmail.com >
2025-07-16 16:50:44 +08:00
Shanshan Shen
06655002c5
[Misc][V0 Deprecation] Remove V0 Worker ( #1821 )
...
### What this PR does / why we need it?
Remove V0 worker.
This PR is a part of
https://github.com/vllm-project/vllm-ascend/issues/1620 .
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/6cbc4d4bea2f0169846accbecead8f0fccebd761
Signed-off-by: shen-shanshan <467638484@qq.com >
2025-07-16 14:07:17 +08:00
Shanshan Shen
b005def0a5
[Misc][V0 Deprecation] Remove Multi-Step Model Runner ( #1820 )
...
### What this PR does / why we need it?
Remove multi-step model runner.
This PR is a part of
https://github.com/vllm-project/vllm-ascend/issues/1620 .
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/34cda778a091d4e1fd204cfde4a0f5e2b5616ac2
Signed-off-by: shen-shanshan <467638484@qq.com >
2025-07-16 14:06:49 +08:00
Shanshan Shen
f9e2e9bb31
[Misc][V0 Deprecation] Remove Draft Model Runner Used for V0 Spec Decode ( #1810 )
...
### What this PR does / why we need it?
Remove draft model runner used for V0 spec decode.
This PR is a part of
https://github.com/vllm-project/vllm-ascend/issues/1620 .
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/34cda778a091d4e1fd204cfde4a0f5e2b5616ac2
---------
Signed-off-by: shen-shanshan <467638484@qq.com >
2025-07-16 10:51:23 +08:00
Shanshan Shen
f96100fad5
[Misc][V0 Deprecation] Remove V0 related codes of test, example, platform ( #1805 )
...
### What this PR does / why we need it?
Remove V0 related codes of test, example, platform.
This PR is a part of
https://github.com/vllm-project/vllm-ascend/issues/1620 .
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/235bfd5dfe0975e42b115cfb910e73eff5c670d8
---------
Signed-off-by: shen-shanshan <467638484@qq.com >
2025-07-15 19:58:55 +08:00
Shanshan Shen
a929699e98
[Misc][V0 Deprecation] Remove multi-step worker ( #1809 )
...
### What this PR does / why we need it?
Remove multi-step worker
This PR is a part of
https://github.com/vllm-project/vllm-ascend/issues/1620 .
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/235bfd5dfe0975e42b115cfb910e73eff5c670d8
---------
Signed-off-by: shen-shanshan <467638484@qq.com >
2025-07-15 19:48:47 +08:00
wangxiyuan
bf2549856f
[CI] Fix changes CI to recover codecov ( #1799 )
...
Add `checkout` action before `dorny/paths-filter` to make it works with
`push` case.
This is a known issue that `dorny/paths-filter` works without `checkout`
in `pull_request` case but failed in `push` case. More detail is here:
https://github.com/dorny/paths-filter/issues/60#issuecomment-1464281021
The push CI works after this PR. The test result is here:
https://github.com/wangxiyuan/vllm-ascend/actions/runs/16285606468/job/45983607539
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/d4d309409f2396e68e4b5a67ede194913502388b
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-15 15:01:13 +08:00
wangxiyuan
787010a637
[Test] Remove VLLM_USE_V1 in example and tests ( #1733 )
...
V1 is enabled by default, no need to set it by hand now. This PR remove
the useless setting in example and tests
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/9ad0a4588ba4e9c979cda0d178dec4fcdb89fd0c
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-15 12:49:57 +08:00
wangxiyuan
eb921d2b6f
[Doc] Fix 404 error ( #1797 )
...
Fix url 404 error in doc
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/9ad0a4588ba4e9c979cda0d178dec4fcdb89fd0c
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-15 11:52:38 +08:00
wangxiyuan
7bdada58eb
[Misc] Remove VLLM_USE_V1 usage in code ( #1764 )
...
We plan to remove V0 code from this version. The first step is to delete
v0 usage.
Related: https://github.com/vllm-project/vllm-ascend/issues/1620
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/61e20828da1639c05a7bb7d1592c4834e10b33b7
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-15 11:52:16 +08:00
wangxiyuan
494b0f474f
[CI]Fix broken CI ( #1773 )
...
This PR fixed the broken CI. It require
https://github.com/vllm-project/vllm/pull/20900 merged first.
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/e8cc53af5e17205470c04f442e67f276e08623a1
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-15 00:54:20 +08:00
Li Wang
afcfe91dfa
[Doc] Fix multi node doc ( #1783 )
...
### What this PR does / why we need it?
### Does this PR introduce _any_ user-facing change?
Pin docker image to latest release
### How was this patch tested?
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/1e9438e0b0f65f2a5cfbd522d35638e7264c8399
Signed-off-by: wangli <wangli858794774@gmail.com >
2025-07-14 17:56:57 +08:00
zhangxinyuehfad
cabfb2bc31
[Test] Resolve vllm-ascend version accuracy test ( #1769 )
...
### What this PR does / why we need it?
Resolve vllm-ascend version for accuracy test
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/66f6fbd393721c98440436ab067304ac4331219c
Signed-off-by: hfadzxy <starmoon_zhang@163.com >
2025-07-14 15:43:37 +08:00
Shanshan Shen
d3c6dd985a
[Misc] Add include dir to .gitignore ( #1771 )
...
### What this PR does / why we need it?
Add `include` dir to `.gitignore`.
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/66f6fbd393721c98440436ab067304ac4331219c
Signed-off-by: shen-shanshan <467638484@qq.com >
2025-07-14 12:05:29 +08:00
Li Wang
9cd4ac76a1
[CI] Remove benchmark patch and increase the scheduler frequency ( #1762 )
...
### What this PR does / why we need it?
This pr purpose to do the following things:
1. Remove `benchmark_datasets.py` patch
2. Increase the scheduler frequency to 2 times per day, due to the
recent large number of daily submissions, we need to increase the
default test time(6h)
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/247102f07f1dda117fef06493292e1925a5fcd31
---------
Signed-off-by: wangli <wangli858794774@gmail.com >
2025-07-13 20:00:35 +08:00
Yikun Jiang
d118bf8a26
Update README.zh.md to fix typo ( #1758 )
...
### What this PR does / why we need it?
Update README.zh.md to fix typo
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
CI passed
Signed-off-by: Yikun Jiang <yikunkero@gmail.com >
2025-07-12 14:01:34 +08:00
Yikun Jiang
eff4b5791c
Recover offline_inference_npu.py to make doctest passed ( #1756 )
...
### What this PR does / why we need it?
Rename offline_inference_npu_v1.py to offline_inference_npu.py to
recover doctest
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
CI passed
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/a8593237c04f4d778c0e48d4d56395240ebe3011
Signed-off-by: Yikun Jiang <yikunkero@gmail.com >
2025-07-12 12:36:35 +08:00
Yikun Jiang
8b3a483269
Add recommend version and refresh readme / contribution.md ( #1757 )
...
### What this PR does / why we need it?
Add recommend version and contribution.md
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
CI passed
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/890323dc1b1141c816637af960208678a30588b6
Signed-off-by: Yikun Jiang <yikunkero@gmail.com >
2025-07-12 12:35:40 +08:00
wangxiyuan
3c404de1b1
[Release]Update release note ( #1753 )
...
There is still issue with pp in some case. such as aclgraph, ray. Remove
the related doc in release note
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-11 17:58:26 +08:00
wangxiyuan
b5b7e0ecc7
[Doc] Add qwen3 embedding 8b guide ( #1734 )
...
1. Add the tutorials for qwen3-embedding-8b
2. Remove VLLM_USE_V1=1 in docs, it's useless any more from 0.9.2
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/5923ab9524e32006ffb9354c5340b6988a45fe3e
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-11 17:40:17 +08:00
wangxiyuan
9c560b009a
[Release] Add 0.9.2rc1 release note ( #1725 )
...
Add release note for 0.9.2rc1, we'll release soon
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/7bd4c37ae7c6f2223c1a031bbdd2e3435d53da94
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-11 17:36:05 +08:00
zhangxinyuehfad
1b4a2f3817
[CI] Add accuracy ci for DP and EP and TP and ETP ( #1140 )
...
### What this PR does / why we need it?
Add accuracy ci for DP and EP and TP
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/35514b682ac737505209eca21a26df86293ed5fb
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com >
2025-07-11 17:25:17 +08:00
Pr0Wh1teGivee
d13fb0766e
[Perf] add patch to optimize apply_topk_topp ( #1732 )
...
### What this PR does / why we need it?
Performance optimization for apply_top_k_top_p
### Does this PR introduce _any_ user-facing change?
Use VLLM_ASCEND_ENABLE_TOPK_TOPP_OPTIMIZATION to enable this feature
### How was this patch tested?
e2e & ut
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/6a9e6b2abf88181f93a1959fe16291c3f1696329
Signed-off-by: Pr0Wh1teGivee <calvin_zhu0210@outlook.com >
2025-07-11 15:32:02 +08:00
weiguihua2
aa4240c67f
Support pipeline parallel in V1 Engine ( #1700 )
...
### What this PR does / why we need it?
This patch supports pipeline parallel in V1 Engine
### Does this PR introduce _any_ user-facing change?
Yes, users can run PP in V1
### How was this patch tested?
Manully test
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/31d5c1797f320b2f407c893673330b3a8766ae47
Signed-off-by: weiguihua2 <weiguihua2@huawei.com >
2025-07-11 15:30:51 +08:00
zhangxinyuehfad
1cd27da5fb
[Test] Remove VLLM_USE_V1 in accuracy test ( #1739 )
...
### What this PR does / why we need it?
Remove VLLM_USE_V1 in accuracy test
Signed-off-by: hfadzxy <starmoon_zhang@163.com >
2025-07-11 15:29:11 +08:00
ttanzhiqiang
ee40d3d850
use npu_moe_gating_top_k_softmax ( #1355 )
...
### What this PR does / why we need it?
The optimization solution for non-deepseek select_experts is to replace
gating_topk_softmax with softmax+topk+to, which is optimized from 37us
to 14us on bf16/fp16 of qwen3-235b
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/1a4f35e2eaa3ebdecb8ef9ff8302b01e289305c9
---------
Signed-off-by: ttanzhiqiang <389825161@qq.com >
2025-07-11 08:55:06 +08:00
ttanzhiqiang
9d16c9982e
rm router logits Improve TTOP 3ms ( #1407 )
...
### What this PR does / why we need it?
The previous code is
router_logits, _ = self.gate(hidden_states)
hidden_states = get_dp_group().all_gather(hidden_states, 0)
router_logits = get_dp_group().all_gather(router_logits, 0)
I want to change the two all_gathers to one, reduce one all_gather
communication, and make it
hidden_states = get_dp_group().all_gather(hidden_states, 0)
router_logits, _ = self.gate(hidden_states)
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
bash examples/run_dp_attention_etp16.sh
bash examples/run_dp_attention_etp16_benmark.sh
gsm8k accuracy verification
<img width="1809" alt="截屏2025-06-24 21 53 24"
src="https://github.com/user-attachments/assets/47eace3b-a86b-41b4-9de8-773f57fea33b "
/>
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/77f77a951e68fc9564685dd8041208c00fc22bb1
---------
Signed-off-by: ttanzhiqiang <389825161@qq.com >
2025-07-11 08:53:17 +08:00
ApsarasX
0fc9b56d40
[Perf] Improve MLA multistream performance ( #1353 )
...
### What this PR does / why we need it?
> Need to merge after PR #1322
According to benchmark results, this PR brings approximately 1%
performance gain.
#### Before Improvement
Profiling
<img width="1147" alt="截屏2025-06-22 14 54 47"
src="https://github.com/user-attachments/assets/4a4dc7f1-5b76-45d5-864d-dd7f8faf993c "
/>
Evaluation
```
# server launch command
python -m vllm.entrypoints.openai.api_server --model=/DeepSeek-R1-W8A8 \
--quantization ascend \
--served-model-name auto \
--trust-remote-code \
--distributed-executor-backend=mp \
--port 8006 \
-tp=16 \
--max-num-seqs 24 \
--max-model-len 32768 \
--max-num-batched-tokens 8192 \
--block-size 128 \
--no-enable-prefix-caching \
--additional-config '{"torchair_graph_config":{"enable_multistream_mla": true,"enabled":true,"use_cached_graph":true,"graph_batch_sizes":[24]},"ascend_scheduler_config":{"enabled":true},"expert_tensor_parallel_size":16}' \
--gpu-memory-utilization 0.96
# client benchmark command
python /root/vllm/benchmarks/benchmark_serving.py --backend vllm --dataset-name random \
--random-input-len 4096 \
--random-output-len 1536 \
--num-prompts 200 \
--ignore-eos \
--model auto \
--tokenizer /DeepSeek-R1-W8A8 \
--port 8006 \
--request-rate 1 \
--max-concurrency 24 \
--save-result \
--skip-initial-test \
--metric-percentiles "50,90,99"
```
```
============ Serving Benchmark Result ============
Successful requests: 200
Benchmark duration (s): 958.59
Total input tokens: 819200
Total generated tokens: 307200
Request throughput (req/s): 0.2086
Output token throughput (tok/s): 320.47
Total Token throughput (tok/s): 1175.05
---------------Time to First Token----------------
Mean TTFT (ms): 942.70
Median TTFT (ms): 713.87
P50 TTFT (ms): 713.87
P90 TTFT (ms): 1363.88
P99 TTFT (ms): 2008.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 68.96
Median TPOT (ms): 69.49
P50 TPOT (ms): 69.49
P90 TPOT (ms): 70.42
P99 TPOT (ms): 70.72
---------------Inter-token Latency----------------
Mean ITL (ms): 68.96
Median ITL (ms): 59.88
P50 ITL (ms): 59.88
P90 ITL (ms): 61.59
P99 ITL (ms): 68.82
==================================================
```
#### After Improvement
Profiling
<img width="1200" alt="截屏2025-06-22 14 55 42"
src="https://github.com/user-attachments/assets/e3eb9dec-0ff0-4e5f-ab94-93c65003e51f "
/>
Evaluation
```
============ Serving Benchmark Result ============
Successful requests: 200
Benchmark duration (s): 948.08
Total input tokens: 819200
Total generated tokens: 307200
Request throughput (req/s): 0.2110
Output token throughput (tok/s): 324.02
Total Token throughput (tok/s): 1188.08
---------------Time to First Token----------------
Mean TTFT (ms): 1019.25
Median TTFT (ms): 714.63
P50 TTFT (ms): 714.63
P90 TTFT (ms): 1367.31
P99 TTFT (ms): 2661.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 68.14
Median TPOT (ms): 68.68
P50 TPOT (ms): 68.68
P90 TPOT (ms): 69.33
P99 TPOT (ms): 70.30
---------------Inter-token Latency----------------
Mean ITL (ms): 68.14
Median ITL (ms): 59.04
P50 ITL (ms): 59.04
P90 ITL (ms): 60.93
P99 ITL (ms): 66.89
==================================================
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/65393ee064036d591bfc55d8c5b9ef4711ecbc70
Signed-off-by: ApsarasX <apsarax@outlook.com >
2025-07-11 08:51:17 +08:00
Mengqing Cao
cc210f46e6
[AscendScheduler][Bugfix] Remove num_draft_tokens while allocating slots ( #1718 )
...
### What this PR does / why we need it?
Now there is no need to calculate `num_draft_tokens` when allocating
slots.
This PR follows the changes in vllm:
https://github.com/vllm-project/vllm/pull/20701
### Does this PR introduce _any_ user-facing change?
N/A
### How was this patch tested?
CI passed with existing test
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/cc876d0f2909ea62f8e96a9fe384162a1680a9c8
---------
Signed-off-by: MengqingCao <cmq0113@163.com >
2025-07-10 18:47:45 +08:00
wangxiyuan
011fd73a48
[CI] Make CI tracker more clear ( #1720 )
...
1. enable lint check for all change
2. only run ut and e2e if it's the code change.
3. only run ut and disable e2e if the change is ut only.
4. disable wheel build for push case
5. run unit test when pr is merged
6. remove useless pytest.ini
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/fdfd409f8f6733823ae99545f96a6b503eea0c06
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-10 16:03:23 +08:00
wangxiyuan
3d1e6a5929
[Doc] Update user doc index ( #1581 )
...
Add user doc index to make the user guide more clear
- vLLM version: v0.9.1
- vLLM main:
https://github.com/vllm-project/vllm/commit/49e8c7ea256bd48a36391b5bc72212af39278b67
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-10 14:26:59 +08:00
Li Wang
c7446438a9
[1/N][CI] Move linting system to pre-commits hooks ( #1256 )
...
### What this PR does / why we need it?
Follow vllm-project/vllm lint way:
https://github.com/vllm-project/vllm/blob/main/.pre-commit-config.yaml
Enable pre-commit to avoid some low level error AMAP.
This pr is one step of #1241 , The purpose is make linting system more
clear and convenient, on this step, Mainly did the following things:
yapf, actionlint, ruff, typos, isort, mypy, png-lint, signoff-commit,
enforce-import-regex-instead-of-re.
TODO:
- clang-format(check for csrc with google style)
need clean code, disable for now
- pymarkdown
need clean code, disable for now
- shellcheck
need clean code, disable for now
### Does this PR introduce _any_ user-facing change?
Only developer UX change:
https://vllm-ascend--1256.org.readthedocs.build/en/1256/developer_guide/contributing.html#run-lint-locally
```
pip install -r requirements-lint.txt && pre-commit install
bash format.sh
```
### How was this patch tested?
CI passed with new added/existing test.
Co-authored-by: Yikun [yikunkero@gmail.com ](mailto:yikunkero@gmail.com )
Co-authored-by: wangli
[wangli858794774@gmail.com ](mailto:wangli858794774@gmail.com )
- vLLM version: v0.9.1
- vLLM main:
https://github.com/vllm-project/vllm/commit/5358cce5ffbd4011f8fea2188995a249b43b8bfe
---------
Signed-off-by: wangli <wangli858794774@gmail.com >
2025-07-10 14:17:15 +08:00
ApsarasX
643e6f5486
[Bugfix] Fix accuracy problem caused by mask pollution ( #1678 )
...
### What this PR does / why we need it?
If a small batch of short requests is sent first, forming a chunk with a
length <128, it will corrupt the `attn_mask_cache`, causing subsequent
requests that do not form a chunk to have accuracy issues.
The root cause of this problem is the use of in-place multiplication.
Modifying it to use out-of-place multiplication will resolve the
accuracy problem.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Yes.
- vLLM version: v0.9.2
- vLLM main:
https://github.com/vllm-project/vllm/commit/ad6c2e1a0b56c29065c7d70ff2e736e4f2fb03af
---------
Signed-off-by: ApsarasX <apsarax@outlook.com >
2025-07-10 14:06:49 +08:00
ttanzhiqiang
60519c71bd
shared_experts+router_experts merge all_reduce(Improve TTOP 5ms) ( #1395 )
...
### What this PR does / why we need it?
When all_reduce_merge is in progress, shared_experts does not do
all_reduce in mlp, but waits until shared_experts+router_experts are
completed before doing all_reduce
In prefill and decode, as long as shared_experts+router_experts are
all_reduce, there will be benefits.
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
bash examples/run_dp_attention_etp16.sh
bash examples/run_dp_attention_etp16_benmark.sh
- vLLM version: v0.9.1
- vLLM main:
https://github.com/vllm-project/vllm/commit/977180c912b1b07153decbeb62c2cef24032a701
---------
Signed-off-by: ttanzhiqiang <389825161@qq.com >
2025-07-10 12:07:05 +08:00
Yikun Jiang
997f156a51
Use ci_vllm_version when recording vLLM commit ( #1689 )
...
### What this PR does / why we need it?
Use ci_vllm_version when recording vllm commit
Followup on https://github.com/vllm-project/vllm-ascend/pull/1623
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Test mannually.
$ python3 docs/source/conf.py | jq .ci_vllm_version | tr -d '"'
v0.9.2
- Test on my local repo: https://github.com/Yikun/vllm-ascend/pull/35
- vLLM version: v0.9.1
- vLLM main:
https://github.com/vllm-project/vllm/commit/49e8c7ea256bd48a36391b5bc72212af39278b67
Signed-off-by: Yikun Jiang <yikunkero@gmail.com >
2025-07-10 11:07:27 +08:00
ApsarasX
89c1a0f006
[Bugfix] Fix memory-leak caused by dist._functional_collectives.reduce_scatter_tensor ( #1380 )
...
### What this PR does / why we need it?
In some cases, `dist._functional_collectives.reduce_scatter_tensor` can
cause its input tensor not to be released immediately after the current
layer ends. Instead, it will only be released when the GPU memory usage
of the current process reaches a certain threshold (approximately every
15 layers each time).
**Before Fix**
<img width="1441" alt="截屏2025-06-24 01 26 13"
src="https://github.com/user-attachments/assets/72d5dbb3-c8c8-4778-bf64-8db7bab8aff0 "
/>
**After Fix**
<img width="1475" alt="截屏2025-06-24 01 23 43"
src="https://github.com/user-attachments/assets/6c69cfcd-a469-4ee5-b8c6-210aeb3a5bdf "
/>
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- vLLM version: v0.9.1
- vLLM main:
https://github.com/vllm-project/vllm/commit/9ff2af6d2ba1757c6e59fe803225a73e61fe526f
---------
Signed-off-by: ApsarasX <apsarax@outlook.com >
2025-07-10 10:57:24 +08:00
Mengqing Cao
b1c66b211f
[CI] Fix lint in CI ( #1712 )
...
### What this PR does / why we need it?
Fix lint in CI
- vLLM version: v0.9.1
- vLLM main:
https://github.com/vllm-project/vllm/commit/49e8c7ea256bd48a36391b5bc72212af39278b67
Signed-off-by: MengqingCao <cmq0113@163.com >
2025-07-10 10:47:18 +08:00
Li Wang
0c4aa2b4f1
[Doc] Add multi node data parallel doc ( #1685 )
...
### What this PR does / why we need it?
add multi node data parallel doc
### Does this PR introduce _any_ user-facing change?
add multi node data parallel doc
### How was this patch tested?
- vLLM version: v0.9.1
- vLLM main:
https://github.com/vllm-project/vllm/commit/805d62ca88e4fb534ab58aa884612228c77cc08d
Signed-off-by: wangli <wangli858794774@gmail.com >
2025-07-10 09:36:37 +08:00
leo-pony
b4b19ea588
[Doc] Add multi-npu qwen3-MoE-32B Tutorials ( #1419 )
...
Signed-off-by: leo-pony <nengjunma@outlook.com >
### What this PR does / why we need it?
Add multi-npu qwen3-MoE-32B Tutorials
Relate RFC: https://github.com/vllm-project/vllm-ascend/issues/1248
- vLLM version: v0.9.1
- vLLM main:
https://github.com/vllm-project/vllm/commit/5358cce5ffbd4011f8fea2188995a249b43b8bfe
---------
Signed-off-by: leo-pony <nengjunma@outlook.com >
2025-07-10 09:06:51 +08:00