ZongYuan Zhan and Yizhou
d8e15dae6c
Optimize some rejectsampler functions to make npu op launch non-blocking ( #4587 )
...
### What this PR does / why we need it?
- Vetorize the loop (but change not output) in some rejectsampler
functions include: `expand_pytorch`, `sample_recovered_tokens_pytorch`,
`rejection_random_sample_pytorch`, `sample_recovered_tokens`.
- Remove synchronize-launch torchnpu operator in them to accelerate
sampling + MTP postprocess.
### Does this PR introduce _any_ user-facing change?
- No
### How was this patch tested?
- We tested this change with the serve&bench command:
```
===== serve =====
vllm serve $LOCAL_CKPT_DIR \
--host 0.0.0.0 \
--port 8000 \
--data-parallel-size 4 \
--data-parallel-size-local 2 \
--data-parallel-address $MASTER_NODE_IP \
--data-parallel-start-rank $((2*VC_TASK_INDEX)) \
--data-parallel-rpc-port 13387 \
--tensor-parallel-size 8 \
--seed 1024 \
--enable-expert-parallel \
--served-model-name $NAME \
--max-model-len 4096 \
--max-num-seqs 16 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
$headless \
--speculative_config '{"method": "deepseek_mtp", "num_speculative_tokens": 1}' \
--additional-config '{"ascend_scheduler_config":{"enabled":false, "enable_chunked_prefill":true, "chunked_prefill_enabled":true}}'
==== bench =====
vllm bench serve --model $LOCAL_CKPT_DIR --served-model-name DeepseekV3ForCausalLM \
--dataset-name spec_bench --spec-bench-output-len 2048 \
--dataset-path question.jsonl \
--top-p 1.0 --temperature 0.8 \
--ignore-eos \
--num-prompts 64 --trust-remote-code --base-url "http://0.0.0.0:8000 " --request-rate 64
```
- In this case, our rj optimization can reduce TPOT from 84.94ms to
64.61ms, about 23% gain.
## before
<img width="1068" height="830" alt="image"
src="https://github.com/user-attachments/assets/278ac878-b49d-4588-b87c-316ca4d537f5 "
/>
## after
<img width="781" height="756" alt="image"
src="https://github.com/user-attachments/assets/0c6d37ad-ed77-40b3-a1be-4933c468365c "
/>
- vLLM version: v0.12.0
- vLLM main:
https://github.com/vllm-project/vllm/commit/ad32e3e19ccf0526cb6744a5fed09a138a5fb2f9
---------
Signed-off-by: ZongYuan Zhan <zhanzy178@gmail.com >
Co-authored-by: Yizhou <136800916+yiz-liu@users.noreply.github.com >
2025-12-29 14:10:39 +08:00
linfeng-yuan and realliujiaxu
515267de22
[perf][bugfix] improve performance of rejection sampler and eliminate HD synchronize in TopKTopPSampler ( #4154 )
...
### What this PR does / why we need it?
1. Use optimized apply_top_k_top_p for NPU platfrom in rejection
sampler; (avoid scatter elements which can reduce ~26ms TPOT with bs=24
per DP)
2. <del>Avoid D2H Synchronization before calling npu_top_k_top_p
introduced by parameter validation which improves inference speed with
`async_scheduling` enabled;</del> In order to elminate the D2H
synchronization introduced by parameter validation before calling
`npu_top_k_top_p`, we directly drop this fused operator since the
performance improvement is not significant compared to async_scheduling
and may bring potential accuracy problem.
3. Refactor the implementation of AscendTopKTopPSampler to align that of
vLLM.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
E2E serving test with combinations of `k=500` and `p=0.95` with
async_scheduling in single node and wide-EP scenarios.
- vLLM version: v0.11.0
- vLLM main:
https://github.com/vllm-project/vllm/commit/83f478bb19489b41e9d208b47b4bb5a95ac171ac
---------
Signed-off-by: linfeng-yuan <1102311262@qq.com >
Co-authored-by: realliujiaxu <realliujiaxu@163.com >
2025-12-24 19:10:33 +08:00
meihanc
592cfb6a6f
[CI] Add Triton Ascend in CI ( #4921 )
...
Add triton-ascend in UT and e2e
- vLLM version: v0.12.0
- vLLM main:
https://github.com/vllm-project/vllm/commit/ad32e3e19ccf0526cb6744a5fed09a138a5fb2f9
---------
Signed-off-by: Meihan-chen <jcccx.cmh@gmail.com >
2025-12-23 12:47:35 +08:00
weijinqian0 and weijinqian_v1
a6ef3ac4e4
[Performance] Pre-issued exponential distribution operator. ( #4908 )
...
Pre-issued exponential distribution operator.
Result:
Single inference saves 200-300 microseconds.
before:
<img width="2257" height="1058" alt="2"
src="https://github.com/user-attachments/assets/c1da19e2-a439-42cb-9d7c-c0218e61fd4c "
/>
After:
<img width="2211" height="342" alt="image"
src="https://github.com/user-attachments/assets/03c84292-c802-4755-949c-4266a9a72fc0 "
/>
- vLLM version: v0.12.0
- vLLM main:
https://github.com/vllm-project/vllm/commit/ad32e3e19ccf0526cb6744a5fed09a138a5fb2f9
---------
Signed-off-by: weijinqian_v1 <weijinqian@huawei.com >
Co-authored-by: weijinqian_v1 <weijinqian@huawei.com >
2025-12-11 23:02:51 +08:00
lio
cd58a643c5
[UT] Fix test_sample_recovered_tokens_pytorch_autoregressive ( #3434 )
...
### What this PR does / why we need it?
This 'test_rejection_sampler' unit test is something wrong.
> def test_sample_recovered_tokens_pytorch_autoregressive(self):
> output_token_ids = torch.empty(2, dtype=torch.int32)
> cu_num_draft_tokens = torch.tensor([1, 1])
> draft_token_ids = torch.tensor([0, 1])
len(draft_token_ids ) = 2, cu_num_draft_tokens should be
torch.tensor([1, 2]) or torch.tensor([2, 2])
I fix it and set cu_num_draft_tokens = torch.tensor([1, 2]). The methods
before and after optimization can pass.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
NA
- vLLM version: v0.11.0rc3
- vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.0
---------
Signed-off-by: lio <1983142975@qq.com >
2025-10-24 11:20:57 +08:00
CaranLic
168ad600b5
[main] add pd transfer for ascend scheduler ( #2753 )
...
### What this PR does / why we need it?
For offline scenarios, adjust the scheduling process to prioritize the
prefill phase of all requests, then process the decode phase of all
requests.
### How was this patch tested?
```
max_num_seqs=24,
additional_config={
"ascend_scheduler_config":{
"enabled": True,
"enable_pd_transfer": True,
"decode_max_num_seqs": 24,
"enable_chunked_prefill": False
}
},
```
| input | output | num prompts | max_num_seqs | dp | tp | scheduler |
tps |
| ------ | ------ | ---------- | ---------------- | ---- | ---- |
---------------- | --------------- |
| dapo-math-17K | 2K | 384 | 24 | 2 | 1 | v1 | 234.06 |
| dapo-math-17K | 2K | 384 | 24 | 2 | 1 | pd transfer | 239.59(+2.4%) |
| dapo-math-17K| 2K | 384 | 24 | 4 | 1 | v1 | 222.85 |
| dapo-math-17K| 2K | 384 | 24 | 4 | 1 | pd transfer | 225.81(+1.3%) |
- vLLM version: v0.10.1.1
- vLLM main:
https://github.com/vllm-project/vllm/commit/6fb27881634d89c2e70e9e5fbad1b918c0d916cf
---------
Signed-off-by: CaranLic <740821011@qq.com >
2025-09-10 08:46:39 +08:00
whx
29aaba5f84
[Perf][MTP] Optimize reject sampler in greedy situation. ( #2137 )
...
This PR port optimization in PR #2002 to main and makes it cleaner.
- vLLM version: v0.10.0
- vLLM main:
https://github.com/vllm-project/vllm/commit/afa5b7ca0b417abadfa85e32f28969b72e58a885
---------
Signed-off-by: whx-sjtu <2952154980@qq.com >
2025-08-11 17:37:49 +08:00
sherie
126cdfc92b
[Test] add rejection sampler ut ( #2084 )
...
### What this PR does / why we need it?
add rejection sampler ut.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
UT passed
- vLLM version: v0.10.0
- vLLM main:
https://github.com/vllm-project/vllm/commit/586f286789a09f5616be74ee8bedde0a9f698a72
Signed-off-by: wangxiaoxin-sherie <wangxiaoxin7@huawei.com >
2025-08-05 19:03:36 +08:00
wangxiyuan
9b67c87b14
[Refactor]Refactor sampler ( #2050 )
...
Refactor Sampler implementation from patch way to inherit from vLLM
Sampler interface.
Next step: Make the op `TopKTopPSampler` in vLLM support custom ops
register mechanism
- vLLM version: v0.10.0
- vLLM main:
https://github.com/vllm-project/vllm/commit/61a6905ab036fd00eafdb1b0ca130d5feccfe686
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2025-07-30 08:47:22 +08:00