515267de22
### What this PR does / why we need it?
1. Use optimized apply_top_k_top_p for NPU platfrom in rejection
sampler; (avoid scatter elements which can reduce ~26ms TPOT with bs=24
per DP)
2. <del>Avoid D2H Synchronization before calling npu_top_k_top_p
introduced by parameter validation which improves inference speed with
`async_scheduling` enabled;</del> In order to elminate the D2H
synchronization introduced by parameter validation before calling
`npu_top_k_top_p`, we directly drop this fused operator since the
performance improvement is not significant compared to async_scheduling
and may bring potential accuracy problem.
3. Refactor the implementation of AscendTopKTopPSampler to align that of
vLLM.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
E2E serving test with combinations of `k=500` and `p=0.95` with
async_scheduling in single node and wide-EP scenarios.
- vLLM version: v0.11.0
- vLLM main:
83f478bb19
---------
Signed-off-by: linfeng-yuan <1102311262@qq.com>
Co-authored-by: realliujiaxu <realliujiaxu@163.com>
12 lines
461 B
Python
12 lines
461 B
Python
from tests.ut.base import TestBase
|
|
from vllm_ascend.sample.sampler import AscendSampler, AscendTopKTopPSampler
|
|
|
|
|
|
class TestAscendSampler(TestBase):
|
|
|
|
def test_init_with_raw_logprobs(self):
|
|
sampler = AscendSampler(logprobs_mode="raw_logprobs")
|
|
self.assertEqual(sampler.logprobs_mode, "raw_logprobs")
|
|
self.assertTrue(hasattr(sampler, 'topk_topp_sampler'))
|
|
self.assertIsInstance(sampler.topk_topp_sampler, AscendTopKTopPSampler)
|