EngineX/xc-llm-ascend
3
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

[v0.11.0] cherry-pick Fix performance degradation when mtp>1 (#3597) (#3630)

Browse Source 
### What this PR does / why we need it?
cherry-pick Fix performance degradation when mtp>1 (#3597)

This PR aims to fix performance degradation when mtp>1. Since mtp>1 may
result in more tokens (i.e. larger batch size) than acl graph maximum
batch size, this will cause draft model to run in eager mode.

### How was this patch tested?
by ci

---------

Signed-off-by: Zetong Li <slippersss@126.com>
This commit is contained in:
Zetong Li
2025-10-22 22:07:39 +08:00
committed by GitHub
parent a989fef5de
commit 6e72bfdc50
2 changed files with 21 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
tests/ut/test_utils.py
View File

@@ -272,6 +272,7 @@ class TestUtils(TestBase):
len(test_vllm_config.compilation_config.cudagraph_capture_sizes))
test_vllm_config.speculative_config = mock.MagicMock()
test_vllm_config.speculative_config.num_speculative_tokens = 2
test_vllm_config.speculative_config.draft_model_config = mock.MagicMock(
)
test_vllm_config.speculative_config.draft_model_config.hf_config = mock.MagicMock(