[MLA][Graph] Improve assertion on Graph mode with MLA (#933)
### What this PR does / why we need it?
Improve assertion on Graph mode with MLA.
When running deepseek with graph mode, the fused MLA op only support
`numHeads / numKvHeads ∈ {32, 64, 128}`, thus we improve the assertion
info here to avoid users confused with this.
### Does this PR introduce _any_ user-facing change?
Adjusting tp size is required when running deepseek-v3/r1 with graph
mode. deepseek-v2-lite is not supported in graph mode.
### How was this patch tested?
Test locally as the CI machine could not run V3 due to the HBM limits.
---------
Signed-off-by: MengqingCao <cmq0113@163.com>
This commit is contained in:
@@ -40,6 +40,8 @@ from vllm_ascend.platform import CUSTOM_OP_ENABLED
|
||||
from vllm_ascend.worker.model_runner import (
|
||||
ModelInputForNPUBuilder, ModelInputForNPUWithSamplingMetadata)
|
||||
|
||||
_ALLOWED_NUM_QUERIES_PER_KV = [32, 64, 128]
|
||||
|
||||
|
||||
def generate_attn_mask(max_seq_len: int, dtype=torch.float16, mask_value=None):
|
||||
# Construct lower triangle matrix.
|
||||
@@ -1005,6 +1007,15 @@ class AscendMLAAttentionBackendImpl(MLAAttentionImpl):
|
||||
ascend_config = get_ascend_config()
|
||||
self.torchair_graph_enabled = ascend_config.torchair_graph_config.enabled
|
||||
|
||||
# TODO: support numHeads / numKvHeads < 16 in MLA kernel
|
||||
if self.torchair_graph_enabled:
|
||||
assert self.num_queries_per_kv in _ALLOWED_NUM_QUERIES_PER_KV, \
|
||||
("The allowed number of queries per kv when enabling both MLA and Graph mode"
|
||||
" only support {32, 64, 128}, Thus this is not supported for DeepSeek-V2-Lite,"
|
||||
" as it only has 16 attention heads. And if you're using DeepSeek-V3 or DeepSeek-R1,"
|
||||
" please make sure after the tensor parallel split, num_heads / num_kv_heads in "
|
||||
"{32, 64, 128}.")
|
||||
|
||||
def exec_kv(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
|
||||
Reference in New Issue
Block a user