[Feature] Add docs of batch invariance and make some extra operators patch (#6910)
### What this PR does / why we need it?
This PR add docs of batch invariance and make some extra operators
according to validation result.
please see https://github.com/vllm-project/vllm-ascend/issues/5487 to
track progress.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- vLLM version: v0.16.0
- vLLM main:
15d76f74e2
---------
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
This commit is contained in:
@@ -124,7 +124,14 @@ class AscendConfig:
|
||||
# npu_fused_infer_attention_score performs better on all scenarios.
|
||||
self.pa_shape_list = additional_config.get("pa_shape_list", [])
|
||||
|
||||
self.enable_async_exponential = bool(additional_config.get("enable_async_exponential", False))
|
||||
# when enable_async_exponential is True, AscendSampler will be different from vllm Sampler,
|
||||
# which make batch_invariant mode not working.
|
||||
# so we disable async exponential when batch_invariant mode is enabled.
|
||||
from vllm.model_executor.layers.batch_invariant import vllm_is_batch_invariant
|
||||
|
||||
self.enable_async_exponential = (
|
||||
bool(additional_config.get("enable_async_exponential", False)) and not vllm_is_batch_invariant()
|
||||
)
|
||||
|
||||
self.enable_kv_nz = additional_config.get("enable_kv_nz", False)
|
||||
if self.enable_kv_nz:
|
||||
|
||||
Reference in New Issue
Block a user