Files
xc-llm-ascend/vllm_ascend
Shanshan Shen 14d9a64047 [ModelRunner][V1] Optimize V1 attention mask (#442)
### What this PR does / why we need it?
Pre-construct a mask matrix to improve the efficiency of attention mask
construction during inference.

Note that the length of the matrix needs to be carefully balanced: a
matrix that is too large will consume excessive VRAM, while a matrix
that is too small will require dynamic concatenation during inference,
leading to performance degradation.

Therefore, an environment variable is added here to dynamically set the
size of the pre-constructed mask matrix based on requirements.

---------

Signed-off-by: shen-shanshan <467638484@qq.com>
Co-authored-by: didongli182 <didongli@huawei.com>
2025-04-02 10:33:53 +08:00
..
2025-03-11 19:20:06 +08:00