37db0844f5
With more and more custom ops merged, disable `COMPILE_CUSTOM_KERNELS `
for vllm ascend seems useless now. Let's enable csrc compile by default.
- vLLM version: v0.12.0
- vLLM main:
ad32e3e19c
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
1.0 KiB
1.0 KiB
LoRA Adapters Guide
Overview
Like vLLM, vllm-ascend supports LoRA as well. The usage and more details can be found in vLLM official document.
You can refer to Supported Models to find which models support LoRA in vLLM.
You can run LoRA with ACLGraph mode now. Please refer to Graph Mode Guide for a better LoRA performance.
Example
We provide a simple LoRA example here, which enables the ACLGraph mode by default.
vllm serve meta-llama/Llama-2-7b \
--enable-lora \
--lora-modules '{"name": "sql-lora", "path": "/path/to/lora", "base_model_name": "meta-llama/Llama-2-7b"}'
Custom LoRA Operators
We have implemented LoRA-related AscendC operators, such as bgmv_shrink, bgmv_expand, sgmv_shrink and sgmv_expand. You can find them under the "csrc/kernels" directory of vllm-ascend repo.