4.3 KiB
FAQs
Version Specific FAQs
General FAQs
1. What devices are currently supported?
Currently, ONLY Atlas A2 series (Ascend-cann-kernels-910b) are supported:
- Atlas A2 Training series (Atlas 800T A2, Atlas 900 A2 PoD, Atlas 200T A2 Box16, Atlas 300T A2)
- Atlas 800I A2 Inference series (Atlas 800I A2)
Below series are NOT supported yet:
- Atlas 300I Duo、Atlas 300I Pro (Ascend-cann-kernels-310p) might be supported on 2025.Q2
- Atlas 200I A2 (Ascend-cann-kernels-310b) unplanned yet
- Ascend 910, Ascend 910 Pro B (Ascend-cann-kernels-910) unplanned yet
From a technical view, vllm-ascend support would be possible if the torch-npu is supported. Otherwise, we have to implement it by using custom ops. We are also welcome to join us to improve together.
2. How to get our docker containers?
You can get our containers at Quay.io, e.g., vllm-ascend and cann.
If you are in China, you can use daocloud to accelerate your downloading:
- Open
daemon.json:
vi /etc/docker/daemon.json
- Add
https://docker.m.daocloud.iotoregistry-mirrors:
{
"registry-mirrors": [
"https://docker.m.daocloud.io"
]
}
- Restart your docker service:
sudo systemctl daemon-reload
sudo systemctl restart docker
After configuration, you can download our container from m.daocloud.io/quay.io/ascend/vllm-ascend:v0.7.3rc2.
3. What models does vllm-ascend supports?
Currently, we have already fully tested and supported Qwen / Deepseek (V0 only) / Llama models, other models we have tested are shown here. Plus, according to users' feedback, gemma3 and glm4 are not supported yet. Besides, more models need test.
4. How to get in touch with our community?
There are many channels that you can communicate with our community developers / users:
- Submit a GitHub issue.
- Join our weekly meeting and share your ideas.
- Join our WeChat group and ask your quenstions.
- Join our ascend channel in vLLM forums and publish your topics.
5. What features does vllm-ascend V1 supports?
Find more details here.
6. How to solve the problem of "Failed to infer device type" or "libatb.so: cannot open shared object file"?
Basicly, the reason is that the NNAL environment is not sourced. Please try source /usr/local/Ascend/nnal/atb/set_env.sh to solve the problem.
7. Does vllm-ascend support Atlas 300I Duo?
No, vllm-ascend now only supports Atlas A2 series. We are working on it.
8. How does vllm-ascend perform?
Currently, only some models are imporved. Such as Qwen2 VL, Deepseek V3. Others are not good enough. In the future, we will support graph mode and custom ops to improve the performance of vllm-ascend. And when the official release of vllm-ascend is released, you can install mindie-turbo with vllm-ascend to speed up the inference as well.
9. How vllm-ascend work with vllm?
vllm-ascend is a plugin for vllm. Basicly, the version of vllm-ascend is the same as the version of vllm. For example, if you use vllm 0.7.3, you should use vllm-ascend 0.7.3 as well. For main branch, we will make sure vllm-ascend and vllm are compatible by each commit.
10. Does vllm-ascend support Prefill Disaggregation feature?
Currently, only 1P1D is supported by vllm. For vllm-ascend, it'll be done by this PR. For NPND, vllm is not stable and fully supported yet. We will make it stable and supported by vllm-ascend in the future.