7d4833bce9
### What this PR does / why we need it? This PR refactors the tutorial documentation by restructuring it into three categories: Models, Features, and Hardware. This improves the organization and navigation of the tutorials, making it easier for users to find relevant information. - The single `tutorials/index.md` is split into three separate index files: - `docs/source/tutorials/models/index.md` - `docs/source/tutorials/features/index.md` - `docs/source/tutorials/hardwares/index.md` - Existing tutorial markdown files have been moved into their respective new subdirectories (`models/`, `features/`, `hardwares/`). - The main `index.md` has been updated to link to these new tutorial sections. This change makes the documentation structure more logical and scalable for future additions. ### Does this PR introduce _any_ user-facing change? Yes, this PR changes the structure and URLs of the tutorial documentation pages. Users following old links to tutorials will encounter broken links. It is recommended to set up redirects if the documentation framework supports them. ### How was this patch tested? These are documentation-only changes. The documentation should be built and reviewed locally to ensure all links are correct and the pages render as expected. - vLLM version: v0.15.0 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.15.0 Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
114 lines
3.2 KiB
Markdown
114 lines
3.2 KiB
Markdown
# Qwen3-30B-A3B
|
|
|
|
## Run vllm-ascend on Multi-NPU with Qwen3 MoE
|
|
|
|
Run docker container:
|
|
|
|
```{code-block} bash
|
|
:substitutions:
|
|
# Update the vllm-ascend image
|
|
# For Atlas A2 machines:
|
|
# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
|
# For Atlas A3 machines:
|
|
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3
|
|
docker run --rm \
|
|
--name vllm-ascend \
|
|
--shm-size=1g \
|
|
--device /dev/davinci0 \
|
|
--device /dev/davinci1 \
|
|
--device /dev/davinci2 \
|
|
--device /dev/davinci3 \
|
|
--device /dev/davinci_manager \
|
|
--device /dev/devmm_svm \
|
|
--device /dev/hisi_hdc \
|
|
-v /usr/local/dcmi:/usr/local/dcmi \
|
|
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
|
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
|
|
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
|
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
|
-v /root/.cache:/root/.cache \
|
|
-p 8000:8000 \
|
|
-it $IMAGE bash
|
|
```
|
|
|
|
Set up environment variables:
|
|
|
|
```bash
|
|
# Load model from ModelScope to speed up download
|
|
export VLLM_USE_MODELSCOPE=True
|
|
|
|
# Set `max_split_size_mb` to reduce memory fragmentation and avoid out of memory
|
|
export PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:256
|
|
```
|
|
|
|
### Online Inference on Multi-NPU
|
|
|
|
Run the following script to start the vLLM server on Multi-NPU:
|
|
|
|
For an Atlas A2 with 64 GB of NPU card memory, tensor-parallel-size should be at least 2, and for 32 GB of memory, tensor-parallel-size should be at least 4.
|
|
|
|
```bash
|
|
vllm serve Qwen/Qwen3-30B-A3B --tensor-parallel-size 4 --enable_expert_parallel
|
|
```
|
|
|
|
Once your server is started, you can query the model with input prompts.
|
|
|
|
```bash
|
|
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
|
|
"model": "Qwen/Qwen3-30B-A3B",
|
|
"messages": [
|
|
{"role": "user", "content": "Give me a short introduction to large language models."}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 0.95,
|
|
"top_k": 20,
|
|
"max_completion_tokens": 4096
|
|
}'
|
|
```
|
|
|
|
### Offline Inference on Multi-NPU
|
|
|
|
Run the following script to execute offline inference on multi-NPU:
|
|
|
|
```python
|
|
import gc
|
|
import torch
|
|
|
|
from vllm import LLM, SamplingParams
|
|
from vllm.distributed.parallel_state import (destroy_distributed_environment,
|
|
destroy_model_parallel)
|
|
|
|
def clean_up():
|
|
destroy_model_parallel()
|
|
destroy_distributed_environment()
|
|
gc.collect()
|
|
torch.npu.empty_cache()
|
|
|
|
prompts = [
|
|
"Hello, my name is",
|
|
"The future of AI is",
|
|
]
|
|
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=40)
|
|
llm = LLM(model="Qwen/Qwen3-30B-A3B",
|
|
tensor_parallel_size=4,
|
|
distributed_executor_backend="mp",
|
|
max_model_len=4096,
|
|
enable_expert_parallel=True)
|
|
|
|
outputs = llm.generate(prompts, sampling_params)
|
|
for output in outputs:
|
|
prompt = output.prompt
|
|
generated_text = output.outputs[0].text
|
|
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
|
|
|
|
del llm
|
|
clean_up()
|
|
```
|
|
|
|
If you run this script successfully, you can see the info shown below:
|
|
|
|
```bash
|
|
Prompt: 'Hello, my name is', Generated text: " Lucy. I'm from the UK and I'm 11 years old."
|
|
Prompt: 'The future of AI is', Generated text: ' a topic that has captured the imagination of scientists, philosophers, and the general public'
|
|
```
|