init v0.23.0

Signed-off-by: Sun Ruoxi <sunruoxi@4paradigm.com>
2026-08-27 15:11:51 +08:00
parent b582a8e7d1
commit 7f8a1b1f7a
2849 changed files with 712887 additions and 22001 deletions
--- a/docs/source/quick_start.md
+++ b/docs/source/quick_start.md
@@ -1,16 +1,71 @@
 # Quickstart

+## Introduction
+
+This section guides you through container-based environment setup and large model inference, using the Qwen3-0.6B offline single-GPU inference script as an example.
+
+- For details on using different models, see the corresponding model tutorial in the "Model Tutorials" directory, for example, [Qwen3-30B-A3B](../../docs/source/tutorials/models/Qwen3-30B-A3B.md).
+- For details on using different functions, see the corresponding function tutorial in the "Function Tutorials" directory, for example, [Prefill-Decode Disaggregation (DeepSeek)](../../docs/source/tutorials/features/pd_disaggregation_mooncake_multi_node.md).
+
 ## Prerequisites

 ### Supported Devices
- Atlas A2 Training series (Atlas 800T A2, Atlas 900 A2 PoD, Atlas 200T A2 Box16, Atlas 300T A2)
- Atlas 800I A2 Inference series (Atlas 800I A2)
- Atlas A3 Training series (Atlas 800T A3, Atlas 900 A3 SuperPoD, Atlas 9000 A3 SuperPoD)
- Atlas 800I A3 Inference series (Atlas 800I A3)
- [Experimental] Atlas 300I Inference series (Atlas 300I Duo)
+
+- Atlas A2 training series (Atlas 800T A2, Atlas 900 A2 PoD, Atlas 200T A2 Box16, Atlas 300T A2)
+- Atlas 800I A2 inference series (Atlas 800I A2)
+- Atlas A3 training series (Atlas 800T A3, Atlas 900 A3 SuperPoD, Atlas 9000 A3 SuperPoD)
+- Atlas 800I A3 inference series (Atlas 800I A3)
+- Atlas 950DT inference series (Atlas 950DT)
+- Atlas 300I DUO
+- Atlas 200I Pro
+
+## Requirements
+
+:::::{tab-set}
+
+::::{tab-item} Atlas A2/A3/950DT inference products
+
+- OS: Linux
+- Python: >= 3.10, < 3.13
+- Hardware with Ascend NPUs. It's usually the Atlas 800 A2 series.
+- Software:
+
+    | Software      | Supported version                | Note                                      |
+    |---------------|----------------------------------|-------------------------------------------|
+    | Ascend HDK    | Refer to the [CANN 9.1.0 Release Notes](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910/softwareinst/releasenote/9.1.0/release-notes.md) | Required for CANN |
+    | CANN          | == 9.1.0                        | Required for vllm-ascend and TorchNPU    |
+    | TorchNPU      | == 2.10.0.post4                 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
+    | torch         | == 2.10.0                       | Required for TorchNPU and vllm, No need to install manually, it will be auto installed in below steps |
+    | NNAL          | == 9.1.0                        | Required for libatb.so, enables advanced tensor operations |
+
+```{note}
+Atlas 300I DUO uses CANN 9.1.0 and `float16`. Use the `-310p` image suffix for Ubuntu or `-310p-openeuler` for openEuler. Atlas 300I DUO does not support `triton` or `triton-ascend`.
+
+Atlas 300I DUO and Atlas 200I Pro do not support `enable_npugraph_ex`. Set --additional-config '{"ascend_compilation_config": {"enable_npugraph_ex":false}}'.
+
+Atlas 200I Pro requires additional device nodes and driver mounts. See [Set up using Docker](installation.md#set-up-using-docker) for the complete container commands.
+```
+
+::::
+
+::::{tab-item} Atlas 300I DUO
+
+ | Software      | Supported version                | Note                                      |
+ |---------------|----------------------------------|-------------------------------------------|
+ | Ascend HDK    | Refer to the [CANN 9.1.0 Release Notes](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910/softwareinst/releasenote/9.1.0/release-notes.md) | Required for CANN |
+ | CANN          | == 9.1.0                | Required for vllm-ascend and TorchNPU    |
+ | TorchNPU      | == 2.10.0.post4         | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
+ | torch         | == 2.10.0               | Required for TorchNPU and vllm, No need to install manually, it will be auto installed in below steps |
+ | NNAL          | == 9.1.0                 | Required for libatb.so, enables advanced tensor operations |
+ | triton / triton-ascend | Not supported          | Uninstalled in `Dockerfile.310p` |
+
+::::
+:::::

 ## Setup environment using container

+Before using containers, make sure Docker is installed on your system. If Docker is not installed, please refer to the [Docker installation guide](https://docs.docker.com/get-started/get-docker/) for installation instructions.
+
 :::::{tab-set}
 ::::{tab-item} Ubuntu

@@ -20,9 +75,48 @@
 # Update DEVICE according to your device (/dev/davinci[0-7])
 export DEVICE=/dev/davinci0
 # Update the vllm-ascend image
+# Atlas A2:
+# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
+# Atlas A3:
+# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3
+# Atlas 950DT:
+# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-950dt
 export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
 docker run --rm \
 --name vllm-ascend \
+--shm-size=1g \
+--device $DEVICE \
+--device /dev/davinci_manager \
+--device /dev/devmm_svm \
+--device /dev/hisi_hdc \
+-v /usr/local/dcmi:/usr/local/dcmi \
+-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
+-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
+-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
+-v /etc/ascend_install.info:/etc/ascend_install.info \
+-v /root/.cache:/root/.cache \
+-p 8000:8000 \
+-it $IMAGE bash
+# Install curl
+apt-get update -y && apt-get install -y curl
+```
+
+::::
+
+::::{tab-item} Ubuntu (Atlas 300I DUO)
+
+The following command applies to Atlas 300I DUO. For Atlas 200I Pro, use the additional device nodes and driver mounts documented in [Installation](installation.md#set-up-using-docker).
+
+```{code-block} bash
+   :substitutions:
+
+# Update DEVICE according to your device (/dev/davinci[0-7])
+export DEVICE=/dev/davinci0
+# Update the vllm-ascend image
+export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p
+docker run --rm \
+--name vllm-ascend \
+--shm-size=1g \
 --device $DEVICE \
 --device /dev/davinci_manager \
 --device /dev/devmm_svm \
@@ -49,9 +143,48 @@ apt-get update -y && apt-get install -y curl
 # Update DEVICE according to your device (/dev/davinci[0-7])
 export DEVICE=/dev/davinci0
 # Update the vllm-ascend image
+# Atlas A2:
+# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-openeuler
+# Atlas A3:
+# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3-openeuler
+# Atlas 950DT:
+# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-950dt-openeuler
 export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-openeuler
 docker run --rm \
 --name vllm-ascend \
+--shm-size=1g \
+--device $DEVICE \
+--device /dev/davinci_manager \
+--device /dev/devmm_svm \
+--device /dev/hisi_hdc \
+-v /usr/local/dcmi:/usr/local/dcmi \
+-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
+-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
+-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
+-v /etc/ascend_install.info:/etc/ascend_install.info \
+-v /root/.cache:/root/.cache \
+-p 8000:8000 \
+-it $IMAGE bash
+# Install curl
+yum update -y && yum install -y curl
+```
+
+::::
+
+::::{tab-item} openEuler (Atlas 300I DUO)
+
+The following command applies to Atlas 300I DUO. For Atlas 200I Pro, use the additional device nodes and driver mounts documented in [Installation](installation.md#set-up-using-docker).
+
+```{code-block} bash
+   :substitutions:
+
+# Update DEVICE according to your device (/dev/davinci[0-7])
+export DEVICE=/dev/davinci0
+# Update the vllm-ascend image
+export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p-openeuler
+docker run --rm \
+--name vllm-ascend \
+--shm-size=1g \
 --device $DEVICE \
 --device /dev/davinci_manager \
 --device /dev/devmm_svm \
@@ -71,16 +204,16 @@ yum update -y && yum install -y curl
 ::::
 :::::

-The default workdir is `/workspace`, vLLM and vLLM Ascend code are placed in `/vllm-workspace` and installed in [development mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html)(`pip install -e`) to help developer immediately take place changes without requiring a new installation.
+The default workdir is `/workspace`, vLLM and vLLM Ascend code are placed in `/vllm-workspace` and installed in [development mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html) (`pip install -e`) to help developers make changes effective immediately without requiring a new installation.

 ## Usage

-You can use Modelscope mirror to speed up download:
+You can use ModelScope mirror to speed up download:

-<!-- tests/e2e/doctest/001-quickstart-test.sh should be considered updating as well -->
+<!-- tests/e2e/doctests/001-quickstart-test.sh should be considered updating as well -->

 ```bash
-export VLLM_USE_MODELSCOPE=true
+export VLLM_USE_MODELSCOPE=True
 ```

 There are two ways to start vLLM on Ascend NPU:
@@ -88,22 +221,21 @@ There are two ways to start vLLM on Ascend NPU:
 :::::{tab-set}
 ::::{tab-item} Offline Batched Inference

-With vLLM installed, you can start generating texts for list of input prompts (i.e. offline batch inferencing).
+With vLLM installed, you can start generating texts for list of input prompts (i.e. offline batch inference).

-Try to run below Python script directly or use `python3` shell to generate texts:
+Create and run a simple inference test. The `example.py` can be like:

 <!-- tests/e2e/doctest/001-quickstart-test.sh should be considered updating as well -->

 ```python
 from vllm import LLM, SamplingParams
-
 prompts = [
    "Hello, my name is",
    "The future of AI is",
 ]
 sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
 # The first run will take about 3-5 mins (10 MB/s) to download models
-llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct")
+llm = LLM(model="Qwen/Qwen3-0.6B")

 outputs = llm.generate(prompts, sampling_params)

@@ -113,24 +245,75 @@ for output in outputs:
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
 ```

+Then run:
+
+```bash
+python example.py
+```
+
+If you encounter a connection error with Hugging Face (e.g., `We couldn't connect to 'https://huggingface.co' to load the files, and couldn't find them in the cached files.`), run the following commands to use ModelScope as an alternative:
+
+```bash
+export VLLM_USE_MODELSCOPE=True
+pip install modelscope
+python example.py
+```
+
+```{note}
+If you encounter custom-op security verification errors while running inference on Atlas 950DT, refer to [Pooling enables UB and UBoE for 950DT and 950PR](https://gitcode.com/Ascend/memcache/wiki/%E6%B1%A0%E5%8C%96%E4%BD%BF%E8%83%BD950DT%E5%92%8C950PR%E7%9A%84UB%E5%92%8CUBoE.md) and run the following commands:
+
+> Each NPU will prompt for confirmation when running the first command. You must manually enter `Y` for all of them.
+
+```bash
+for i in {0..7}; do npu-smi set -t custom-op-secverify-enable -i $i -d 1; done;
+for i in {0..7}; do npu-smi set -t custom-op-secverify-mode -i $i -d 0; done;
+```
+
+This section shows ascend platform is successfully detected in vllm:
+
+```bash
+INFO 05-27 11:40:38 [__init__.py:44] Available plugins for group vllm.platform_plugins:
+INFO 05-27 11:40:38 [__init__.py:46] - ascend -> vllm_ascend:register
+INFO 05-27 11:40:38 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
+INFO 05-27 11:40:38 [__init__.py:238] Platform plugin ascend is activated
+```
+
+This section shows the final output:
+
+```bash
+Prompt: 'Hello, my name is', Generated text: ' Lucy and I am an 8 year old who loves to draw and write stories'
+Prompt: 'The president of the United States is', Generated text: " a key leader in the federal government, and the president's role in the executive"
+Prompt: 'The capital of France is', Generated text: ' a city. What is the capital of France? The capital of France is Paris'
+Prompt: 'The future of AI is', Generated text: ' a topic that is being discussed in various contexts. In the business world, AI'
+```
+
+This section shows process exits after offline inference, and does not affect actual inference:
+
+```bash
+(EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1201] Shutdown initiated (timeout=0)
+(EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1224] Shutdown complete
+ERROR 05-12 11:36:01 [core_client.py:704] Engine core proc EngineCore died unexpectedly, shutting down client.
+sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
+```
+
 ::::

 ::::{tab-item} OpenAI Completions API

 vLLM can also be deployed as a server that implements the OpenAI API protocol. Run
 the following command to start the vLLM server with the
-[Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) model:
+[Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) model:

 <!-- tests/e2e/doctest/001-quickstart-test.sh should be considered updating as well -->

 ```bash
 # Deploy vLLM server (The first run will take about 3-5 mins (10 MB/s) to download models)
-vllm serve Qwen/Qwen2.5-0.5B-Instruct &
+vllm serve Qwen/Qwen3-0.6B &
 ```

-If you see log as below:
+If you see a log as below:

-```
+```shell
 INFO:     Started server process [3594]
 INFO:     Waiting for application startup.
 INFO:     Application startup complete.
@@ -139,7 +322,7 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

 Congratulations, you have successfully started the vLLM server!

-You can query the list the models:
+You can query the list of models:

 <!-- tests/e2e/doctest/001-quickstart-test.sh should be considered updating as well -->

@@ -155,32 +338,31 @@ You can also query the model with input prompts:
 curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
-        "model": "Qwen/Qwen2.5-0.5B-Instruct",
+        "model": "Qwen/Qwen3-0.6B",
        "prompt": "Beijing is a",
-        "max_tokens": 5,
+        "max_completion_tokens": 5,
        "temperature": 0
    }' | python3 -m json.tool
 ```

-vLLM is serving as background process, you can use `kill -2 $VLLM_PID` to stop the background process gracefully,
-it's equal to `Ctrl-C` to stop foreground vLLM process:
+vLLM is serving as a background process, you can use `kill -2 $VLLM_PID` to stop the background process gracefully, which is similar to `Ctrl-C` for stopping the foreground vLLM process:

 <!-- tests/e2e/doctest/001-quickstart-test.sh should be considered updating as well -->

 ```bash
-  VLLM_PID=$(pgrep -f "vllm serve")
-  kill -2 "$VLLM_PID"
+VLLM_PID=$(pgrep -f "vllm serve")
+kill -2 "$VLLM_PID"
 ```

-You will see output as below:
+The output is as below:

-```
+```shell
 INFO:     Shutting down FastAPI HTTP server.
 INFO:     Shutting down
 INFO:     Waiting for application shutdown.
 INFO:     Application shutdown complete.
 ```

-Finally, you can exit container by using `ctrl-D`.
+Finally, you can exit the container by using `ctrl-D`.
 ::::
 :::::