enginex-ascend-910-vllm/docs/source/tutorials/models/Qwen3.5-Dense.md

Qwen3.5-Dense (Qwen3.5-2B/4B/9B)

1 Introduction

Qwen3.5-2B, Qwen3.5-4B, and Qwen3.5-9B are dense hybrid Mamba-Transformer language models in the Qwen3.5 family. They share the same hybrid attention design (GDN + full attention) and are suitable for general-purpose text generation tasks such as dialogue, content creation, and code generation.

This document describes deployment and verification of these models on Atlas 300I DUO and Atlas 200I Pro, including environment preparation, Docker installation, single-node online deployment, functional verification, and tuning notes.

It is strongly recommended to use the latest release candidate (rc) version or the latest official version of vllm-ascend. Support for Qwen3.5-2B/4B/9B on Atlas 300I DUO and Atlas 200I Pro starts from vllm-ascend:v0.23.0rc1.

2 Supported Features

Please refer to the Supported Features List for the model support matrix.

Please refer to the Feature Guide for feature configuration information.

3 Prerequisites

3.1 Model Weight

Model	Version	Hardware Requirement	Download
Qwen3.5-2B	FP16	Atlas 300I DUO or Atlas 200I Pro	Download
Qwen3.5-4B	FP16	Atlas 300I DUO or Atlas 200I Pro	Download
Qwen3.5-9B	FP16	Atlas 300I DUO or Atlas 200I Pro	Download

It is recommended to download the model weight to a local directory such as /root/.cache/ or /home/data/.

4 Installation

4.1 Docker Image Installation

Select an image based on your machine type and start the docker image on your node, refer to using docker.

It is recommended to use the latest release candidate (rc) version or the latest official version of the vllm-ascend image. For Atlas 300I DUO and Atlas 200I Pro on Ubuntu, use vllm-ascend:nightly-releases-v0.23.0-310p (or a later -310p image). For Atlas 200I Pro on openEuler, use vllm-ascend:nightly-releases-v0.23.0-310p-openeuler (or a later -310p-openeuler image).

:::::::{tab-set}

::::::{tab-item} Atlas 300I DUO

Start the docker image on each node.

  :substitutions:
export IMAGE=m.daocloud.io/quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -p 8080:8080 \
    -it $IMAGE bash

::::::

::::::{tab-item} Atlas 200I Pro

Start the docker image on each node. Adjust --device=/dev/davinci0 according to the NPU ID you want to use.

:::::{tab-set}

::::{tab-item} Ubuntu 24.04 :selected:

  :substitutions:
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p

docker run --rm \
--privileged \
--name vllm-ascend \
--shm-size=10g \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci_manager \
--device=/dev/ascend_manager \
--device=/dev/user_config \
-v /etc/sys_version.conf:/etc/sys_version.conf \
-v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
-v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
-v /var/dmp_daemon:/var/dmp_daemon \
-v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
-v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
-v /usr/lib/aarch64-linux-gnu/libyaml-0.so.2:/usr/lib64/libyaml-0.so.2 \
-v /etc/slog.conf:/etc/slog.conf \
-v /var/slogd:/var/slogd \
-v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
-v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
-v /root/.cache:/root/.cache \
-p 8080:8080 \
-it $IMAGE bash

::::

::::{tab-item} openEuler 24.03

  :substitutions:
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p-openeuler

docker run --rm \
--privileged \
--name vllm-ascend \
--shm-size=10g \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci_manager \
--device=/dev/ascend_manager \
--device=/dev/user_config \
-v /etc/sys_version.conf:/etc/sys_version.conf \
-v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
-v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
-v /var/dmp_daemon:/var/dmp_daemon \
-v /usr/lib64/libsemanage.so.2:/usr/lib64/libsemanage.so.2 \
-v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
-v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
-v /usr/lib64/libyaml-0.so.2.0.9:/usr/lib64/libyaml-0.so.2 \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
-v /etc/slog.conf:/etc/slog.conf \
-v /var/slogd:/var/slogd \
-v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
-v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
-v /root/.cache:/root/.cache \
-p 8080:8080 \
-it $IMAGE bash

:::: :::::

:::::: :::::::

After a successful docker run, you can verify the running container service by executing the docker ps command. The expected result is that the container vllm-ascend is listed with status Up, confirming the docker installation is successful.

4.2 Source Code Installation

If you don't want to use the docker image as above, you can also build all from source:

1. Clone the repository and install vllm-ascend from source:

   git clone https://github.com/vllm-project/vllm-ascend.git
   cd vllm-ascend
   pip install -e .
   

   For the complete installation steps, refer to installation.

   On Atlas 300I DUO and Atlas 200I Pro, you may need to uninstall `triton-ascend` and `triton` to avoid dependency conflicts:
   
   ```bash
   pip uninstall -y triton-ascend triton
   ```
   

To verify the source code installation, run the following command and confirm the displayed version matches the one you installed:

pip show vllm-ascend

Expected result: The version information of vllm-ascend is displayed, confirming a successful installation.

5 Online Service Deployment

5.1 Single-Node Online Deployment

Single-node deployment completes both Prefill and Decode within the same node. Qwen3.5-2B, Qwen3.5-4B, and Qwen3.5-9B can be deployed on Atlas 300I DUO or Atlas 200I Pro.

Parallelism note: These platforms currently support the TP scenario. Choose TP=1 or TP=2 according to the available devices. On Atlas 200I Pro with a single visible NPU, use TP=1.

The following examples use FP16 weights from ModelScope. Replace MODEL_PATH with your local directory if needed.

:::::{tab-set}

::::{tab-item} Qwen3.5-2B

Startup Command:

#!/bin/sh
# Load model from ModelScope to speed up download
export VLLM_USE_MODELSCOPE=True

# Model weight path; can be a ModelScope model id or a local directory path
export MODEL_PATH=Qwen/Qwen3.5-2B

vllm serve $MODEL_PATH \
--host 127.0.0.1 \
--port 1025 \
--tensor-parallel-size 1 \
--served-model-name qwen3.5 \
--max-num-seqs 32 \
--max-model-len 16384 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--mamba-ssm-cache-dtype float16 \
--dtype float16 \
--speculative-config '{"method": "qwen3_5_mtp","num_speculative_tokens":1}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [2,4,8,16]}' \
--additional-config '{"ascend_compilation_config": {"enable_npugraph_ex": false}}'

::::

::::{tab-item} Qwen3.5-4B

Startup Command:

#!/bin/sh
# Load model from ModelScope to speed up download
export VLLM_USE_MODELSCOPE=True

# Model weight path; can be a ModelScope model id or a local directory path
export MODEL_PATH=Qwen/Qwen3.5-4B

vllm serve $MODEL_PATH \
--host 127.0.0.1 \
--port 1025 \
--tensor-parallel-size 1 \
--served-model-name qwen3.5 \
--max-num-seqs 32 \
--max-model-len 16384 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--mamba-ssm-cache-dtype float16 \
--dtype float16 \
--speculative-config '{"method": "qwen3_5_mtp","num_speculative_tokens":1}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [2,4,8,16]}' \
--additional-config '{"ascend_compilation_config": {"enable_npugraph_ex": false}}'

::::

::::{tab-item} Qwen3.5-9B

Startup Command:

#!/bin/sh
# Load model from ModelScope to speed up download
export VLLM_USE_MODELSCOPE=True

# Model weight path; can be a ModelScope model id or a local directory path
export MODEL_PATH=Qwen/Qwen3.5-9B

vllm serve $MODEL_PATH \
--host 127.0.0.1 \
--port 1025 \
--tensor-parallel-size 1 \
--served-model-name qwen3.5 \
--max-num-seqs 32 \
--max-model-len 16384 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--mamba-ssm-cache-dtype float16 \
--dtype float16 \
--speculative-config '{"method": "qwen3_5_mtp","num_speculative_tokens":1}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [2,4,8,16]}' \
--additional-config '{"ascend_compilation_config": {"enable_npugraph_ex": false}}'

:::: :::::

Key Parameter Descriptions:

• --tensor-parallel-size sets the tensor parallel size. Prefer TP=1 on Atlas 200I Pro. On Atlas 300I DUO, TP=1 and TP=2 are both supported; choose according to the available devices.
• --max-model-len represents the context length (input plus output for a single request). On Atlas 300I DUO and Atlas 200I Pro, configure this value according to the actual device memory; setting it too high may cause OOM.
• --max-num-seqs indicates the maximum number of requests that can be processed concurrently. On Atlas 300I DUO and Atlas 200I Pro, configure this value according to the actual device memory; setting it too high may cause OOM.
• --gpu-memory-utilization represents the proportion of HBM that vLLM will use for actual inference. On Atlas 300I DUO and Atlas 200I Pro, configure this value according to the actual device memory; setting it too high may cause OOM. The default value is 0.9.
• --dtype float16 must be set on Atlas 300I DUO and Atlas 200I Pro. These devices only support the FP16 data type.
• --mamba-ssm-cache-dtype sets the data type of the Mamba SSM cache. On Atlas 300I DUO and Atlas 200I Pro, only float16 is supported.
• --speculative-config uses qwen3_5_mtp for Qwen3.5 Dense models that include an MTP head. It is recommended to set num_speculative_tokens to 1.
• --compilation-config contains configurations related to the aclgraph graph mode:
  • "cudagraph_mode": "FULL_DECODE_ONLY" is recommended.
  • "cudagraph_capture_sizes": when tensor parallelism (TP) is enabled, hardware event-id constraints allow at most two capture sizes (for example, [1, 8]). With MTP enabled, calculate each capture size as n * (num_speculative_tokens + 1), where n is a capture size for the deployment without MTP. For example, when num_speculative_tokens is 1, the non-MTP sizes [1,2,4,8] become [2,4,8,16].
• --additional-config with "ascend_compilation_config": {"enable_npugraph_ex": false} is required because enable_npugraph_ex is not supported on these platforms.

Common Issues Tip: If you encounter issues, please refer to the Public FAQ for troubleshooting.

Service Verification:

If the service starts successfully, the following startup log will be displayed:

(APIServer pid=<pid>) INFO:     Started server process [<pid>]
(APIServer pid=<pid>) INFO:     Waiting for application startup.
(APIServer pid=<pid>) INFO:     Application startup complete.

6 Functional Verification

After the service is started, the model can be invoked by sending a prompt. Two API interfaces are supported: completions and chat.completions. Use the --served-model-name you configured (for example, qwen3.5). If you used --port 1025 or -p 8080:8080, adjust the URL accordingly.

Completions API:

curl http://127.0.0.1:1025/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen3.5",
        "prompt": "The future of AI is",
        "max_completion_tokens": 50,
        "temperature": 0
    }'

Chat Completions API:

curl http://127.0.0.1:1025/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen3.5",
        "messages": [
            {"role": "user", "content": "The future of AI is"}
        ],
        "max_completion_tokens": 1024,
        "temperature": 0.7,
        "top_p": 0.95
    }'

Expected Result: The service returns HTTP 200 OK. The JSON response contains the choices field with generated text.

7 Accuracy Evaluation

Using AISBench

1. Refer to Using AISBench for details.

2. After execution, you can get the result. Here are the accuracy results of Qwen3.5-2B, Qwen3.5-4B, and Qwen3.5-9B on Atlas 300I DUO for reference only.

Accuracy Evaluation Config File:

# Example configuration: benchmarks/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.model_postprocessors import extract_non_reasoning_content

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChat,
        abbr="vllm-api-general-chat",
        path="your_model_path",
        model="qwen3.5",
        request_rate=0,
        retry=2,
        host_ip="127.0.0.1",
        host_port=1025,
        max_out_len=4096,
        batch_size=16,
        trust_remote_code=False,
        generation_kwargs=dict(
            temperature=0.0,
            ignore_eos=False,
            chat_template_kwargs = {"enable_thinking": False},
        ),
        pred_postprocessor=dict(type=extract_non_reasoning_content)
    )
]

Model	dataset	version	metric	mode	vllm-api-general-chat
Qwen3.5-2B	gsm8k	-	accuracy	gen	77.71
Qwen3.5-2B	textvqa	-	accuracy	gen	76.09
Qwen3.5-4B	gsm8k	-	accuracy	gen	93.18
Qwen3.5-4B	textvqa	-	accuracy	gen	79.08
Qwen3.5-9B	gsm8k	-	accuracy	gen	95.30
Qwen3.5-9B	textvqa	-	accuracy	gen	82.33

8 Performance Evaluation

Using AISBench

Refer to Using AISBench for performance evaluation for details.

9 Performance Tuning

9.1 Recommended Configurations

Note

: The following configurations are for reference only. The optimal configuration depends on model size, maximum input/output length, and actual device memory.

Atlas 300I DUO / Atlas 200I Pro: Currently only the TP scenario is supported. Prefer TP=1 on Atlas 200I Pro. On Atlas 300I DUO, TP=1 and TP=2 are both supported; choose according to the available devices. Configure --max-model-len, --max-num-seqs, and --gpu-memory-utilization based on the actual device memory; setting them too high may cause OOM.

9.2 Tuning Guidelines

Please refer to the Public Performance Tuning Documentation for tuning methods.

Please refer to the Feature Guide for detailed feature descriptions.

10 FAQ

For common environment, installation, and general parameter issues, please refer to the vLLM-Ascend Public FAQ.