From 571edc58fa56f89cc4e9a57cff5a1121e32751b7 Mon Sep 17 00:00:00 2001 From: Wangbei25 Date: Thu, 23 Apr 2026 23:48:03 +0800 Subject: [PATCH] [Doc]Update DeepSeekOCR2.md for releases/v0.18.0 (#8604) ### What this PR does / why we need it? Update DeepSeekOCR2.md for releases/v0.18.0 ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? vLLM version: v0.18.0 vLLM main: https://github.com/vllm-project/vllm/commit/bcf2be96120005e9aea171927f85055a6a5c0cf6 --------- Signed-off-by: Wangbei25 Signed-off-by: Wangbei25 Co-authored-by: Wangbei25 --- docs/source/tutorials/models/DeepSeekOCR2.md | 189 ++++++++++++++++++ docs/source/tutorials/models/index.md | 1 + .../support_matrix/supported_models.md | 1 + 3 files changed, 191 insertions(+) create mode 100644 docs/source/tutorials/models/DeepSeekOCR2.md diff --git a/docs/source/tutorials/models/DeepSeekOCR2.md b/docs/source/tutorials/models/DeepSeekOCR2.md new file mode 100644 index 00000000..d1b30ee4 --- /dev/null +++ b/docs/source/tutorials/models/DeepSeekOCR2.md @@ -0,0 +1,189 @@ +# DeepSeek-OCR-2 + +## Introduction + +DeepSeekOCR2 is a model to investigate the role of vision encoders from an LLM-centric viewpoint. + +The `DeepSeek-OCR-2` model is first supported in `vllm-ascend:v0.16.0` and can stably run in v0.16.0 and later version. + +This document will show the main verification steps of the model, including supported features, feature configuration, environment preparation, single-node deployment, accuracy and performance evaluation. + +## Supported Features + +Refer to [supported features](../../user_guide/support_matrix/supported_models.md) to get the model's supported feature matrix. + +Refer to [feature guide](../../user_guide/feature_guide/index.md) to get the feature's configuration. + +## Environment Preparation + +### Model Weight + +- `DeepSeek-OCR-2`: [Download model weight](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2). + +It is recommended to download the model weight to the shared directory of multiple nodes, such as `/root/.cache/`. + +### Verify Multi-node Communication(Optional) + +If you want to deploy multi-node environment, you need to verify multi-node communication according to [verify multi-node communication environment](../../installation.md#verify-multi-node-communication). + +### Installation + +You can use our official docker image to run `DeepSeek-OCR-2` directly. + +Select an image based on your machine type and start the docker image on your node, refer to [using docker](../../installation.md#set-up-using-docker). + +```{code-block} bash + :substitutions: +# Update --device according to your device (Atlas A2: /dev/davinci[0-7] Atlas A3:/dev/davinci[0-15]). +# Update the vllm-ascend image according to your environment. +# Note you should download the weight to /root/.cache in advance. +# Update the vllm-ascend image +export IMAGE=m.daocloud.io/quay.io/ascend/vllm-ascend:|vllm_ascend_version| +export NAME=vllm-ascend + +# Run the container using the defined variables +# Note: If you are running bridge network with docker, please expose available ports for multiple nodes communication in advance. +docker run --rm \ +--name $NAME \ +--net=host \ +--shm-size=1g \ +--device /dev/davinci0 \ +--device /dev/davinci1 \ +--device /dev/davinci2 \ +--device /dev/davinci3 \ +--device /dev/davinci4 \ +--device /dev/davinci5 \ +--device /dev/davinci6 \ +--device /dev/davinci7 \ +--device /dev/davinci_manager \ +--device /dev/devmm_svm \ +--device /dev/hisi_hdc \ +-v /usr/local/dcmi:/usr/local/dcmi \ +-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \ +-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ +-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ +-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ +-v /etc/ascend_install.info:/etc/ascend_install.info \ +-v /root/.cache:/root/.cache \ +-it $IMAGE bash +``` + +If you want to deploy multi-node environment, you need to set up environment on each node. + +## Deployment + +### Single-node Deployment + +- `DeepSeek-OCR-2` can be deployed on 1 Atlas 800 A2. + +Run the following script to execute online inference. + +```shell +#!/bin/sh + +export VLLM_USE_V1=1 +export VLLM_ASCEND_ENABLE_NZ=0 +export TOKENIZERS_PARALLELISM=false +export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True" +export TASK_QUEUE_ENABLE=1 +export TOKENIZERS_PARALLELISM=false + +vllm serve /root/.cache/DeepSeek-OCR-2 \ + --served-model-name deepseekocr2 \ + --trust-remote-code \ + --tensor-parallel-size 1 \ + --port 1055 \ + --max_model_len 8192 \ + --no-enable-prefix-caching \ + --gpu-memory-utilization 0.8 \ + --allowed-local-media-path / \ + --async-scheduling \ + --additional-config '{ + "enable_cpu_binding": true, + "multistream_overlap_shared_expert": true, + "ascend_compilation_config": {"fuse_qknorm_rope": false} + }' \ + --mm-processor-cache-gb 0 +``` + +**Notice:** +The parameters are explained as follows: + +- `--max-model-len` specifies the maximum context length - that is, the sum of input and output tokens for a single request. +- `--no-enable-prefix-caching` indicates that prefix caching is disabled. To enable it, remove this option. +- `--gpu-memory-utilization` represents the proportion of HBM that vLLM will use for actual inference. Its essential function is to calculate the available kv_cache size. During the warm-up phase (referred to as profile run in vLLM), vLLM records the peak GPU memory usage during an inference process with an input size of `--max-num-batched-tokens`. The available kv_cache size is then calculated as: `--gpu-memory-utilization` * HBM size - peak GPU memory usage. Therefore, the larger the value of `--gpu-memory-utilization`, the more kv_cache can be used. However, since the GPU memory usage during the warm-up phase may differ from that during actual inference (e.g., due to uneven EP load), setting `--gpu-memory-utilization` too high may lead to OOM (Out of Memory) issues during actual inference. The default value is `0.9`. +- `--async-scheduling` Asynchronous scheduling is a technique used to optimize inference efficiency. It allows non-blocking task scheduling to improve concurrency and throughput, especially when processing large-scale models. + +### Multi-node Deployment + +Single-node deployment is recommended. + +### Prefill-Decode Disaggregation + +We don't need to Prefill-Decode disaggregation + +## Functional Verification + +If your service start successfully, you can see the info shown below: + +```bash +INFO: Started server process [87471] +INFO: Waiting for application startup. +INFO: Application startup complete. +``` + +Once your server is started, you can query the model with input prompts: + +```shell +curl http://:/v1/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "deepseekocr2", + "prompt": "The future of AI is", + "max_completion_tokens": 50, + "temperature": 0 + }' +``` + +## Accuracy Evaluation + +Here is an accuracy evaluation methods. + +### Using AISBench + +1. Refer to [Using AISBench](../../developer_guide/evaluation/using_ais_bench.md) for details. + +2. After execution, you can get the result, here is the result of `DeepSeek-OCR-2` for reference only. + +| dataset | version | metric | mode | vllm-api-general-chat | note | +|----- | ----- | ----- | ----- | -----| ----- | +| textvqa | - | accuracy | gen | 50.28 | 1 Atlas 800 A2 | +| ominidocbench | - | accuracy | gen | 66.86 | 1 Atlas 800 A2 | + +## Performance + +### Using AISBench + +Refer to [Using AISBench for performance evaluation](../../developer_guide/evaluation/using_ais_bench.md#execute-performance-evaluation) for details. + +The performance result is: + +**Hardware**: A2-313T, 1 node + +**Input/Output**: 1080P/256 + +**Performance**: TTFT = 2s, TPOT = 200ms, Average performance of each card is 864 TPS (Token Per Second). + +## Best Practices + +In this chapter, we recommend best practices. for details about best practices, see the "Single-node Deployment" section. + +## FAQ + +- **Q: Startup fails with HCCL port conflicts (address already bound). What should I do?** + + A: Clean up old processes and restart: `pkill -f vLLM*`. + +- **Q: How to handle OOM or unstable startup?** + + A: Reduce `--max-num-seqs` and `--max-model-len` first. If needed, reduce concurrency and load-testing pressure (e.g., `max-concurrency` / `num-prompts`). diff --git a/docs/source/tutorials/models/index.md b/docs/source/tutorials/models/index.md index 0a12654f..de8c0060 100644 --- a/docs/source/tutorials/models/index.md +++ b/docs/source/tutorials/models/index.md @@ -27,6 +27,7 @@ Qwen3.5-397B-A17B.md DeepSeek-V3.1.md DeepSeek-V3.2.md DeepSeek-R1.md +DeepSeekOCR2.md GLM4.x.md GLM5.md Kimi-K2-Thinking.md diff --git a/docs/source/user_guide/support_matrix/supported_models.md b/docs/source/user_guide/support_matrix/supported_models.md index 2bd84d16..6fd19e1c 100644 --- a/docs/source/user_guide/support_matrix/supported_models.md +++ b/docs/source/user_guide/support_matrix/supported_models.md @@ -20,6 +20,7 @@ Get the latest info here: