enginex-ascend-910-vllm/docs/source/user_guide/feature_guide/recompute_cpu_offload.md

# Recompute CPU Offload Guide

## Overview

`RecomputeCPUOffloadConnector` preserves the KV cache of requests that are
preempted by the Decode-side recompute scheduler. When HBM KV blocks are not
enough, `RecomputeScheduler` may preempt a running Decode request. Without this
connector, the request falls back to the original recompute path and may be sent
back to the Prefill node to run prefill again. With recompute CPU offload
enabled, the already-computed KV blocks are copied from HBM to CPU DRAM before
the HBM blocks are reused, and copied back to HBM when the request is scheduled
again.

This feature is designed for online P/D disaggregation workloads where Decode
nodes are tuned for decode throughput and cannot efficiently recompute long
prefills after preemption. It is opt-in and focuses on correctness first.

In a typical Decode deployment, `max_num_batched_tokens` is often sized around
`max_num_seqs * (1 + num_spec_tokens)` so that the Decode node mainly schedules
one new token, plus speculative tokens if MTP is enabled, per request. This is
efficient for decode but too small for recomputing long prompts after
preemption. Recompute CPU offload avoids sending such requests back through the
full prompt recompute path when their KV state can be preserved locally.

## Key Concepts

* **Recompute preemption**: When Decode-side HBM KV cache is exhausted,
  `RecomputeScheduler` can preempt a running request and later resume it.
* **CPU DRAM preservation**: `RecomputeCPUOffloadConnector` stores the
  preempted request's computed KV blocks in CPU memory.
* **H2D restore**: When the preempted request is scheduled again, the connector
  restores the preserved KV blocks before model forward.
* **Fallback behavior**: If the connector is not configured, the recompute
  scheduler is not enabled, or CPU offload capacity is insufficient, vLLM
  falls back to the original recompute behavior.
* **`MultiConnector` integration**: In P/D disaggregation, use
  `MultiConnector` to combine the P/D connector, such as `MooncakeConnectorV1`,
  with `RecomputeCPUOffloadConnector` on Decode nodes.

## Configuration Parameters

`RecomputeCPUOffloadConnector` is configured through `kv-transfer-config`.

| Parameter | Description |
| :--- | :--- |
| `kv_connector` | Must be set to `RecomputeCPUOffloadConnector`. |
| `kv_role` | Set to `kv_consumer` on Decode nodes. |
| `cpu_bytes_to_use_per_rank` | Optional and recommended. CPU memory budget in bytes used by each rank/card for recompute offload. If set, it overrides `cpu_bytes_to_use / world_size`. |
| `cpu_bytes_to_use` | Optional. Total CPU memory budget in bytes for this vLLM instance. The connector divides it by `world_size` to get the per-rank budget. This is less direct than `cpu_bytes_to_use_per_rank` and is easier to misconfigure in DP deployments. The default is 8 GiB total. |
| `enable_offload_prefix_caching` | Optional. Enables CPU block sharing for full hashed blocks with the same prefix-cache hash. The default is `false`; keep it disabled unless explicitly testing prefix sharing. |

Prefer `cpu_bytes_to_use_per_rank` when you want every rank/card to use the
same offload capacity. For example, set `cpu_bytes_to_use_per_rank` to
`17179869184` for 16 GiB per rank/card.

If you use `cpu_bytes_to_use`, remember that it is divided by `world_size`. In
typical P/D Decode deployments, this means the value is divided by the active
Decode DP size. For example, when starting a DP2TP8 Decode service, setting
`cpu_bytes_to_use` to 16 GiB gives each DP rank's cards about 8 GiB of recompute
offload space. To avoid ambiguity, use `cpu_bytes_to_use_per_rank` for new
deployments.

`recompute_scheduler_enable` must also be enabled in `additional-config` on
P/D-disaggregated Decode nodes:

```bash
--additional-config '{"recompute_scheduler_enable":true}'
```

```{note}
`recompute_scheduler_enable` is only valid in P/D-disaggregated mode
(`kv_role` is `kv_producer` or `kv_consumer`). Do not enable it in PD-mixed
mode (`kv_role` is `kv_both`).
```

## Docker Shared Memory

Recompute CPU offload allocates pinned CPU tensors for the offloaded KV blocks.
When running in Docker, make sure the container's shared memory is large enough.
If `--shm-size` is too small, a large per-card offload budget, such as 16 GiB
per card, can fail to allocate CPU tensors and may cause the service to OOM or
hang during startup.

For typical A3 deployments with a large recompute-offload budget, set
`--shm-size=1024g` when starting the container. The following snippet follows
the Docker style used by the DeepSeek-V4-Flash tutorial:

```bash

export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3
docker run --rm \
    --name vllm-ascend \
    --shm-size=1024g \
    --net=host \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci8 \
    --device /dev/davinci9 \
    --device /dev/davinci10 \
    --device /dev/davinci11 \
    --device /dev/davinci12 \
    --device /dev/davinci13 \
    --device /dev/davinci14 \
    --device /dev/davinci15 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /etc/hccn.conf:/etc/hccn.conf \
    -it $IMAGE bash
```

## Usage with P/D Disaggregation

On Decode nodes, configure `MultiConnector` with both the P/D connector and
`RecomputeCPUOffloadConnector`. The P/D connector handles KV transfer from
Prefill to Decode, while `RecomputeCPUOffloadConnector` handles Decode-side
preemption preservation and restore.

The following example uses `MooncakeConnectorV1` for P/D KV transfer.

```bash
python3 -m vllm.entrypoints.openai.api_server \
    --model /path/to/model \
    --port 8200 \
    --trust-remote-code \
    --enforce-eager \
    --tensor-parallel-size 1 \
    --data-parallel-size 1 \
    --max-model-len 32768 \
    --block-size 128 \
    --max-num-batched-tokens 4096 \
    --additional-config '{"recompute_scheduler_enable":true}' \
    --kv-transfer-config \
    '{
      "kv_connector": "MultiConnector",
      "kv_role": "kv_consumer",
      "kv_connector_extra_config": {
        "connectors": [
          {
            "kv_connector": "MooncakeConnectorV1",
            "kv_role": "kv_consumer",
            "kv_port": "28000",
            "kv_connector_extra_config": {
              "prefill": {
                "dp_size": 1,
                "tp_size": 1
              },
              "decode": {
                "dp_size": 1,
                "tp_size": 1
              }
            }
          },
          {
            "kv_connector": "RecomputeCPUOffloadConnector",
            "kv_role": "kv_consumer",
            "kv_connector_extra_config": {
              "cpu_bytes_to_use_per_rank": 17179869184,
            }
          }
        ]
      }
    }'
```

For the Prefill node, keep the normal P/D connector configuration, for example
`MooncakeConnectorV1` with `kv_role` set to `kv_producer`.

When `kv_load_failure_policy` is also needed for the P/D connector, configure
it on the top-level `MultiConnector` `kv-transfer-config`, not inside child
connectors.

## Standalone Connector Example

`RecomputeCPUOffloadConnector` must be used together with the vLLM-Ascend
`RecomputeScheduler`. Because `RecomputeScheduler` is only supported on
P/D-disaggregated Decode nodes, recompute CPU offload is currently only
available for P/D-disaggregated Decode nodes as well.

The following standalone connector configuration is intended only as a minimal
configuration fragment for validating the recompute-offload path on a
P/D-disaggregated Decode node. It is not a PD-mixed deployment mode. In
PD-mixed or normal non-P/D deployments, do not enable recompute CPU offload; the
engine uses the normal vLLM recompute behavior instead.

```python
from vllm.config import KVTransferConfig

kv_transfer_config = KVTransferConfig(
    kv_connector="RecomputeCPUOffloadConnector",
    kv_role="kv_consumer",
    kv_connector_extra_config={
        "cpu_bytes_to_use_per_rank": 17179869184,
        "enable_offload_prefix_caching": False,
    },
)
```

For online serving:

```bash
vllm serve /path/to/model \
    --additional-config '{"recompute_scheduler_enable":true}' \
    --kv-transfer-config '{
        "kv_connector": "RecomputeCPUOffloadConnector",
        "kv_role": "kv_consumer",
        "kv_connector_extra_config": {
            "cpu_bytes_to_use_per_rank": 17179869184,
            "enable_offload_prefix_caching": false
        }
    }'
```

## How It Works

1. When `RecomputeScheduler` cannot allocate enough HBM KV blocks, it selects a
   Decode-side running request for preemption.
2. Before the request's HBM blocks are reused, the scheduler calls the
   connector's preemption hook. If enough CPU blocks are available, the
   connector records the HBM block to CPU block mapping.
3. The model runner calls `handle_preemptions()` before updating worker state.
   The worker copies the selected KV blocks from HBM to pinned CPU tensors.
4. After all workers report that the store is complete, the preempted request
   becomes restorable.
5. When the request is scheduled again, the connector reports the number of
   tokens that can be restored from CPU memory.
6. The scheduler allocates new HBM blocks and the connector builds the CPU block
   to HBM block reload mapping.
7. Before model forward, the worker copies the restored KV blocks back to HBM.
   Forward then continues from the restored KV state.

## Sliding-Window and MTP Support

Sliding-window models can contain logical block tables with null block IDs for
tokens outside the attention window. The connector preserves logical block
positions instead of compressing non-zero block IDs, so a block table such as
`[0, 0, 20, 21]` remains aligned as `[0, 0, cpu_20, cpu_21]` on the CPU side.
Only non-zero blocks are transferred.

With MTP or speculative decode enabled, the scheduler may allocate lookahead
blocks. The reload path clips the H2D range to the GPU block table actually
allocated for the resumed request.

## Notes and Limitations

* This feature requires the vLLM V1 engine and the vLLM-Ascend recompute
  scheduler.
* The feature is only intended for Decode nodes in P/D disaggregation. It is not
  supported in PD-mixed or normal non-P/D deployments.
* Do not enable `recompute_scheduler_enable` in PD-mixed deployments. Without
  P/D-disaggregated Decode-side `RecomputeScheduler`, recompute CPU offload is
  not active and vLLM follows the normal recompute path.
* When running in Docker with a large per-rank offload budget, reserve enough
  shared memory. For typical A3 deployments with 16 GiB per-card offload, use
  `--shm-size=1024g`.
* `enable_offload_prefix_caching` is experimental and disabled by default.
* Current D2H and H2D transfers use basic torch copy operations. They are
  correctness-oriented and not yet optimized for transfer throughput.
* Qwen3.5 with async scheduling is not fully supported yet. Disable async
  scheduling when using recompute CPU offload with Qwen3.5 models.
* If CPU memory capacity is insufficient, the connector skips offload for that
  request and the scheduler falls back to the original recompute behavior.

## FAQ

### How much CPU memory should I configure?

Start with a budget that can hold the expected number of preempted request
blocks. Prefer `cpu_bytes_to_use_per_rank` because it directly controls the
offload budget for each rank/card. For example,
`cpu_bytes_to_use_per_rank=17179869184` gives each rank/card 16 GiB.

`cpu_bytes_to_use` is divided by `world_size`. In a DP2TP8 Decode service,
setting `cpu_bytes_to_use=17179869184` gives each DP rank's cards about 8 GiB
of offload space. Increase the budget if logs show that CPU cache free blocks
are insufficient.

### How do I know the offload path is active?

Look for logs similar to:

```text
Recompute preemption offload enabled for request ...
Created recompute offload state for request ...
Prepared recompute offload H2D load for request ...
```

If offload cannot be prepared, logs may show that CPU cache free blocks are
insufficient, and the request will fall back to the original recompute path.

### Is this the same as KV Cache CPU Offload?

No. `KV Cache CPU Offload` is a prefix-cache offload path for inactive KV cache
blocks. `RecomputeCPUOffloadConnector` is specifically for preserving KV blocks
of requests preempted by the Decode-side recompute scheduler.

## Reference

For design background and implementation details, see
[#10820: Recompute CPU Offload Connector](https://github.com/vllm-project/vllm-ascend/issues/10820).