312
docs/source/user_guide/feature_guide/recompute_cpu_offload.md
Normal file
312
docs/source/user_guide/feature_guide/recompute_cpu_offload.md
Normal file
@@ -0,0 +1,312 @@
|
||||
# Recompute CPU Offload Guide
|
||||
|
||||
## Overview
|
||||
|
||||
`RecomputeCPUOffloadConnector` preserves the KV cache of requests that are
|
||||
preempted by the Decode-side recompute scheduler. When HBM KV blocks are not
|
||||
enough, `RecomputeScheduler` may preempt a running Decode request. Without this
|
||||
connector, the request falls back to the original recompute path and may be sent
|
||||
back to the Prefill node to run prefill again. With recompute CPU offload
|
||||
enabled, the already-computed KV blocks are copied from HBM to CPU DRAM before
|
||||
the HBM blocks are reused, and copied back to HBM when the request is scheduled
|
||||
again.
|
||||
|
||||
This feature is designed for online P/D disaggregation workloads where Decode
|
||||
nodes are tuned for decode throughput and cannot efficiently recompute long
|
||||
prefills after preemption. It is opt-in and focuses on correctness first.
|
||||
|
||||
In a typical Decode deployment, `max_num_batched_tokens` is often sized around
|
||||
`max_num_seqs * (1 + num_spec_tokens)` so that the Decode node mainly schedules
|
||||
one new token, plus speculative tokens if MTP is enabled, per request. This is
|
||||
efficient for decode but too small for recomputing long prompts after
|
||||
preemption. Recompute CPU offload avoids sending such requests back through the
|
||||
full prompt recompute path when their KV state can be preserved locally.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
* **Recompute preemption**: When Decode-side HBM KV cache is exhausted,
|
||||
`RecomputeScheduler` can preempt a running request and later resume it.
|
||||
* **CPU DRAM preservation**: `RecomputeCPUOffloadConnector` stores the
|
||||
preempted request's computed KV blocks in CPU memory.
|
||||
* **H2D restore**: When the preempted request is scheduled again, the connector
|
||||
restores the preserved KV blocks before model forward.
|
||||
* **Fallback behavior**: If the connector is not configured, the recompute
|
||||
scheduler is not enabled, or CPU offload capacity is insufficient, vLLM
|
||||
falls back to the original recompute behavior.
|
||||
* **`MultiConnector` integration**: In P/D disaggregation, use
|
||||
`MultiConnector` to combine the P/D connector, such as `MooncakeConnectorV1`,
|
||||
with `RecomputeCPUOffloadConnector` on Decode nodes.
|
||||
|
||||
## Configuration Parameters
|
||||
|
||||
`RecomputeCPUOffloadConnector` is configured through `kv-transfer-config`.
|
||||
|
||||
| Parameter | Description |
|
||||
| :--- | :--- |
|
||||
| `kv_connector` | Must be set to `RecomputeCPUOffloadConnector`. |
|
||||
| `kv_role` | Set to `kv_consumer` on Decode nodes. |
|
||||
| `cpu_bytes_to_use_per_rank` | Optional and recommended. CPU memory budget in bytes used by each rank/card for recompute offload. If set, it overrides `cpu_bytes_to_use / world_size`. |
|
||||
| `cpu_bytes_to_use` | Optional. Total CPU memory budget in bytes for this vLLM instance. The connector divides it by `world_size` to get the per-rank budget. This is less direct than `cpu_bytes_to_use_per_rank` and is easier to misconfigure in DP deployments. The default is 8 GiB total. |
|
||||
| `enable_offload_prefix_caching` | Optional. Enables CPU block sharing for full hashed blocks with the same prefix-cache hash. The default is `false`; keep it disabled unless explicitly testing prefix sharing. |
|
||||
|
||||
Prefer `cpu_bytes_to_use_per_rank` when you want every rank/card to use the
|
||||
same offload capacity. For example, set `cpu_bytes_to_use_per_rank` to
|
||||
`17179869184` for 16 GiB per rank/card.
|
||||
|
||||
If you use `cpu_bytes_to_use`, remember that it is divided by `world_size`. In
|
||||
typical P/D Decode deployments, this means the value is divided by the active
|
||||
Decode DP size. For example, when starting a DP2TP8 Decode service, setting
|
||||
`cpu_bytes_to_use` to 16 GiB gives each DP rank's cards about 8 GiB of recompute
|
||||
offload space. To avoid ambiguity, use `cpu_bytes_to_use_per_rank` for new
|
||||
deployments.
|
||||
|
||||
`recompute_scheduler_enable` must also be enabled in `additional-config` on
|
||||
P/D-disaggregated Decode nodes:
|
||||
|
||||
```bash
|
||||
--additional-config '{"recompute_scheduler_enable":true}'
|
||||
```
|
||||
|
||||
```{note}
|
||||
`recompute_scheduler_enable` is only valid in P/D-disaggregated mode
|
||||
(`kv_role` is `kv_producer` or `kv_consumer`). Do not enable it in PD-mixed
|
||||
mode (`kv_role` is `kv_both`).
|
||||
```
|
||||
|
||||
## Docker Shared Memory
|
||||
|
||||
Recompute CPU offload allocates pinned CPU tensors for the offloaded KV blocks.
|
||||
When running in Docker, make sure the container's shared memory is large enough.
|
||||
If `--shm-size` is too small, a large per-card offload budget, such as 16 GiB
|
||||
per card, can fail to allocate CPU tensors and may cause the service to OOM or
|
||||
hang during startup.
|
||||
|
||||
For typical A3 deployments with a large recompute-offload budget, set
|
||||
`--shm-size=1024g` when starting the container. The following snippet follows
|
||||
the Docker style used by the DeepSeek-V4-Flash tutorial:
|
||||
|
||||
```bash
|
||||
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--shm-size=1024g \
|
||||
--net=host \
|
||||
--device /dev/davinci0 \
|
||||
--device /dev/davinci1 \
|
||||
--device /dev/davinci2 \
|
||||
--device /dev/davinci3 \
|
||||
--device /dev/davinci4 \
|
||||
--device /dev/davinci5 \
|
||||
--device /dev/davinci6 \
|
||||
--device /dev/davinci7 \
|
||||
--device /dev/davinci8 \
|
||||
--device /dev/davinci9 \
|
||||
--device /dev/davinci10 \
|
||||
--device /dev/davinci11 \
|
||||
--device /dev/davinci12 \
|
||||
--device /dev/davinci13 \
|
||||
--device /dev/davinci14 \
|
||||
--device /dev/davinci15 \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
--device /dev/hisi_hdc \
|
||||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||||
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
|
||||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||||
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
|
||||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /etc/hccn.conf:/etc/hccn.conf \
|
||||
-it $IMAGE bash
|
||||
```
|
||||
|
||||
## Usage with P/D Disaggregation
|
||||
|
||||
On Decode nodes, configure `MultiConnector` with both the P/D connector and
|
||||
`RecomputeCPUOffloadConnector`. The P/D connector handles KV transfer from
|
||||
Prefill to Decode, while `RecomputeCPUOffloadConnector` handles Decode-side
|
||||
preemption preservation and restore.
|
||||
|
||||
The following example uses `MooncakeConnectorV1` for P/D KV transfer.
|
||||
|
||||
```bash
|
||||
python3 -m vllm.entrypoints.openai.api_server \
|
||||
--model /path/to/model \
|
||||
--port 8200 \
|
||||
--trust-remote-code \
|
||||
--enforce-eager \
|
||||
--tensor-parallel-size 1 \
|
||||
--data-parallel-size 1 \
|
||||
--max-model-len 32768 \
|
||||
--block-size 128 \
|
||||
--max-num-batched-tokens 4096 \
|
||||
--additional-config '{"recompute_scheduler_enable":true}' \
|
||||
--kv-transfer-config \
|
||||
'{
|
||||
"kv_connector": "MultiConnector",
|
||||
"kv_role": "kv_consumer",
|
||||
"kv_connector_extra_config": {
|
||||
"connectors": [
|
||||
{
|
||||
"kv_connector": "MooncakeConnectorV1",
|
||||
"kv_role": "kv_consumer",
|
||||
"kv_port": "28000",
|
||||
"kv_connector_extra_config": {
|
||||
"prefill": {
|
||||
"dp_size": 1,
|
||||
"tp_size": 1
|
||||
},
|
||||
"decode": {
|
||||
"dp_size": 1,
|
||||
"tp_size": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"kv_connector": "RecomputeCPUOffloadConnector",
|
||||
"kv_role": "kv_consumer",
|
||||
"kv_connector_extra_config": {
|
||||
"cpu_bytes_to_use_per_rank": 17179869184,
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
For the Prefill node, keep the normal P/D connector configuration, for example
|
||||
`MooncakeConnectorV1` with `kv_role` set to `kv_producer`.
|
||||
|
||||
When `kv_load_failure_policy` is also needed for the P/D connector, configure
|
||||
it on the top-level `MultiConnector` `kv-transfer-config`, not inside child
|
||||
connectors.
|
||||
|
||||
## Standalone Connector Example
|
||||
|
||||
`RecomputeCPUOffloadConnector` must be used together with the vLLM-Ascend
|
||||
`RecomputeScheduler`. Because `RecomputeScheduler` is only supported on
|
||||
P/D-disaggregated Decode nodes, recompute CPU offload is currently only
|
||||
available for P/D-disaggregated Decode nodes as well.
|
||||
|
||||
The following standalone connector configuration is intended only as a minimal
|
||||
configuration fragment for validating the recompute-offload path on a
|
||||
P/D-disaggregated Decode node. It is not a PD-mixed deployment mode. In
|
||||
PD-mixed or normal non-P/D deployments, do not enable recompute CPU offload; the
|
||||
engine uses the normal vLLM recompute behavior instead.
|
||||
|
||||
```python
|
||||
from vllm.config import KVTransferConfig
|
||||
|
||||
kv_transfer_config = KVTransferConfig(
|
||||
kv_connector="RecomputeCPUOffloadConnector",
|
||||
kv_role="kv_consumer",
|
||||
kv_connector_extra_config={
|
||||
"cpu_bytes_to_use_per_rank": 17179869184,
|
||||
"enable_offload_prefix_caching": False,
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
For online serving:
|
||||
|
||||
```bash
|
||||
vllm serve /path/to/model \
|
||||
--additional-config '{"recompute_scheduler_enable":true}' \
|
||||
--kv-transfer-config '{
|
||||
"kv_connector": "RecomputeCPUOffloadConnector",
|
||||
"kv_role": "kv_consumer",
|
||||
"kv_connector_extra_config": {
|
||||
"cpu_bytes_to_use_per_rank": 17179869184,
|
||||
"enable_offload_prefix_caching": false
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
1. When `RecomputeScheduler` cannot allocate enough HBM KV blocks, it selects a
|
||||
Decode-side running request for preemption.
|
||||
2. Before the request's HBM blocks are reused, the scheduler calls the
|
||||
connector's preemption hook. If enough CPU blocks are available, the
|
||||
connector records the HBM block to CPU block mapping.
|
||||
3. The model runner calls `handle_preemptions()` before updating worker state.
|
||||
The worker copies the selected KV blocks from HBM to pinned CPU tensors.
|
||||
4. After all workers report that the store is complete, the preempted request
|
||||
becomes restorable.
|
||||
5. When the request is scheduled again, the connector reports the number of
|
||||
tokens that can be restored from CPU memory.
|
||||
6. The scheduler allocates new HBM blocks and the connector builds the CPU block
|
||||
to HBM block reload mapping.
|
||||
7. Before model forward, the worker copies the restored KV blocks back to HBM.
|
||||
Forward then continues from the restored KV state.
|
||||
|
||||
## Sliding-Window and MTP Support
|
||||
|
||||
Sliding-window models can contain logical block tables with null block IDs for
|
||||
tokens outside the attention window. The connector preserves logical block
|
||||
positions instead of compressing non-zero block IDs, so a block table such as
|
||||
`[0, 0, 20, 21]` remains aligned as `[0, 0, cpu_20, cpu_21]` on the CPU side.
|
||||
Only non-zero blocks are transferred.
|
||||
|
||||
With MTP or speculative decode enabled, the scheduler may allocate lookahead
|
||||
blocks. The reload path clips the H2D range to the GPU block table actually
|
||||
allocated for the resumed request.
|
||||
|
||||
## Notes and Limitations
|
||||
|
||||
* This feature requires the vLLM V1 engine and the vLLM-Ascend recompute
|
||||
scheduler.
|
||||
* The feature is only intended for Decode nodes in P/D disaggregation. It is not
|
||||
supported in PD-mixed or normal non-P/D deployments.
|
||||
* Do not enable `recompute_scheduler_enable` in PD-mixed deployments. Without
|
||||
P/D-disaggregated Decode-side `RecomputeScheduler`, recompute CPU offload is
|
||||
not active and vLLM follows the normal recompute path.
|
||||
* When running in Docker with a large per-rank offload budget, reserve enough
|
||||
shared memory. For typical A3 deployments with 16 GiB per-card offload, use
|
||||
`--shm-size=1024g`.
|
||||
* `enable_offload_prefix_caching` is experimental and disabled by default.
|
||||
* Current D2H and H2D transfers use basic torch copy operations. They are
|
||||
correctness-oriented and not yet optimized for transfer throughput.
|
||||
* Qwen3.5 with async scheduling is not fully supported yet. Disable async
|
||||
scheduling when using recompute CPU offload with Qwen3.5 models.
|
||||
* If CPU memory capacity is insufficient, the connector skips offload for that
|
||||
request and the scheduler falls back to the original recompute behavior.
|
||||
|
||||
## FAQ
|
||||
|
||||
### How much CPU memory should I configure?
|
||||
|
||||
Start with a budget that can hold the expected number of preempted request
|
||||
blocks. Prefer `cpu_bytes_to_use_per_rank` because it directly controls the
|
||||
offload budget for each rank/card. For example,
|
||||
`cpu_bytes_to_use_per_rank=17179869184` gives each rank/card 16 GiB.
|
||||
|
||||
`cpu_bytes_to_use` is divided by `world_size`. In a DP2TP8 Decode service,
|
||||
setting `cpu_bytes_to_use=17179869184` gives each DP rank's cards about 8 GiB
|
||||
of offload space. Increase the budget if logs show that CPU cache free blocks
|
||||
are insufficient.
|
||||
|
||||
### How do I know the offload path is active?
|
||||
|
||||
Look for logs similar to:
|
||||
|
||||
```text
|
||||
Recompute preemption offload enabled for request ...
|
||||
Created recompute offload state for request ...
|
||||
Prepared recompute offload H2D load for request ...
|
||||
```
|
||||
|
||||
If offload cannot be prepared, logs may show that CPU cache free blocks are
|
||||
insufficient, and the request will fall back to the original recompute path.
|
||||
|
||||
### Is this the same as KV Cache CPU Offload?
|
||||
|
||||
No. `KV Cache CPU Offload` is a prefix-cache offload path for inactive KV cache
|
||||
blocks. `RecomputeCPUOffloadConnector` is specifically for preserving KV blocks
|
||||
of requests preempted by the Decode-side recompute scheduler.
|
||||
|
||||
## Reference
|
||||
|
||||
For design background and implementation details, see
|
||||
[#10820: Recompute CPU Offload Connector](https://github.com/vllm-project/vllm-ascend/issues/10820).
|
||||
Reference in New Issue
Block a user