dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

ref(upstream): FULL TREE — Deep-Spark xllm (1470) + ds_vllm csrc/models (703)

Browse Source 
Replaces cherry-picked upstream_ref with complete source trees.

xllm/ — Iluvatar official C++ inference engine (15MB, 1470 files)
  Complete: kernels → layers → models → runtime → scheduler → api
  Excluded: .git, binary images, third_party submodule checkouts

ds_vllm/ — Iluvatar official vllm fork (8MB, 703 files)
  Included: csrc/ (ALL CUDA kernels), fused_moe/, qwen3_5 model, _custom_ops
  Excluded: tests, benchmarks, docs, examples (not needed for reference)

Critical call chains now fully traceable:
  MoE: moe_topk_softmax_kernels.cuh → ixformer.h → fused_moe.cpp → layer
  GDN: qwen3_gated_delta_net_base.cpp → qwen3_5_gated_delta_net.cpp
  Attention: ixformer.h → xllm_paged_attention → attention.cpp
This commit is contained in:
EX Engine
2026-08-10 02:53:54 +00:00
parent 9e4fb3712f
commit 002f9879b2
2179 changed files with 494021 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
upstream_ref/xllm/docs/en/.readthedocs.yaml Normal file
View File

@@ -0,0 +1,28 @@
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the OS, Python version, and other tools you might need
build:
os: ubuntu-24.04
tools:
python: "3.13"
jobs:
pre_build:
# - cp -r docs/en/* docs/
- mv docs/en/* docs/
- rm -rf docs/zh/
- find docs/ -name "*.md" -exec sed -i 's#../assets/#assets/#g' {} \;
# Build documentation with Mkdocs
mkdocs:
configuration: mkdocs_en.yml
# Optionally, but recommended,
# declare the Python requirements required to build your documentation
# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: docs/requirements.txt

59
upstream_ref/xllm/docs/en/accuracy_test.md Normal file
View File

@@ -0,0 +1,59 @@
# 1. LLM Accuracy Test
## 1.1 Setup ais_bench
```bash
# Create a virtual environment for ais_bench using conda or uv
conda create --name ais_bench python=3.10 -y
conda activate ais_bench
# Clone ais_bench and install dependencies
git clone https://gitee.com/aisbench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517
# Download the dataset and copy it to the ais_bench directory
cp -r /path/to/dataset /path/to/benchmark/ais_bench/datasets
```
## 1.2 Modify Configuration
Modify the accuracy test configuration file according to your actual situation: `/path/to/benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py`. It is recommended to set the sampling parameters as follows:
```python
models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr='vllm-api-general-chat',
path="/path/to/model/Qwen3-8B", # Model path
model="Qwen3-8B", # Model name
request_rate = 0,
retry = 2,
host_ip = "127.0.0.1",
host_port = 19000, # xllm server port
max_out_len = 32768, # Limit maximum model length
batch_size=32,
trust_remote_code=False,
generation_kwargs = dict(
temperature = 0.6,
# top_k = -1,
top_p = 0.95,
# seed = None,
# repetition_penalty = 1,
),
pred_postprocessor=dict(type=extract_non_reasoning_content)
)
]
```
## 1.3 Launch ais_bench
Before using ais_bench, you need to start the xllm server first. Use `ais_bench -h` to get parameter descriptions. The launch commands for gsm8k and ceval datasets are as follows:
```bash
# Using gsm8k dataset
ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt --dump-eval-details
# Using ceval dataset
ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt --merge-ds --dump-eval-details
```
We will integrate ais_bench and datasets (ceval and gsm8k) into the development image in the future. The ais_bench documentation and datasets are as follows:
* [ais_bench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/index.html)
* [Datasets](https://ais-bench-benchmark.readthedocs.io/en/latest/base_tutorials/all_params/datasets.html)

83
upstream_ref/xllm/docs/en/cli_reference.md Normal file
View File

@@ -0,0 +1,83 @@
---
hide:
- navigation
---
# Service Startup Parameters
xLLM uses gflags to manage service startup parameters. The specific parameter meanings are as follows:
## Common Parameters
| Parameter Name | Data Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `master_node_addr` | `string` | "127.0.0.1:19888" | ip:port | The listening address of the master node's rpc server | [Details](./features/basics.md) |
| `host` | `string` | "" | The machine IP where the current device is located | The host IP used by the current device for communication. An rpc server is started on each device for multi-device communication. | |
| `port` | `int32` | 8010 | Any available port | Used in conjunction with the `host` parameter. The combination is used for rpc communication between devices. | |
| `model` | `string` | "" | | Path to the model. | |
| `devices` | `string` | "npu:0" | | Specifies the NPU devices used by the current process. | |
| `nnodes` | `int32` | 1 | | The total number of devices used by the current service. | |
| `node_rank` | `int32` | 0 | 0 ~ (total devices - 1) | The rank id of each device. | |
| `max_memory_utilization` | `double` | 0.8 | Between 0-1 | The maximum proportion of device memory available for model weights and KV Cache combined. | |
| `max_tokens_per_batch` | `int32` | 10240 | | The maximum number of tokens that can be computed per step. | |
| `max_seqs_per_batch` | `int32` | 1024 | | The maximum number of sequences that can be computed per step. | |
| `enable_chunked_prefill` | `bool` | true | false | Whether to enable chunked prefill. | |
| `enable_prefill_sp` | `bool` | false | true | Whether to enable prefill-only sequence parallel. | `enable_chunked_prefill=true` is supported only for prefill-only batches (`PREFILL` / `CHUNKED_PREFILL`); `MIXED` and `DECODE` batches do not run with sequence parallel. |
| `enable_schedule_overlap` | `bool` | false | true | Whether to enable asynchronous scheduling. | [Details](./features/async_schedule.md) |
| `enable_prefix_cache` | `bool` | true | false | Whether to enable prefix cache (not supported by DeepSeek currently). | |
| `communication_backend` | `string` | "hccl" | "lccl" | The backend used for communication operations. | |
| `block_size` | `int32` | 128 | | The block size for KV Cache storage. | |
| `task` | `string` | "generate" | "embed", "mm_embed" | Service type: generation, embedding, or multimodal embedding. | |
| `max_cache_size` | `int64` | 0 | | The usable KV Cache size in bytes. | |
| `kv_cache_dtype` | `string` | "auto" | "int8" | KV Cache data type. "auto" aligns with model dtype (no quantization), "int8" enables INT8 quantization to save ~50% memory. MLU backend only. | |
## MoE Model Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `dp_size` | `int32` | 1 | Power of 2 | The dp scale size for the Attention part. | |
| `ep_size` | `int32` | 1 | Power of 2 | The ep scale size for the MoE part. | |
| `expert_parallel_degree` | `int32` | 0 | 1,2 | Parameter related to ep parallelism. Defaults to 0 when ep is not used, and to 1 when ep is enabled. Can be set to 2 when `ep_size` equals the total number of devices (uses all2all communication). | |
## P-D Separation Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `enable_disagg_pd` | `bool` | false | true | Whether to enable P-D separation. | [Details](./features/disagg_pd.md) |
| `disagg_pd_port` | `int32` | 7777 | Any available port | Configuration when P-D separation is enabled. Corresponds to the listening port number of the pd separation rpc server started on each card. | |
| `instance_role` | `string` | DEFAULT | PREFILL, DECODE, MIX | Defaults to DEFAULT. Must be configured as PREFILL, DECODE, or MIX when P-D separation is enabled. | |
| `kv_cache_transfer_mode` | `string` | "PUSH" | "PULL" | The mode for transferring KV Cache in P-D separation. PUSH mode: Prefill transmits layer by layer to Decode; PULL mode: Decode pulls the KV Cache from Prefill in one go. | |
| `transfer_listen_port` | `int32` | 26000 | Any available port | Configuration when P-D separation is enabled. Corresponds to the listening port for KV Cache Transfer on each card. | |
## MTP Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `draft_model` | `string` | "" | | Path to the MTP model. | [Details](./features/mtp.md) |
| `draft_devices` | `string` | "npu:0" | Same format as `devices`, e.g. `npu:0` or `npu:0,npu:1` | Should be set consistently with the `devices` parameter. | |
| `num_speculative_tokens` | `int32` | 0 | Any integer, suggestion 1 or 2 | The number of tokens output by the MTP model per step. | |
## Graph Execution Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `enable_graph` | `bool` | false | true | Whether to enable graph execution mode to optimize decode phase performance. Only applied during decode phase and does not take effect during prefill phase. Supports ACL Graph (NPU), and MLU Graph. | [Details](./features/graph_mode.md) |
| `enable_graph_mode_decode_no_padding` | `bool` | false | true | Builds decode graphs with the actual `num_tokens` instead of the padded shape. | |
| `enable_prefill_piecewise_graph` | `bool` | false | true | Whether to enable piecewise graph for prefill phase. Attention runs eagerly while other ops are captured into CUDA graphs. | |
| `max_tokens_for_graph_mode` | `int32` | 2048 | Any integer greater than or equal to 0 | Maximum number of tokens for graph execution. If 0, no limit is applied. | |
## Parameters for Use with xLLM-service
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `etcd_addr` | `string` | "" | ip:port | The listening address of the etcd's rpc server. | |
| `enable_service_routing` | `bool` | false | true | Whether the request from the xllm service, use this when enable the xllm service. | |
## Other Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `max_concurrent_requests` | `int32` | 200 | Any integer greater than or equal to 0 | For rate limiting, restricts the total number of requests being processed in the instance. Set to 0 for no limit. | |
| `model_id` | `string` | "" | | Model name, not a path. | |
| `num_request_handling_threads` | `int32` | 4 | Any integer greater than 0 | The thread pool size for handling input requests. | |
| `prefill_scheduling_memory_usage_threshold` | `double` | 0.95 | Value between 0-1 | When kv cache usage reaches this threshold, scheduling of prefill requests is paused. | |
| `num_response_handling_threads` | `int32` | 4 | Any integer greater than 0 | The thread pool size for handling outputs. | |
| `rank_tablefile` | `string` | "" | | Configuration file for creating the communication domain. Required for multi-node scenarios. | |

1062
upstream_ref/xllm/docs/en/design/generative_recommendation_design.md Normal file
View File

File diff suppressed because it is too large Load Diff

474
upstream_ref/xllm/docs/en/design/graph_mode_design.md Normal file
View File

@@ -0,0 +1,474 @@
# Graph Mode Design Document
## Overview
xLLM's Graph Mode supports multiple graph execution backends. Its goal is to turn the original Host-driven stream of fine-grained kernel launches into a capture-then-replay execution flow in inference serving, thereby reducing Host scheduling overhead, reducing device-side bubbles, and improving throughput and latency stability.
This document is intended for developers who need to understand the implementation principles and key design choices. It focuses on:
- the basic Graph Mode mechanism and how xLLM applies it
- dynamic dimension parameterization
- Piecewise Graph
- a multi-shape reusable memory pool, including input tensor reuse
This document focuses on the unified Graph Mode design in xLLM and does not expand on backend-specific platform differences.
The design goals of this document are:
- provide a unified Graph Mode abstraction across xLLM backends
- explain the three key designs: dynamic dimension parameterization, Piecewise Graph, and multi-shape memory reuse
- clarify what problem each design solves, what assumptions it depends on, and where its boundary lies
The non-goals of this document are:
- full adaptation details for every operator or every model
- replacing feature documentation for flags and usage examples
Related design documents:
- for a recommendation-oriented case study that focuses on fixed scheduling, multi-step execution, and custom operators, see: [Generative Recommendation Design Document](generative_recommendation_design.md)
## 1. Graph Mode Fundamentals
### 1.1 Capture / Replay Basics
In traditional eager execution, one forward pass launches many kernels, memory copies, and synchronization operations from the Host. For decode-like workloads, where each step is small but requests are frequent, Host scheduling overhead becomes significant and device-side bubbles become more visible.
The core idea of Graph Mode is:
1. **Capture**: when a shape bucket is seen for the first time, run one forward pass on a dedicated stream and record the kernel launches, memory operations, and dependencies into a graph.
2. **Replay**: for later requests that hit the same bucket, replay the recorded graph instead of launching kernels one by one from the Host.
![Graph Mode](../../assets/graph_mode.png)
This mechanism usually requires:
- **Stable execution path**
- graph capture records one concrete execution path; once capture completes, the control flow, launch shape, and dependencies on that path are fixed
- therefore, the capture path cannot switch to a different dynamic branch during replay
- **Stable addresses for key tensors**
- key tensor addresses written into the graph during capture must still be valid at replay time
- **Stable operator behavior and result semantics**
- replay correctness depends on operators along the path being Graph Mode compatible and not changing semantics because of runtime condition changes
### 1.2 xLLM GraphMode Runtime Foundations
Starting from the three basic requirements in 1.1, the first two are mainly handled by the xLLM runtime: turning dynamic requests into execution units that can be replayed stably, and ensuring replay still accesses the fixed addresses recorded during capture. For that reason, xLLM introduces a unified Graph Executor in Graph Mode to centralize bucketing, persistent buffers, graph cache management, and the capture / replay lifecycle.
xLLM's runtime-side foundation work mainly includes:
- **Graph selection and execution scheduling around execution-path stability**
- requests must first be grouped into buckets by `num_tokens` or a nearby shape
- graph cache is maintained per bucket: replay directly on a hit, otherwise capture first and cache the graph
- use full graphs for paths that are fully capturable, and switch to Piecewise Graph when part of the path breaks graph capture
- **Persistent buffer and graph-instance management around address stability**
- dynamic inputs such as tokens, positions, seq_lens, and block_tables cannot be reallocated to arbitrary new addresses before replay; xLLM must write them into persistent buffers first and then update content at fixed addresses
- tensors allocated temporarily during model graph construction also need to be retained with the corresponding graph instance instead of being reclaimed when their local scope ends
- when the shared memory pool is enabled, captures of different shapes can reuse the same underlying physical memory, while the virtual addresses seen by the graph still remain stable
- **A unified Graph Executor abstraction**
- capture, cache, replay, graph-instance management, and backend abstraction all need to be handled centrally by the runtime
- the point of this layer is to pull Graph Mode runtime orchestration out of model code rather than scattering it across layers or operators
In concrete execution, the Graph Executor runtime flow can be summarized as:
1. **Bucket the request**: group the request into a bucket by `num_tokens` or a nearby shape.
2. **Prepare inputs**: write tokens, positions, seq_lens, and block tables into persistent buffers, and update runtime metadata such as `attn_metadata` and `plan_info`.
3. **Choose capture or replay**: replay directly if the bucket already has a cached graph; otherwise capture, cache, and then enter the replay path.
4. **Execute the graph**: run a full graph or a piecewise graph depending on the scenario.
The third requirement in 1.1, "stable operator behavior and result semantics", falls more on model-side and operator-side adaptation, which is covered in 1.3.
### 1.3 Model Adaptation Required for Graph Mode
After the xLLM runtime takes care of bucketing, buffers, and graph-instance management, the model side still needs to satisfy the third requirement from 1.1: operator behavior and result semantics must remain stable across capture / replay and must not break because of host participation, dispatch changes, or launch-shape changes.
The required model adaptations mainly include:
- **Remove operations that are not capture-safe**
- General requirement: the capture path must not contain operations that trigger host-side decisions, implicit host-device synchronization, or extra control-flow changes, such as stream synchronization, logic branches driven by host tensors, or implicit host-device interactions
- Typical example 1: operators may depend on host tensor contents to decide task partitioning or execution mode. For example, ATB [PA](https://www.hiascend.com/document/detail/zh/canncommercial/83RC1/API/ascendtbapi/ascendtb_01_0197.html) and [MLA](https://www.hiascend.com/document/detail/zh/canncommercial/83RC1/API/ascendtbapi/ascendtb_01_0314.html) in Ascend CANN commercial 8.3 can choose different task layouts based on host-side `kv_seq_lens` and `q_seq_lens`
- Typical example 2: a host scalar participates in tensor computation, or the host directly reads tensor data, for example:
```cpp
Tensor a;
Tensor b = a * 0.5; // the scalar is implicitly transferred to the tensor's device, causing a synchronized H2D
torch::Tensor max_of_seq = torch::max(input_params.kv_seq_lens);
max_seq_len_ = std::max(max_of_seq.item<int>(), max_seq_len_); // host reads tensor data
```
- **Adapt operators**
- General requirement: the kernel execution path must remain stable for the same input shape. For one input shape, graph execution should choose the same kernel every time, without mixing prefill, chunked prefill, and decode behavior. At the same time, `grid_dim`, `block_dim`, task count, workspace shape, and tiling content must also remain stable
- Typical example: in Ascend CANN commercial 8.3, ATB PA may decide at runtime whether to enable flash-decoding long-sequence mode based on sequence length and batch size. The long-sequence and short-sequence modes use different tiling keys, and the actual dispatched `kernel_name` is also different. This kind of behavior, where the same business path triggers different kernels under different runtime conditions, must be converged before entering Graph Mode
## 2. Dynamic Dimension Parameterization
### 2.1 Problem
Graph capture records a fixed kernel sequence and the launch parameters of every kernel. At replay time, the runtime can only repeat that recorded launch shape. It cannot re-decide `grid_dim`, `block_dim`, task count, or the tiling path inside the same graph.
As a result, not every dynamic request dimension can continue to vary once the request enters Graph Mode. For attention, the real dynamic factors usually include more than `num_tokens`: they also include `batch_size`, `q_seq_lens`, `kv_seq_lens`, and `block_tables_size`. If these dimensions further affect task partitioning, workspace layout, `tiling_params`, or kernel path, replay may reuse the stale capture-time configuration and become incorrect.
The core question of this chapter is therefore: which dynamic information can still be updated after the graph has been selected, and which dimensions must instead be handled by finer-grained bucketing or by building another graph.
### 2.2 Solution
The goal of dynamic dimension parameterization is not to make one graph accept arbitrary `num_tokens`. Instead, once the graph has already been selected by `bucket_num_tokens`, the goal is to let the same bucket safely cover more real dynamic requests.
#### 2.2.1 Parameterization Boundary
Whether a dimension can still vary at replay time does not depend on whether it is dynamic in the request. It depends on whether that dimension has already been folded into the execution shape during capture.
Dynamic factors can be divided into two categories:
1. **Dimensions that determine graph shape**
- once a dimension enters `grid_dim`, `block_dim`, task count, workspace layout, tiling key, or `tiling_params`, it becomes part of the capture result
- after that, it usually cannot be changed inside the same graph
2. **Dimensions that can still be updated after graph selection**
- such as `batch_size`, `q_seq_lens`, `kv_seq_lens`, `block_tables`, `new_cache_slots`, and `plan_info`
- they are suitable parameterization targets only if they do not further rewrite execution shape
A simplified norm-kernel example is:
```cpp
int grid_dim = num_tokens;
NormKernel<<<grid_dim, block_dim>>>(x, y, ...);
```
Here, `grid_dim` and `block_dim` are both launch parameters. Once `num_tokens` participates in launch configuration, it becomes part of graph shape; after capture, replay can no longer change `grid_dim = 128` into `grid_dim = 256` within the same graph.
So the boundary is straightforward: any dynamic factor that changes launch shape, workspace, or tiling path should not be handled only by updating parameters before replay.
#### 2.2.2 How xLLM Handles It
xLLM handles this in two layers.
The first layer handles `num_tokens`: xLLM uses bucketing and padding to normalize the raw request into a fixed `bucket_num_tokens`, and then selects the graph accordingly.
The second layer handles the remaining dynamic information inside that bucket: adapt operators that depend on host tensors, host planning, or host-side tiling so that request-varying information is stored in device-side persistent buffers, or otherwise externalized into data that can be refreshed before replay, instead of letting replay stay bound to the old host-side planning result from capture.
Take NPU attention as an example. A common pre-adaptation problem is that `q_seq_lens` and `kv_seq_lens` first participate in host-side planning, which produces task, workspace, or tiling-related data before launching the kernel:
```cpp
// Host side
auto plan_info = PlanAttention(q_seq_lens_host, kv_seq_lens_host, ...);
WritePlanToWorkspace(attention_workspace, plan_info);
AttentionKernel<<<grid_dim, block_dim>>>(
q, k, v, out,
attention_workspace,
tiling_params);
```
In this kind of implementation, graph replay does not automatically redo planning, so it can only repeat the launch shape and workspace layout from capture time. The adapted direction is to write request-varying information such as `q_seq_lens`, `kv_seq_lens`, `block_tables_size`, and `plan_info` into device-side persistent buffers under a fixed launch shape, and let the kernel read them directly. For example, a parameterized `kv_seq_lens` path may look like:
```cpp
struct AttentionLaunchArgs {
void* q;
void* k;
void* v;
void* out;
int32_t batch_size;
int32_t* kv_seq_lens; // device buffer
void* attention_workspace;
void* tiling_params;
};
LaunchAttentionKernel(args);
```
If a dynamic factor changes task count, array length, workspace shape, or tiling path as soon as it varies, xLLM does not force it into "parameterization updates". Instead, it uses one of two approaches:
1. expand the bucketing dimensions and include those factors in the graph-selection key, for example a combination of `num_tokens` and `num_comm_tokens`
2. keep that dynamic-shape-sensitive part outside the full graph and handle it with Piecewise Graph
### 2.3 Result
Dynamic dimension parameterization directly brings the following result: one `num_tokens` bucket can safely cover more real requests, and under a stable execution shape, operators such as attention can still see the correct dynamic information at replay time, including `batch_size`, `seq_lens`, `block_tables_size`, and `plan_info`.
One thing to keep in mind is that parameterization does not eliminate bucketing. The dynamic nature of `num_tokens` is still mainly absorbed by graph selection; if a bucket still contains other factors that rewrite tiling keys, task count, communication scale, or kernel path, those factors must still enter the bucketing key or move to Piecewise Graph. Communication-heavy scenarios are a typical example. In Attention DP + MoE EP, communication scale may depend on the maximum DP data size rather than strictly matching local `num_tokens` on one device.
## 3. Piecewise Graph
### 3.1 Problem
In prefill and chunked prefill scenarios, the full forward path is usually more complex than decode. Some operators may:
- fail to be captured stably
- be highly sensitive to dynamic metadata
- rely on extra runtime preparation during capture
If the requirement is that the entire path must be capturable as one full graph, Graph Mode coverage becomes very limited. Once one critical operator breaks graph capture, the whole path loses graph execution.
### 3.2 Solution
#### 3.2.1 Underlying Logic
One common question in implementation is: why can decode use a full graph while chunked prefill usually needs Piecewise Graph?
The key difference is attention.
Under current decode semantics, one step usually means one generated token per sequence, so `num_tokens` and `batch_size` are almost one-to-one. For attention, that means:
- when `num_tokens` grows, `batch_size` grows with it
- the pattern of `q_seq_lens` stays relatively stable
- task partitioning, workspace requirements, and launch arguments can mostly be bucketed together with `num_tokens`
Therefore, when building buckets by `num_tokens`, decode satisfies the reuse conditions for a full graph.
In chunked prefill, however, `num_tokens` and `batch_size` are no longer tied together. The same `num_tokens` may correspond to very different request combinations, for example:
- `num_tokens = 128, batch_size = 2, q_seq_lens = [64, 64]`
- `num_tokens = 128, batch_size = 4, q_seq_lens = [32, 32, 32, 32]`
- `num_tokens = 128, batch_size = 8, q_seq_lens = [16, 16, 16, 16, 16, 16, 16, 16]`
For attention, these requests share the same `num_tokens` but they are not the same shape. Attention task partitioning, local indexing, the lengths of `q_seq_lens` and `kv_seq_lens`, and the corresponding `plan_info` all directly depend on `batch_size`. If the system reuses a previously captured graph only by `num_tokens`, then replay may keep using the old batch partitioning and old task partitioning from capture, which can make the result incorrect.
`batch_size` is hard to parameterize here because it changes more than just one scalar value. It changes the structure of the launch arguments themselves:
- the lengths of `q_seq_lens` and `kv_seq_lens` change
- the task count and task boundaries of attention change
- `plan_info` and workspace layout change
- the indexing and partition logic inside the kernel change
In other words, in chunked prefill, `batch_size` is not a dimension that can be handled by updating a few parameters. It changes the execution shape of the attention part itself. Unless `num_tokens × batch_size`, or even more fine-grained `seq_lens` combinations, are added into bucketing, selecting graphs only by `num_tokens` is not safe.
That is why the current design is better suited to:
- **decode**: attention can be captured together with the full graph
- **chunked prefill**: attention stays outside the graph while the more stable surrounding parts enter Piecewise Graph
#### 3.2.2 Core Idea
The goal of Piecewise Graph is not to force full-graph capture. It is to split one full execution path into several pieces:
- graph-friendly parts: captured into subgraphs
- parts that are not suitable for graph capture: kept in eager mode
At replay time, those pieces are executed in the same order recorded during capture.
This preserves as much Graph Mode benefit as possible even when full-graph capture is not feasible.
The following diagram uses a three-layer Qwen3 decoder as an example. Attention stays as an independent runner, while the non-attention operators between attention calls are packed as tightly as possible into continuous graph pieces.
![Piecewise Graph Diagram](../../assets/piecewise_graph_diagram.svg)
#### 3.2.3 How xLLM Implements It
In xLLM, Piecewise Graph is mainly used for prefill-like scenarios and is controlled by `enable_prefill_piecewise_graph`.
Its core implementation is:
1. use `PiecewiseGraphs` to maintain the replay instruction sequence
2. store capturable segments as graph objects
3. record non-graph attention segments as runners through `AttentionRunner`
4. replay in the original order as `graph -> runner -> graph -> ...`
The key point is how attention is handled:
- during piecewise capture, attention temporarily ends the current graph capture
- attention itself is not executed during capture; instead, the tensors, workspace, and parameters required for attention are recorded
- at replay time, attention runs with the latest `plan_info`, `q_cu_seq_lens`, and `kv_cu_seq_lens`
The direct reason is that, in chunked prefill, attention has dynamic dimensions beyond `num_tokens`, while current graph bucketing is mainly organized by `num_tokens`. Keeping attention outside the graph avoids replay errors where the graph hits by `num_tokens`, but `batch_size` and `seq_lens` have already changed.
So the essence of Piecewise Graph is not to support all dynamic attention logic inside the graph. It is to:
- put the stably capturable parts into graphs
- keep the more dynamic-shape-sensitive attention as a separate execution unit
- preserve the original execution order through one unified instruction sequence
### 3.3 Result
The results and boundaries of this design are reflected in the following aspects.
#### 3.3.1 Applicable Scenarios
- prefill
- chunked prefill
- scenarios with local graph breaks where graph execution benefit is still desired for the overall path
#### 3.3.2 Benefits
- more flexible than all-or-nothing full-graph capture
- significantly expands Graph Mode coverage
- fits model structures where attention is highly dynamic while parts such as MLP remain relatively stable
#### 3.3.3 Limitations
- currently focused mainly on prefill-like paths
- replay depends on correctly updated `attn_metadata.plan_info`
## 4. Multi-Shape Reusable Memory Pool
### 4.1 Problem
In multi-shape scenarios, Graph Mode faces two problems at the same time:
1. **capture-time memory grows linearly with the number of shapes**
- each new shape entering capture usually gets its own independent memory buffer pool
- because Graph Mode requires stable addresses between capture and replay, these buffers often cannot be released in time
- memory usage grows from the expected `max(shape)` to `sum(shape)`
2. **input-side memory is also repeatedly duplicated**
- if each shape keeps its own tokens, positions, seq_lens, block_tables, and related tensors
- then even if the graph itself can be reused, input-side memory still accumulates by shape
So the reusable memory pool is not just about saving memory. It is about supporting dynamic new shapes while making memory usage converge from `sum(shape)` to `max(shape)`.
The concrete goals include:
1. **capture-time memory reuse**: different graphs share the same underlying physical memory
2. **input-side memory reuse**: replays of different shapes share one set of persistent input buffers
3. **no address conflicts**: every captured graph still keeps its own stable virtual address view
4. **on-demand support for new shapes**: a new shape can still be captured without invalidating existing graphs
5. **no extra replay overhead**: most reuse happens during capture and input preparation, while replay stays as unchanged as possible
#### 4.1.1 Root Cause
- on the CUDA path, the allocator typically allocates an independent memory buffer pool for every graph capture
- Graph Mode then requires stable addresses between capture and replay, so buffers from old shapes cannot be reclaimed like ordinary eager temporary memory
- as more shapes enter capture, memory usage grows from the expected `max(shape)` to `sum(shape)`
#### 4.1.2 Constraints
- under the current assumption, graphs of different shapes do not replay simultaneously
- dynamically appearing new shapes must still be supported through on-demand capture
- existing graph caches should not be torn down just because a new shape appears
- replay should not become more complex because of memory reuse
#### 4.1.3 Failed Approach and Technical Challenge
A more direct idea is to reset the allocation pointer before each capture so that different shapes reuse the same virtual address space.
But this is not safe in practice. Allocators usually track allocated blocks by address. If a new capture reuses an old address, it may overwrite the previous address record. Then, when tensors belonging to an old graph are later destructed or freed, the address record may already be gone, which can trigger errors such as `invalid device pointer`.
This reveals the real challenge: what Graph Mode truly needs to keep stable is the address view seen by each graph, while what really needs to be reused is the underlying physical memory. So the problem is not just memory saving. It is how to satisfy all of the following at the same time:
- the virtual address spaces of captured graphs do not conflict
- the underlying physical memory can still be shared across shapes
- adding a new shape into capture does not break replayability of already captured graphs
### 4.2 Solution
The core idea is not to let different shapes share one virtual address range. Instead, it is to let them share one set of physical memory while each captured graph keeps its own virtual address view.
The reason is simple: if different shapes are forced to reuse the same virtual address space, allocator address tracking will conflict, and old graphs may no longer find their original address records when they are released or destroyed. In other words, the thing that must be reused is the underlying physical memory, not the virtual address seen by the graph itself.
To show why xLLM chooses "shared physical memory + independent virtual address spaces", the common solutions can be compared as follows:
| Solution | Memory reuse mechanism | Physical memory outcome | Address-conflict risk / mitigation | Main limitation |
|----------|------------------------|-------------------------|------------------------------------|-----------------|
| vLLM | Shared graph memory pool + capture larger shapes first | may still approach `sum(shape)` | relies on a shared memory pool and free-block management, usually without explicit address conflicts | reuse efficiency depends on capture order and allocator behavior |
| SGLang | Global graph memory pool + capture larger shapes first | may still approach `sum(shape)` | relies on global pool or graph-private pool management to avoid conflicts | memory still grows when there are many distinct shapes |
| xLLM | Map multiple virtual address spaces to one set of physical memory | converges to `max(shape)` | every graph keeps an independent address view, avoiding address-record conflicts at the root | depends on VMM and allocator integration support from the runtime |
xLLM does not reuse the same virtual address space. Instead, each graph keeps its own address view, while the truly shareable part is pushed down into underlying physical memory.
#### 4.2.1 Multi-Shape Input Tensor Reuse
xLLM first solves repeated input-side memory usage. Instead of allocating independent input tensors for each shape, it pre-allocates one set of persistent buffers for the maximum shape and lets different shapes share them.
These buffers usually include:
- `tokens`
- `positions`
- `q_seq_lens`
- `kv_seq_lens`
- `block_tables`
- `new_cache_slots`
- `hidden_states`
Replay-time handling is straightforward:
1. write the actual input of the current request into the prefix of the buffer
2. zero-fill padding regions when needed
3. construct slice views for the actual shape
As a result, input-side memory no longer accumulates by shape and stays at one `max(shape)` allocation.
#### 4.2.2 Capture-Time Memory Reuse
After solving input duplication, xLLM still needs to solve the memory accumulation caused by graph capture itself.
The core design is:
- every capture uses a **new virtual address space**
- different virtual address spaces map to the **same physical memory**
This satisfies two requirements at once:
1. **stable addresses from the graph's point of view**: each captured graph sees its own fixed virtual addresses
2. **only one physical-memory footprint**: the underlying physical memory grows by the maximum demand instead of accumulating by shape
Compared with reusing one virtual address range, this design avoids address-tracking conflicts and therefore works better with existing allocator mechanisms.
This design relies on a unified virtual memory management capability, including:
- reserving a new virtual address space
- creating or expanding underlying physical memory
- mapping the same physical memory into multiple virtual address spaces
- performing unmap, address-space release, and access-permission operations when needed
The key property is that the same physical memory can be mapped into multiple virtual address spaces. That is exactly what makes "different graphs have independent addresses while still sharing the same physical memory" possible.
#### 4.2.2.1 Address Mapping Relation
The core mapping relation for capture-time memory reuse is shown below:
![Shared virtual-to-physical mapping](../../assets/shared_vmm_mapping_diagram.svg)
This diagram shows that, as the shape grows, each new capture switches to a new virtual address space, while all three virtual spaces reuse the same physical memory pool. Different shapes only use different portions of that pool, and the total physical memory converges to `max(shape)`.
### 4.3 Results
#### 4.3.1 Memory Efficiency Metrics
| Metric | Without memory reuse | With memory reuse |
|--------|----------------------|-------------------|
| Physical memory size | `sum(shape)` | `max(shape)` ✅ |
| Virtual memory | multiple copies | multiple copies |
| Stability | address-conflict risk | stable ✅ |
The key change is that physical memory converges to `max(shape)`, while virtual address space still grows with the number of captured shapes.
#### 4.3.2 Performance Impact
- **capture stage**: switching virtual address spaces and extending mappings introduces extra overhead
- **replay stage**: no extra graph execution overhead, and behavior is essentially the same as normal Graph Mode replay
- **input preparation stage**: one extra write into persistent buffers plus slice-view construction, which is still typically cheaper than repeated allocation
#### 4.3.3 Virtual Address Space Overhead
One cost of this design is that virtual address space grows as more shapes are captured. But this growth is address-space usage, not physical memory usage.
On 64-bit systems, virtual address space is usually much larger than single-device memory capacity, so it is typically not the primary bottleneck for this kind of graph capture scenario. In other words, this design uses abundant address space to bring physical memory down from `sum(shape)` to `max(shape)`.
#### 4.3.4 Result Validation
- captures of different shapes can share the same underlying physical memory
- every capture still keeps an independent virtual address space instead of reusing the same address view
- address-conflict issues are no longer exposed through the failed "reuse one virtual address space" path
- memory usage converges from `sum(shape)` to `max(shape)`
#### 4.3.5 Practical Effects
- in multi-shape scenarios, memory behavior changes from "grows linearly with captures" to "physical memory converges to the maximum shape"
- graph reuse becomes more stable because direct reuse of one virtual address space is avoided
- the design supports on-demand capture for new shapes without tearing down existing graph caches
- replay stays unchanged or nearly unchanged, while most added complexity is concentrated in capture and input preparation
#### 4.3.6 Assumptions and Boundaries
- virtual address space still grows with the number of captured shapes, even though physical memory only grows with the maximum demand
- the design depends on runtime support for virtual address management, physical memory mapping, and allocator integration
- it currently assumes that graphs of different shapes do not replay simultaneously; that assumption is what makes physical memory sharing safe
- the current implementation still keeps physical memory and virtual spaces for the full lifecycle instead of doing fine-grained dynamic release
## 5. References
- [ACLGraph Capture / Replay mechanism](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1alpha002/appdevg/acldevg/aclcppdevg_000519.html)
- [CudaGraph capture mechanism](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs)
- [vLLM graph execution reference](https://github.com/vllm-project/vllm/blob/main/vllm/v1/worker/gpu/cudagraph_utils.py)
- [SGLang graph execution reference](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/cuda_graph_runner.py)

27
upstream_ref/xllm/docs/en/dev_guide/code_arch.md Normal file
View File

@@ -0,0 +1,27 @@
# Code Architecture
```
├── xllm/
| : main source folder
│ ├── api_service/ # code for api services
│ ├── core/
│ │ : xllm core features folder
│ │ ├── common/
│ │ ├── distributed_runtime/ # code for distributed and pd serving
│ │ ├── framework/ # code for execution orchestration
│ │ ├── kernels/ # adaption for npu kernels adaption
│ │ ├── layers/ # model layers impl
│ │ ├── platform/ # adaption for various platform
│ │ ├── runtime/ # code for worker and executor
│ │ ├── scheduler/ # code for batch and pd scheduler
│ │ └── util/
│ ├── function_call # code for tool call parser
│ ├── models/ # models impl
│ ├── processors/ # code for vlm pre-processing
│ ├── proto/ # communication protocol
│ ├── pybind/ # code for python bind
| └── server/ # xLLM server
├── examples/ # examples of calling xLLM
├── tools/ # code for npu time generations
└── xllm.cpp # entrypoint of xLLM
```

396
upstream_ref/xllm/docs/en/dev_guide/tilelang_ascend_kernel_dev.md Normal file
View File

@@ -0,0 +1,396 @@
# xLLM Ascend TileLang Kernel Development Guide
This document explains how to add or modify an Ascend TileLang kernel in xLLM. The examples use the current `rope` kernel throughout.
Relevant directories:
- Python kernel definitions: `xllm/xllm/compiler/tilelang/targets/ascend/kernels`
- NPU runtime wrappers: `xllm/xllm/core/kernels/npu/tilelang`
Builds and tests should be run inside the NPU container.
## 1. First Decide What Kind of Change You Are Making
- Add a `specialization`
- Add one more compiled parameter combination to an existing kernel
- Reuse the same wrapper, the same runtime dispatch fields, and the same C ABI
- Typical changes are updates to `DISPATCH_SCHEMA` or `SPECIALIZATIONS`
- Add a `kernel`
- Add a new logical operator
- Typical changes are a new Python kernel file, a new wrapper C++ file, and one CMake registration
For `rope`:
- Adding one more item like `{"variant_key": "...", "head_dim": ..., "rope_dim": ..., "dtype": ...}` to `SPECIALIZATIONS` means adding a new `specialization`
- Adding a new external interface such as `xxx_wrapper.cpp` means adding a new `kernel`
## 2. Development Order
The recommended order is:
1. Implement the TileLang kernel in a Python file such as `rope.py`
2. Implement `generate_source(...)` to lower the kernel into Ascend-C source
3. Declare `DISPATCH_SCHEMA` and `SPECIALIZATIONS`
4. Generate `registry.inc` once and inspect it
5. Then write or update the runtime specialization construction logic in the wrapper
6. Wire it into CMake and run tests
The key idea behind this order is:
- implement the kernel itself first
- then fix the runtime dispatch schema
- then write the wrapper against the generated `registry.inc`
## 3. Write the Python Kernel
Using `rope.py` as the example, the Python side can be understood in three layers:
- `build_rope_kernel(...)`: kernel implementation
- `generate_source(...)`: AOT export
- `RopeKernel`: kernel registration plus dispatch schema and compiled instance declaration
### 3.1 Implement `build_rope_kernel(...)`
`build_rope_kernel(...)` is the actual TileLang kernel implementation. This is where you write:
- `@T.prim_func`
- input and output tensor shapes
- parallel task organization under `with T.Kernel(...)`
- UB allocation and the actual compute logic
The simplified structure in `rope.py` looks like this:
```python
def build_rope_kernel(
head_dim: int,
rope_dim: int,
vec_core_num: int,
ub_buffer_bytes: int,
):
task_num = vec_core_num
m_num = vec_core_num // 2
@T.prim_func
def rope_in_place_kernel(...):
with T.Kernel(m_num, is_npu=True) as (cid, vid):
task_id = cid * 2 + vid
...
return rope_in_place_kernel
```
Here, `head_dim` and `rope_dim` are the compile-time parameters that this implementation actually depends on.
Vector kernels such as `rope` must also follow the fixed-task convention used by the current AOT path. In the current AOT flow, the kernel launch `block_num` is fixed at compile time, which means:
- runtime input shapes do not change the kernel launch `block_num`
- runtime input shapes only change workload splitting across the fixed tasks
The convention in the current `rope.py` is:
```python
task_num = vec_core_num
m_num = vec_core_num // 2
with T.Kernel(m_num, is_npu=True) as (cid, vid):
task_id = cid * 2 + vid
```
This means:
- `cid` ranges over `[0, vec_core_num // 2)`
- `vid` ranges over `[0, 2)`
- the total task count is fixed as `task_num = vec_core_num`
As a result, `rope.py` also derives the compile-time token count for one specialization using the fixed task count:
```python
max_rows_num_in_ub = _derive_max_rows_num_in_ub(...)
compile_num_tokens = task_num * max_rows_num_in_ub
```
### 3.2 Implement `generate_source(...)`
`generate_source(...)` lowers the TileLang kernel above into the final source code. The export layer takes one specialization and turns it into compilable Ascend-C source.
For `rope`, the core logic is:
```python
@staticmethod
def generate_source(head_dim: int, rope_dim: int, dtype: str) -> str:
vec_core_num = detect_vec_core_num()
tilelang_kernel = build_rope_kernel(
head_dim=head_dim,
rope_dim=rope_dim,
vec_core_num=vec_core_num,
ub_buffer_bytes=FIXED_UB_BUFFER_BYTES,
)
with tilelang.tvm.transform.PassContext(...):
kernel = tilelang.engine.lower(tilelang_kernel)
return kernel.kernel_source
```
The rules here are:
- the inputs to `generate_source(...)` come from the current `SPECIALIZATIONS` entry
- `generate_source(...)` calls `build_rope_kernel(...)`
- the return value is the lowered source string
### 3.3 Declare `DISPATCH_SCHEMA` and `SPECIALIZATIONS`
After the kernel implementation and export layer are done, use an `@register_kernel` class to attach the kernel to the framework.
The current minimal template in `rope.py` is:
```python
from ....common.spec import DispatchField, TilelangKernel, register_kernel
@register_kernel
class RopeKernel(TilelangKernel):
DISPATCH_SCHEMA = [
DispatchField("head_dim", "int32"),
DispatchField("rope_dim", "int32"),
DispatchField("dtype", "dtype"),
]
SPECIALIZATIONS = [
{
"variant_key": "hd128_rd128_bf16",
"head_dim": 128,
"rope_dim": 128,
"dtype": "bf16",
},
{
"variant_key": "hd576_rd64_bf16",
"head_dim": 576,
"rope_dim": 64,
"dtype": "bf16",
},
]
@staticmethod
def generate_source(head_dim: int, rope_dim: int, dtype: str) -> str:
...
```
There are two concepts to distinguish here:
- `DISPATCH_SCHEMA`
- defines the field names, order, and types of the runtime specialization
- is the single source of truth for the C++ specialization struct, builder, and lookup interface
- `SPECIALIZATIONS`
- represents the set of instances that will actually be compiled
- each item corresponds to one variant
The rules are:
- every field in `DISPATCH_SCHEMA` must appear in every `SPECIALIZATIONS` item
- `SPECIALIZATIONS` may contain extra fields; those fields are passed into `generate_source(...)`, but do not enter the runtime dispatch schema
- `variant_key` is the unique identifier for that specialization
- `DISPATCH_SCHEMA` and `SPECIALIZATIONS` must match the runtime specialization one-to-one
For `rope`, the runtime dispatch dimensions are:
- `head_dim`
- `rope_dim`
- `dtype`
So these three fields must appear in both:
- `DISPATCH_SCHEMA`
- every `SPECIALIZATIONS` item
At build time, Ascend build resolves the actual `bisheng_arch` from the `--device a2|a3` value passed by the main build path.
### 3.4 Inspect the Generated Ascend-C Source
When debugging the implementation details in `build_rope_kernel(...)`, or comparing how different kernel styles affect the final code generation, use the common `compile-kernels` entry to regenerate artifacts and inspect the Ascend-C source for the specialization you care about.
For `rope`, you can fix:
- `head_dim=576`
- `rope_dim=64`
- `dtype=bf16`
Then regenerate the `rope` artifacts:
```bash
python xllm/compiler/tilelang_launcher.py compile-kernels \
--target ascend \
--device a3 \
--output-root /tmp/tilelang_debug \
--kernels rope \
--force
```
It is recommended to keep `--force` so the source and object files are regenerated from the current code instead of reusing an old cache hit.
This command uses an isolated debug output directory, `/tmp/tilelang_debug`, so only the debug artifacts for `rope` are generated there and they do not get mixed with artifacts from other kernels in the main build directory.
After that, you can directly inspect the generated source for the specialization, including the entry function, UB allocation, and vector compute logic:
```bash
sed -n '1,200p' \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
rg -n 'extern "C"|__global__|alloc_ub|alloc_shared|g_tilingKey' \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
```
To compare two kernel implementations, keep the specialization fixed, run `compile-kernels --force` before and after the change, then diff the generated `.cpp` file:
```bash
cp /tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp \
/tmp/rope_before.cpp
diff -u /tmp/rope_before.cpp \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
```
This helps isolate specialization changes from kernel implementation changes.
After generation, the main files to inspect are:
- `/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp`
- `/tmp/tilelang_debug/targets/ascend/rope/registry.inc`
- `/tmp/tilelang_debug/targets/ascend/rope/manifest.json`
These correspond to:
- the final Ascend-C source for one specialization
- the runtime dispatch interface directly included by the wrapper
- the full compiled artifact record for the current kernel
The recommended debugging sequence is:
1. run `compile-kernels --force` to regenerate the current kernel artifacts
2. inspect the `.cpp` for the specialization and analyze the code generation result
3. inspect `registry.inc` and `manifest.json` to confirm they match expectations
4. finally run `rope_wrapper_test` to check end-to-end behavior and performance
## 4. Update the Wrapper
When adding a new `kernel`, you need a new wrapper. When adding a new `specialization`, the wrapper only needs an update if the runtime specialization semantics change.
For `rope_wrapper.cpp`, the manually written parts should remain:
- tensor shape, dtype, and layout validation
- reshaping inputs into `x_rows / sin_rows / cos_rows`
- constructing the runtime specialization from tensors
- assembling launch arguments and calling `entry->fn(...)`
### 4.1 What `registry.inc` Generates Automatically
`registry.inc` is generated automatically from the Python-side `DISPATCH_SCHEMA`, `SPECIALIZATIONS`, and the exported Ascend-C ABI.
For `rope`, the generated content includes:
- `RopeSpecialization`
- `RopeHeadDim`
- `RopeRopeDim`
- `RopeDType`
- `RopeKernelFn`
- `make_rope_specialization(...)`
- `find_rope_kernel_entry(...)`
- `available_rope_variant_keys()`
For `rope_wrapper.cpp`, `registry.inc` directly provides dispatch-related definitions such as `RopeSpecialization`, `operator==(...)`, and `RopeKernelFn`. Dtype conversion uses the shared helper `to_tilelang_dtype(...)`.
### 4.2 What the Wrapper Actually Needs to Write
The most important handwritten logic in `rope_wrapper.cpp` is constructing the runtime specialization from the tensors. The current code looks like this:
```cpp
RopeSpecialization build_runtime_specialization(const torch::Tensor& x_rows) {
return make_rope_specialization(
RopeHeadDim{static_cast<int32_t>(x_rows.stride(0))},
RopeRopeDim{static_cast<int32_t>(x_rows.size(1))},
RopeDType{to_tilelang_dtype(x_rows.scalar_type())});
}
```
For `rope`:
- `head_dim` maps to `x_rows.stride(0)`, which is the `x_stride` used by the kernel
- `rope_dim` maps to `x_rows.size(1)`
- `dtype` maps to `x_rows.scalar_type()`
The runtime path is:
1. the wrapper reshapes the inputs into `x_rows / sin_rows / cos_rows`
2. `build_runtime_specialization(...)` constructs a specialization from `x_rows`
3. `find_rope_kernel_entry(...)` performs an exact match in the static registry
4. after a match, `entry->fn(...)` calls the actual compiled symbol
The current lookup strategy is a linear scan with exact matching. If any of `head_dim`, `rope_dim`, or `dtype` differs, the lookup will miss.
So when you add a new `specialization`, the main things to cross-check are:
- the field semantics in Python-side `DISPATCH_SCHEMA`
- the field values in Python-side `SPECIALIZATIONS`
- the field values constructed by `build_runtime_specialization(...)` in the wrapper
All three must match exactly.
### 4.3 Generate and Inspect `registry.inc` First
Before writing or modifying wrapper code, generate `registry.inc` once and inspect it. Focus on:
- whether the generated field order in `RopeSpecialization` matches expectations
- whether the generated wrapped field type names match expectations
- the parameter order of `make_rope_specialization(...)`
- the generated entry symbol names
`registry.inc` is the direct contract for the wrapper. Inspect it first, then write the wrapper against it.
## 5. Update CMake
When adding a new `kernel`, register it in `xllm/xllm/core/kernels/npu/tilelang/CMakeLists.txt`.
CMake registration is unified through the high-level helper:
- `tilelang_register_runtime_kernel(NAME <kernel> WRAPPER_SRCS <srcs...>)`
Using `rope` as the example, the minimal template is:
```cmake
tilelang_register_runtime_kernel(
NAME rope
WRAPPER_SRCS rope_wrapper.cpp
)
```
This helper will:
- derive the manifest path as `TILELANG_GENERATED_ROOT/targets/ascend/<kernel>/manifest.json`
- import the manifest
- add the wrapper source and compiled objects into `tilelang_kernels`
- append the `XLLM_TL_<KERNEL>_REGISTRY_INC=...` compile definition automatically
So when adding a new runtime kernel, the CMake-side work mainly consists of two things:
1. make sure the Python side can already generate the manifest for that kernel
2. add one `tilelang_register_runtime_kernel(...)` entry in the TileLang CMakeLists
For day-to-day kernel additions, add one `tilelang_register_runtime_kernel(...)` line directly in CMake. `tilelang_import_kernel_manifest(...)` stays underneath as the implementation base for that higher-level helper.
## 6. Validate
The recommended validation order is:
1. compile the TileLang kernel and inspect the generated `registry.inc`
2. then run the full wrapper test
Common commands:
```bash
python xllm/compiler/tilelang_launcher.py compile-kernels \
--target ascend \
--device a3 \
--output-root build/cmake.linux-aarch64-cpython-311/xllm/compiler/tilelang \
--kernels rope
python setup.py test --test-name rope_wrapper_test --device a3
```
The first command generates `manifest.json`, `registry.inc`, and the object files. The second command validates the full integration path.

29
upstream_ref/xllm/docs/en/features/async_schedule.md Normal file
View File

@@ -0,0 +1,29 @@
# Async schedule
## Background
The inference process of large language models can be divided into three sequential stages:1) CPU-side scheduling (preparing model inputs), 2) Device computation (GPU/TPU execution), 3) CPU-side post-processing (output handling).
Due to the sequential nature of decoding operations, the input for step-i+1 depends on the output of step-i. This forces strict serial execution of all three stages, creating device idle periods (“bubbles”) during CPU-bound stages 1 and 3, leading to suboptimal resource utilization.
## Introduction
xLLM addresses this at the framework level by supporting asynchronous scheduling, where the CPU proactively executes scheduling operations for step-i+1 while the device is computing step-i. This allows the device to immediately begin computing step-i+1 upon completing step-i, thereby eliminating bubbles. Specifically, after initiating the computation call for step-i, the CPU does not wait for the device to finish computing. Instead, it constructs fake tokens for the step-i request, uses these fake tokens to perform scheduling operations for step-i+1 (such as allocating KV Cache), and replaces them with the true tokens computed in step-i when launching step-i+1 computation to ensure correctness. Meanwhile, the CPU processes the results of step-i in a separate thread and returns them to the client.
In the overall architecture, stages 1 and 3 on the CPU side are handled by different thread pools, and RPC function calls employ non-blocking C++ future and promise mechanisms to achieve a fully asynchronous runtime.
![Async schedule](../../assets/async_schedule_architecture.jpg)
## Usage
xLLM provides the gflags parameter `enable_schedule_overlap`, which defaults to false. To enable this feature, set it to true in xLLM's service startup script:
```shell
--enable_schedule_overlap=true
```
## Performance
- With asynchronous scheduling enabled, the device idle time between two steps is approximately 200us - comparable to a single kernel launch duration.
- On the DeepSeek-R1-Distill-Qwen-1.5B model with TPOT constrained to 50ms, this achieves 17% throughput improvement.
## Notice
The asynchronous scheduling feature requires the server to compute one additional step. For use cases involving limited output tokens (e.g., few-token generation) or single-output scenarios like embedding models, enabling this feature is not recommended as it may reduce server-side throughput, thus hard-disabled internally.
The VLM model is currently being adapted, will be temporarily disabled.

5
upstream_ref/xllm/docs/en/features/basics.md Normal file
View File

@@ -0,0 +1,5 @@
# Basics
- xLLM uses a one-device-per-process architecture. Across multiple devices, RPC is used for function calls, and data communication during model computation uses device collective communication libraries.
- HCCL/LCCL are high-performance collective communication frameworks that provide data-parallel and model-parallel collective communication for both single-node multi-device and multi-node multi-device scenarios.

19
upstream_ref/xllm/docs/en/features/chunked_scheduler.md Normal file
View File

@@ -0,0 +1,19 @@
# ChunkedPrefill Scheduler
## Feature Introduction
xLLM supports the chunked prefill scheduling strategy. Chunked prefill is a technique that optimizes large language model inference by splitting long prompts into smaller chunks for batch processing, rather than processing the entire prompt at once.
This method can effectively reduce peak GPU memory usage, improve device utilization, and better schedule and mix processing with requests from the decode stage.
## Usage
The aforementioned strategy has been implemented in xLLM and is exposed through gflags parameters to control the feature's on/off state.
- Enable chunked prefill and set the chunked size, if not set chunked size, its default value is equal to max_tokens_per_batch.
```bash
--enable_chunked_prefill=true
--max_tokens_per_chunk_for_prefill=20480 # optional
```
## Performance Impact
After enabling chunked prefill, on the Qwen3-8B model with a TPOT constraint of 50ms, the TTFT latency **decreased by 46%**.

7
upstream_ref/xllm/docs/en/features/continuous_scheduler.md Normal file
View File

@@ -0,0 +1,7 @@
# Continuous Scheduler
## Feature Introduction
xLLM implements a scheduling strategy that supports continuous batching. Continuous batching is a dynamic batching strategy that does not wait for a batch to be filled. Instead, it starts processing as soon as requests are available, while continuously accepting new requests and adding them to the currently executing batch. This approach significantly reduces latency while maintaining high throughput.
## Usage
xLLM implements the continuous batching scheduling strategy. And, the default scheduler is chunked prefill, `enable_chunked_prefill=true` by default.

83
upstream_ref/xllm/docs/en/features/disagg_pd.md Normal file
View File

@@ -0,0 +1,83 @@
# Disaggregated PD
## Background
LLM online inference services typically need to meet two performance metrics: TTFT and TPOT. Traditional Contiguous Batching scheduling strategies mix Prefill and Decode requests during scheduling, causing Prefill and Decode phases to compete for computational resources. This prevents maximized utilization of computing resources and impacts performance metrics. To resolve this conflict, the Prefill and Decode phases are split to run on independent computational resources, enabling parallel execution. This simultaneously reduces TTFT and TPOT while improving throughput.
## Introduction
The xLLM PD Separation feature is primarily implemented through the following three modules:
- **etcd**: Stores metadata such as instance information.
- **xLLM Service**: Schedules requests and manages all computing instances.
- **xLLM**: Handles request computation instances.
The overall architecture is shown below:
![xLLM PD Separation Architecture](../../assets/pd_architecture.jpg)
## Usage
### Preparation
#### Install Dependencies
- **xLLM**: Refer to [Installation && Compilation](../getting_started/quick_start.md)
- **xLLM Service**: Refer to [PD disaggregation](../getting_started/disagg_pd.md)
#### Obtain Environment Information
Deploying Disaggregated PD Service requires obtaining the Device IP of the machine to create communication resources. Execute the command `cat /etc/hccn.conf | grep address` on the current AI Server to get the Device IP, for example:
```
address_0=xx.xx.xx.xx
address_1=xx.xx.xx.xx
```
`address_xx` represents the Device IP.
### Start Disaggregated PD Service
1. Start etcd
```bash
./etcd
```
2. Start xLLM Service
```bash
ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
3. Start xLLM
- Taking Qwen2-7B as an example
- Start Prefill Instance
```bash
/path/to/xllm --model=Qwen2-7B-Instruct \
--port=8010 \
--devices="npu:0" \
--master_node_addr="127.0.0.1:18888" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=PREFILL \
--etcd_addr="127.0.0.1:12389" \
--transfer_listen_port=26000 \
--disagg_pd_port=7777 \
--node_rank=0 \
--nnodes=1
```
- Start Decode Instance
```bash
/path/to/xllm --model=Qwen2-7B-Instruct \
--port=8020 \
--devices="npu:1" \
--master_node_addr="127.0.0.1:18898" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=DECODE \
--etcd_addr="127.0.0.1:12389" \
--transfer_listen_port=26100 \
--disagg_pd_port=7787 \
--node_rank=0 \
--nnodes=1
```
Important notes:
- PD disaggregation requires reading the `/etc/hccn.conf` file. Make sure this file on the physical machine is mapped into the container.
- `etcd_addr` must match the `etcd_addr` of `xllm_service`
## Notice
Disaggregated PD **does not support** enabling prefix cache or chunked prefill. These features must be disabled using the following parameters:
```shell
--enable_prefix_cache=false
--enable_chunked_prefill=false
```

30
upstream_ref/xllm/docs/en/features/eplb.md Normal file
View File

@@ -0,0 +1,30 @@
# MoE Load Balancing (EPLB)
## Background
MoE models rely on dynamic token routing to distribute tokens among experts. However, in real-world deployments, uneven data distribution leads to expert load imbalance (some overloaded while others idle). Expert redundancy adjustment (e.g., adding/removing replicas) consumes additional GPU memory and may impact inference latency due to weight migration, posing significant implementation challenges. To address this, we employ an expert redundancy strategy (replicating hot experts) combined with hierarchical and global dynamic load balancing to achieve dynamic MoE load balancing.
## Features
The xLLM MoE Load Balancing (EPLB) functionality is implemented through three main modules:
- **EPLB Manager**: Responsible for monitoring expert loads, collecting and managing expert distribution updates. It uses a layer-by-layer update mechanism, determining whether to update each layer based on expert load changes.
- **EPLB Executor**: The actual executor for expert distribution updates.
- **EPLB Policy**: Strategy for generating new expert load tables.
Overall architecture diagram:
![xLLM eplb](../../assets/eplb_architecture.png)
## Usage
Simply add the following gflag parameters when launching xLLM:
(Replace with actual number of devices. `ep_size` must match the number of devices)
- xLLM provides the gflag parameter `enable_eplb` (default: false). Set to true in the xLLM service startup script to enable dynamic expert load balancing.
- `expert_parallel_degree` and `ep_size` are MoE-related parameters. `expert_parallel_degree` should be set to `2`, and `ep_size` must match the actual number of NPU/GPU devices. See [moe_params](./moe_params.md)
- `eplb_update_interval` sets the expert distribution update interval in seconds (default: 1000).
- The expert distribution update uses a layer-by-layer mechanism based on expert load. When the similarity between consecutive loads for a layer is below `eplb_update_threshold`, that layer is updated (default: 1, range: 0-1).
```bash
--enable_eplb=true --expert_parallel_degree=2 --ep_size=16 --eplb_update_interval=2000 --eplb_update_threshold=0.9

42
upstream_ref/xllm/docs/en/features/global_kvcache.md Normal file
View File

@@ -0,0 +1,42 @@
# Global Multi-Level KV Cache
## Background
In the decoding phase of large language models (LLMs), frequent access to historical KV cache due to autoregressive generation creates a bottleneck in memory bandwidth. As model sizes and context windows expand (e.g., 128K Tokens consuming over 40GB of memory), the pressure on single-device memory increases dramatically. Existing solutions (such as vLLM) exhibit significant limitations in long-context scenarios: prefill time surges, severe memory bandwidth contention during decoding, and the need for excessive resource reservation to meet SLO requirements (TTFT < 2s, TBT < 100ms). This often results in GPU utilization below 40% and difficulties in leveraging cross-server resources. To address this, we propose a distributed global multi-level KV cache management system, adopting a memory-compute integrated architecture to break through single-machine resource constraints.
## Feature Introduction
The xLLM Global KV Cache feature is primarily implemented through the following three modules:
- **etcd**: For cluster service registration, load information synchronization, and global cache state management.
- **xLLM Service**: For scheduling requests and managing all compute instances.
- **xLLM**: The compute instances handling requests.
The overall architecture is shown in the diagram below:
![xLLM Global Multi-Level KV Cache](../../assets/globalkvcache_architecture.png)
## Usage Example
### Preparation
#### Install Dependencies
- **xLLM**: Refer to [Quick Start](../getting_started/quick_start.md)
- **xLLM Service**: Refer to [PD disaggregation](../getting_started/disagg_pd.md)
### Usage Instructions
1. **etcd Startup Configuration:**
```bash
./etcd --listen-peer-urls=http://0.0.0.0:10999 --listen-client-urls=http://0.0.0.0:10998
```
2. **xLLM Service Startup Configuration:**
```bash
./xllm_master_serving --etcd_addr="127.0.0.1:10998" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
3. **xLLM Startup Configuration:**
Add the following gflag parameters when starting xLLM:
```bash
--enable_service_routing=true
--enable_cache_upload=true
# PD separation currently does not support Global KVCache Management
--enable_disagg_pd=false
```

79
upstream_ref/xllm/docs/en/features/graph_mode.md Normal file
View File

@@ -0,0 +1,79 @@
# Graph Mode
## Overview
xLLM supports Graph Mode: computation graphs are pre-captured and replayed in subsequent runs to reduce CPU overhead and improve inference performance. Graph Mode has corresponding implementations on different hardware platforms.
## Feature Description
To optimize Host-side scheduling, graph mode submits a large task from the CPU once and then executes small kernels in a streaming manner on the device, significantly reducing startup time and device bubbles.
In the xLLM engine, Graph Mode provides the following:
### Dynamic Shape Parameterization
- Key dynamic dimensions other than num_tokens are treated as whole-graph input parameters, including batch_size, kv_seq_lens, q_seq_lens, block_table_size, and the like, for flexibility. During memory allocation and kernel configuration. At graph launch, the actual values of these parameters are passed so kernels use the correct strides to access data.
### Piecewise Graph
- When some operators do not support graph capture and thus break the full graph, each segment (piece) after the break is captured as a separate graph. This maximizes graph-mode benefits even when the full graph cannot be captured, and is commonly used for prefill and chunked prefill.
### Multi-Shape Reusable Memory Pool
- To avoid waste from separate memory buffers (input, output, and intermediate tensors) per shape, we use an expandable memory pool. Multiple shapes share the pool base address, with different shapes using different offsets from that base.
## Usage
These capabilities are implemented inside the xLLM engine and are generally controlled through gflags.
The minimal configuration only needs `enable_graph` to turn on Graph Mode for the decode phase:
```shell
--enable_graph=true
```
Common companion flags include:
- `enable_graph`: enables the base Graph Mode capability for the decode phase
- `enable_prefill_piecewise_graph`: enables Piecewise Graph for the prefill phase
- `enable_graph_mode_decode_no_padding`: builds decode graphs with the actual `num_tokens` instead of the padded shape
- `max_tokens_for_graph_mode`: limits the maximum number of tokens covered by Graph Mode; `0` means no limit
If you want to enable both decode Graph and prefill Piecewise Graph, use:
```shell
--enable_graph=true \
--enable_prefill_piecewise_graph=true \
--max_tokens_for_graph_mode=2048
```
If you need decode graph capture without padding, add:
```shell
--enable_graph=true \
--enable_graph_mode_decode_no_padding=true
```
For a more complete description of the flags, see [CLI Reference](../cli_reference.md).
## Performance Impact
- With Graph Mode enabled, decode-phase throughput **improves by about 8%–10%** on models such as Qwen3-0.6B and Qwen3-1.7B.
## Model Support
The following table lists each model’s support on ACLGraph, CudaGraph, and MLUGraph.
| Model | ACLGraph | CudaGraph | MLUGraph |
|------|----------|-----------|----------|
| Qwen3/Qwen3-MoE | ✅ | ✅ | ✅ |
| DeepseekV3.2 | ✅ | | ✅ |
| GLM4.5/4.6/4.7 | ✅ | | |
| Qwen2.5-VL | | | ✅ |
| Qwen3-VL/Qwen3-VL-MoE | ✅ | | |
| GLM4V | ✅ | | |
| GLM4V-MoE | ✅ | | |
!!! warning "Adding Graph Mode support for new models"
Ensure that the kernels used in the computation implement dynamic dimension parameterization; otherwise the graph may break and kernels may need to be re-implemented.
## Related Documentation
- For more detailed Graph Mode design and implementation notes, including ACL Graph / CUDA Graph fundamentals, dynamic dimension parameterization, Piecewise Graph, and multi-shape memory reuse, see: [Graph Mode Design Document](../design/graph_mode_design.md)

40
upstream_ref/xllm/docs/en/features/groupgemm.md Normal file
View File

@@ -0,0 +1,40 @@
# GroupGEMM Operator Optimization
## Background
The Mixture of Experts (MoE) architecture has become an important paradigm for scaling large language models. Its core idea is to dynamically route input tokens to different expert sub-networks for processing. During inference, the GroupGEMM operator is a key computational unit in the MoE architecture, responsible for efficiently executing parallel computations of multiple expert matrix multiplications, which dominate the overall inference time.
## Function Introduction
Considering the current performance bottleneck of GroupGEMM being I/O constrained, an optimization scheme is proposed that replaces data copying with index reordering, eliminating multiple copies of the token vector and instead maintaining an index table for expert allocation. Through this line number index, tokens are directly mapped to the corresponding expert computation units, and the allocation scheduling of tokens is fused with matrix multiplication into a single kernel.
## User Interface
### Operator Direct Call API
```c++
aclnnStatus aclnnIndexGroupMatmulGetWorkspaceSize(
const aclTensorList *x,
const aclTensorList *weight,
const aclTensorList *scale,
const aclTensorList *perTokenScale,
const aclTensor *groupList,
const aclTensorList *out,
uint64_t *workspaceSize,
aclOpExecutor **executor);
aclnnStatus aclnnIndexGroupMatmul(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream);
```
- `x`: The input tensor list containing the data to be processed.
- `weight`: The weight tensor containing the model parameters.
- `scale`: The scaling factor used to adjust the values of the input tensors.
- `perTokenScale`: The scaling factor for each token, used for dynamic adjustment.
- `groupList`: The list of expert groups indicating which experts participate in the computation.
- `out`: The output tensor list that stores the computation results.
## Performance Effects
![groupmatmul](../../assets/groupmatmul_performance.png)
* The optimized GroupMatmul operator shows significant advantages in computation time, especially when \( k = 128 \) and \( m = 64 \). As shown in the figure, the computation delay of the optimized operator is **reduced by 50%**.

24
upstream_ref/xllm/docs/en/features/moe_params.md Normal file
View File

@@ -0,0 +1,24 @@
# EP Parallelism
## Background
When deploying the DeepSeek-R1 671B parameter-scale model, traditional distributed deployment faces core bottlenecks including low GPU memory utilization, high communication overhead, and expensive hardware costs. Therefore, Expert Parallelism (EP) is introduced.
Key advantages:
+ With the same resources, fewer Experts per GPU means more memory available for KV Cache, allowing more tokens to be cached.
+ Due to MLA characteristics, smaller TP Size with the same resources means less redundant KV Cache, enabling more tokens to be cached.
+ Large-scale EP deployment can concentrate token computations for the same expert on the same device, improving hardware utilization.
## Parameter Configuration
+ **dp_size**: Sets the data parallelism scale for the Attention part. Default: 1, can be set to powers of 2. When dp_size doesn't equal the total number of devices, TP parallelism is used within the DP group.
+ **ep_size**: Sets the expert parallelism scale for the MoE part. Default: 1, can be set to powers of 2. When ep_size doesn't equal the total number of devices, TP parallelism is used within the DP group.
+ **expert_parallel_degree**: EP-related parameter. Default is 0 when EP is disabled. When EP is enabled, default is 1 (EP Level 1). When ep_size equals the total number of devices, can be set to 2 to enable EP Level 2.
## Solution Design
+ When EP is enabled (default EP Level 1), after computing both Attention and MoE parts, data is sent to the next stage via All Gather communication across all devices. Example for 64 devices (Attention: dp32tp2, MoE: ep32tp2):
![Alt text](../../assets/moe_eplevel1.jpg)
+ When ep_size equals the total number of devices, EP Level 2 can be enabled. Communication between Attention and MoE parts changes to ALL2ALL, sending data only to required devices to reduce communication volume and overhead. Example for 64 devices:
![Alt text](../../assets/moe_eplevel2.jpg)

149
upstream_ref/xllm/docs/en/features/mtp.md Normal file
View File

@@ -0,0 +1,149 @@
# MTP Speculative Inference
## Background
MTP (Multi-Token Prediction) is an innovative inference acceleration technique that addresses efficiency bottlenecks in large language model generation. By incorporating specialized pre-training designs, MTP provides efficient draft token prediction capabilities during inference, significantly improving generation speed. Its core value lies in balancing inference efficiency with output quality, offering an optimal solution for long-sequence generation problems in LLMs, ultimately optimizing inference performance.
## Key Features
MTP offers the following core acceleration capabilities:
- **Efficient Draft Generation**: Uses a lightweight MTP architecture to rapidly generate draft tokens that serve as input for the main model's verification, dramatically reducing computation overhead compared to traditional autoregressive generation.
- **Batch Verification Mechanism**: The main model can simultaneously verify multiple MTP-generated draft tokens in batch, rather than processing them sequentially, significantly boosting inference speed.
- **High Sampling Accuracy**: MTP solves the critical pain point of low token acceptance rates in post-training draft modules (like Eagle and Medusa). By optimizing draft generation during pre-training, MTP produces tokens with higher accuracy, reducing the verification burden on the main model.
- **Reduced Inference Latency**: By pre-generating multiple potential subsequent tokens, MTP effectively decreases cumulative latency during long-text generation, creating a smoother user experience.
- **Optimized Resource Consumption**: Compared to other inference acceleration techniques, MTP maintains acceleration effects while requiring fewer additional computational resources, making it suitable for deployment in resource-constrained environments.
MTP technology provides a novel efficiency optimization solution for LLM inference, particularly well-suited for real-time applications requiring rapid responses, representing an important direction in language model inference optimization.
!!! note "Model Support"
Currently supports MTP structure export for the following models:
- DeepSeek-V3 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v3_mtp)
- DeepSeek-V3.2 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v32_mtp)
- DeepSeek-R1 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v3_mtp)
- GLM4 MoE (e.g., GLM-4.5-Air, exported MTP model_type: glm4_moe_mtp)
Note:
- DeepSeek V3 and R1 both have input model_type "deepseek_v3", and the exported MTP model will have model_type "deepseek_v3_mtp"
- DeepSeek V3.2 has input model_type "deepseek_v3" (but can be auto-detected by index_head_dim fields), and the exported MTP model will have model_type "deepseek_v32_mtp"
## Usage Example
### Export Model
The script will automatically detect the model type, or you can manually specify it.
#### DeepSeek-V3
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-V3 \
--output-dir /path/to/DeepSeek-V3-mtp
```
#### DeepSeek-V3.2
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-V3.2 \
--output-dir /path/to/DeepSeek-V3.2-mtp
```
#### DeepSeek-R1
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-R1 \
--output-dir /path/to/DeepSeek-R1-mtp
```
#### GLM4 MoE
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/GLM-4.5-Air \
--output-dir /path/to/GLM-4.5-Air-mtp
```
#### Manually Specify Model Type
If auto-detection fails, you can manually specify the model type:
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/model \
--output-dir /path/to/model-mtp \
--model-type deepseek_v3 # Options: deepseek_v3 (for V3/R1), deepseek_v32 (for V3.2), glm4_moe
```
Input model references:
- [DeepSeek-V3](https://huggingface.co/deepseek-ai/DeepSeek-V3)
- [DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)
- [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1)
- [GLM-4.5-Air](https://huggingface.co/zai-org/GLM-4.5-Air)
### Launch Script
When using MTP for inference, you need to specify both the main model and the draft model (MTP model).
#### DeepSeek-V3/V3.2/R1 Launch Example
```bash
MODEL_PATH="/models/DeepSeek-V3"
DRAFT_MODEL_PATH="/models/DeepSeek-V3-mtp"
MASTER_NODE_ADDR="127.0.0.1:42123"
START_PORT=13222
START_DEVICE=0
LOG_DIR="log"
NNODES=16
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
nohup ./xllm \
--model $MODEL_PATH \
--devices="npu:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--draft_model $DRAFT_MODEL_PATH \
--draft_devices="npu:$DEVICE" \
--num_speculative_tokens 1 \
--max_memory_utilization=0.90 \
--max_tokens_per_batch=10000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--ep_size=1 \
--dp_size=1 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--node_rank=$i > $LOG_FILE 2>&1 &
sleep 0.5
done
```
#### GLM4 MoE Launch Example
```bash
MODEL_PATH="/models/GLM-4.5-Air"
DRAFT_MODEL_PATH="/models/GLM-4.5-Air-mtp"
# ... same other configurations
```
# Performance Data
Based on ShareGPT dataset with input length=2500, output length=1500, total requests=80.
| method | Concurrency | Mean TPOT(ms) | Mean TTFT(ms) | Output Tokens/s | Total Tokens/s |
|:---------:|:-----------:|:-------------:|:-------------:|:---------------:|:--------------:|
| baseline | 1 | 40.61 | 141.80 | 24.20 | 65.77 |
| mtp | 1 | 28.33 | 142.35 | 35.19 | 95.52 |
| baseline | 2 | 42.69 | 178.59 | 45.16 | 122.74 |
| mtp | 2 | 29.81 | 187.97 | 64.75 | 175.78 |
| baseline | 4 | 46.18 | 172.34 | 79.83 | 216.96 |
| mtp | 4 | 33.54 | 194.22 | 111.18 | 301.81 |
| baseline | 8 | 53.16 | 181.49 | 110.68 | 300.81 |
| mtp | 8 | 40.99 | 203.37 | 154.46 | 419.34 |
| baseline | 16 | 68.50 | 213.89 | 143.81 | 390.84 |
| mtp | 16 | 57.04 | 254.99 | 201.89 | 548.04 |
| baseline | 20 | 74.72 | 228.80 | 154.77 | 420.65 |
| mtp | 20 | 61.73 | 264.34 | 206.24 | 559.84 |
| baseline | 40 | 119.68 | 559.32 | 180.22 | 489.80 |
| mtp | 40 | 105.70 | 544.54 | 252.91 | 686.74 |
| baseline | 80 | 180.89 | 2996.21 | 192.09 | 522.06 |
| mtp | 80 | 152.19 | 2163.72 | 278.07 | 755.12 |

31
upstream_ref/xllm/docs/en/features/multi_streams.md Normal file
View File

@@ -0,0 +1,31 @@
# Multi-stream parallel
## Background
In distributed inference scenarios for large-scale models, additional communication operations are required to aggregate computation results from different devices. Taking large-scale MoE models like Deepseek as an example, the distributed scale is typically substantial, leading to increased communication overhead.
If both computation and communication are performed on the same stream, the device’s computing resources will remain idle while waiting for communication to complete, resulting in wasted computational capacity before subsequent calculations can begin.
## Introduction
xLLM implements multi-stream parallelism at the model layer, where the input batch is split into 2 micro-batches. One stream handles computation for the first micro-batch, another concurrently executes communication for the second micro-batch.
This overlap of computation and communication effectively hides the communication latency.![multi_streams_parallel](../../assets/multi_streams_architecture.jpg)
## Usage
xLLM provides the gflags parameter `enable_multi_stream_parallel`, which defaults to false. To enable this feature, set it to true in xLLM’s service startup script, as:
```shell
--enable_multi_stream_parallel=true
```
## Performance
With prefill dual-stream parallelism enabled, it can effectively mask over 75% of communication overhead.
On the DeepSeek-R1 model, when generating just 1 token, this achieves:
- **7%** reduction in TTFT.
- **7%** throughput improvement.
## Notice
The dual-stream parallelism currently only supports the prefill phase, with greater performance benefits observed for longer input requests.
Only Support DeepSeek, Qwen3 dense(non-MoE) models.

51
upstream_ref/xllm/docs/en/features/overview.md Normal file
View File

@@ -0,0 +1,51 @@
# Overall Architecture
## Backgroud
In recent years, with the groundbreaking progress of large language models (LLMs) ranging from tens of billions to trillions of parameters (such as GPT, Claude, DeepSeek, LLaMA, etc.) in the fields of natural language processing and multimodal interaction, the industry has an urgent need for efficient inference engines and service systems. How to reduce cluster inference costs and improve inference efficiency has become a key challenge for achieving large-scale commercial deployment.
Although a number of optimization engines for large model inference have emerged, several technical bottlenecks remain in practical deployment:
* **Hardware Adaptability Challenges:** Existing inference engines lack sufficient support for the architectural characteristics of specialized accelerators like domestic chips, making it difficult to fully exploit the performance potential of heterogeneous computing hardware, leading to low computational resource utilization.
* **MoE Architecture Optimization Difficulties:** The token distribution process in expert parallelism mechanisms generates significant All-to-All communication overhead, while dynamic routing strategies cause expert load imbalance, severely constraining system scalability.
* **Long Context Management Bottlenecks:** As model context windows continue to expand, the efficiency of optimizations in KV cache handling—such as memory fragmentation management and cross-node synchronization—directly impacts overall inference throughput performance.
* **Hybrid Deployment Efficiency Limitations:** Existing inference clusters struggle to simultaneously guarantee service quality (SLO) and optimize resource utilization when handling both online services and offline tasks.
* **Insufficient Dynamic PD Adaptation:** When input/output sequence lengths fluctuate drastically, static PD resource partitioning lacks the ability to adjust PD resource configurations in real-time. This can lead to idle GPU resources and poses a risk of SLO violations.
To address these challenges, we present xLLM—an efficient and user-friendly open-source intelligent inference framework that provides enterprise-grade service guarantees and high-performance engine computing capabilities for model inference on domestic chips.
## Feature Introduction
xLLM provides intelligent computing capabilities, implementing joint inference acceleration across multiple computational system layers and algorithm-driven layers:
### Computational System Layer
#### Multi-Level Pipeline Execution Orchestration
Asynchronizes CPU scheduling at the framework layer to pipeline it with chip inference computation, reducing computation bubbles; at the model graph layer, splits single batches to create pipelines between micro-batches, overlapping computation and communication; at the operator kernel layer, pipelines different computing units, overlapping computation and memory access.
#### Dynamic Shape Graph Execution Optimization
Addressing the static graph adaptation problem for large language models processing dynamic inputs (e.g., variable sequence lengths and batch sizes), xLLM achieves dynamic adaptation through parametric design capturing input dimensions. It combines a multi-graph caching scheme to reduce compilation overhead and uses a managed memory pool instead of absolute addresses to ensure safe reuse, ultimately achieving high execution efficiency while maintaining high flexibility.
#### Operator Optimization
xLLM implements specific optimizations for key operators in LLMs on domestic hardware chips, including GroupMatmul, Chunked Prefill, and others.
#### xTensor Memory Management
The xTensor memory management framework employs a method of *pre-allocated physical memory page pools + contiguous virtual address mapping*. It achieves efficient dynamic memory management through dynamic on-demand mapping of physical pages, reuse of reusable memory pages (Reusable), and optimized scheduling with asynchronous pre-mapping. Combined with NPU operator adaptation (such as virtual address FlashMLA), it results in improved memory utilization and reduced latency.
### Algorithm-Driven Layer
#### PD Separation
xLLM fully supports PD separation scenarios, enabling efficient management of PD instances, communication between PD instances, and KV cache transfer.
#### Global Scheduling
xLLM provides intelligent, full-lifecycle resource scheduling management for requests and instances.
##### Instance Scheduling
We have implemented various instance scheduling strategies to select how to assign instances to more suitable ones. These include a simple Round Robin strategy, a prefix cache-aware strategy based on the prefix cache hit rate of requests on each instance, and a KV cache-aware strategy based on the free memory level of instances. Furthermore, for PD separation scenarios, where static PD ratios often struggle with traffic fluctuations and sudden changes in request input/output lengths, we implemented an adaptive PD dynamic scheduler responsible for global instance allocation for online requests and runtime PD dynamic adjustment.
##### Request Scheduling
We have implemented various request scheduling strategies supporting continuous batching, including chunked prefill, prefill priority, decode priority, and other batch strategies, all while fully supporting PD separation scenarios.
#### Global KV Cache Management
Utilizes ETCD as a metadata service middleware at the global level for cluster service registration, load information synchronization, and global cache state management. Each compute instance maintains a local multi-level cache pool. Regarding scheduling strategy, the system adopts a dynamic decision-making mechanism based on KV cache: it first performs prefix matching detection, calculates the KV cache reuse rate of candidate nodes, and finally selects the node with the optimal comprehensive performance for processing, achieving dynamic offloading and migration of KV cache.
#### Speculative Inference
xLLM incorporates an optimized speculative inference algorithm that generates multiple tokens at once to boost throughput. xLLM reduces communication costs by sinking the speculative module and optimizes speculative inference computation through methods like overlapping scheduling and computation timelines and reducing operator data movement in speculative scenarios.
#### MoE Load Balancing
xLLM implements expert weight updates based on historical expert load statistics for MoE models. During inference, it achieves effective dynamic load balancing through efficient expert load statistics and double-buffered, seamless expert weight updates.
### Multimodal Support
xLLM provides comprehensive support for various multimodal models, including Qwen2-VL and MiniCPMV.

36
upstream_ref/xllm/docs/en/features/ppmatmul.md Normal file
View File

@@ -0,0 +1,36 @@
# PpMatmul Operator Optimization
## Background
In the inference of large models, matrix multiplication accounts for a high proportion and takes a long time. We have optimized the implementation of the matrix multiplication operator.
## Feature Introduction
The PpMatmul operator uses a Tiling strategy to decompose matrix multiplication into multiple smaller matrix multiplication tasks. However, when the number of tiles is small, tasks cannot be evenly distributed across all NPU cores, leading to the tail effect problem, which affects computational efficiency. We optimize the performance of the PpMatmul operator by prefetching memory or redistributing tasks.
## User Interface
### Operator Direct Call API
```cpp
aclnnStatus aclnnPpMatmulOptGetWorkspaceSize(
const aclTensor *a,
const aclTensor *b,
const aclTensor *out,
uint64_t *workspaceSize,
aclOpExecutor **executor);
aclnnStatus aclnnPpMatmulOpt(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream);
```
- `a`: Input matrix A.
- `b`: Input matrix B.
- `out`: Output matrix, storing the computation result.
## Performance Effect
For cases with a small number of tiles (e.g., when M is small, corresponding to a small batch size), there is an **18%** performance improvement of the operator compared to before optimization when (TP=4).

19
upstream_ref/xllm/docs/en/features/prefix_cache.md Normal file
View File

@@ -0,0 +1,19 @@
# Prefix Cache Optimization
## Feature Introduction
xLLM supports prefix cache matching. The prefix cache is based on `murmur_hash` and uses an LRU eviction policy, delivering superior matching efficiency and increased prefix cache hit rates.
Additionally, the prefix cache has been optimized to support the `continuous_scheduler`, `chunked_scheduler`, and `zero_evict_scheduler`. The cache is updated immediately after prefill operations, enhancing matching timeliness. For the `chunked_scheduler`, multi-stage chunked prefill matching is supported, reducing computational overhead and minimizing KV cache usage as much as possible.
## Usage
The prefix cache is implemented in xLLM and exposed through gflags parameters to control its functionality.
- Enable prefix cache with specific policy and settings:
```
--enable_prefix_cache=true
```
## Performance Impact
After enabling prefix cache, on the Qwen3-8B model with a TPOT constraint of 50ms, the E2E latency **decreased by 10%**.
!!! warning "Note"
PD separation scheduler is not currently supported.

27
upstream_ref/xllm/docs/en/features/topk_topp.md Normal file
View File

@@ -0,0 +1,27 @@
# Topk & Topp Operator Optimization
## Background
In natural language generation tasks, the topK and topP sampling strategies are widely used to control the diversity and quality of generated text. However, in small models, the computation time for these two strategies is relatively long. This is mainly due to the fewer parameters in small models, which leads to reduced efficiency in sorting and filtering when processing probability distributions, thereby affecting generation speed. Therefore, optimizing the implementation of topK and topP in small models can enhance their sampling efficiency.
## Feature Introduction
The implementation of the topKtopP operator merges multiple small operators, such as sorting, topK, softmax, and topP, into a single large operator, thereby improving computational efficiency and performance.
## User Interface
### Operator Call API
```c++
void top_k_top_p(torch::Tensor& logits,
const torch::Tensor& topK,
const torch::Tensor& topP);
```
- `logits`: The input logits tensor containing the model's output scores.
- `topK`: The threshold tensor for selecting the top K probabilities.
- `topP`: The threshold tensor for selecting the cumulative probabilities.
## Performance Effect
* After using the topKtopP fused operator, in the qwen2-0.5B model, TTOT **decreased by 37%**, and TTFT **increased by 10%**.

54
upstream_ref/xllm/docs/en/features/xllm_service_overview.md Normal file
View File

@@ -0,0 +1,54 @@
# xLLM Service
<p align="right">[:simple-github: xLLM Service](https://github.com/jd-opensource/xllm-service)</p>
## Project Overview
**xLLM-service** is a service-layer framework developed based on the **xLLM** inference engine, providing efficient, fault-tolerant, and flexible LLM inference services for clustered deployment.
xLLM-service targets to address key challenges in enterprise-level service scenarios:
- How to ensure the SLA of online services and improve resource utilization of offline tasks in a hybrid online-offline deployment environment.
- How to react to changing request loads in actual businesses, such as fluctuations in input/output lengths.
- Resolving performance bottlenecks of multimodal model requests.
- Ensuring high reliability of computing instances.
#### Background
LLM with parameter scales ranging from tens of billions to trillions are being rapidly deployed in core business scenarios such as intelligent customer service, real-time recommendation, and content generation. Efficient support for domestic computing hardware has become a core requirement for low-cost inference deployment. Existing inference engines struggle to effectively adapt to the architectural characteristics of dedicated accelerators like domestic chips. Performance issues such as low utilization of computing units, load imbalance and communication overhead bottlenecks under the MoE architecture, and difficulties in kv cache management have restricted the efficient inference of requests and the scalability of the system. The xLLM-service + xLLM inference engine improves the efficiency of the entire performance link and it provides crucial technical support for the large-scale implementation of LLM in real-world business scenarios.
---
## Overall Architecture
The overall architecture of xLLM-service is shown in the figure below:
![1](../../assets/service_arch.png)
## Core Components
### ETCD Cluster
It is used for metadata management, including the storage and management of metadata such as models, xllm instances, and requests. It also provides xllm node registration and discovery services.
### Fault Tolerance
xLLM-service provides fault tolerance management to ensure service quality and stability.
### Global Scheduler
It implements globally aware scheduling. Based on the current system status, it accurately dispatches requests to the optimal instances for execution, effectively improving the overall service response efficiency and resource utilization.
### Global KV Cache Manager
It is responsible for global KV Cache management. Its core capabilities include distributed KV cache awareness, Prefix matching, and dynamic migration of KV Cache, which optimize the efficiency of cache resource usage.
### Instance Manager
It focuses on the full-lifecycle management of instances. All xllm instances must register to service after startup. Based on preset policies, the module provides support for instances such as scheduling adaptation and fault tolerance handling.
### Event Plane
As the metrics and event hub, it receives Metrics data reported by various instances, uniformly collects and organizes statistical indicators, and provides data support for decisions such as service scheduling, fault tolerance, and scaling.
### Planner
It undertakes the functions of strategy analysis and decision-making. Based on the Metrics data reported by the Event Plane (including instance runtime indicators, machine load indicators, etc.), it analyzes the service scaling needs and the necessity of expanding hot instances, and outputs resource adjustment and instance optimization strategies.

17
upstream_ref/xllm/docs/en/features/zero_evict_scheduler.md Normal file
View File

@@ -0,0 +1,17 @@
# Zero Evict Scheduler
## Feature Introduction
xLLM supports the zero evict scheduling strategy. The zero evict scheduling strategy is an algorithm designed to minimize request eviction rates, reducing the need for prefill computation on evicted requests and consequently improving TPOT (Time Per Output Token).
This scheduling algorithm employs simulation rounds to detect whether a request can be scheduled without causing the eviction of other requests.
## Usage
The aforementioned strategy has been implemented in xLLM and is exposed through gflags parameters to control the feature's on/off state.
- Enable the zero evict strategy and set the maximum decode tokens per sequence.
```
--use_zero_evict=true
--max_decode_token_per_sequence=256
```
## Performance Impact
After enabling zero evict, on the Qwen3-8B model with an E2E latency constraint, the TPOT latency **decreased by 27%**.

99
upstream_ref/xllm/docs/en/getting_started/disagg_pd.md Normal file
View File

@@ -0,0 +1,99 @@
# PD disaggregation
`xllm` supports PD disaggregation deployment, which requires integration with our other open-source library [xllm service](https://github.com/jd-opensource/xllm-service).
## xLLM Service Dependencies
First, download and install `xllm service`, similar to installing and compiling `xllm`:
```bash
git clone https://github.com/jd-opensource/xllm-service
cd xllm_service
git submodule init
git submodule update
```
### etcd Installation
`xllm_service` compilation and operation depend on [etcd](https://github.com/etcd-io/etcd).Use the [installation script](https://github.com/etcd-io/etcd/releases) provided by etcd for installation. The default installation path provided by the script is `/tmp/etcd-download-test/etcd`. You can either manually modify the installation path in the script or manually migrate after running the script:
```bash
mv /tmp/etcd-download-test/etcd /path/to/your/etcd
```
### xLLM Service Compilation
Apply patch:
```bash
sh prepare.sh
```
Then execute the compilation:
```bash
mkdir -p build
cd build
cmake ..
make -j 8
cd ..
```
!!! warning "Potential Errors"
You may encounter installation errors related to `boost-locale` and `boost-interprocess`: `vcpkg-src/packages/boost-locale_x64-linux/include: No such file or directory`, `/vcpkg-src/packages/boost-interprocess_x64-linux/include: No such file or directory`
Reinstall these packages using `vcpkg`:
```bash
/path/to/vcpkg remove boost-locale boost-interprocess
/path/to/vcpkg install boost-locale:x64-linux
/path/to/vcpkg install boost-interprocess:x64-linux
```
## PD Disaggregation Execution
Start etcd:
```bash
./etcd-download-test/etcd --listen-peer-urls 'http://localhost:2390' --listen-client-urls 'http://localhost:2389' --advertise-client-urls 'http://localhost:2391'
```
Start xllm service:
```bash
ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
Taking Qwen2-7B as an example:
- Start Prefill Instance
```bash
/path/to/xllm --model=path/to/Qwen2-7B-Instruct \
--port=8010 \
--devices="npu:0" \
--master_node_addr="127.0.0.1:18888" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=PREFILL \
--etcd_addr=127.0.0.1:12389 \
--transfer_listen_port=26000 \
--disagg_pd_port=7777 \
--node_rank=0 \
--nnodes=1
```
- Start Decode Instance
```bash
/path/to/xllm --model=path/to/Qwen2-7B-Instruct \
--port=8020 \
--devices="npu:1" \
--master_node_addr="127.0.0.1:18898" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=DECODE \
--etcd_addr=127.0.0.1:12389 \
--transfer_listen_port=26100 \
--disagg_pd_port=7787 \
--node_rank=0 \
--nnodes=1
```
Important notes:
- PD disaggregation requires reading the `/etc/hccn.conf` file. Make sure this file on the physical machine is mapped into the container.
- `etcd_addr` must match the `etcd_addr` of `xllm_service`
The test command is similar to above. Note that the `PORT` in `curl http://localhost:{PORT}/v1/chat/completions ...` should be the `port` of the `http_server_port` of xLLM service.

125
upstream_ref/xllm/docs/en/getting_started/launch_xllm.md Normal file
View File

@@ -0,0 +1,125 @@
# Launch xllm
Taking Qwen3 as an example, the script for launching xllm is as follows. The provided script is suitable for both single-node single-device and single-node multi-device scenarios. When using multiple devices on a single node, you need to modify `NNODES` (one device represents one node), as well as environment variables such as `ASCEND_RT_VISIBLE_DEVICES`, `CUDA_VISIBLE_DEVICES`, or `MLU_VISIBLE_DEVICES`.
## NPU
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_RT_VISIBLE_DEVICES=0
export HCCL_IF_BASE_PORT=43432 # HCCL communication base port
MODEL_PATH="/path/to/model/Qwen3-8B" # Model path
MASTER_NODE_ADDR="127.0.0.1:9748" # Master node address (must be globally consistent)
START_PORT=18000 # Service starting port
START_DEVICE=0 # Starting logical device number
LOG_DIR="log" # Log directory
NNODES=1 # Number of nodes (current script launches 1 process)
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="npu:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--block_size=128 \
--communication_backend="hccl" \
--enable_prefix_cache=false \
--enable_chunked_prefill=true \
--enable_schedule_overlap=true \
--enable_shm=true \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
## NVIDIA GPU
```bash
#!/bin/bash
set -e
rm -rf core.*
export CUDA_VISIBLE_DEVICES=0
# for debug
# export CUDA_LAUNCH_BLOCKING=1
MODEL_PATH="/path/to/model/Qwen3-8B"
MASTER_NODE_ADDR="127.0.0.1:9748"
START_PORT=18000
START_DEVICE=0
LOG_DIR="log"
NNODES=1
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="cuda:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--block_size=32 \
--max_memory_utilization=0.8 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_schedule_overlap=true \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
## MLU
```bash
#!/bin/bash
set -e
rm -rf core.*
export MLU_VISIBLE_DEVICES=0
MODEL_PATH="/path/to/model/Qwen3-8B"
MASTER_NODE_ADDR="127.0.0.1:9748"
START_PORT=18000
START_DEVICE=0
LOG_DIR="log"
NNODES=1
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="mlu:$DEVICE" \
--port $PORT \
--nnodes=$NNODES \
--master_node_addr=$MASTER_NODE_ADDR \
--block_size=16 \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```

109
upstream_ref/xllm/docs/en/getting_started/multi_machine.md Normal file
View File

@@ -0,0 +1,109 @@
# Multi-Node Deployment
This example demonstrates how to launch a 32-GPU (NPU) deployment across 2 machines.
Launching Services on the First Machine:
```shell
bash start_deepseek_machine_1.sh
```
The start_deepseek_machine_1.sh script is as follows:
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export HCCL_IF_BASE_PORT=43432 # HCCL communication base port
# 4. Start distributed service
MODEL_PATH="/path/to/your/DeepSeek-R1" # Model path
MASTER_NODE_ADDR="123.123.123.123:9748" # Master node address (must be globally consistent)
LOCAL_HOST=123.123.123.123 # Local IP for service launch
START_PORT=18000 # Service starting port
START_DEVICE=0 # Starting NPU logical device number
LOCAL_NODES=16 # Number of local processes (this script launches 16 processes)
LOG_DIR="log" # Log directory
NNODES=32 # Total number of GPUs/NPUs (32 in this 2-machine example)
mkdir -p $LOG_DIR
for (( i=0; i<$LOCAL_NODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--max_tokens_per_batch=40000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--communication_backend="hccl" \
--enable_schedule_overlap=true \
--rank_tablefile=./ranktable_2s_32p.json \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
Launching Services on the Second Machine:
```shell
bash start_deepseek_machine_2.sh
```
The start_deepseek_machine_2.sh script is as follows:
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export HCCL_IF_BASE_PORT=43432 # HCCL communication base port
MODEL_PATH="/path/to/your/DeepSeek-R1" # Model path
MASTER_NODE_ADDR="123.123.123.123:9748" # Master node address (must be globally consistent)
LOCAL_HOST=456.456.456.456 # Local IP for service launch
START_PORT=18000 # Service starting port
START_DEVICE=0 # Starting NPU logical device number
LOCAL_NODES=16 # Number of local processes (this script launches 16 processes)
LOG_DIR="log" # Log directory
NNODES=32 # Total number of GPUs/NPUs (32 in this 2-machine example)
mkdir -p $LOG_DIR
for (( i=0; i<$LOCAL_NODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--max_tokens_per_batch=40000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--communication_backend="hccl" \
--enable_schedule_overlap=true \
--rank_tablefile=./ranktable_2s_32p.json \
--node_rank=$((i + LOCAL_NODES)) \ > $LOG_FILE 2>&1 &
done
```
This example uses 2 machines. You can set the total number of GPUs/NPUs via `--nnodes`, where `--node_rank` specifies the global rank ID for each node.
The `--rank_tablefile=./ranktable_2s_32p.json`parameter points to the configuration file required for establishing the NPU communication domain. For instructions on generating this file, refer to [Ranktable Generation](https://gitee.com/mindspore/models/blob/master/utils/hccl_tools/README.md).

15
upstream_ref/xllm/docs/en/getting_started/offline_service.md Normal file
View File

@@ -0,0 +1,15 @@
# Offline Inference
To facilitate users in quickly using xLLM for offline inference, we provide Python script examples for launching offline inference.
## LLM
LLM inference example: [:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate.py)
## Embedding
Generate embedding example: [:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate_embedding.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate_embedding.py)
## VLM
VLM inference example: [:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py)

225
upstream_ref/xllm/docs/en/getting_started/online_service.md Normal file
View File

@@ -0,0 +1,225 @@
# Online Service
First, start the xllm service according to the [xllm launch documentation](launch_xllm.md). Below are examples of client calls for LLM and VLM. Please modify the parameters according to your actual situation.
## LLM Client Calls
### HTTP Call
Chat mode:
```bash
curl http://localhost:9977/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"max_tokens": 10,
"temperature": 0,
"stream": true,
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "hello xllm"
}
]
}'
```
Completions mode:
```bash
curl http://127.0.0.1:9977/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"prompt": "hello xllm",
"max_tokens": 10,
"temperature": 0,
"stream": true
}'
```
Sample mode:
```bash
curl http://127.0.0.1:9977/v1/sample \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"prompt": "Question: <emb_0> matched or not. Conclusion: <emb_0>",
"selector": {
"type": "literal",
"value": "<emb_0>"
},
"logprobs": 5,
"request_id": "sample-demo-001"
}'
```
Typical response:
```json
{
"id": "sample-demo-001",
"object": "sample_completion",
"created": 1773369600,
"model": "Qwen2-7B-Instruct",
"choices": [
{
"index": 0,
"text": "True",
"logprobs": {
"tokens": ["True", "False"],
"token_ids": [3456, 7890],
"token_logprobs": [-0.12, -2.31]
},
"finish_reason": "selector_match"
},
{
"index": 1,
"text": "",
"logprobs": {
"tokens": [],
"token_ids": [],
"token_logprobs": []
},
"finish_reason": "empty_logprobs"
}
],
"usage": {
"prompt_tokens": 20,
"completion_tokens": 2,
"total_tokens": 22
}
}
```
`/v1/sample` notes:
- Only `--backend=llm` is supported. VLM/DiT/Rec are not supported yet.
- `selector.type` is currently fixed to `literal`. `selector.value` is matched against prompt text in full and in order.
- `logprobs` defaults to `5`, with an allowed range of `[1, 5]`.
- `choices[i].index` is the matched `sample_id`, corresponding one-to-one with the matched order in prompt.
- If no selector match is found, the service returns `200` with `choices=[]`. If a matched position has no available logprobs, it returns `finish_reason="empty_logprobs"`.
- Service logs only summary fields such as `request_id`, `sample_id`, `match_count`, and `model`, and do not log the full prompt.
`/v1/sample` common error semantics:
- Missing `model/prompt/selector/selector.value`, `selector.type != literal`, or out-of-range `logprobs` returns `INVALID_ARGUMENT`.
- If the model does not exist or the backend is not `llm`, it returns `UNKNOWN`.
- When concurrency reaches the upper limit, it returns `RESOURCE_EXHAUSTED`.
- When the model is in sleep state, it returns `UNAVAILABLE`.
### Python Call
```python
import requests
import json
url = f"http://localhost:9977/v1/chat/completions"
messages = [
{'role': 'user', 'content': "List three countries and their capitals."}
]
request_data = {
"model": "Qwen2-7B-Instruct",
"messages": messages,
"stream": False,
"temperature": 0.6,
"max_tokens": 2048,
}
response = requests.post(url, json=request_data)
if response.status_code != 200:
print(response.status_code, response.text)
else:
ans = json.loads(response.text)["choices"]
print(ans[0]['message'])
```
## VLM Client Calls
### HTTP API
```python
import base64
import requests
api_url = "http://localhost:12345/v1/chat/completions"
image_url = ""
def encode_image(url: str) -> str:
with requests.get(url) as response:
response.raise_for_status()
result = base64.b64encode(response.content).decode("utf-8")
return result
image_base64 = encode_image(image_url)
payload = {
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{image_base64}"},
},
],
}
],
"model": "Qwen2.5-VL-7B-Instruct",
"max_completion_tokens": 128,
}
response = requests.post(
api_url,
json=payload,
headers={"Content-Type": "application/json"}
)
print(response.json())
```
### OpenAI API
```python
from openai import OpenAI
import base64
import requests
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:12345/v1"
image_url = ""
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
def encode_image(url: str) -> str:
with requests.get(url) as response:
response.raise_for_status()
result = base64.b64encode(response.content).decode("utf-8")
return result
image_base64 = encode_image(image_url)
chat_completion = client.chat.completions.create(
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{image_base64}"},
},
],
}
],
model="Qwen2.5-VL-7B-Instruct",
max_completion_tokens=128,
)
result = chat_completion.choices[0].message.content
print("Chat completion output:", result)
```

107
upstream_ref/xllm/docs/en/getting_started/quick_start.md Normal file
View File

@@ -0,0 +1,107 @@
# Quick Start
## Environment Setup
All images are stored [here](https://quay.io/repository/jd_xllm/xllm-ai?tab=tags). The docker startup command below uses the dev image as an example.
### NPU
Below are our pre-built dev image.
```bash
# A2 x86
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a2-x86-20260429
# A2 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a2-arm-20260429
# A3 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a3-arm-20260429
```
Container startup command:
```bash
docker run -it \
--ipc=host \
-u 0 \
--name xllm-npu \
--privileged \
--network=host \
--device=/dev/davinci0 \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/local/sbin/:/usr/local/sbin/ \
-v /var/log/npu/conf/slog/slog.conf:/var/log/npu/conf/slog/slog.conf \
-v /var/log/npu/slog/:/var/log/npu/slog \
-v /var/log/npu/profiling/:/var/log/npu/profiling \
-v /var/log/npu/dump/:/var/log/npu/dump \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
### NVIDIA GPU
We provide a [Dockerfile](../../../docker/Dockerfile.cuda) for NVIDIA GPU usage, which can be used to build custom image. Of course, you can also use dev image we built based on the default Dockerfile:
```bash
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-cuda-x86
```
Container startup command:
```bash
sudo docker run -it \
--privileged \
--shm-size '128gb' \
--ipc=host \
--net=host \
--pid=host \
--name=xllm-cuda \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
### MLU
We cannot provide MLU image. If you already have the dev image, you can start the container with the following command:
```bash
sudo docker run -it \
--privileged \
--shm-size '128gb' \
--ipc=host \
--net=host \
--pid=host \
--name xllm-mlu \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
## Build xllm
If you download a release image, i.e., an image with a version number in the tag, you can skip this step because the release image comes with a pre-compiled xllm binary, located at `/usr/local/bin/xllm`.
Download xllm and dependencies:
```bash
git clone https://github.com/jd-opensource/xllm
cd xllm
# Install pre-commit for the first time
pip install pre-commit
pre-commit install
git submodule update --init --recursive
```
The compiled binary file is located at `/path/to/xllm/build/xllm/core/server/xllm`. In a new image, the first compilation of xllm takes a long time because all dependencies in vcpkg need to be compiled, but subsequent compilations will be much faster.
```bash
python setup.py build
```
## Launch xllm
Please refer to [How to Launch xllm](launch_xllm.md).

57
upstream_ref/xllm/docs/en/index.md Normal file
View File

@@ -0,0 +1,57 @@
---
hide:
- navigation
---
<style>
.md-content h1:first-of-type {
display: none;
}
</style>
<div style="text-align:center">
<img src="../assets/logo_with_llm.png" alt="xLLM" style="width:50%; height:auto;">
</div>
## Project Overview
**xLLM** is an efficient and user-friendly LLM intelligent inference framework that provides enterprise-level service guarantees and high-performance engine computing capabilities for model inference on domestic AI accelerators.
#### Background
LLM with parameter scales ranging from tens of billions to trillions are being rapidly deployed in core business scenarios such as intelligent customer service, real-time recommendation, and content generation. Efficient support for domestic computing hardware has become a core requirement for low-cost inference deployment. Existing inference engines struggle to effectively adapt to the architectural characteristics of dedicated accelerators like domestic chips. Performance issues such as low utilization of computing units, load imbalance and communication overhead bottlenecks under the MoE architecture, and difficulties in kv cache management have restricted the efficient inference of requests and the scalability of the system. The xLLM inference engine improves the resource efficiency of the entire "communication-computation-storage" performance link and it provides crucial technical support for the large-scale implementation of LLM in real-world business scenarios.
## Core Features
**xLLM** delivers robust intelligent computing capabilities. By leveraging hardware system optimization and algorithm-driven decision control, it jointly accelerates the inference process, enabling high-throughput, low-latency distributed inference services.
**Full Graph Pipeline Execution Orchestration**
- Asynchronous decoupled scheduling at the requests scheduling layer, to reduce computational bubbles.
- Asynchronous parallelism of computation and communication at the model graph layer, overlapping computation and communication.
- Pipelining of heterogeneous computing units at the operator kernel layer, overlapping computation and memory access.
**Graph Optimization for Dynamic Shapes**
- Dynamic shape adaptation based on parameterization and multi-graph caching methods to enhance the flexibility of static graph.
- Controlled tensor memory pool to ensure address security and reusability.
- Integration and adaptation of performance-critical custom operators (e.g., *PageAttention*, *AllReduce*).
**MoE Kernel Optimization**
- *GroupMatmul* optimization to improve computational efficiency.
- Chunked Prefill optimization to support long-sequence inputs.
**Efficient Memory Optimization**
- Mapping management between discrete physical memory and continuous virtual memory.
- On-demand memory allocation to reduce memory fragmentation.
- Intelligent scheduling of memory pages to increase memory reusability.
- Adaptation of corresponding operators for domestic accelerators.
**Global KV Cache Management**
- Intelligent offloading and prefetching of KV in hierarchical caches.
- KV cache-centric distributed storage architecture.
- Intelligent KV routing among computing nodes.
**Algorithm-driven Acceleration**
- Speculative decoding optimization to improve efficiency through multi-core parallelism.
- Dynamic load balancing of MoE experts to achieve efficient adjustment of expert distribution.

49
upstream_ref/xllm/docs/en/supported_models.md Normal file
View File

@@ -0,0 +1,49 @@
# Model Support List
## LLM
| | NPU | MLU | ILU |
| ------------------------ | :---: | :---: | :---: |
| DeepSeek-V3/R1/V3.1 | ✅ | ✅ | ❌ |
| DeepSeek-V3.2 | ✅ | ✅ | ❌ |
| DeepSeek-R1-Distill-Qwen | ✅ | ❌ | ❌ |
| Qwen2/2.5/QwQ | ✅ | ✅ | ✅ |
| Qwen3 | ✅ | ✅ | ✅ |
| Qwen3 Moe | ✅ | ✅ | ✅ |
| Kimi-k2 | ✅ | ❌ | ❌ |
| Llama2/3 | ✅ | ❌ | ✅ |
| GLM4.5 | ✅ | ❌ | ❌ |
| GLM4.6 | ✅ | ❌ | ❌ |
| GLM-4.7 | ✅ | ❌ | ❌ |
| GLM-5 | ✅ | ❌ | ❌ |
## VLM
| | NPU | MLU | ILU |
| ------------ | :---: | :---: | :---: |
| MiniCPM-V | ✅ | ❌ | ❌ |
| MiMo-VL | ✅ | ❌ | ❌ |
| Qwen2.5-VL | ✅ | ✅ | ❌ |
| Qwen3-VL | ✅ | ✅ | ❌ |
| Qwen3-VL-MoE | ✅ | ✅ | ❌ |
| GLM-4.6V | ✅ | ❌ | ❌ |
| VLM-R1 | ✅ | ❌ | ❌ |
## Rerank
| | NPU | MLU | ILU |
| -------------- | :---: | :---: | :---: |
| Qwen3-Reranker | ✅ | ❌ | ❌ |
## DiT
| | NPU | MLU | ILU |
| ---- | :---: | :---: | :---: |
| Flux | ✅ | ❌ | ❌ |
## Rec
| | NPU | MLU | ILU |
| --- | :---: | :---: | :---: |
| OneRec | ✅ | ❌ | ❌ |
| Qwen2 | ✅ | ❌ | ❌ |
| Qwen2.5 | ✅ | ❌ | ❌ |
| Qwen3 | ✅ | ❌ | ❌ |

19
upstream_ref/xllm/docs/mkdocs/javascripts/mathjax.js Normal file
View File

@@ -0,0 +1,19 @@
window.MathJax = {
tex: {
inlineMath: [["\\(", "\\)"]],
displayMath: [["\\[", "\\]"]],
processEscapes: true,
processEnvironments: true
},
options: {
ignoreHtmlClass: ".*|",
processHtmlClass: "arithmatex"
}
};
document$.subscribe(() => {
MathJax.startup.output.clearCache()
MathJax.typesetClear()
MathJax.texReset()
MathJax.typesetPromise()
})

1
upstream_ref/xllm/docs/mkdocs/overrides/.icons/email-fill.svg Normal file
View File

@@ -0,0 +1 @@
<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1755326167427" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="13611" xmlns:xlink="http://www.w3.org/1999/xlink" width="256" height="256"><path d="M926.47619 355.644952V780.190476a73.142857 73.142857 0 0 1-73.142857 73.142857H170.666667a73.142857 73.142857 0 0 1-73.142857-73.142857V355.644952l304.103619 257.828572a170.666667 170.666667 0 0 0 220.745142 0L926.47619 355.644952zM853.333333 170.666667a74.044952 74.044952 0 0 1 26.087619 4.778666 72.704 72.704 0 0 1 30.622477 22.186667 73.508571 73.508571 0 0 1 10.678857 17.67619c3.169524 7.509333 5.12 15.652571 5.607619 24.210286L926.47619 243.809524v24.380952L559.469714 581.241905a73.142857 73.142857 0 0 1-91.306666 2.901333l-3.632762-2.925714L97.52381 268.190476v-24.380952a72.899048 72.899048 0 0 1 40.155428-65.292191A72.97219 72.97219 0 0 1 170.666667 170.666667h682.666666z" p-id="13612" data-spm-anchor-id="a313x.search_index.0.i13.7cb13a81l9Qfa6" class="selected" fill="#acacac"></path></svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.1 KiB

1
upstream_ref/xllm/docs/mkdocs/overrides/.icons/gitcodeai.svg Normal file
View File

@@ -0,0 +1 @@
<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1755324951889" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="5508" width="200" height="200" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M660.48 212.352c5.952-3.392 12.352-6.912 19.2-10.88l2.368 12.032c1.28 6.272 2.368 11.456 2.88 16.512 4.032 44.288 26.304 76.928 57.856 84.224 46.144 10.752 89.664-7.04 113.088-46.08 28.16-46.848 16-104.32-32-138.88C690.56 33.216 544.128 6.72 386.304 53.056 46.144 153.344-70.272 571.264 171.2 827.52c103.296 109.632 234.432 156.928 383.36 153.344 190.72-4.48 328-99.52 415.744-264.256 62.08-116.864-5.44-244.608-134.848-271.168a837.376 837.376 0 0 0-224.768-14.4c-24.96 2.432-49.28 9.792-71.424 21.76-24.768 12.8-31.872 39.488-29.12 65.92 2.56 24.064 21.056 38.528 43.008 42.176a1837.44 1837.44 0 0 0 133.76 14.592c12.928 1.088 25.984 1.28 39.04 1.344 18.752 0.192 37.376 0.384 55.68 3.392 52.032 8.576 69.888 50.816 43.136 96-6.592 10.88-14.208 21.056-22.848 30.336a259.392 259.392 0 0 1-131.392 77.76c-92.416 22.592-184.896 23.872-276.8-5.12-104.704-33.088-167.168-109.952-169.344-213.504-0.832-63.872 15.36-126.72 46.976-182.272 14.272-25.856 22.08-52.48 19.84-81.856-0.896-12.48-1.408-24.96-1.92-38.4a2748.8 2748.8 0 0 0-1.024-22.4c10.624 2.176 21.12 4.992 31.36 8.32 40.064 16.128 79.488 23.488 122.88 11.648A222.08 222.08 0 0 1 517.76 256a188.608 188.608 0 0 0 115.648-28.288c8.64-5.056 17.344-9.92 26.944-15.36z" fill="#ACACAC" p-id="5509"></path></svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.6 KiB

27
upstream_ref/xllm/docs/mkdocs/stylesheets/extra.css Normal file
View File

@@ -0,0 +1,27 @@
:root > * {
--md-primary-fg-color: #F1002B;
--md-primary-fg-color--light: #F1002B;
--md-primary-fg-color--dark: #af0510;
--md-accent-fg-color: #F1002B;
--md-accent-fg-color--light: #F1002B;
--md-accent-fg-color--dark: #af0510;
}
/* :root > * {
--md-footer-bg-color: var(--md-primary-fg-color);
--md-footer-fg-color: var(--md-primary-bg-color);
--md-footer-fg-color--light: var(--md-primary-bg-color--light);
--md-footer-fg-color--lighter: var(--md-primary-bg-color--lighter);
} */
[data-md-color-scheme="jd"] {
--md-primary-fg-color: #FB002B;
--md-primary-fg-color--light: #FB002B;
--md-primary-fg-color--dark: #af0510;
--md-accent-fg-color: #FB002B;
--md-accent-fg-color--light: #FB002B;
--md-accent-fg-color--dark: #af0510;
}

173
upstream_ref/xllm/docs/project/README_zh.md Normal file
View File

@@ -0,0 +1,173 @@
<!-- Copyright 2022 JD Co.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this project except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. -->
[English](../../README.md) | [中文](./README_zh.md)
<div align="center">
<img src="../assets/logo_with_llm.png" alt="xLLM" style="width:50%; height:auto;">
[![Document](https://img.shields.io/badge/Document-black?logo=html5&labelColor=grey&color=red)](https://xllm.readthedocs.io/zh-cn/latest/) [![Docker](https://img.shields.io/badge/Docker-black?logo=docker&labelColor=grey&color=%231E90FF)](https://hub.docker.com/r/xllm/xllm-ai) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen?labelColor=grey)](https://opensource.org/licenses/Apache-2.0) [![report](https://img.shields.io/badge/Technical%20Report-red?logo=arxiv&logoColor=%23B31B1B&labelColor=%23F0EBEB&color=%23D42626)](https://arxiv.org/abs/2510.14686) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/jd-opensource/xllm)
</div>
---------------------
<p align="center">
| <a href="https://xllm.readthedocs.io/zh-cn/latest/"><b>Documentation</b></a> | <a href="https://arxiv.org/abs/2510.14686"><b>Technical Report</b></a> |
</p>
### 📢 新闻
- 2026-04-24: 🎉 我们 day-0 支持了[DeepSeek-V4](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) 模型的推理服务，部署请参考[部署文档](https://github.com/jd-opensource/xllm/blob/preview/deepseek-v4-mlu/testspace/run_deepseek_v4.sh)。
- 2026-02-12: 🎉 我们 day-0 支持了最新的[GLM-5](https://github.com/zai-org/GLM-5) 模型的高效推理服务，部署请参考[部署文档](https://github.com/zai-org/GLM-5/blob/main/example/ascend.md)。
- 2025-12-21: 🎉 我们在第一时间内支持了[GLM-4.7](https://github.com/zai-org)模型的高效推理。
- 2025-12-08: 🎉 我们在第一时间内支持了[GLM-4.6V](https://github.com/zai-org/GLM-V)模型的高效推理。
- 2025-12-05: 🎉 我们支持了[GLM-4.5/GLM-4.6](https://github.com/zai-org/GLM-4.5/blob/main/README_zh.md)系列模型.
- 2025-12-05: 🎉 我们支持了[VLM-R1](https://github.com/om-ai-lab/VLM-R1) 模型.
- 2025-12-05: 🎉 我们基于[Mooncake](https://github.com/kvcache-ai/Mooncake)构建了混合 KV 缓存管理机制，支持具备智能卸载与预取能力的全局 KV 缓存管理。
- 2025-10-16: 🎉 我们最近在 arXiv 上发布了我们的 [xLLM 技术报告](https://arxiv.org/abs/2510.14686)，提供了全面的技术蓝图和实施见解。
## 简介
**xLLM** 是一个高效的开源大模型推理框架，专为**国产芯片**优化设计，提供企业级的服务部署，使得性能更高、成本更低。该框架采用**服务-引擎分离的推理架构**，通过服务层的在离线请求弹性调度、动态PD分离、EPD混合机制及高可用容错设计，结合引擎层的多流并行计算、图融合优化、投机推理、动态负载均衡及全局KV缓存管理，实现推理效率突破性提升。xLLM整体架构和功能如下图所示：
<div align="center">
<img src="../assets/xllm_arch.png" alt="xllm_arch" style="width:90%; height:auto;">
</div>
**xLLM** 已支持主流大模型（如 *DeepSeek-V3.1*，*Qwen2/3*等）在国产芯片上的高效部署，助力企业实现高性能、低成本的 AI 大模型应用落地。xLLM已全面落地京东零售核心业务，涵盖智能客服、风控、供应链优化、广告推荐等多种场景。
## 核心特性
xLLM 提供了强大的智能计算能力，通过硬件系统的算力优化与算法驱动的决策控制，联合加速推理过程，实现高吞吐、低延迟的分布式推理服务。
**全图化/多层流水线执行编排**
- 框架调度层的异步解耦调度，减少计算空泡；
- 模型图层的计算和通信异步并行，重叠计算与通信；
- 算子内核层的异构计算单元深度流水，重叠计算与访存。
**动态shape的图执行优化**
- 基于参数化与多图缓存方法的动态尺寸适配，提升静态图灵活性；
- 受管控的显存池，保证地址安全可复用；
- 集成适配性能关键的自定义算子（如 *PageAttention*, *AllReduce*）。
**高效显存优化**
- 离散物理内存与连续虚拟内存的映射管理；
- 按需分配内存空间，减少内存碎片与浪费；
- 智能调度内存空间，增加内存页复用，减小分配延迟；
- 国产芯片相应算子适配。
**全局多级KV Cache管理**
- 多级缓存的kv智能卸载与预取；
- 以kv cache为中心的分布式存储架构；
- 多节点间kv的智能传输路由。
**算法优化**
- 投机推理优化，多核并行提升效率；
- MoE专家的动态负载均衡，实现专家分布的高效调整。
---
## 硬件支持
| 硬件类型 | 型号 | 备注 |
| -------- | ------ | --------------- |
| NPU | A2, A3 | HDK Driver 25.2.0 + |
| MLU | | |
| ILU | BI150 | |
| MUSA | S5000 | |
此外，请在[模型支持列表](../zh/supported_models.md)查看不同硬件上的模型支持情况。
---
## 快速开始
请参考[快速开始文档](../zh/getting_started/quick_start.md)。
---
## 成为贡献者
您可以通过以下方法为 xLLM 作出贡献:
1. 在Issue中报告问题
2. 提供改进建议
3. 补充文档
+ Fork仓库
+ 修改文档
+ 提出pull request
4. 修改代码
+ Fork仓库
+ 创建新分支
+ 加入您的修改
+ 提出pull request
感谢您的贡献！ 🎉🎉🎉
如果您在开发中遇到问题，请参阅**[xLLM中文指南](https://xllm.readthedocs.io/zh-cn/latest)**
---
## 社区支持
如果你在xLLM的开发或使用过程中遇到任何问题，欢迎在项目的Issue区域提交可复现的步骤或日志片段。
如果您有企业内部Slack，请直接联系xLLM Core团队。另外，我们建立了官方微信群，可以访问以下二维码加入。欢迎沟通和联系我们:
<div align="center">
<img src="../assets/wechat_qrcode.png" alt="qrcode3" width="50%" />
</div>
---
## 致谢
本项目的实现得益于以下开源项目:
- [ScaleLLM](https://github.com/vectorch-ai/ScaleLLM) - 采用了ScaleLLM中构图方式和借鉴Runtime执行。
- [Mooncake](https://github.com/kvcache-ai/Mooncake) - 依赖构建了多级KV Cache管理机制。
- [brpc](https://github.com/apache/brpc) - 依赖brpc构建了高性能http service。
- [tokenizers-cpp](https://github.com/mlc-ai/tokenizers-cpp) - 依赖tokenizers-cpp构建了c++ tokenizer。
- [safetensors](https://github.com/huggingface/safetensors) - 依赖其c binding safetensors能力。
- [Partial JSON Parser](https://github.com/promplate/partial-json-parser) - xLLM的C++版本JSON解析器，参考Python与Go实现的设计思路。
- [concurrentqueue](https://github.com/cameron314/concurrentqueue) - 高性能无锁Queue.
感谢以下合作的高校实验室：
- [THU-MIG](https://ise.thss.tsinghua.edu.cn/mig/projects.html)（清华大学软件学院、北京信息科学与技术国家研究中心）
- USTC-Cloudlab（中国科学技术大学云计算实验室）
- [Beihang-HiPO](https://github.com/buaa-hipo)（北京航空航天大学HiPO研究组）
- PKU-DS-LAB（北京大学数据结构实验室）
- PKU-NetSys-LAB（北京大学网络系统实验室）
- [TJU-TANKLab](https://flashserve.org/) (天津大学TANK实验室)
感谢以下为xLLM作出贡献的[开发者](https://github.com/jd-opensource/xllm/graphs/contributors)
<a href="https://github.com/jd-opensource/xLLM/graphs/contributors">
<img src="https://contrib.rocks/image?repo=jd-opensource/xllm" />
</a>
---
## 许可证
[Apache License](LICENSE)
#### xLLM 由 JD.com 提供
#### 感谢您对xLLM的关心与贡献!
## 引用
如果你觉得这个仓库对你有帮助，欢迎引用我们：
```
@article{liu2025xllm,
title={xLLM Technical Report},
author={Liu, Tongxuan and Peng, Tao and Yang, Peijun and Zhao, Xiaoyang and Lu, Xiusheng and Huang, Weizhe and Liu, Zirui and Chen, Xiaoyu and Liang, Zhiwei and Xiong, Jun and others},
journal={arXiv preprint arXiv:2510.14686},
year={2025}
}
```

208
upstream_ref/xllm/docs/project/RELEASE.md Normal file
View File

@@ -0,0 +1,208 @@
# Release xllm 0.9.0
## **Major Features and Improvements**
### Model Support
#### NPU
- Support GLM-5 model.
- Support GLM4.7-Flash model.
- Support Qwen3-next model.
- Support OneRec model.
- Support Qwen3.5/Qwen3.5-MoE model.
#### CUDA
- Support LongCat-Image model.
- Support LongCat-Image-Edit model.
#### MLU
- Support DeepSeek-V3.2 W4A8 MoE model.
- Support GLM-5 W8A8 model.
#### ILU
- Support Qwen3-8B model.
- Support Qwen3-30B-MoE model.
### Feature
- Adapt NPU builds to CANN 8.5 and PyTorch 2.7.1.
- Support graph mode for the LLM part of VLM models on NPU devices.
- Support context parallelism for NPU DeepSeek-V3.2 / GLM-5.
- Support DeepSeek-V3.2 prefill sequence parallel on MLU devices.
- Support rolling weight loading and loading model weights with varied prefixes.
- Support dynamic and scalable multi-model serving.
- Support bidirectional remote-host to local-device KV cache transfer and batch offload.
- Support Qwen3 xattention on NPU devices.
- Support prefix cache for DeepSeek-V3.2.
- Support chunked prefill on CUDA devices.
- Support embedding interface for all generate LLM models.
- Support Anthropic Messages API.
- Support the new `v1/sample` interface.
- Support a single xLLM instance connecting to multiple xLLM services.
- Support startup progress bar, worker health check, and unified request statistics logging.
- Optimize Qwen3 MoE performance on NPU devices.
- Add CUDA Graph Executor and piecewise prefill graph.
- Support KV cache quantization on MLU devices.
- Add VMM-based allocators to reuse graph buffers and physical memory.
- Improve FP8 GEMM, fused RMSNorm, fused MoE, xattention, and activation kernel performance.
### Bugfix
- Support the new `compressed-tensors` FP8 config and fix Qwen2 prompt length.
- Fix Qwen3 MoE VL parameter settings on MLU devices.
- Fix Qwen VL issues on MLU devices and Qwen2.5 chunked-prefill accuracy on NPU.
- Fix DeepSeek tool-call, prefix-cache, DP/MTP, and PD-disagg related issues.
- Fix GLM-4.7 streaming function call issues and GLM detector stability issues.
- Fix graph mode, schedule overlap, KV cache, and REC multi-round stability issues.
- Fix multiple compile, link, env setup, and worker lifecycle issues.
# Release xllm 0.8.0
## **Major Features and Improvements**
### Model Support
#### NPU
- Support DeepSeek-v3.2 model.
- Support GLM4.7 model.
- Support GLM4.6Vmodel.
- Support GME-Qwen2-VL model.
- Support FluxControl model.
#### CUDA
- Support Qwen2/3 Dense model.
#### MLU
- Support DeepSeek-v3.2 model.
- Support Qwen2_5_vl/Qwen3_vl/Qwen3_vl_moe model.
#### ILU
- Support Qwen3-0.6B model.
### Feature
- Implement chunked prefill and prefix cache for Qwen3 MoE.
- Support GLM-4.6V model.
- Add wrappers for ATB and ACLNN fused operators.
- Optimize prefetch from kv cache store.
- Support Qwen2-VL & GME-Qwen2-VL model on npu device.
- Fix hang issue when enable schedule overlap.
- Add GLM-4.7 detector implementation and update tool call parser.
- Adapt hierarchy block manager for disagg PD.
- Support deepseek-v3.2-Exp for npu.
- Support acl_graph for qwen3/qwen3_moe.
- Support prefix cache for deepseek-v3/r1 models.
- Support disagg PD for MTP.
- Add mooncake kv cache transfer.
- Add GLM-4.7 support to reasoning detector registry.
- Support nd-to-nz continuous memory copy.
- Support RPC-based link/unlink for PD disaggregation.
- Support IntraLayerAddNorm, aclgraph, etc for DeepSeek V3.2.
- Add activation, norm and rope ops for cuda device.
- Support fused norm for Qwen3 and DeepSeek for cuda device.
- Build deepseek v2 decoder layer and related model files for mlu device.
- Support qwen2_5_vl/qwen3_vl/qwen3_vl_moe on mlu device.
- Add moe all2all kernels and deep ep layer on mlu device.
- Support deepseek mtp on mlu device.
- Support graph executor on mlu device.
- Support dp+ep moe and all2all computation on mlu device.
- Support parallelized shared experts in fused moe on mlu device.
- Support qwen3 0.6B model on iluvatar device.
- Add rec proto,serivce and utils for rec framework
- Support C api for llm inference.
- Add constrained decoding for generative recommendation.
- Add rec scheduler master and engine for rec framework.
- Add rec_type and onerec batch input builder for rec framework.
- Add onerec worker impl for rec framework.
- Add qwen3/LlmRec support in rec framework.
### Bugfix
- Reslove core dump of stream chat completion request when backend is VLM.
- Resolve duplicate content in multi-turn tool call conversations.
- Fix core dump issue triggered by client disconnection.
- Fix the memory leak issue in the completions interface.
- Fix wrong positons of validate input when enable MTP.
- Resolve kv_cache_num mismatch in ChunkedPrefill due to H2D block copy.
- Fix the missing index shape in the allocate kv cache transfer.
- Fix MiMo-VL weights loading crash on NPU device.
- Fix inaccurate metrics issue when enabling schedule overlap.
- Fix potential out-of-range and block leaks during deallocate in D2H copy.
- Fix allocation failure in HierarchyBlockManagerPool::allocate.
- Fix deepseek accuracy issues with prefix cache enabled.
- Resolve Deepseek execution failure caused by invalid input.
- Fix DeepSeek failing to run when enabling DP.
- Fix the rate_limit bug for stream and non-stream request in PD disagg and refactor some callback logics.
- Correct attn mask when prefix cache and MTP are both enabled in deepseek.
- Correct precision loss when enabling prefixcache with disagg pd.
- Fix incorrect async implementation in rerank interface.
- Fix acl_graph_executor not handling q_cu_seq_lens parameter for deepseekv3.2.
- Fix precision issue when enabling MTP in PD disaggregation mode.
- Fix mrope calculation in the multimodal situation.
- Fix core dump of large beam width.
# Release xllm 0.7.0
## **Major Features and Improvements**
### Model Support
- Support GLM-4.5.
- Support Qwen3-Embedding.
- Support Qwen3-VL.
- Support FluxFill.
### Feature
- Support MLU backend, currently supports Qwen3 series models.
- Support dynamic disaggregated PD, with dynamic switching between P and D phases based on strategy.
- Support multi-stream parallel overlap optimization.
- Support beam-search capability in generative models.
- Support virtual memory continuous kv-cache capability.
- Support ACL graph executor.
- Support unified online-offline co-location scheduling in disaggregated PD scenarios.
- Support PrefillOnly Scheduler.
- Support v1/rerank model service interface.
- Support communication between devices via shared memory instead of RPC on a single machine.
- Support function call.
- Support reasoning output in chat interface.
- Support top-k+add fusion in the router component of MoE models.
- Support offline inference for LLM, VLM, and Embedding models.
- Optimized certain runtime performance.
### Bugfix
- Skip cancelled requests when processing stream output.
- Resolve segmentation fault during qwen3 quantized inference.
- Fix the alignment of monitoring metrics format for Prometheus.
- Clear outdated tensors to save memory when loading model weights.
- Fix attention mask to support long sequence requests.
- Fix bugs caused by enabling scheduler overlap.
# Release xllm 0.6.0
## **Major Features and Improvements**
### Model Support
- Support DeepSeek-V3/R1.
- Support DeepSeek-R1-Distill-Qwen.
- Support Kimi-k2.
- Support Llama2/3.
- Support Qwen2/2.5/QwQ.
- Support Qwen3/Qwen3-MoE.
- Support MiniCPM-V.
- Support MiMo-VL.
- Support Qwen2.5-VL .
### Feature
- Support KV cache store.
- Support Expert Parallelism Load Balance.
- Support multi-priority on/offline scheduler.
- Support latency-aware scheduler.
- Support serving early stop.
- Optimize ppmatmul kernel.
- Support image url input for VLM.
- Support disaggregated prefill and decoding.
- Support large-scale EP parallelism.
- Support Hash-based PrefixCache matching.
- Support Multi-Token Prediction for DeepSeek.
- Support asynchronous scheduling, allowing the scheduling and computational pipeline to execute in parallel.
- Support EP, DP, TP model parallel.
- Support multiple process and multiple nodes.
### Docs
- Add getting started docs.
- Add features docs.

43
upstream_ref/xllm/docs/requirements.txt Normal file
View File

@@ -0,0 +1,43 @@
# mkdocs-material
# mkdocs-minify-plugin
# python-markdown-math
# regex
# ruff
# jinja2~=3.1
# markdown~=3.2
# mkdocs~=1.6
# mkdocs-material-extensions~=1.3
# pygments~=2.16
# pymdown-extensions~=10.2
# # Requirements for plugins
# babel~=2.10
# colorama~=0.4
# paginate~=0.5
# backrefs~=5.7.post1
# requests~=2.26
# Requirements for core
jinja2
markdown
mkdocs
mkdocs-material
mkdocs-material-extensions
pygments
pymdown-extensions
mkdocs-minify-plugin
python-markdown-math
mkdocs-git-revision-date-localized-plugin
# Requirements for plugins
babel
colorama
paginate
backrefs
requests
# Temporarily pin click until this is resolved in MkDocs, see
# https://github.com/mkdocs/mkdocs/issues/4014#issuecomment-3146508306
click

28
upstream_ref/xllm/docs/zh/.readthedocs.yaml Normal file
View File

@@ -0,0 +1,28 @@
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the OS, Python version, and other tools you might need
build:
os: ubuntu-24.04
tools:
python: "3.13"
jobs:
pre_build:
# - cp -r docs/zh/* docs/
- mv docs/zh/* docs/
- rm -rf docs/en/
- find docs/ -name "*.md" -exec sed -i 's#../assets/#assets/#g' {} \;
# Build documentation with Mkdocs
mkdocs:
configuration: mkdocs_zh.yml
# Optionally, but recommended,
# declare the Python requirements required to build your documentation
# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: docs/requirements.txt

58
upstream_ref/xllm/docs/zh/accuracy_test.md Normal file
View File

@@ -0,0 +1,58 @@
# 1. LLM精度测试
## 1.1 设置ais_bench
```bash
# 使用conda或uv为ais_bench创建虚拟环境
conda create --name ais_bench python=3.10 -y
conda activate ais_bench
# 下载ais_bench并安装依赖
git clone https://gitee.com/aisbench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517
# 下载数据集并复制到ais_bench目录下
cp -r /path/to/dataset /path/to/benchmark/ais_bench/datasets
```
## 1.2 修改配置
根据实际情况修改精度测试配置文件：`/path/to/benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py`，采样参数建议按如下代码设置：
```python
models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr='vllm-api-general-chat',
path="/path/to/model/Qwen3-8B", # 模型路径
model="Qwen3-8B", # 模型名称
request_rate = 0,
retry = 2,
host_ip = "127.0.0.1",
host_port = 19000, # xllm服务端端口
max_out_len = 32768, # 限制模型最大长度
batch_size=32,
trust_remote_code=False,
generation_kwargs = dict(
temperature = 0.6,
# top_k = -1,
top_p = 0.95,
# seed = None,
# repetition_penalty = 1,
),
pred_postprocessor=dict(type=extract_non_reasoning_content)
)
]
```
## 1.3 启动ais_bench
在使用ais_bench前需要先启动xllm服务。使用`ais_bench -h`能够获取参数含义，对于gsm8k和ceval数据集的启动命令如下：
```bash
# 使用gsm8k数据集
ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt --dump-eval-details
# 使用ceval数据集
ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt --merge-ds --dump-eval-details
```
我们会在未来将ais_bench和数据集（ceval和gsm8k）集成进开发镜像，ais_bench文档和数据集如下：
* [ais_bench文档](https://ais-bench-benchmark.readthedocs.io/zh-cn/latest/index.html)
* [数据集](https://ais-bench-benchmark.readthedocs.io/zh-cn/latest/base_tutorials/all_params/datasets.html)

85
upstream_ref/xllm/docs/zh/cli_reference.md Normal file
View File

@@ -0,0 +1,85 @@
---
hide:
- navigation
---
# 服务启动参数
xLLM使用gflags来管理服务启动参数，具体的参数含义如下：
## 常用参数
| 参数名称 | 类型 | 默认值 | 其他值 | 参数含义 | 其他 |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `master_node_addr` | `string` | "127.0.0.1:19888" | ip:port | master节点rpc server的监听地址 | [详情](./features/basics.md) |
| `host` | `string` | "" | 当前device所在的机器ip | 当前device用于通信的host ip，每个device上会启动一个rpc server，用于多卡之间通信 | |
| `port` | `int32` | 8010 | 任意可用的端口 | 与host参数配套使用，组合后用于device间的rpc通信 | |
| `model` | `string` | "" | | 模型所在的路径 | |
| `devices` | `string` | "npu:0" | | 指定当前进程使用的NPU设备 | |
| `nnodes` | `int32` | 1 | | 当前服务所使用的device总数 | |
| `node_rank` | `int32` | 0 | 0 ~ device总数减1 | 每个device的rank id | |
| `max_memory_utilization` | `double` | 0.8 | 0-1之间 | 模型权重和KV Cache一起可用的最大device memory占比 | |
| `max_tokens_per_batch` | `int32` | 10240 | | 每个step可计算的最大token数量 | |
| `max_seqs_per_batch` | `int32` | 1024 | | 每个step可计算的最大sequence数量 | |
| `enable_chunked_prefill` | `bool` | true | false | 是否开启chunked prefill | |
| `enable_prefill_sp` | `bool` | false | true | 是否开启 prefill 阶段的 sequence parallel | 支持 `enable_chunked_prefill=true`，但仅限纯 prefill batch（`PREFILL` / `CHUNKED_PREFILL`）；`MIXED` 与 `DECODE` batch 不会进入 sequence parallel。 |
| `enable_schedule_overlap` | `bool` | false | true | 是否开启异步调度 | [详情](./features/async_schedule.md) |
| `enable_prefix_cache` | `bool` | true | false | 是否开启prefix cache（DeepSeek暂不支持） | |
| `communication_backend` | `string` | "hccl" | "lccl" | 通信操作采用的后端 | |
| `block_size` | `int32` | 128 | | KV Cache存储的block size大小 | |
| `task` | `string` | "generate" | "embed", "mm_embed" | 服务类型，生成式、embedding或多模态embedding | |
| `max_cache_size` | `int64` | 0 | | 可使用的KV Cache大小，单位byte | |
| `kv_cache_dtype` | `string` | "auto" | "int8" | KV Cache数据类型。"auto"表示与模型dtype对齐（不量化），"int8"启用INT8量化以节省约50%显存。仅MLU后端支持 | |
## MOE模型相关参数
| 参数名称 | 类型 | 默认值 | 其他值 | 参数含义 | 其他 |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `dp_size` | `int32` | 1 | 2的指数 | Attention部分的dp规模大小 | |
| `ep_size` | `int32` | 1 | 2的指数 | MoE部分的ep规模大小 | |
| `expert_parallel_degree` | `int32` | 0 | 1,2 | ep并行相关参数，gflag默认值为0；当`ep_size > 1`且未显式配置时，部分NPU MoE实现会按EP Level 1处理；当`ep_size=devices`总数时可设置为2（使用all2all通信） | |
支持 MLA 的模型会自动开启 MLA，不再需要单独配置 CLI 参数。
## PD分离相关参数
| 参数名称 | 类型 | 默认值 | 其他值 | 参数含义 | 其他 |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `enable_disagg_pd` | `bool` | false | true | 是否启动PD分离 | [详情](./features/disagg_pd.md) |
| `disagg_pd_port` | `int32` | 7777 | 任意可用的端口 | 启用PD分离后配置，对应每张卡上启动的pd分离rpc server的监听端口号 | |
| `instance_role` | `string` | DEFAULT | PREFILL DECODE MIX | 默认情况下为DEFAULT，开启PD分离后需要配置为PREFILL、DECODE或者MIX | |
| `kv_cache_transfer_mode` | `string` | "PUSH" | "PULL" | PD分离传输KV Cache的模式。PUSH模式：Prefill逐层向Decode传输；PULL模式：Decode一次性拉取Prefill的KV Cache | |
| `transfer_listen_port` | `int32` | 26000 | 任意可用的端口 | 启用PD分离后配置，对应每张卡上KV Cache Transfer的监听端口 | |
## MTP相关参数
| 参数名称 | 类型 | 默认值 | 其他值 | 参数含义 | 其他 |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `draft_model` | `string` | "" | | MTP模型所在的路径 | [详情](./features/mtp.md) |
| `draft_devices` | `string` | "npu:0" | 与`devices`格式保持一致，如`npu:0`或`npu:0,npu:1` | 与devices设置保持一致 | |
| `num_speculative_tokens` | `int32` | 0 | 任意整数，建议1或者2 | MTP模型每次step输出token的个数 | |
## 图执行相关参数
| 参数名称 | 类型 | 默认值 | 其他值 | 参数含义 | 其他 |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `enable_graph` | `bool` | false | true | 是否启用图执行模式优化decode阶段性能。仅用于decode阶段，对prefill阶段不生效。支持ACL Graph（NPU）、MLU Graph。 | [详情](./features/graph_mode.md) |
| `enable_graph_mode_decode_no_padding` | `bool` | false | true | decode阶段按实际`num_tokens`建图，而不是按padding后的shape建图 | |
| `enable_prefill_piecewise_graph` | `bool` | false | true | 是否启用prefill阶段的分段Graph。attention以eager执行，其他算子进入图捕获。 | |
| `max_tokens_for_graph_mode` | `int32` | 2048 | 任意大于等于0的整数 | 图执行模式最大token数。为0表示不限制。 | |
## 配套xLLM-service使用的参数
| 参数名称 | 类型 | 默认值 | 其他值 | 参数含义 | 其他 |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `etcd_addr` | `string` | "" | ip:port | etcd的rpc server监听地址 | |
| `enable_service_routing` | `bool` | false | true | 请求是否来自xllm service，当使用xllm service管理xllm实例时使用 | |
## 其他参数
| 参数名称 | 类型 | 默认值 | 其他值 | 参数含义 | 其他 |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `max_concurrent_requests` | `int32` | 200 | 任意大于等于0的整数 | 限流用，限制实例中正在处理的总请求数；设置为0表示不限流 | |
| `model_id` | `string` | "" | | 模型名称，非路径 | |
| `num_request_handling_threads` | `int32` | 4 | 任意大于0的整数 | 处理输入请求的线程池大小 | |
| `num_response_handling_threads` | `int32` | 4 | 任意大于0的整数 | 处理输出的线程池大小 | |
| `prefill_scheduling_memory_usage_threshold` | `double` | 0.95 | 0-1之间的值 | 当kv cache使用量达到该阈值时，暂停prefill请求的调度 | |
| `rank_tablefile` | `string` | "" | | 创建通信域的配置文件,多机场景需要 | |

1076
upstream_ref/xllm/docs/zh/design/generative_recommendation_design.md Normal file
View File

File diff suppressed because it is too large Load Diff

475
upstream_ref/xllm/docs/zh/design/graph_mode_design.md Normal file
View File

@@ -0,0 +1,475 @@
# Graph Mode 设计文档
## 概述
xLLM 的 Graph Mode 覆盖多种图执行后端。其目标是在推理服务场景中，将原本由 Host 频繁发起的小粒度 kernel 启动，转化为先捕获、后重放的图执行流程，从而降低 Host 调度开销、减少设备气泡，并提升吞吐和时延稳定性。
本文档面向需要理解实现原理和关键设计的开发者，重点介绍以下内容：
- Graph Mode 的原理与在 xLLM 中的落地方式
- 动态维度参数化
- Piecewise Graph
- 多 Shape 复用的显存池（含输入 Tensor 复用）
本文聚焦 Graph Mode 的统一设计，不展开不同后端之间的差异。
本文档的设计目标包括：
- 统一 xLLM 在不同图执行后端上的 Graph Mode 抽象
- 解释动态维度参数化、Piecewise Graph 与多 Shape 显存复用三项关键设计
- 说明这些设计分别解决什么问题、依赖什么前提、边界在哪里
本文档的非目标包括：
- 不覆盖所有算子或所有模型的适配细节
- 不替代功能文档中的参数说明与使用示例
相关设计文档：
- 若希望看一个更偏业务推理场景、并且聚焦固定调度、多步执行和定制算子的案例，可参考：[生成式推荐设计文档](generative_recommendation_design.md)
## 1. Graph Mode 原理和在 xLLM 中的落地
### 1.1 Graph Capture / Replay 的基本原理
传统 eager 执行模式下，模型一次 forward 会由 Host 连续发起大量 kernel、memcpy 和同步操作。对于 decode 这类单 step 计算较小但请求频繁的场景，Host 调度开销会比较显著，设备端也更容易出现执行气泡。
Graph Mode 的基本思路是：
1. **Capture 阶段**：第一次遇到某个 shape bucket 时，在专用 stream 上执行一次 forward，并把这条执行路径上的 kernel 启动、内存操作和依赖关系记录成图。
2. **Replay 阶段**：后续相同 bucket 的请求，不再由 Host 逐个下发 kernel，而是直接重放已捕获的图。
![alt text](../../assets/graph_mode.png)
这类机制通常要求：
- **执行路径稳定**
- graph capture 记录的是一次具体执行路径；capture 完成后，这条路径上的控制流、launch 形态和依赖关系就被固化下来
- 因此，capture 路径中无法在 replay 时再切换到另一套动态条件分支
- **关键 Tensor 地址稳定**
- capture 期间写入图中的关键 Tensor 地址，在 replay 时必须仍然可用
- **算子与结果语义稳定**
- replay 的正确性依赖路径上的算子与 Graph Mode 兼容，且不会因为运行时条件变化而改变语义
### 1.2 xLLM GraphMode 基础工作
从 1.1 的三条基本要求出发，前两项主要由 xLLM 运行时负责：一是把动态请求整理成可稳定 replay 的执行单元，二是保证 replay 时仍然访问 capture 阶段记录下来的固定地址。为此，xLLM 在 Graph Mode 下引入了统一的 Graph Executor，负责收敛分桶、持久化 buffer、graph cache 以及 capture / replay 生命周期管理。
xLLM 在运行时侧需要完成的基础工作，主要包括：
- **围绕执行路径稳定的图选择与执行调度**
- 请求需要先按 `num_tokens` 或相近 shape 归入 bucket
- bucket 维度维护 graph cache：已命中则直接 replay，未命中则先 capture 再缓存
- 对可整图捕获的路径执行完整图；存在 break graph 的路径则切换到 Piecewise Graph
- **围绕地址稳定的持久化 buffer 与 graph instance 管理**
- tokens、positions、seq_lens、block_tables 等动态输入不能在 replay 前重新分配到任意新地址， xLLM 需要先把这些输入写入持久化 buffer，再在固定地址上更新内容
- 模型计算图过程临时分配的Tensor都需要持久化保存对对应的graph instance中，不能随着作用域结束触发Tensor回收
- 启用共享内存池后，不同 shape 的 capture 可以复用同一组底层物理内存，但 graph 看到的虚拟地址仍需保持稳定
- **统一的 Graph Executor 封装**
- capture、cache、replay、图实例管理以及后端差异屏蔽，都需要由运行时统一处理
- 这层封装的目的，是把 Graph Mode 的运行时组织逻辑从模型实现中剥离出来，而不是把它们散落到各个 layer 或算子里
在具体执行上，Graph Executor 的运行时流程可以概括为：
1. **请求分桶**：按 `num_tokens` 或相近 shape 归类到 bucket
2. **输入准备**：将 tokens、positions、seq_lens、block_tables 等写入持久化 buffer，并更新 `attn_metadata`、`plan_info` 等动态元数据
3. **capture 或 replay 决策**：bucket 已命中则直接 replay；未命中则先 capture、缓存图，再进入 replay 路径
4. **执行图**：按场景执行完整图或 Piecewise Graph
而 1.1 中第三项“算子与结果语义稳定”，更多落在模型和算子本身的适配上，见 1.3。
### 1.3 模型适配 Graph Mode 需要的改造
在 xLLM 运行时负责分桶、buffer 和 graph instance 管理之后，模型侧仍然需要解决 1.1 中第三项要求：保证 capture / replay 期间的算子行为与结果语义稳定，不会因为 host 参与、dispatch 变化或 launch 形态变化而失效。
模型适配 Graph Mode 需要的改造主要包括：
- **去掉不支持 capture 的操作**
- 泛化要求：capture 路径中不能包含会触发 host 参与决策、host-device 隐式同步或额外控制流切换的操作，例如 stream 同步、读取 host tensor 决定逻辑分支、隐式 host-device 同步交互等
- 典型示例 1：算子内部依赖 host tensor 内容决定任务划分或执行模式。例如商发 8.3 的 ATB [PA](https://www.hiascend.com/document/detail/zh/canncommercial/83RC1/API/ascendtbapi/ascendtb_01_0197.html) 和 [MLA](https://www.hiascend.com/document/detail/zh/canncommercial/83RC1/API/ascendtbapi/ascendtb_01_0314.html)，会根据 host 侧kv_seq_lens和q_seq_lens来做不同的任务排列。
- 典型示例 2：host 标量参与 Tensor 计算，或 host 侧直接读取 Tensor 数据，例如：
```cpp
Tensor a;
Tensor b = a * 0.5; // 标量会被隐式传到 tensor 所在设备，产生同步 h2d
torch::Tensor max_of_seq = torch::max(input_params.kv_seq_lens);
max_seq_len_ = std::max(max_of_seq.item<int>(), max_seq_len_); // host 读取 tensor
```
- **算子适配**
- 泛化要求：需要保证计算图的 kernel 执行路径稳定。对于同一个输入 shape，每次执行计算图选择的 kernel 应保持一致，不发生 prefill、chunked prefill、decode 混杂；同时 `grid_dim`、`block_dim`、task 数量、workspace 形态和 tiling 内容也不能变化
- 典型示例：Ascned Cann商发8.3版本 的 ATB PA 会根据序列长度和 batch size 实时决定是否启用 flash-decoding 长序列模式；长序列模式和短序列模式的 tiling key 不同，dispatch 后实际 launch 的 `kernel_name` 也不同。这种“同一业务路径、不同运行时条件触发不同 kernel”的行为，在进入 Graph Mode 前需要先收敛
## 2. 动态维度参数化
### 2.1 问题
Graph Capture 记录的是固定的 kernel 执行序列和 launch 形态。Replay 时，运行时只能重复这套已经记录下来的 `grid_dim`、`block_dim`、task 数量和 tiling 路径，不能在同一张图里重新决定它们。
因此，请求里的动态维度进入 Graph Mode 后，并不都能在 replay 时继续变化。对 attention 来说，真实动态因素通常不止 `num_tokens`，还包括 `batch_size`、`q_seq_lens`、`kv_seq_lens`、`block_tables_size` 等。如果这些维度进一步影响 task 划分、workspace 布局、`tiling_params` 或 kernel 路径，那么 replay 就会沿用 capture 时的旧配置，从而出错。
本章要解决的问题就是：哪些动态信息可以在图选定后继续更新，哪些必须通过分桶或重新建图处理。
### 2.2 解法
动态维度参数化的解法，不是“让一张图接受任意 `num_tokens`”，而是在**图已经按 `bucket_num_tokens` 选定之后**，让同一个 bucket 尽量覆盖更多真实动态请求。
#### 2.2.1 参数化边界
一个维度能不能在 replay 时变化，关键不在于它是不是“请求里的动态值”，而在于它有没有在 capture 阶段被折叠进图的执行形态。
可以把动态因素分成两类：
1. **决定图形态的维度**
- 一旦进入 `grid_dim`、`block_dim`、task 数量、workspace 布局、tiling key 或 `tiling_params`，它就成为 capture 结果的一部分
- 这类维度变化后，通常不能在同一张 graph 内直接修改
2. **图选定后仍可更新的维度**
- 例如 `batch_size`、`q_seq_lens`、`kv_seq_lens`、`block_tables`、`new_cache_slots`、`plan_info`
- 它们只有在不改写执行形态时，才适合作为参数化对象
一个简化后的 norm kernel 例子是：
```cpp
int grid_dim = num_tokens;
NormKernel<<<grid_dim, block_dim>>>(x, y, ...);
```
这里的 `grid_dim`、`block_dim` 都属于 launch params。只要 `num_tokens` 已经参与了 launch 配置，它就属于图形态的一部分；capture 完成后，replay 就不能在同一张图里把 `grid_dim = 128` 改成 `grid_dim = 256`。
因此，动态维度参数化的判断标准很直接：凡是会改变 launch、workspace 或 tiling 路径的动态因素，都不应只靠 replay 前改参数解决。
#### 2.2.2 xLLM 中的处理方式
xLLM 的处理方式分两层。
第一层先处理 `num_tokens`：通过 bucket 化和 padding，把原始请求归一化到固定的 `bucket_num_tokens`，再据此选定图。
第二层再处理 bucket 内剩余的动态信息：对依赖 host tensor、host planning 或 host 侧 tiling 的算子做改造，把真正随请求变化的信息放到设备侧持久化 buffer 中，或者显式外置为 replay 前可更新的数据，而不是让 replay 继续绑定 capture 时那次 host 侧 planning 结果。
以 NPU attention 为例，改造前常见的问题是：`q_seq_lens`、`kv_seq_lens` 先在 Host 侧参与 planning，生成 task、workspace 或 tiling 相关数据，再发射 kernel：
```cpp
// Host side
auto plan_info = PlanAttention(q_seq_lens_host, kv_seq_lens_host, ...);
WritePlanToWorkspace(attention_workspace, plan_info);
AttentionKernel<<<grid_dim, block_dim>>>(
q, k, v, out,
attention_workspace,
tiling_params);
```
这类实现里，graph replay 不会自动重做 planning，因此只能重复 capture 时那次 launch 和那次 workspace 布局。改造后的方向是，在固定 launch 形态下，把 `q_seq_lens`、`kv_seq_lens`、`block_tables_size`、`plan_info` 等动态信息写入设备侧持久化 buffer，再由 kernel 在执行时读取，例如`kv_seq_lens`的参数化示例：
```cpp
struct AttentionLaunchArgs {
void* q;
void* k;
void* v;
void* out;
int32_t batch_size;
int32_t* kv_seq_lens; // device buffer
void* attention_workspace;
void* tiling_params;
};
LaunchAttentionKernel(args);
```
如果某个动态因素一旦变化就会改写任务数量、数组长度、workspace 形态或 tiling 路径，那么 xLLM 不会强行把它塞进“参数化更新”里，而是采用两种处理方式之一：
1. 扩大分桶维度，把这些因素纳入选图键，例如 `num_tokens` 和 `num_comm_tokens` 的组合
2. 把这段对动态 shape 敏感的执行留在 full graph 之外，交给 Piecewise Graph 处理
### 2.3 结果
动态维度参数化的直接结果是：同一个 `num_tokens` bucket 可以安全覆盖更多真实请求；在执行形态稳定的前提下，attention 等算子也能在 replay 时看到正确的 `batch_size`、`seq_lens`、`block_tables_size`、`plan_info` 等动态信息。
需要注意的是，参数化并不等于取消分桶。`num_tokens` 的动态性仍主要通过选图吸收；如果同一个 bucket 下还有其他会改写 tiling key、任务数量、通信规模或 kernel 路径的因素，那么这些因素仍然必须进入分桶键，或者转为 Piecewise Graph。通信相关场景就是典型例子，例如 Attention DP + MoE EP 下，通信规模可能取决于最大 DP Data Size，而不完全等于单卡本地的 `num_tokens`。
## 3. Piecewise Graph
### 3.1 问题
在 prefill、chunked prefill 等场景中，整条 forward 路径往往比 decode 更复杂。某些算子可能：
- 无法被稳定 capture
- 对动态元数据高度敏感
- 在 capture 模式下依赖额外的运行时准备逻辑
如果要求“整图都必须可捕获”，Graph Mode 的适用范围会非常受限。只要有一个关键算子 break graph，整条路径都无法使用图执行。
### 3.2 解法
#### 3.2.1 底层逻辑
在实现上，一个经常被问到的问题是：为什么 decode 可以使用 full graph，而 chunked prefill 往往只能使用 Piecewise Graph？
关键差别在 attention。
在当前 decode 语义下，单步通常是“每个序列生成一个 token”，因此 `num_tokens` 与 `batch_size` 基本是一一对应的。对 attention 来说，这意味着：
- `num_tokens` 增长时，`batch_size` 也同步增长
- `q_seq_lens` 的模式相对稳定
- 任务划分、workspace 需求和 launch args 变化主要可以跟着 `num_tokens` 一起分桶
因此，按 `num_tokens` 建 bucket 时，decode 路径满足 full graph 的复用条件。
但在 chunked prefill 中，`num_tokens` 和 `batch_size` 不再绑定。相同的 `num_tokens`，可能对应完全不同的请求组合，例如：
- `num_tokens = 128, batch_size = 2, q_seq_lens = [64, 64]`
- `num_tokens = 128, batch_size = 4, q_seq_lens = [32, 32, 32, 32]`
- `num_tokens = 128, batch_size = 8, q_seq_lens = [16, 16, 16, 16, 16, 16, 16, 16]`
对 attention 算子来说，这些请求虽然 `num_tokens` 相同，但并不是同一种 shape。attention 的任务数量划分、本地索引、`q_seq_lens` / `kv_seq_lens` 的长度、以及对应的 `plan_info` 都和 `batch_size` 有直接关系。如果系统只按 `num_tokens` 去复用之前记录的 graph，那么 attention 在 replay 时就可能沿用 capture 阶段那套“旧的 batch 划分”和“旧的任务分块”，从而导致结果错误。
这里的 `batch_size` 之所以难以简单参数化，是因为它影响的不只是某个标量值，而是整组 launch args 的结构：
- `q_seq_lens` / `kv_seq_lens` 数组长度会变
- attention 的任务数量和任务边界会变
- `plan_info` 和 workspace 布局会变
- kernel 内部使用的索引和分块逻辑会变
换句话说，`batch_size` 在 chunked prefill 中不是“改几个参数就行”的维度，而是会改变 attention 这一段图的执行形态。除非把 `num_tokens × batch_size` 甚至更细粒度的 `seq_lens` 组合都纳入分桶，也就是做高维笛卡尔积建图，否则仅靠 `num_tokens` 选图并不安全。
这也是为什么当前实现更适合：
- **decode**：attention 可以随 full graph 一起捕获
- **chunked prefill**：attention 留在图外，其他稳定部分进入 Piecewise Graph
#### 3.2.2 核心思路
Piecewise Graph 的目标不是强行整图捕获，而是把一条完整执行路径拆成多个 piece：
- 可图化的部分：捕获为多个子图
- 不适合图化的部分：保留为 eager 形式
在 replay 时，再按 capture 期间记录的顺序，把这些 piece 串起来执行。
这样可以在“无法整图”的前提下，尽量保留 Graph Mode 的收益。
下图以三层 Qwen3 decoder 为例展开了 Piecewise Graph 的执行顺序：attention 保留为独立 runner，而两次 attention 之间的非 attention 算子会被尽量压紧并组织成连续的 graph piece。
![Piecewise Graph 示意图](../../assets/piecewise_graph_diagram.svg)
#### 3.2.3 xLLM 中的实现方式
在 xLLM 中，Piecewise Graph 主要用于 prefill 类场景，由 `enable_prefill_piecewise_graph` 控制。
它的核心实现方式是：
1. 使用 `PiecewiseGraphs` 维护 replay 指令序列
2. 可捕获片段以图对象形式保存
3. attention 这类不直接进入图的部分，通过 `AttentionRunner` 记录为 runner
4. replay 时按原始顺序执行 `graph -> runner -> graph -> ...`
这套机制的关键点在于 attention 的处理：
- 在 piecewise capture 期间，attention 会暂时结束当前图捕获
- attention 本身不在 capture 时执行，而是把运行 attention 所需的张量、workspace 和参数保存下来
- replay 时，再基于最新的 `plan_info`、`q_cu_seq_lens`、`kv_cu_seq_lens` 执行 attention
这样做的直接原因就是：chunked prefill 下 attention 的动态维度不只由 `num_tokens` 决定，而当前图分桶主要按 `num_tokens` 组织。把 attention 留在图外，才能避免“同样的 `num_tokens` 命中了 graph，但 `batch_size` 和 `seq_lens` 已经变了”的 replay 错误。
因此，Piecewise Graph 的本质不是“图内支持 attention 的所有动态逻辑”，而是：
- 让可稳定 capture 的部分进入图
- 让对动态 shape 更敏感的 attention 保持单独执行
- 通过统一指令序列保证整条执行路径顺序不变
### 3.3 结果
这套做法带来的结果和边界主要体现在以下几个方面。
#### 3.3.1 适用场景
- prefill
- chunked prefill
- 局部 break graph，但整条路径仍希望获得图执行收益的场景
#### 3.3.2 优点
- 比“整图要么全成功、要么全放弃”更灵活
- 能显著扩大 Graph Mode 的覆盖范围
- 适合 attention 动态性强、而 MLP 等部分较稳定的模型结构
#### 3.3.3 限制
- 当前主要覆盖 prefill 类路径
- replay 依赖正确更新的 `attn_metadata.plan_info`
## 4. 多 Shape 复用的显存池
### 4.1 问题
Graph Mode 在多 shape 场景下面临两个同时出现的问题：
1. **捕获期显存随 shape 数量线性增长**
- 每次新的 shape 进入 capture，运行时通常会为它分配独立的 memory buffer pool
- 由于图执行要求 capture 和 replay 之间地址稳定，这些缓冲区往往不能及时释放
- 结果是显存占用从期望的 `max(shape)` 变成了 `sum(shape)`
2. **输入侧也会重复占用显存**
- 如果每个 shape 都维护一套独立的 tokens、positions、seq_lens、block_tables 等输入 tensor
- 那么即使图本身可以复用，输入侧显存也会继续按 shape 累积
因此，多 Shape 复用显存池要解决的问题，不只是节省显存，而是要在支持动态新 shape 的同时，把显存行为从 `sum(shape)` 收敛到 `max(shape)`。
具体目标包括：
1. **捕获期显存复用**：不同 shape 的 graph 共享同一组底层物理内存
2. **输入侧显存复用**：不同 shape 的 replay 共享同一组持久化输入 buffer
3. **地址不冲突**：每个已捕获 graph 仍然保有自己稳定的虚拟地址视图
4. **支持按需新建 shape**：新 shape 出现时可以继续 capture，而不必推倒已缓存的 graph
5. **不增加 replay 开销**：内存复用主要发生在 capture 和输入准备阶段，replay 路径尽量保持不变
#### 4.1.1 根本原因
- 以 CUDA 路径为例，allocator 往往会为每次 graph capture 分配独立的 memory buffer pool
- Graph Mode 又要求 capture 和 replay 之间地址稳定，因此旧 shape 对应的 buffer 不能像普通 eager 临时内存那样及时回收
- 当不同 shape 持续进入 capture 时，显存占用就会从期望的 `max(shape)` 累积到 `sum(shape)`
#### 4.1.2 约束条件
- 不同 shape 的 graph 在当前前提下不会同时 replay
- 需要支持动态出现的新 shape，按需进行 capture
- 不希望为了新 shape 推倒已缓存 graph，或重新捕获已经可用的旧 shape
- replay 路径不希望因为显存复用而引入额外的图执行复杂度
#### 4.1.3 失败方案与技术难点
一个更直接的想法是：每次 capture 前重置分配指针，让不同 shape 复用同一段 virtual address space。
但这条路在工程上并不安全。allocator 通常按地址跟踪已经分配的 block；如果新 capture 复用了旧地址，就可能覆盖原来的地址记录。这样一来，旧 graph 后续在 tensor 析构或释放时，就可能遇到地址记录缺失，甚至触发 `invalid device pointer` 一类错误。
这也暴露了问题的关键：Graph Mode 真正需要稳定的是“每张 graph 看到的地址视图”，而真正需要复用的是“底层 physical memory”。因此，技术难点不只是节省显存，而是要同时满足：
- 已捕获 graph 的 virtual address space 彼此不冲突
- 底层 physical memory 又能在多 shape 之间共享
- 新 shape 进入 capture 时，不破坏已有 graph 的可 replay 性
### 4.2 解法
多 Shape 复用的底层思路，不是让不同 shape 共用同一段虚拟地址，而是让它们共享同一组物理内存，同时为每个已捕获 graph 保留独立的虚拟地址视图。
这样设计的原因是：如果强行让不同 shape 复用同一段虚拟地址空间，allocator 的地址跟踪会发生冲突，旧 graph 释放或析构时可能找不到原来的地址记录，最终导致错误。换句话说，真正需要复用的是底层物理显存，而不是 graph 看到的虚拟地址。
为了说明为什么 xLLM 选择“共享 physical memory + 独立 virtual address space”，可以把几种方案对比如下：
| 方案 | 内存复用机制 | 物理内存结果 | 地址冲突风险 / 规避方式 | 主要局限 |
|------|--------------|--------------|--------------------------|----------|
| vLLM | 共享 graph memory pool + 大 shape 优先 capture | 可能继续接近 `sum(shape)` | 依赖共享 memory pool 和空闲块管理，通常无显式地址冲突 | 复用效果受捕获顺序和 allocator 行为影响 |
| SGLang | 全局 graph memory pool + 大 shape 优先 capture | 可能继续接近 `sum(shape)` | 依赖全局 pool / graph-private pools 管理规避冲突 | 不同 shape 较多时显存仍会继续增长 |
| xLLM | 多 virtual address space 映射同一组 physical memory | 收敛到 `max(shape)` | 每张 graph 保留独立地址视图，从根源避免地址记录冲突 | 依赖底层运行时提供 VMM 和 allocator 接入能力 |
xLLM 的选择不是复用同一段 virtual address space，而是让每个 graph 保留自己的地址视图，同时把真正可共享的部分下沉到底层 physical memory。
#### 4.2.1 输入 Tensor 多 Shape 复用
xLLM 先解决输入侧显存重复占用的问题。做法是：不再为每个 shape 分配独立输入 tensor，而是预分配一组“最大 shape”对应的持久化 buffer，然后让不同 shape 共享这组 buffer。
这组 buffer 通常包括：
- `tokens`
- `positions`
- `q_seq_lens`
- `kv_seq_lens`
- `block_tables`
- `new_cache_slots`
- `hidden_states`
replay 前的处理方式也比较直接：
1. 把当前请求的实际输入写入 buffer 前缀
2. 必要时对 padding 区域补零
3. 用 slice 视图构造当前 actual shape 的输入
这样做的结果是：输入侧显存不再按 shape 累加，而是稳定在一份 `max(shape)`。
#### 4.2.2 捕获期显存复用
解决输入侧之后，还需要解决 graph capture 本身的显存累积问题。
xLLM 的核心做法是：
- 每次 capture 使用**新的虚拟地址空间**
- 不同虚拟地址空间映射到**同一组物理内存**
这样可以同时满足两件事：
1. **对 graph 来说地址稳定**：每个已捕获 graph 都看到自己的固定虚拟地址
2. **对物理显存来说只保留一份**：底层物理内存按最大需求扩张，不再按 shape 叠加
相比“复用同一虚拟地址”的思路，这种做法规避了地址跟踪冲突问题，因此更适合和现有 allocator 机制协作。
这套方案依赖统一的虚拟内存管理能力，核心操作包括：
- 保留一段新的虚拟地址空间
- 创建或扩张底层物理内存
- 将同一组物理内存映射到多个虚拟地址空间
- 在需要时执行 unmap、释放地址空间和访问权限设置
其中最关键的特性是：**同一组物理内存可以同时映射到多个虚拟地址空间**。这正是“不同 shape 的 graph 地址互不冲突，但底层物理显存仍然共享”的基础。
#### 4.2.2.1 地址映射关系
Graph 捕获期显存复用的核心映射关系如下：
![多虚拟地址空间映射同一物理内存](../../assets/shared_vmm_mapping_diagram.svg)
这张图表达的是：随着 shape 逐步增大，每次 capture 都切换到新的虚拟地址空间，但三个 virtual space 复用的是同一个 physical memory pool。不同 shape 只会使用其中不同比例的页，而底层物理显存总占用收敛到 `max(shape)`。
### 4.3 结果
#### 4.3.1 内存效率指标
| 指标 | 显存不复用方案 | 显存复用方案 |
|------|----------------|--------------|
| 物理内存大小 | `sum(shape)` | `max(shape)` ✅ |
| 虚拟内存 | 多份 | 多份 |
| 稳定性 | 地址冲突风险 | 稳定 ✅ |
这里的关键变化是：物理显存收敛到 `max(shape)`，而 virtual address space 仍会随着已捕获 shape 数量增长。
#### 4.3.2 性能影响
- **Capture 阶段**：切换虚拟地址空间和扩张映射会带来额外开销
- **Replay 阶段**：无额外图执行开销，与常规 Graph Mode replay 基本一致
- **输入准备阶段**：增加一次写入持久化 buffer 和 slice 视图构造，但这部分通常远小于重复分配带来的成本
#### 4.3.3 虚拟地址空间开销
这套方案的主要代价之一，是 virtual address space 会随着已捕获 shape 数量持续增长。但这部分增长对应的是地址空间占用，而不是物理显存占用。
在 64 位系统下，virtual address space 通常远大于单卡显存规模，因此对当前这类 graph capture 场景来说，它通常不是首要瓶颈。换句话说，这套方案是用更充足的地址空间，换取物理显存从 `sum(shape)` 收敛到 `max(shape)`。
#### 4.3.4 结果验证
- 不同 shape 的 capture 可以共享同一组底层 physical memory
- 每次 capture 仍保有独立的 virtual address space，而不是复用同一组地址视图
- 地址冲突类问题不再通过“复用同一虚拟地址”这条路径暴露出来
- 显存占用从按 shape 累积的 `sum(shape)` 收敛到 `max(shape)`
#### 4.3.5 实际效果
- 多 shape 场景下，显存行为从“随 capture 次数线性累积”转为“物理显存按最大 shape 收敛”
- Graph 复用的稳定性提升，规避了直接复用同一虚拟地址空间带来的地址冲突问题
- 设计上支持动态新 shape 按需 capture，而不必推倒已有 graph 缓存
- replay 路径保持不变或基本不变，主要复杂度集中在 capture 和输入准备阶段
#### 4.3.6 适用前提与边界
- 虚拟地址空间会随着已捕获 shape 数量增长而增长，但物理显存只按最大需求扩张
- 方案依赖底层运行时提供虚拟地址空间管理、物理内存映射以及 allocator 接入能力
- 当前假设不同 shape 的 graph 不会同时 replay；正是这个前提让底层物理内存能够安全共享
- 当前实现仍以生命周期内统一持有物理内存和虚拟空间为主，不做细粒度动态释放
## 5. 参考文档
- [ACLGraph Capture / Replay 机制说明](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1alpha002/appdevg/acldevg/aclcppdevg_000519.html)
- [CudaGraph Capture 机制参考](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs)
- [vLLM 图执行实现参考](https://github.com/vllm-project/vllm/blob/main/vllm/v1/worker/gpu/cudagraph_utils.py)
- [SGLang 图执行实现参考](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/cuda_graph_runner.py)

27
upstream_ref/xllm/docs/zh/dev_guide/code_arch.md Normal file
View File

@@ -0,0 +1,27 @@
# 代码结构
```
├── xllm/
| : 主代码目录
│ ├── api_service/ # api服务化实现
│ ├── core/
│ │ : xllm核心功能代码目录
│ │ ├── common/
│ │ ├── distributed_runtime/ # 分布式PD服务实现
│ │ ├── framework/ # 引擎执行模块实现
│ │ ├── kernels/ # 国产芯片kernels适配实现
│ │ ├── layers/ # 模型层实现
│ │ ├── platform/ # 多平台兼容层
│ │ ├── runtime/ # worker/executor角色实现
│ │ ├── scheduler/ # 批调度与PD调度实现
│ │ └── util/
│ ├── function_call # function call实现
│ ├── models/ # 模型实现
│ ├── processors/ # 多模态模型预处理实现
│ ├── proto/ # 通信协议
│ ├── pybind/ # python接口
| └── server/ # xLLM服务实例
├── examples/ # 服务调用示例
├── tools/ # NPU Timeline生成工具
└── xllm.cpp # xLLM启动入口
```

396
upstream_ref/xllm/docs/zh/dev_guide/tilelang_ascend_kernel_dev.md Normal file
View File

@@ -0,0 +1,396 @@
# xLLM Ascend TileLang Kernel 开发指南
本文说明在 xLLM 中新增或修改 Ascend TileLang kernel 的开发方式。示例全程使用当前的 `rope` kernel。
相关目录：
- Python kernel 定义：`xllm/xllm/compiler/tilelang/targets/ascend/kernels`
- NPU runtime wrapper：`xllm/xllm/core/kernels/npu/tilelang`
构建和测试应在 NPU 容器中执行。
## 1. 先判断修改类型
- 新增 `specialization`
- 给现有 kernel 增加一组新的编译参数组合
- 仍复用同一个 wrapper、同一套 runtime dispatch 字段和同一套 C ABI
- 典型动作是修改 `DISPATCH_SCHEMA` 或 `SPECIALIZATIONS`
- 新增 `kernel`
- 新增一个新的逻辑算子
- 典型动作是新增 Python kernel 文件、wrapper C++ 文件和一条 CMake 接线
对 `rope` 来说：
- 给 `SPECIALIZATIONS` 增加一项 `{"variant_key": "...", "head_dim": ..., "rope_dim": ..., "dtype": ...}`，这是新增 `specialization`
- 新增一个新的 `xxx_wrapper.cpp` 对外接口，这是新增 `kernel`
## 2. 开发顺序
推荐按下面顺序开发：
1. 在 `rope.py` 这类 Python 文件里先写 TileLang kernel 实现
2. 实现 `generate_source(...)`，把 kernel lower 成 Ascend-C 源码
3. 声明 `DISPATCH_SCHEMA` 和 `SPECIALIZATIONS`
4. 先生成一次 `registry.inc` 并查看内容
5. 再写或修改 wrapper 里的 runtime specialization 构造逻辑
6. 接入 CMake 并运行测试
这个顺序的重点是：
- 先把 kernel 本身实现出来
- 再把 runtime dispatch schema 固定下来
- 最后根据生成出来的 `registry.inc` 写 wrapper
## 3. 编写 Python Kernel
以 `rope.py` 为例，Python 侧可以按三层理解：
- `build_rope_kernel(...)`：kernel 实现
- `generate_source(...)`：AOT 导出
- `RopeKernel`：注册 kernel，并声明 dispatch schema 与编译实例
### 3.1 实现 `build_rope_kernel(...)`
`build_rope_kernel(...)` 才是 TileLang kernel 的实现主体。这里负责写：
- `@T.prim_func`
- 输入输出张量 shape
- `with T.Kernel(...)` 下的并行任务组织
- UB 分配和实际计算逻辑
`rope.py` 的精简骨架如下：
```python
def build_rope_kernel(
head_dim: int,
rope_dim: int,
vec_core_num: int,
ub_buffer_bytes: int,
):
task_num = vec_core_num
m_num = vec_core_num // 2
@T.prim_func
def rope_in_place_kernel(...):
with T.Kernel(m_num, is_npu=True) as (cid, vid):
task_id = cid * 2 + vid
...
return rope_in_place_kernel
```
这里的 `head_dim`、`rope_dim` 是这一组实现真正依赖的编译参数。
`rope` 这类 vector kernel 还要遵守当前 AOT 使用方式下的固定任务约定。当前路径是 AOT 编译，kernel launch 的 `block_num` 会在编译时固定下来，因此：
- 运行时输入 shape 不影响 kernel launch 的 `block_num`
- 运行时输入 shape 只影响固定任务之间的 workload 切分
当前 `rope.py` 的约定是：
```python
task_num = vec_core_num
m_num = vec_core_num // 2
with T.Kernel(m_num, is_npu=True) as (cid, vid):
task_id = cid * 2 + vid
```
这表示：
- `cid` 范围是 `[0, vec_core_num // 2)`
- `vid` 范围是 `[0, 2)`
- 总任务数固定为 `task_num = vec_core_num`
因此，`rope.py` 在推导单个 specialization 的编译 token 数时，也按固定任务数计算：
```python
max_rows_num_in_ub = _derive_max_rows_num_in_ub(...)
compile_num_tokens = task_num * max_rows_num_in_ub
```
### 3.2 实现 `generate_source(...)`
`generate_source(...)` 负责把上面的 TileLang kernel lower 成最终源码。导出层的职责，是把一组 specialization 参数转换成可编译的 Ascend-C 源码。
对 `rope` 来说，核心逻辑如下：
```python
@staticmethod
def generate_source(head_dim: int, rope_dim: int, dtype: str) -> str:
vec_core_num = detect_vec_core_num()
tilelang_kernel = build_rope_kernel(
head_dim=head_dim,
rope_dim=rope_dim,
vec_core_num=vec_core_num,
ub_buffer_bytes=FIXED_UB_BUFFER_BYTES,
)
with tilelang.tvm.transform.PassContext(...):
kernel = tilelang.engine.lower(tilelang_kernel)
return kernel.kernel_source
```
这里的规则是：
- `generate_source(...)` 的输入来自当前这组 `SPECIALIZATIONS`
- `generate_source(...)` 内部调用 `build_rope_kernel(...)`
- 返回值是 lower 后的源码字符串
### 3.3 声明 `DISPATCH_SCHEMA` 与 `SPECIALIZATIONS`
当 kernel 实现和导出层写完后，再通过 `@register_kernel` 类把它接入框架。
`rope.py` 当前的最小模板如下：
```python
from ....common.spec import DispatchField, TilelangKernel, register_kernel
@register_kernel
class RopeKernel(TilelangKernel):
DISPATCH_SCHEMA = [
DispatchField("head_dim", "int32"),
DispatchField("rope_dim", "int32"),
DispatchField("dtype", "dtype"),
]
SPECIALIZATIONS = [
{
"variant_key": "hd128_rd128_bf16",
"head_dim": 128,
"rope_dim": 128,
"dtype": "bf16",
},
{
"variant_key": "hd576_rd64_bf16",
"head_dim": 576,
"rope_dim": 64,
"dtype": "bf16",
},
]
@staticmethod
def generate_source(head_dim: int, rope_dim: int, dtype: str) -> str:
...
```
这里要分清楚两个概念：
- `DISPATCH_SCHEMA`
- 定义 runtime specialization 的字段名、顺序和类型
- 是 C++ 侧 specialization struct、builder 和查表接口的单一真相源
- `SPECIALIZATIONS`
- 表示要实际编译出的实例集合
- 每一项都对应一个 variant
规则如下：
- `DISPATCH_SCHEMA` 中每个字段都必须出现在每一项 `SPECIALIZATIONS` 里
- `SPECIALIZATIONS` 中可以有额外字段，这些字段会传给 `generate_source(...)`，但不会进入 runtime dispatch schema
- `variant_key` 是这一组 specialization 的唯一标识
- `DISPATCH_SCHEMA` 和 `SPECIALIZATIONS` 必须与 runtime specialization 一一对应
对 `rope` 来说，runtime dispatch 维度是：
- `head_dim`
- `rope_dim`
- `dtype`
所以这三个字段必须同时出现在：
- `DISPATCH_SCHEMA`
- 每一项 `SPECIALIZATIONS`
构建时，Ascend build 会根据主构建路径传入的 `--device a2|a3` 解析实际使用的 `bisheng_arch`。
### 3.4 查看生成的 Ascend-C 源码
在调试 `build_rope_kernel(...)` 的实现细节，或者比较不同 kernel 写法对最终代码生成的影响时，建议通过公共入口 `compile-kernels` 重新生成产物，再查看对应 specialization 的 Ascend-C 源码。
对 `rope` 来说，可以先固定：
- `head_dim=576`
- `rope_dim=64`
- `dtype=bf16`
然后重新生成 `rope` 的编译产物：
```bash
python xllm/compiler/tilelang_launcher.py compile-kernels \
--target ascend \
--device a3 \
--output-root /tmp/tilelang_debug \
--kernels rope \
--force
```
这里建议带上 `--force`，保证源码和 object 会按当前修改重新生成，不直接命中旧 cache。
这里把 `--output-root` 指到独立的调试目录 `/tmp/tilelang_debug`，这样当前命令只会在这个目录下生成 `rope` 的调试产物，不会和主构建目录里的其他 kernel 产物混在一起。
执行后，可以直接查看对应 specialization 的源码中的入口函数、UB 分配和向量计算逻辑：
```bash
sed -n '1,200p' \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
rg -n 'extern "C"|__global__|alloc_ub|alloc_shared|g_tilingKey' \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
```
如果要比较两种 kernel 写法的差异，做法是保持 specialization 不变，在修改前后各执行一次 `compile-kernels --force`，再对生成的 `.cpp` 做 diff：
```bash
cp /tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp \
/tmp/rope_before.cpp
diff -u /tmp/rope_before.cpp \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
```
这样可以把“specialization 变化”和“kernel 实现变化”分开看。
执行后，可以重点查看这些文件：
- `/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp`
- `/tmp/tilelang_debug/targets/ascend/rope/registry.inc`
- `/tmp/tilelang_debug/targets/ascend/rope/manifest.json`
这三类文件分别对应：
- 某个 specialization 的最终 Ascend-C 源码
- wrapper 会直接包含的 runtime dispatch 接口
- 当前 kernel 的全部编译产物记录
调试顺序建议是：
1. 先执行 `compile-kernels --force` 重新生成当前 kernel 的产物
2. 查看对应 specialization 的 `.cpp`，分析代码生成结果
3. 再看 `registry.inc` 和 `manifest.json` 是否符合预期
4. 最后通过 `rope_wrapper_test` 看完整接入后的结果和性能
## 4. 修改 Wrapper
新增 `kernel` 时，需要新增 wrapper。新增 `specialization` 时，只有 runtime specialization 语义变化，才需要同步修改 wrapper。
对 `rope_wrapper.cpp` 来说，人工需要保留的内容是：
- tensor shape、dtype、layout 校验
- 把输入整理成 `x_rows / sin_rows / cos_rows`
- 从 tensor 构造 runtime specialization
- 组装 launch 参数并调用 `entry->fn(...)`
### 4.1 `registry.inc` 会自动生成什么
`registry.inc` 由 Python 侧的 `DISPATCH_SCHEMA`、`SPECIALIZATIONS` 和导出出来的 Ascend-C ABI 自动生成。
对 `rope` 来说，生成内容包括：
- `RopeSpecialization`
- `RopeHeadDim`
- `RopeRopeDim`
- `RopeDType`
- `RopeKernelFn`
- `make_rope_specialization(...)`
- `find_rope_kernel_entry(...)`
- `available_rope_variant_keys()`
对 `rope_wrapper.cpp` 来说，`registry.inc` 会直接提供 `RopeSpecialization`、`operator==(...)`、`RopeKernelFn` 等 dispatch 相关定义。`dtype` 转换统一使用公共函数 `to_tilelang_dtype(...)`。
### 4.2 wrapper 里真正要写的东西
`rope_wrapper.cpp` 中最关键的人工逻辑，是从 tensor 构造 runtime specialization。当前写法如下：
```cpp
RopeSpecialization build_runtime_specialization(const torch::Tensor& x_rows) {
return make_rope_specialization(
RopeHeadDim{static_cast<int32_t>(x_rows.stride(0))},
RopeRopeDim{static_cast<int32_t>(x_rows.size(1))},
RopeDType{to_tilelang_dtype(x_rows.scalar_type())});
}
```
对 `rope` 而言：
- `head_dim` 对应 `x_rows.stride(0)`，也就是 kernel 使用的 `x_stride`
- `rope_dim` 对应 `x_rows.size(1)`
- `dtype` 对应 `x_rows.scalar_type()`
运行时路径是：
1. wrapper 把输入整理成 `x_rows / sin_rows / cos_rows`
2. `build_runtime_specialization(...)` 从 `x_rows` 构造 specialization
3. `find_rope_kernel_entry(...)` 在静态 registry 中精确匹配
4. 命中后通过 `entry->fn(...)` 调用实际编译出来的符号
当前查找策略是线性扫描加精确匹配。只要 `head_dim`、`rope_dim`、`dtype` 有一个不一致，就不会命中。
所以新增 `specialization` 时，要重点核对的是：
- Python 侧 `DISPATCH_SCHEMA` 的字段语义
- Python 侧 `SPECIALIZATIONS` 的字段值
- wrapper 里 `build_runtime_specialization(...)` 构造出的字段值
这三者必须完全对齐。
### 4.3 先生成并查看 `registry.inc`
在写或修改 wrapper 之前，先生成一次 `registry.inc` 并查看内容。重点看：
- 生成出来的 `RopeSpecialization` 字段顺序是否符合预期
- 生成出来的字段包装类型名是否符合预期
- `make_rope_specialization(...)` 的参数顺序是什么
- 生成出来的 entry symbol 名称是什么
`registry.inc` 是 wrapper 的直接契约，先看它，再写 wrapper。
## 5. 修改 CMake
新增 `kernel` 时，在 `xllm/xllm/core/kernels/npu/tilelang/CMakeLists.txt` 中接入这个 kernel。
CMake 接入统一通过高层 helper 完成：
- `tilelang_register_runtime_kernel(NAME <kernel> WRAPPER_SRCS <srcs...>)`
以 `rope` 为例，最小模板如下：
```cmake
tilelang_register_runtime_kernel(
NAME rope
WRAPPER_SRCS rope_wrapper.cpp
)
```
这条 helper 会完成：
- 按 `TILELANG_GENERATED_ROOT/targets/ascend/<kernel>/manifest.json` 推导 manifest 路径
- 导入 manifest
- 把该 kernel 的 wrapper source 和 compiled objects 加入 `tilelang_kernels`
- 自动追加 `XLLM_TL_<KERNEL>_REGISTRY_INC=...` compile definition
因此，新增一个 runtime kernel 时，CMake 侧主要就是两件事：
1. 保证 Python 侧已经能生成该 kernel 的 manifest
2. 在 `tilelang` 的 CMakeLists 里新增一条 `tilelang_register_runtime_kernel(...)`
日常新增 kernel 时，直接在 CMake 中增加一条 `tilelang_register_runtime_kernel(...)`。`tilelang_import_kernel_manifest(...)` 作为这条高层 helper 的实现基础，保留在底层。
## 6. 验证
推荐按下面顺序验证：
1. 先编译 TileLang kernel，并查看生成的 `registry.inc`
2. 再跑完整的 wrapper 测试
常用命令：
```bash
python xllm/compiler/tilelang_launcher.py compile-kernels \
--target ascend \
--device a3 \
--output-root build/cmake.linux-aarch64-cpython-311/xllm/compiler/tilelang \
--kernels rope
python setup.py test --test-name rope_wrapper_test --device a3
```
第一条命令用于生成 `manifest.json`、`registry.inc` 和 object；第二条命令用于验证完整接入路径。

33
upstream_ref/xllm/docs/zh/features/async_schedule.md Normal file
View File

@@ -0,0 +1,33 @@
# 异步调度
## 背景
大模型推理过程可划分为3个阶段，包括CPU执行调度准备模型输入阶段，device计算阶段，CPU处理输出阶段。
由于解码操作的序列性，step-i+1 的输入需要依赖 step-i 的输出结果，
上述3个阶段需要按顺序串行执行，导致在CPU执行阶段1和3的时候，device侧空闲等待出现空泡，资源利用不充分。
## 功能介绍
xLLM在框架层支持了异步调度功能，在device执行 step-i 计算的同时提前让CPU执行 step-i+1 的调度操作，device在完成 step-i 计算后可立即开始 step-i+1 的计算，从而消除空泡。
具体地，CPU在发起 step-i 计算调用后，不等待device计算完成，为 step-i 的请求构造fake token，使用fake token执行 step-i+1 的调度操作，分配KV Cache等；device在启动 step-i+1 的计算时，用 step-i 计算出来的true token替换fake token，保证计算的正确性。CPU在另外的线程中同步处理 step-i 的结果返回给client。
整体架构如图，实现中CPU侧执行阶段1和阶段3的操作分别采用了不同的线程池，rpc等函数调用采用C++ future和promise非阻塞调用，实现全异步runtime。![异步调度](../../assets/async_schedule_architecture.jpg)
## 使用方式
xLLM中提供了gflags参数`enable_schedule_overlap`，默认false，如需开启在xLLM的服务启动脚本中设置为true即可，示例如下：
```shell
--enable_schedule_overlap=true
```
## 性能效果
- 异步调度开启后，两个step之间的device空闲时在200us左右，基本类似一个kernel launch的时间。
- 在DeepSeek-R1-Distill-Qwen-1.5B模型上，限制TPOT 50ms，吞吐 **提升17%**。
!!! warning "注意"
- 异步调度功能会在服务端额外计算一个step，当使用场景中输出token数量较少，或是类似embedding模型只一次性输出的场景，会影响服务端吞吐，所以强制关闭异步调度。
- VLM模型正在适配中，暂时会强制关闭异步调度。

9
upstream_ref/xllm/docs/zh/features/basics.md Normal file
View File

@@ -0,0 +1,9 @@
# 基础知识
- xLLM使用一卡一进程模式，多卡之间使用rpc进行函数调用，模型计算过程中的数据通信使用device集合通信库。
- HCCL/LCCL是高性能集合通信，提供单机多卡以及多机多卡间的数据并行、模型并行集合通信方案。

17
upstream_ref/xllm/docs/zh/features/chunked_scheduler.md Normal file
View File

@@ -0,0 +1,17 @@
# ChunkedPrefill调度器
## 功能介绍
xLLM支持chunked prefill调度策略。Chunked prefill是一种优化大语言模型推理的技术，将长prompt分割成多个较小的chunk进行分批处理，而不是一次性处理整个prompt。
这种方法可以有效降低显存峰值使用量，提高Device利用率，并且能够更好地与decode阶段的请求进行调度和混合处理。
## 使用方式
上述策略已在xLLM实现，并向外暴露gflag参数，控制功能的开关。
- 开启chunked prefill，并设置chunked_size，如果不手动设置chunked size，则默认等于max_tokens_per_batch。
```bash
--enable_chunked_prefill=true
--max_tokens_per_chunk_for_prefill=20480 # optional
```
## 性能效果
开启chunked_prefill之后，在Qwen3-8B模型上，限制TPOT 50ms，TTFT时延 **下降46%**。

7
upstream_ref/xllm/docs/zh/features/continuous_scheduler.md Normal file
View File

@@ -0,0 +1,7 @@
# Continuous调度器
## 功能介绍
xLLM实现了支持continuous batching的调度策略，continuous_batch是一种动态批处理策略，它不等待批次填满，而是在有请求时就开始处理，同时持续接收新请求并将其加入正在执行的批次中，从而在保持高吞吐量的同时显著降低延迟。
## 使用方式
xLLM提供了continuous batching调度策略。目前`enable_chunked_prefill`默认值为true，开箱即用时默认调度器是chunked prefill调度器。

74
upstream_ref/xllm/docs/zh/features/disagg_pd.md Normal file
View File

@@ -0,0 +1,74 @@
# PD分离
## 背景
LLM在线推理服务通常需要满足TTFT和TPOT两项性能指标，而传统的Contiguous Batching调度策略将Prefill和Decode请求混合在一起调度，导致P和D会互相抢占计算资源，影响性能指标无法最大程度的利用计算资源。为解决上述矛盾，将Prefill和Decode两阶段拆分到独立的计算资源并行执行，从而同时降低TTFT和TPOT并提升吞吐量。
## 功能介绍
xLLM PD分离功能主要通过以下三个模块实现：
- **etcd**: 存储实例信息等元数据
- **xLLM Service**: 调度请求和管理所有计算实例
- **xLLM**: 请求计算实例
整体架构图如下：
![xLLM PD分离架构图](../../assets/pd_architecture.jpg)
## 功能使用示例
### 使用准备
#### 安装相关依赖
- **xLLM**: 参见[安装编译](../getting_started/quick_start.md)
- **xLLM Service**: 参见[PD分离部署](../getting_started/disagg_pd.md)
### 启动PD分离服务
1. 启动etcd
```
./etcd
```
2. 启动xLLM Service
```bash
ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
3. 启动xLLM
4. 以Qwen2-7B为例
- 启动Prefill实例
```bash
/path/to/xllm --model=Qwen2-7B-Instruct \
--port=8010 \
--devices="npu:0" \
--master_node_addr="127.0.0.1:18888" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=PREFILL \
--etcd_addr=127.0.0.1:12389 \
--transfer_listen_port=26000 \
--disagg_pd_port=7777 \
--node_rank=0 \
--nnodes=1
```
- 启动Decode实例
```bash
/path/to/xllm --model=Qwen2-7B-Instruct \
--port=8020 \
--devices="npu:1" \
--master_node_addr="127.0.0.1:18898" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=DECODE \
--etcd_addr=127.0.0.1:12389 \
--transfer_listen_port=26100 \
--disagg_pd_port=7787 \
--node_rank=0 \
--nnodes=1
```
需要注意：
- PD分离需要读取`/etc/hccn.conf`文件，确保将物理机上的该文件映射到了容器中
- `etcd_addr`需与`xllm_service`的`etcd_addr`相同
!!! warning "注意事项"
PD分离目前不支持开启prefix cache及chunked prefill功能，需要通过以下参数关闭
``` shell
--enable_prefix_cache=false
--enable_chunked_prefill=false
```

35
upstream_ref/xllm/docs/zh/features/eplb.md Normal file
View File

@@ -0,0 +1,35 @@
# MoE负载均衡（EPLB）
## 背景介绍
MoE模型依赖动态路由分配tokens给专家，但实际部署中因数据分布不均，导致专家负载失衡（部分过载、部分闲置）。专家冗余调整（如新增/删除副本）需要消耗额外显存，并可能因权重迁移影响推理延迟，如何高效、平滑地完成是一大挑战。为此，采用专家冗余策略（复制热点专家）结合分层和全局动态负载均衡实现了动态的MOE负载均衡。
## 功能介绍
xLLM MoE负载均衡（EPLB）功能主要通过以下三个模块实现：
- eplb manager: 负责专家负载并收集并管理专家分布更新更新，采用逐层更新机制，根据专家负载变化情况判断是否更新该层。
- eplb excutor: 实际专家分布更新执行器。
- eplb policy: 新专家负载表生成策略。
整体架构图如下：
![xLLM eplb](../../assets/eplb_architecture.png)
## 使用方式
只需在启动 xLLM 时加上下面的 gflag 参数即可：
替换为实际的Device个数 ep_size要与device个数保持一致
- xLLM中提供了gflags参数`enable_eplb`，默认false，如需开启动态专家负载均衡，在xLLM的服务启动脚本中设置为true即可。
- `expert_parallel_degree`与`ep_size`为moe相关参数，`expert_parallel_degree`需要设置为`2`，`ep_size`要与实际NPU/GPU卡个数保持一致。参考 [moe_params](./moe_params.md)
- `eplb_update_interval`为专家分布更新时间间隔，单位为妙，默认值为1000.
- 专家分布更新采用根据专家负载的逐层更新机制，当某一层专家的前后两次的负载相似度小于`eplb_update_interval`时选择更新该层，默认值为1，取之范围为(0,1)。
```bash
--enable_eplb=true
--expert_parallel_degree=2
--ep_size=16
--eplb_update_interval=2000
--eplb_update_threshold=0.9
```
## 未来工作
* 采用更加细粒度的专家更新机制。
* 与调度层结合，通过请求batch的重组实现更好的负载均衡。

34
upstream_ref/xllm/docs/zh/features/global_kvcache.md Normal file
View File

@@ -0,0 +1,34 @@
# 全局多级KV Cache
## 背景
大型语言模型（LLM）解码阶段因自回归生成需频繁访问历史KV缓存，导致显存带宽成为瓶颈。随着模型规模与上下文窗口扩大（如128K Token消耗超40GB显存），单卡显存压力剧增。现有方案（如vLLM）在长上下文场景下存在明显局限：预填充耗时激增、解码阶段显存带宽争抢严重，为满足SLO（TTFT<2s, TBT<100ms）常需过量预留资源，致使GPU利用率不足40%，且难以利用跨服务器资源。为此，我们提出分布式全局多级KV缓存管理系统，采用存算一体架构以突破单机资源限制。
## 功能介绍
xLLM 全局KV Cache功能主要通过以下三个模块实现：
- etcd: 集群服务注册、负载信息同步及全局缓存状态管理
- xLLM Service: 调度请求和管理所有计算实例
- xLLM: 请求计算实例
整体架构图如下：
![xLLM 全局多级KV Cache](../../assets/globalkvcache_architecture.png)
## 功能使用示例
### 使用准备
#### 安装相关依赖
- **xLLM**: 参见[快速开始](../getting_started/quick_start.md)
- **xLLM Service**: 参见[PD分离部署](../getting_started/disagg_pd.md)
### 使用方式
1. etcd启动配置：
```bash
./etcd --listen-peer-urls=http://0.0.0.0:10999 --listen-client-urls=http://0.0.0.0:10998
```
2. xLLM Service启动配置：
```bash
./xllm_master_serving --etcd_addr="127.0.0.1:10998" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
3. xLLM启动添加上下面的 gflag 参数即可：
```bash
--enable_service_routing=true
--enable_cache_upload=true
# PD分离暂时不支持全局KVCache管理
--enable_disagg_pd=false
```

77
upstream_ref/xllm/docs/zh/features/graph_mode.md Normal file
View File

@@ -0,0 +1,77 @@
# Graph Mode
## 概述
xLLM 支持 Graph Mode，通过预捕获计算图并在后续执行中重放，减少 CPU 开销并提高推理性能。Graph Mode 在不同硬件平台上均有对应实现。
## 功能介绍
为了优化 Host 侧调度性能，图模式通过在 CPU 一次提交大任务后，设备内部流式执行小 kernel，显著降低启动时间和设备气泡。
在 xLLM 引擎中，Graph Mode 实现了以下特性：
### 动态维度参数化
- 将除 num_tokens 以外的关键动态维度作为整图输入参数，包括 batch_size、kv_seq_lens、q_seq_lens、block_table_size 等，从而提高灵活性。在进行图的内存分配和内核配置时，利用这些动态参数计算实际所需值。在图启动阶段，将上述实际参数传入，以确保 kernel 能够使用正确的 stride 访问数据。
### Piecewise Graph
- 当部分算子不支持 graph 导致整图无法捕获（break graph）时，对 break 之后的各段（piece）分别捕获 graph。这样在无法整图捕获的情况下，仍能尽可能获得 graph mode 的收益，常用于 prefill、chunked_prefill 等场景。
### 多 shape 复用的显存池
- 为了避免不同 shape 的 graph capture 分别占用独立显存，我们让不同 capture 使用不同虚拟地址空间，并共享同一组底层物理内存；同时，输入 tensor 通过持久化 buffer 与 slice 方式复用。
## 使用方式
上述功能已在 xLLM 引擎内部实现，通常通过 gflags 参数控制。
最小配置只需要开启 `enable_graph`，用于打开 decode 阶段的 Graph Mode：
```shell
--enable_graph=true
```
常见的配套开关包括：
- `enable_graph`：开启 decode 阶段的 Graph Mode 基础能力
- `enable_prefill_piecewise_graph`：开启 prefill 阶段的 Piecewise Graph
- `enable_graph_mode_decode_no_padding`：decode 阶段按实际 `num_tokens` 建图，而不是按 padding 后的 shape 建图
- `max_tokens_for_graph_mode`：限制 Graph Mode 覆盖的最大 token 数；`0` 表示不限制
如果希望同时开启 decode Graph 和 prefill Piecewise Graph，示例如下：
```shell
--enable_graph=true \
--enable_prefill_piecewise_graph=true \
--max_tokens_for_graph_mode=2048
```
如果需要在 decode 阶段启用无 padding 建图，可额外开启：
```shell
--enable_graph=true \
--enable_graph_mode_decode_no_padding=true
```
更完整的参数说明可参考 [CLI 参数说明](../cli_reference.md)。
## 性能效果
- 开启 Graph Mode 后，在 Qwen3-0.6B 和 Qwen3-1.7B 等模型上，decode 阶段吞吐 **提升约 8%–10%**。
## 模型支持
下表列出目前各模型在 ACLGraph、CudaGraph、MLUGraph 上的支持情况。
| 模型 | ACLGraph | CudaGraph | MLUGraph |
|------|----------|-----------|----------|
| Qwen3/Qwen3-MoE | ✅ | ✅ | ✅ |
| DeepseekV3.2 | ✅ | | ✅ |
| GLM4.5/4.6/4.7 | ✅ | | |
| Qwen2.5-VL | | | ✅ |
| Qwen3-VL/Qwen3-VL-MoE | ✅ | | |
| GLM4V | ✅ | | |
| GLM4V-MoE | ✅ | | |
## 相关文档
- 更详细的 Graph Mode 设计与实现说明（含 ACL Graph / CUDA Graph 基本原理、动态维度参数化、Piecewise Graph 与多 shape 复用内存方案）见：[Graph Mode 设计文档](../design/graph_mode_design.md)

41
upstream_ref/xllm/docs/zh/features/groupgemm.md Normal file
View File

@@ -0,0 +1,41 @@
# GroupGEMM算子优化
# 背景
混合专家(Mixture of Experts, MoE)架构已成为扩展大规模语言模型的重要范式，其核心思想是将输入token动态路由至不同的专家子网络进行处理。在推理过程中，GroupGEMM算子是MoE架构的关键计算单元，负责高效执行多个专家矩阵乘法的并行计算，且在整个推理耗时中占据主导地位。
## 功能介绍
结合当前GroupGEMM的性能瓶颈为I/O受限，提出了一种优化方案，通过索引重排替代数据拷贝，取消了对token向量的多次复制，改为维护专家分配的索引表。通过该行号索引，直接将token映射到相应的专家计算单元，并将token的分配调度与矩阵乘法融合为一个单一的kernel。
## 用户接口
### 算子直调API
```c++
aclnnStatus aclnnIndexGroupMatmulGetWorkspaceSize(
const aclTensorList *x,
const aclTensorList *weight,
const aclTensorList *scale,
const aclTensorList *perTokenScale,
const aclTensor *groupList,
const aclTensorList *out,
uint64_t *workspaceSize,
aclOpExecutor **executor);
aclnnStatus aclnnIndexGroupMatmul(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream);
```
- `x`: 输入的张量列表，包含待处理的数据。
- `weight`: 权重张量，包含模型的参数。
- `scale`: 缩放因子，用于调整输入张量的值。
- `perTokenScale`:每个token的缩放因子，用于动态调整。
- `groupList`: 专家组列表，指示哪些专家参与计算。
- `out`: 输出张量列表，存储计算结果。
## 性能效果
![groupmatmul](../../assets/groupmatmul_performance.png)
* 优化后的GroupMatmul算子在计算时间上表现出明显的优势，尤其是在k为128，m为64情况下，如图所示，优化后算子计算延时 **减少50%**。

17
upstream_ref/xllm/docs/zh/features/moe_params.md Normal file
View File

@@ -0,0 +1,17 @@
# EP并行
## 背景介绍
在部署DeepSeek-R1 671B参数规模模型时传统分布式部署面临、显存利用率低、通信开销大、硬件成本高昂等核心瓶颈，因此需要引入ep并行。
+ 在同等资源下，单张卡上的Expert越少，可用于KV Cache的显存越多，可Cache的token个数越多。
+ 因MLA的特性，同等资源下TP Size越小，冗余的KV Cache就越少，可Cache的token个数越多。
+ 采用大规模ep并行部署，可以将同一个expert的token计算集中到同一设备上，提高硬件利用率
## 参数设置
+ dp_size：设置Attention部分的dp规模大小，默认值为1，可设置为2的指数倍，当dp_size不等于卡数时，dp组内为tp并行.
+ ep_size：设置MoE部分的ep规模大小，默认值为1，可设置为2的指数倍，当ep_size不等于卡数时，dp组内为tp并行.
+ expert_parallel_degree ：ep并行相关参数，不开启ep时默认设置为0，开启ep时默认为1，此时为ep level1，当ep_size等于卡数时可以设置为2开启ep level2.
支持 MLA 的模型会自动开启 MLA，不再需要手动配置。
## 方案设计
+ 当开启ep时，默认为ep level1，此时attn与moe部分计算完成后，通过All Gather全卡通讯将数据发送到下一阶段，以64卡attn部分dp32tp2 moe部分ep32tp2为例，执行流程如下：
![Alt text](../../assets/moe_eplevel1.jpg)
+ 当ep_size设置为卡数时，可以开启ep level2，此时attn部分与moe部分之间通讯变为ALL2ALL，只向需要的卡发送数据，降低通讯量与通讯开销，以64卡部署为例，执行流程如下：
![Alt text](../../assets/moe_eplevel2.jpg)

149
upstream_ref/xllm/docs/zh/features/mtp.md Normal file
View File

@@ -0,0 +1,149 @@
# MTP投机推理
## 背景
MTP是一种创新的推理阶段加速技术，专注于解决大语言模型生成过程中的效率瓶颈。MTP的本质是通过预训练阶段的特殊设计，为推理阶段提供高效的草稿token预测能力，从而显著提升模型的生成速度。其核心价值在于平衡推理效率与输出质量，为大语言模型的长序列生成问题提供了一种高效的解决方案，最终实现推理性能的优化。
## 功能介绍
MTP在推理加速方面具有以下核心功能：
- **高效草稿生成**：使用低成本的MTP结构快速生成草稿token，这些草稿token作为主模型验证的基础，大幅减少了传统自回归生成的计算开销。
- **批量验证机制**：主模型能够同时批量验证多个MTP生成的草稿token，而不必逐个生成和验证，显著提升了推理速度。
- **高采样准确率**：MTP解决了Eagle、Medusa等现有推理加速方法中的关键痛点——训练后生成的draft模块token采样率低的问题。由于MTP在预训练阶段就优化了草稿生成能力，其生成的token具有更高的准确率，减少了主模型的验证负担。
- **推理延迟降低**：通过预先生成多个可能的后续token，MTP有效降低了模型生成长文本时的累积延迟，使用户体验更加流畅。
- **资源消耗优化**：相比其他推理加速技术，MTP在保持加速效果的同时，对计算资源的额外需求更少，适合在资源受限环境下部署。
MTP技术为大语言模型的推理阶段提供了一种全新的效率优化方案，特别适合需要快速响应的实时应用场景，代表了语言模型推理优化的重要发展方向。
!!! note "模型支持"
目前支持以下模型的MTP结构导出：
- DeepSeek-V3 (输入 model_type: deepseek_v3, 导出 MTP model_type: deepseek_v3_mtp)
- DeepSeek-V3.2 (输入 model_type: deepseek_v3, 导出 MTP model_type: deepseek_v32_mtp)
- DeepSeek-R1 (输入 model_type: deepseek_v3, 导出 MTP model_type: deepseek_v3_mtp)
- GLM4 MoE (如 GLM-4.5-Air, 导出 MTP model_type: glm4_moe_mtp)
注意：
- DeepSeek V3 和 R1 的输入 model_type 都是 "deepseek_v3"，导出的 MTP 模型 model_type 为 "deepseek_v3_mtp"
- DeepSeek V3.2 的输入 model_type 是 "deepseek_v3"（但可通过 index_head_dim 等字段自动识别），导出的 MTP 模型 model_type 为 "deepseek_v32_mtp"
## 使用示例
### 导出模型
脚本会自动检测模型类型，也可以手动指定。
#### DeepSeek-V3
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-V3 \
--output-dir /path/to/DeepSeek-V3-mtp
```
#### DeepSeek-V3.2
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-V3.2 \
--output-dir /path/to/DeepSeek-V3.2-mtp
```
#### DeepSeek-R1
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-R1 \
--output-dir /path/to/DeepSeek-R1-mtp
```
#### GLM4 MoE
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/GLM-4.5-Air \
--output-dir /path/to/GLM-4.5-Air-mtp
```
#### 手动指定模型类型
如果自动检测失败，可以手动指定模型类型：
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/model \
--output-dir /path/to/model-mtp \
--model-type deepseek_v3 # 可选: deepseek_v3 (用于V3/R1), deepseek_v32 (用于V3.2), glm4_moe
```
输入模型参考:
- [DeepSeek-V3](https://huggingface.co/deepseek-ai/DeepSeek-V3)
- [DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)
- [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1)
- [GLM-4.5-Air](https://huggingface.co/zai-org/GLM-4.5-Air)
### 启动脚本
使用MTP进行推理时，需要同时指定主模型和草稿模型（MTP模型）。
#### DeepSeek-V3/V3.2/R1 启动示例
```bash
MODEL_PATH="/models/DeepSeek-V3"
DRAFT_MODEL_PATH="/models/DeepSeek-V3-mtp"
MASTER_NODE_ADDR="127.0.0.1:42123"
START_PORT=13222
START_DEVICE=0
LOG_DIR="log"
NNODES=16
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
nohup ./xllm \
--model $MODEL_PATH \
--devices="npu:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--draft_model $DRAFT_MODEL_PATH \
--draft_devices="npu:$DEVICE" \
--num_speculative_tokens 1 \
--max_memory_utilization=0.90 \
--max_tokens_per_batch=10000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--ep_size=1 \
--dp_size=1 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--node_rank=$i > $LOG_FILE 2>&1 &
sleep 0.5
done
```
#### GLM4 MoE 启动示例
```bash
MODEL_PATH="/models/GLM-4.5-Air"
DRAFT_MODEL_PATH="/models/GLM-4.5-Air-mtp"
# ... 其他配置相同
```
# 性能数据
基于sharegpt数据集，输入长度2500，输出长度1500，请求总数80。
| method | Concurrency | Mean TPOT(ms) | Mean TTFT(ms) | Output Tokens/s | Total Tokens/s |
|:---------:|:-----------:|:-------------:|:-------------:|:---------------:|:--------------:|
| baseline | 1 | 40.61 | 141.80 | 24.20 | 65.77 |
| mtp | 1 | 28.33 | 142.35 | 35.19 | 95.52 |
| baseline | 2 | 42.69 | 178.59 | 45.16 | 122.74 |
| mtp | 2 | 29.81 | 187.97 | 64.75 | 175.78 |
| baseline | 4 | 46.18 | 172.34 | 79.83 | 216.96 |
| mtp | 4 | 33.54 | 194.22 | 111.18 | 301.81 |
| baseline | 8 | 53.16 | 181.49 | 110.68 | 300.81 |
| mtp | 8 | 40.99 | 203.37 | 154.46 | 419.34 |
| baseline | 16 | 68.50 | 213.89 | 143.81 | 390.84 |
| mtp | 16 | 57.04 | 254.99 | 201.89 | 548.04 |
| baseline | 20 | 74.72 | 228.80 | 154.77 | 420.65 |
| mtp | 20 | 61.73 | 264.34 | 206.24 | 559.84 |
| baseline | 40 | 119.68 | 559.32 | 180.22 | 489.80 |
| mtp | 40 | 105.70 | 544.54 | 252.91 | 686.74 |
| baseline | 80 | 180.89 | 2996.21 | 192.09 | 522.06 |
| mtp | 80 | 152.19 | 2163.72 | 278.07 | 755.12 |

29
upstream_ref/xllm/docs/zh/features/multi_streams.md Normal file
View File

@@ -0,0 +1,29 @@
# 多流并行
## 背景
大模型分布式推理场景中需要引入额外的通信操作，将不同设备上的计算结果聚合在一起。以Deepseek这类大规模的MoE模型为例，分布式规模通常较大，通信开销也会随之变大。计算和通信都采用同一个stream的话，在通信的同时，device计算资源会出现浪费，一直等待通信完成才能开始后面的计算。
## 功能介绍
xLLM在模型图层支持了多流并行功能，将输入的batch拆分成2个micro batches，一个流执行一个micro batch的计算操作，另一个流执行另一个micro batch的通信操作，计算和通信同时执行，从而掩盖通信开销。
![异步调度](../../assets/multi_streams_architecture.jpg)
## 使用方式
xLLM中提供了gflags参数`enable_multi_stream_parallel`，默认false，如需开启在xLLM的服务启动脚本中设置为true即可，示例如下：
```shell
--enable_multi_stream_parallel=true
```
## 性能效果
prefill双流并行开启后，基本可掩盖75以上的通信开销，在DeepSeek-R1模型上，只输出1个token的情况下
- TTFT下降 **7%**
- 吞吐 **提升7%**
!!! warning "注意"
双流并行目前只支持prefill阶段，请求输入越长，收益越大。
目前仅支持DeepSeek、Qwen3 dense（非MoE）模型。

19
upstream_ref/xllm/docs/zh/features/multimodal.md Executable file
View File

@@ -0,0 +1,19 @@
# 多模态支持
本文档主要介绍xLLM推理引擎中多模态的支持进展，包括支持模型及模态类型，以及离在线接口等。
## 支持模型
- Qwen2.5-VL: 包括7B/32B/72B。
- Qwen3-VL: 包括2B/4B/8B/32B。
- Qwen3-VL-MoE: 包括A3B/A22B。
- MiniCPM-V-2_6: 7B。
## 模态类型
- 图片: 支持单图、多图的输入，以及图片+Prompt组合、纯文本Promot等输入方式。
!!! warning "注意事项"
- 目前多模态后端不支持prefix cache以及chunk prefill，正在支持中。
- 目前，xLLM统一基于JinJa渲染ChatTemplate，部署MiniCPM-V-2_6，模型目录需提供ChatTemplate文件。
- 图片支持Base64输入以及图片Url。
- 目前多模态模型主要支持了图片模态，视频、音频等模态正在推进中。

57
upstream_ref/xllm/docs/zh/features/overview.md Normal file
View File

@@ -0,0 +1,57 @@
# 整体架构
## 背景
近年来，随着百亿至万亿参数规模的大语言模型（如GPT、Claude、DeepSeek、LLaMA等）在自然语言处理和多模态交互领域取得突破性进展，产业界对高效推理引擎与服务体系的构建提出了迫切需求。如何降低集群推理成本、提升推理效率已成为实现规模化商业落地的关键挑战。
尽管当前已涌现出一批面向大模型推理的优化引擎，但在实际部署过程中仍面临诸多技术瓶颈：
- 硬件适配性挑战：现有推理引擎对国产芯片等专用加速器的架构特性支持不足，难以充分发挥异构计算硬件的性能潜力，导致计算资源利用率低下；
- MoE架构优化难题：专家并行机制中的令牌分发过程产生显著的All-to-All通信开销，同时动态路由策略引发的专家负载不均衡问题严重制约了系统的可扩展性；
- 长上下文管理瓶颈：随着模型上下文窗口持续扩展，KV缓存在内存碎片化处理、跨节点同步等方面的优化效率直接影响整体推理吞吐性能；
- 混合部署效能局限：现有推理集群在同时处理在线服务和离线任务时，难以兼顾服务质量（SLO）保障与资源利用率优化。
- 动态PD适配不足：当输入/输出序列长度出现剧烈波动时，静态PD资源划分缺乏实时调整PD资源配置的能力，既可能导致GPU资源闲置，又存在SLO违约风险。
为此，我们提出了xLLM——高效且易用的开源智能推理框架，为模型在国产芯片上的推理提供企业级服务保障与高性能引擎计算能力。
## 功能介绍
xLLM提供智能计算能力，我们实现了多种计算系统层和算法驱动层的联合推理加速：
### 计算系统层
#### 多层流水线执行编排
在框架层异步化CPU调度，使其与芯片推理计算形成流水线，减少计算空泡；在模型图层，切分单个batch，形成两个micro-batches之间的流水线，重叠计算通信；在算子内核层，不同计算单元间流水，重叠计算访存。
#### 动态shape的图执行优化
针对大语言模型处理动态输入（如可变序列长度和批大小）时面临的静态图适配问题，xLLM通过参数化设计捕获输入维度实现动态适配，并结合多图缓存方案减小编译开销，使用受管控的显存池替代绝对地址保障安全复用，最终在保持高灵活性的同时获得较高的执行效率。
#### 算子优化
xLLM实现了LLM中的关键算子在国产硬件芯片上的特定优化，包括GroupMatmul、Chunked Prefill等。
#### xTensor显存管理
xTensor 显存管理框架采用 物理内存页池预分配 + 虚拟地址连续性映射 的方法，通过动态按需映射物理页、复用可重用内存页（Reusable）及异步预映射优化调度，结合 NPU 算子适配（如虚拟地址化 FlashMLA），实现了高效动态内存管理，取得了内存利用率提升以及延迟降低。
### 算法驱动层
#### PD分离
xLLM全面支持PD分离场景，实现了高效的PD实例的管理以及PD实例之间的通信以及kv cache传输。
#### 全局调度
xLLM对请求和实例做全周期的资源调度智能管理。
##### 实例调度
我们实现了多种实例调度策略来选择如何将实例分配到更适合的实例。包括简单的Round Robin策略，基于请求在各实例上的 prefix cache 命中率来选择的 prefix cache-aware 策略，基于实例上的显存空闲程度的 KV Cache-aware 策略。另外，针对PD分离场景，由于静态的PD比例往往无法很好应对流量以及请求输入输出长度突变的场景，我们实现了一种自适应的PD动态调度器，负责在线请求的全局实例分配与运行时PD动态调整。
##### 请求调度
我们实现了多种请求调度策略，支持continuous batching，包括chunked prefill，prefill优先和decode优先等batch策略，同时全面支持PD分离场景。
#### 全局kv cache管理
在全局层面采用ETCD作为元数据服务中间件，实现集群服务注册、负载信息同步及全局缓存状态管理。每个计算实例维护本地多级缓存池。在调度策略方面，系统采用基于 KV Cache 的动态决策机制：首先进行前缀匹配检测，计算各候选节点的 KV Cache 复用率，最终选择综合性能最优的节点进行处理，实现 KV Cache 的动态卸载与迁移。
#### 投机推理
xLLM内置优化后的投机推理算法，一次生成多个tokens提升吞吐。xLLM通过投机模块下沉减少通信成本，并使用调度和计算时序重叠优化、减少投机场景算子数据搬运等方式优化投机推理计算。
#### MOE负载均衡
xLLM针对MoE模型实现了基于历史专家负载统计的专家权重更新，在推理时通过高效专家负责统计和双缓冲无感知的专家权重更新实现有效的动态负载均衡。
### 多模态支持
xLLM对包括Qwen2-VL，MiniCPMV在内的多种多模态模型提供全面的支持。
### 相关设计文档
- [Graph Mode 设计文档](../design/graph_mode_design.md)
- [生成式推荐设计文档](../design/generative_recommendation_design.md)

36
upstream_ref/xllm/docs/zh/features/ppmatmul.md Normal file
View File

@@ -0,0 +1,36 @@
# PpMatmul 算子优化
## 背景
针对大模型推理中矩阵乘法占比高、耗时长的问题，优化了矩阵乘法算子的实现。
## 功能介绍
PpMatmul 算子使用 Tiling 切分策略，将矩阵乘法分解为多个小的矩阵乘法任务。然而当 tile 数量较小时任务无法被均匀分配到所有 npu 核心上，导致 tail effect 问题，影响计算效率。我们通过预取内存或重新划分任务的方式，优化 PpMatmul 算子的性能。
## 用户接口
### 算子直调 API
```cpp
aclnnStatus aclnnPpMatmulOptGetWorkspaceSize(
const aclTensor *a,
const aclTensor *b,
const aclTensor *out,
uint64_t *workspaceSize,
aclOpExecutor **executor);
aclnnStatus aclnnPpMatmulOpt(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream);
```
- `a`: 输入矩阵 A。
- `b`: 输入矩阵 B。
- `out`: 输出矩阵，存储计算结果。
## 性能效果
对于 tile 数量较小的情况（例如 M 较小，对应于 batch size 较小的情况），在（TP=4）时，算子较优化前有 **18%** 的性能提升。

20
upstream_ref/xllm/docs/zh/features/prefix_cache.md Normal file
View File

@@ -0,0 +1,20 @@
# Prefix Cache 优化
## 功能介绍
xLLM支持prefix_cache匹配。prefix_cache基于mermer_hash，使用lru淘汰策略，提供更极致的匹配效率，同时提高prefix_cache命中率。
同时对prefix_cache进行了优化，支持continuous_scheduler、chunked_scheduler和zero_evict_scheduler，在prefill之后即更新
prefix_cache，提高匹配时效性，同时对于chunked_scheduler，支持多阶段chunked_prefill匹配，减少计算量并尽可能减少kv_cache占用。
## 使用方式
prefix_cache已在xLLM实现，并向外暴露gflag参数，控制功能的开关。
- 开启zero_evict策略，并设置max_decode_token_per_sequence。
```
--enable_prefix_cache=true
```
## 性能效果
开启prefix_cache之后，在Qwen3-8B模型上，限制TPOT50ms，E2E时延 **下降10%**。
!!! warning "注意"
暂不支持PD分离调度器

27
upstream_ref/xllm/docs/zh/features/topk_topP.md Normal file
View File

@@ -0,0 +1,27 @@
# Topk&Topp算子优化
## 背景
在自然语言生成任务中，topK和topP采样策略被广泛应用于控制生成文本的多样性和质量。然而，在小模型中，这两种策略的计算耗时相对较长。这主要是由于小模型的参数较少，导致在处理概率分布时，排序和筛选的效率降低，从而影响了生成速度。因此，优化小模型中topK和topP的实现，可以提升其采样效率。
## 功能介绍
topKtopP算子的实现将排序、topK、softmax和topP等多个小算子融合为一个大算子，从而提高了计算效率和性能。
## 用户接口
### 算子调用API
```c++
void top_k_top_p(torch::Tensor& logits,
const torch::Tensor& topK,
const torch::Tensor& topP);
```
- `logits`: 输入的logits张量，包含模型的输出分数。
- `topK`: 用于选择的前K个概率的阈值张量。
- `topP`: 用于选择的累积概率的阈值张量。
## 性能效果
* 使用topKtopP融合算子后，在qwen2-0.5B模型中，TTOT **下降37%**,TTFT **提升10%**。

48
upstream_ref/xllm/docs/zh/features/xllm_service_overview.md Normal file
View File

@@ -0,0 +1,48 @@
# xLLM Service
[:simple-github: xLLM Service](https://github.com/jd-opensource/xllm-service)
## 简介
**xLLM-service** 是一个基于 xLLM 推理引擎开发的服务层框架，为集群化部署提供高效率、高容错、高灵活性的大模型推理服务。
xLLM-service 旨在解决企业级服务场景中的关键挑战：
- 如何于在离线混合部署环境中，保障在线服务的SLA，提升离线任务的资源利用率。
- 如何适应实际业务中动态变化的请求负载，如输入/输出长度出现剧烈波动。
- 解决多模态模型请求的性能瓶颈。
- 保障集群计算实例的高可靠性。
#### 背景
当前，百亿至万亿参数规模的大语言模型正快速部署于智能客服、实时推荐、内容生成等核心业务场景，对国产计算硬件的高效支持已成为低成本推理部署的核心需求。现有推理引擎难以有效适配国产芯片等专用加速器的架构特性，硬件计算单元利用率低、MoE 架构下的负载不均衡与通信开销瓶颈、kv 缓存管理困难等问题，制约了请求的高效推理与系统的可扩展性。xLLM-service + xLLM推理引擎提升了全链路效率，为大语言模型在实际业务中的规模化落地提供了关键技术支撑。
---
## 整体架构
xLLM-service 整体架构如图所示:
![1](../../assets/service_arch.png)
## 核心组件
### ETCD Cluster
用于元信息管理，包括模型，xllm实例，请求等元信息的存储与管理。同时提供xllm节点注册与发现服务。
### Fault Tolerance
xLLM-service 提供容错管理，保障服务质量以及稳定性。
### Global Scheduler
实现全局感知调度，根据当前系统状态，将请求精准调度至最优实例执行，有效提升整体服务响应效率与资源利用率。
### Global KV Cache Manager
负责全局 KV Cache 管理，核心能力包括分布式 KV 缓存感知、Prefix 前缀匹配、KV Cache 动态迁移等，优化缓存资源使用效率。
### Instance Manager
聚焦实例全生命周期管理，所有 xllm 实例启动后需向本模块注册，模块基于预设策略，为实例提供调度适配、容错处理等支持。
### Event Plane
作为指标与事件中枢，接收各实例上报的 Metrics 数据，对统计指标进行统一收集与整理，为服务调度、容错、扩缩容等决策提供数据支撑。
### Planner
承担策略分析与决策职能，基于 Event Plane 上报的 Metrics 数据（含实例运行时指标、机器负载指标等），分析服务扩缩容需求、热点实例扩展必要性，输出资源调整与实例优化策略。

17
upstream_ref/xllm/docs/zh/features/zero_evict_scheduler.md Normal file
View File

@@ -0,0 +1,17 @@
# Zero Evict调度器
## 功能介绍
xLLM支持zero_evict调度策略。zero_evict调度策略是一种尽可能减少请求淘汰率的调度算法，可以减少淘汰请求的prefill计算，减少TPOT。
这种调度算法通过模拟轮次，检测请求是否调度可以被调度且不导致其它请求被淘汰。
## 使用方式
上述策略已在xLLM实现，并向外暴露gflag参数，控制功能的开关。
- 开启zero_evict策略，并设置max_decode_token_per_sequence。
```
--use_zero_evict=true
--max_decode_token_per_sequence=256
```
## 性能效果
开启zero_evict之后，在Qwen3-8B模型上，限制E2E时延，TPOT时延 **下降27%**。

88
upstream_ref/xllm/docs/zh/getting_started/disagg_pd.md Normal file
View File

@@ -0,0 +1,88 @@
# PD分离部署
`xllm`支持PD分离部署，这需要与我们的另一个开源库[xllm service](https://github.com/jd-opensource/xllm-service)配套使用。
## xLLM Service依赖
首先，我们下载安装`xllm service`，与安装编译`xllm`类似：
```bash
git clone https://github.com/jd-opensource/xllm-service
cd xllm_service
git submodule init
git submodule update
```
### etcd安装
`xllm_service`依赖[etcd](https://github.com/etcd-io/etcd)，使用etcd官方提供的[安装脚本](https://github.com/etcd-io/etcd/releases)进行安装，其脚本提供的默认安装路径是`/tmp/etcd-download-test/etcd`，我们可以手动修改其脚本中的安装路径，也可以运行完脚本之后手动迁移：
```bash
mv /tmp/etcd-download-test/etcd /path/to/your/etcd
```
### xLLM Service编译
先应用patch:
```bash
sh prepare.sh
```
再执行编译:
```bash
mkdir -p build
cd build
cmake ..
make -j 8
cd ..
```
!!! warning "可能的错误"
这里能会遇到关于`boost-locale`和`boost-interprocess`的安装错误：`vcpkg-src/packages/boost-locale_x64-linux/include: No such file or directory`,`/vcpkg-src/packages/boost-interprocess_x64-linux/include: No such file or directory`
我们使用`vcpkg`重新安装这些包:
```bash
/path/to/vcpkg remove boost-locale boost-interprocess
/path/to/vcpkg install boost-locale:x64-linux
/path/to/vcpkg install boost-interprocess:x64-linux
```
## PD分离运行
启动etcd:
```bash
./etcd-download-test/etcd --listen-peer-urls 'http://localhost:2390' --listen-client-urls 'http://localhost:2389' --advertise-client-urls 'http://localhost:2391'
```
启动xllm service:
```bash
ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
以Qwen2-7B为例
- 启动Prefill实例
```bash
/path/to/xllm --model=path/to/Qwen2-7B-Instruct \
--port=8010 \
--devices="npu:0" \
--master_node_addr="127.0.0.1:18888" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=PREFILL \
--etcd_addr=127.0.0.1:12389 \
--transfer_listen_port=26000 \
--disagg_pd_port=7777 \
--node_rank=0 \
--nnodes=1
```
- 启动Decode实例
```bash
/path/to/xllm --model=path/to/Qwen2-7B-Instruct \
--port=8020 \
--devices="npu:1" \
--master_node_addr="127.0.0.1:18898" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=DECODE \
--etcd_addr=127.0.0.1:12389 \
--transfer_listen_port=26100 \
--disagg_pd_port=7787 \
--node_rank=0 \
--nnodes=1
```
需要注意：
- PD分离需要读取`/etc/hccn.conf`文件，确保将物理机上的该文件映射到了容器中
- `etcd_addr`需与`xllm_service`的`etcd_addr`相同
测试命令和上面类似，注意`curl http://localhost:{PORT}/v1/chat/completions ...`的`PORT`选择为启动xLLM service的`http_server_port`。

125
upstream_ref/xllm/docs/zh/getting_started/launch_xllm.md Normal file
View File

@@ -0,0 +1,125 @@
# 启动xllm
以Qwen3为例，启动xllm的脚本如下，给出的脚本适用于单机单卡和单机多卡，当使用单机多卡时，需要修改`NNODES`（一张卡就代表一个node），以及`ASCEND_RT_VISIBLE_DEVICES`或`CUDA_VISIBLE_DEVICES`或`MLU_VISIBLE_DEVICES`等环境变量。
## NPU
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_RT_VISIBLE_DEVICES=0
export HCCL_IF_BASE_PORT=43432 # HCCL 通信基础端口
MODEL_PATH="/path/to/model/Qwen3-8B" # 模型路径
MASTER_NODE_ADDR="127.0.0.1:9748" # Master 节点地址（需全局一致）
START_PORT=18000 # 服务起始端口
START_DEVICE=0 # 起始逻辑设备号
LOG_DIR="log" # 日志目录
NNODES=1 # 节点数（当前脚本启动 1 个进程）
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="npu:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--block_size=128 \
--communication_backend="hccl" \
--enable_prefix_cache=false \
--enable_chunked_prefill=true \
--enable_schedule_overlap=true \
--enable_shm=true \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
## NVIDIA GPU
```bash
#!/bin/bash
set -e
rm -rf core.*
export CUDA_VISIBLE_DEVICES=0
# for debug
# export CUDA_LAUNCH_BLOCKING=1
MODEL_PATH="/path/to/model/Qwen3-8B"
MASTER_NODE_ADDR="127.0.0.1:9748"
START_PORT=18000
START_DEVICE=0
LOG_DIR="log"
NNODES=1
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="cuda:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--block_size=32 \
--max_memory_utilization=0.8 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_schedule_overlap=true \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
## MLU
```bash
#!/bin/bash
set -e
rm -rf core.*
export MLU_VISIBLE_DEVICES=0
MODEL_PATH="/path/to/model/Qwen3-8B"
MASTER_NODE_ADDR="127.0.0.1:9748"
START_PORT=18000
START_DEVICE=0
LOG_DIR="log"
NNODES=1
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="mlu:$DEVICE" \
--port $PORT \
--nnodes=$NNODES \
--master_node_addr=$MASTER_NODE_ADDR \
--block_size=16 \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```

106
upstream_ref/xllm/docs/zh/getting_started/multi_machine.md Normal file
View File

@@ -0,0 +1,106 @@
# 多机部署
该示例为两机32卡启动示例，第一台机器服务:
```shell
bash start_deepseek_machine_1.sh
```
start_deepseek_machine_1.sh 脚本如下:
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export HCCL_IF_BASE_PORT=43432 # HCCL 通信基础端口
MODEL_PATH="/path/to/your/DeepSeek-R1" # 模型路径
MASTER_NODE_ADDR="123.123.123.123:9748" # Master 节点地址（需全局一致）
LOCAL_HOST=123.123.123.123 # 本机服务启动IP
START_PORT=18000 # 服务起始端口
START_DEVICE=0 # 起始 NPU 逻辑设备号
LOG_DIR="log" # 日志目录
LOCAL_NODES=16 # 单机节点数（当前脚本启动 16 个进程）
NNODES=32 # 总卡数（该示例为2机32卡）
mkdir -p $LOG_DIR
for (( i=0; i<$LOCAL_NODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--max_tokens_per_batch=40000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--communication_backend="hccl" \
--enable_schedule_overlap=true \
--rank_tablefile=./ranktable_2s_32p.json \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
启动第二台机器服务:
```shell
bash start_deepseek_machine_2.sh
```
start_deepseek_machine_2.sh 脚本如下:
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export HCCL_IF_BASE_PORT=43432 # HCCL 通信基础端口
MODEL_PATH="/path/to/your/DeepSeek-R1" # 模型路径
MASTER_NODE_ADDR="123.123.123.123:9748" # Master 节点地址（需全局一致）
LOCAL_HOST=456.456.456.456 # 本机服务启动IP
START_PORT=18000 # 服务起始端口
START_DEVICE=0 # 起始 NPU 逻辑设备号
LOG_DIR="log" # 日志目录
LOCAL_NODES=16 # 单机节点数（当前脚本启动 16 个进程）
NNODES=32 # 总卡数（该示例为2机32卡）
mkdir -p $LOG_DIR
for (( i=0; i<$LOCAL_NODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--max_tokens_per_batch=40000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--communication_backend="hccl" \
--enable_schedule_overlap=true \
--rank_tablefile=./ranktable_2s_32p.json \
--node_rank=$((i + LOCAL_NODES)) \ > $LOG_FILE 2>&1 &
done
```
这里使用了两台机器，可以通过 `--nnodes`设置总卡数，`--node_rank`为全局rank id。
`--rank_tablefile=./ranktable_2s_32p.json`为构建npu通信域所需文件，可参考[ranktable 生成](https://gitee.com/mindspore/models/blob/master/utils/hccl_tools/README.md)生成。

16
upstream_ref/xllm/docs/zh/getting_started/offline_service.md Normal file
View File

@@ -0,0 +1,16 @@
# 离线推理
为了方便用户快速使用xLLM进行离线推理，我们提供了启动离线推理的python脚本例子
## LLM
LLM推理示例：[:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate.py)
## Embedding
生成Embedding示例：[:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate_embedding.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate_embedding.py)
## VLM
VLM推理示例：[:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py)

224
upstream_ref/xllm/docs/zh/getting_started/online_service.md Normal file
View File

@@ -0,0 +1,224 @@
# 在线服务
先按照[xllm启动文档](launch_xllm.md)启动xllm服务。下面给出LLM和VLM的客户端调用示例，需要根据实际情况修改其中的参数。
## LLM 客户端调用
### HTTP 调用
chat模式：
```bash
curl http://localhost:9977/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"max_tokens": 10,
"temperature": 0,
"stream": true,
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "hello xllm"
}
]
}'
```
completions模式：
```bash
curl http://127.0.0.1:9977/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"prompt": "hello xllm",
"max_tokens": 10,
"temperature": 0,
"stream": true
}'
```
sample模式：
```bash
curl http://127.0.0.1:9977/v1/sample \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"prompt": "问题：<emb_0> 是否命中。结论：<emb_0>",
"selector": {
"type": "literal",
"value": "<emb_0>"
},
"logprobs": 5,
"request_id": "sample-demo-001"
}'
```
典型响应：
```json
{
"id": "sample-demo-001",
"object": "sample_completion",
"created": 1773369600,
"model": "Qwen2-7B-Instruct",
"choices": [
{
"index": 0,
"text": "True",
"logprobs": {
"tokens": ["True", "False"],
"token_ids": [3456, 7890],
"token_logprobs": [-0.12, -2.31]
},
"finish_reason": "selector_match"
},
{
"index": 1,
"text": "",
"logprobs": {
"tokens": [],
"token_ids": [],
"token_logprobs": []
},
"finish_reason": "empty_logprobs"
}
],
"usage": {
"prompt_tokens": 20,
"completion_tokens": 2,
"total_tokens": 22
}
}
```
`/v1/sample` 使用说明：
- 仅支持 `--backend=llm`，当前不支持 VLM/DiT/Rec。
- `selector.type` 当前固定为 `literal`，`selector.value` 按 prompt 文本顺序全文匹配。
- `logprobs` 默认值为 `5`，允许范围为 `[1, 5]`。
- `choices[i].index` 即该命中的 `sample_id`，与 prompt 中命中顺序一一对应。
- selector 无命中时返回 `200` 且 `choices=[]`；某命中位点无可用 logprobs 时返回 `finish_reason="empty_logprobs"`。
- 服务日志只记录 `request_id`、`sample_id`、`match_count`、`model` 等摘要字段，不记录完整 prompt。
`/v1/sample` 常见错误语义：
- 缺少 `model/prompt/selector/selector.value`、`selector.type != literal` 或 `logprobs` 越界时返回 `INVALID_ARGUMENT`。
- 模型不存在或后端不是 `llm` 时返回 `UNKNOWN`。
- 并发达到上限时返回 `RESOURCE_EXHAUSTED`。
- 模型处于 sleep 状态时返回 `UNAVAILABLE`。
### Python调用
```python
import requests
import json
url = f"http://localhost:9977/v1/chat/completions"
messages = [
{'role': 'user', 'content': "列出三个国家和他的首都。"}
]
request_data = {
"model": "Qwen2-7B-Instruct",
"messages": messages,
"stream": False,
"temperature": 0.6,
"max_tokens": 2048,
}
response = requests.post(url, json=request_data)
if response.status_code != 200:
print(response.status_code, response.text)
else:
ans = json.loads(response.text)["choices"]
print(ans[0]['message'])
```
## VLM 客户端调用
### HTTP API
```python
import base64
import requests
api_url = "http://localhost:12345/v1/chat/completions"
image_url = ""
def encode_image(url: str) -> str:
with requests.get(url) as response:
response.raise_for_status()
result = base64.b64encode(response.content).decode("utf-8")
return result
image_base64 = encode_image(image_url)
payload = {
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "介绍下这张图片"},
{
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{image_base64}"},
},
],
}
],
"model": "Qwen2.5-VL-7B-Instruct",
"max_completion_tokens": 128,
}
response = requests.post(
api_url,
json=payload,
headers={"Content-Type": "application/json"}
)
print(response.json())
```
### OpenAI API
```python
from openai import OpenAI
import base64
import requests
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:12345/v1"
image_url = ""
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
def encode_image(url: str) -> str:
with requests.get(url) as response:
response.raise_for_status()
result = base64.b64encode(response.content).decode("utf-8")
return result
image_base64 = encode_image(image_url)
chat_completion = client.chat.completions.create(
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "介绍下这张图片"},
{
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{image_base64}"},
},
],
}
],
model="Qwen2.5-VL-7B-Instruct",
max_completion_tokens=128,
)
result = chat_completion.choices[0].message.content
print("Chat completion output:", result)
```

106
upstream_ref/xllm/docs/zh/getting_started/quick_start.md Normal file
View File

@@ -0,0 +1,106 @@
# 快速开始
## 环境设置
所有的镜像都存放在[这里](https://quay.io/repository/jd_xllm/xllm-ai?tab=tags)，下面的docker启动命令以开发镜像为例。
### NPU
下面是我们构建好的开发镜像。
```bash
# A2 x86
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a2-x86-20260429
# A2 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a2-arm-20260429
# A3 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a3-arm-20260429
```
容器启动命令如下：
```bash
docker run -it \
--ipc=host \
-u 0 \
--name xllm-npu \
--privileged \
--network=host \
--device=/dev/davinci0 \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/local/sbin/:/usr/local/sbin/ \
-v /var/log/npu/conf/slog/slog.conf:/var/log/npu/conf/slog/slog.conf \
-v /var/log/npu/slog/:/var/log/npu/slog \
-v /var/log/npu/profiling/:/var/log/npu/profiling \
-v /var/log/npu/dump/:/var/log/npu/dump \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
### NVIDIA GPU
我们提供了NVIDIA GPU使用的[Dockerfile](../../../docker/Dockerfile.cuda)，可以构建自定义镜像，当然也可以使用我们根据默认Dockerfile构建的开发镜像：
```bash
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-cuda-x86
```
容器启动命令如下：
```bash
sudo docker run -it \
--privileged \
--shm-size '128gb' \
--ipc=host \
--net=host \
--pid=host \
--name=xllm-cuda \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
### MLU
我们无法提供MLU镜像，如果您已经拥有了相应的开发镜像，那么可以根据下面的命令启动容器：
```bash
sudo docker run -it \
--privileged \
--shm-size '128gb' \
--ipc=host \
--net=host \
--pid=host \
--name xllm-mlu \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
## 编译xllm
如果下载的是release镜像，即tag中带有版本号的镜像，可以跳过此步，因为release镜像自带编译好的xllm二进制文件，路径为`/usr/local/bin/xllm`。
下载xllm及依赖
```bash
git clone https://github.com/jd-opensource/xllm
cd xllm
# 第一次需要进行pre-commit安装
pip install pre-commit
pre-commit install
git submodule update --init --recursive
```
编译生成的二进制文件位于`/path/to/xllm/build/xllm/core/server/xllm`，在新镜像中，第一次编译xllm耗时较长，因为需要编译vcpkg中的所有依赖，但是后续编译会很快。
```bash
python setup.py build
```
## 启动xllm
请参考 [xllm启动方式](launch_xllm.md)。

583
upstream_ref/xllm/docs/zh/getting_started/quick_start_GLM5.md Normal file
View File

@@ -0,0 +1,583 @@
# 使用 xLLM 在 Ascend A3设备 推理 GLM-5.0-W8A8 基座模型
+ 源码地址：https://github.com/jd-opensource/xllm
+ 国内可用: https://gitcode.com/xLLM-AI/xllm
+ 权重下载: [modelscope-GLM-5-W8A8](https://www.modelscope.cn/models/Eco-Tech/GLM-5-W8A8-xLLM/files)
## 1.拉取镜像环境
首先下载xLLM提供的镜像：
```bash
# A2 x86
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a2-x86-20260429
# A2 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a2-arm-20260429
# A3 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a3-arm-20260429
```
**注意**: A2 机器性能未进行压测。
然后创建对应的容器
```bash
sudo docker run -it --ipc=host -u 0 --privileged --name mydocker --network=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /var/log/npu/conf/slog/slog.conf:/var/log/npu/conf/slog/slog.conf \
-v /var/log/npu/slog/:/var/log/npu/slog \
-v ~/.ssh:/root/.ssh \
-v /var/log/npu/profiling/:/var/log/npu/profiling \
-v /var/log/npu/dump/:/var/log/npu/dump \
-v /runtime/:/runtime/ -v /etc/hccn.conf:/etc/hccn.conf \
-v /export/home:/export/home \
-v /home/:/home/ \
-w /export/home \
quay.io/jd_xllm/xllm-ai:xllm-dev-hb-rc2-x86
```
## 2.拉取源码并编译
下载官方仓库与模块依赖：
```bash
git clone https://github.com/jd-opensource/xllm
cd xllm
git checkout preview/glm-5
git submodule init
git submodule update
```
下载安装依赖:
```bash
pip install --upgrade pre-commit
yum install numactl
```
执行编译，在`build/`下生成可执行文件`build/xllm/core/server/xllm`：
```bash
python setup.py build
```
## 3.启动模型
### 若机器为重启后初次拉起服务，需先执行以下脚本对device进行初始化
#若不执行且npu未初始化可能导致xllm进程拉起失败
```bash
python -c "import torch_npu
for i in range(16):torch_npu.npu.set_device(i)"
```
### 环境变量
```bash
##### 1， 配置依赖路径相关环境变量
# export PYTHON_INCLUDE_PATH="$(python3 -c 'from sysconfig import get_paths; print(get_paths()["include"])')"
# export PYTHON_LIB_PATH="$(python3 -c 'from sysconfig import get_paths; print(get_paths()["include"])')"
# export PYTORCH_NPU_INSTALL_PATH=/usr/local/libtorch_npu/
# export PYTORCH_INSTALL_PATH="$(python3 -c 'import torch, os; print(os.path.dirname(os.path.abspath(torch.__file__)))')"
# export LIBTORCH_ROOT="$(python3 -c 'import torch, os; print(os.path.dirname(os.path.abspath(torch.__file__)))')"
# export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/xllm/op_api/lib/:$LD_LIBRARY_PATH
# export LD_LIBRARY_PATH=/usr/local/libtorch_npu/lib:$LD_LIBRARY_PATH
export LD_PRELOAD=/usr/lib64/libjemalloc.so.2:$LD_PRELOAD
# source /usr/local/Ascend/ascend-toolkit/set_env.sh
# source /usr/local/Ascend/nnal/atb/set_env.sh
##### 2， 配置日志相关环境变量
rm -rf /root/ascend/log/
rm -rf core.*
##### 3. 配置性能、通信相关环境变量
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export NPU_MEMORY_FRACTION=0.96
export ATB_WORKSPACE_MEM_ALLOC_ALG_TYPE=3
export ATB_WORKSPACE_MEM_ALLOC_GLOBAL=1
export OMP_NUM_THREADS=12
export ALLOW_INTERNAL_FORMAT=1
export ATB_LAYER_INTERNAL_TENSOR_REUSE=1
export ATB_LLM_ENABLE_AUTO_TRANSPOSE=0
export ATB_CONVERT_NCHW_TO_AND=1
export ATB_LAUNCH_KERNEL_WITH_TILING=1
export ATB_OPERATION_EXECUTE_ASYNC=2
export ATB_CONTEXT_WORKSPACE_SIZE=0
export INF_NAN_MODE_ENABLE=1
export HCCL_EXEC_TIMEOUT=300
export HCCL_CONNECT_TIMEOUT=300
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_BASE_PORT=2864
```
## 启动命令 - GLM-5 （W8A8权重可单机拉起）
```bash
BATCH_SIZE=256
#推理最大batch数量
XLLM_PATH="./myxllm/xllm/build/xllm/core/server/xllm"
#推理入口文件路径（上一步中编译产物）
MODEL_PATH=/path/to/GLM-5-W8A8/
#模型路径（此处为int8量化的Glm-5）
DRAFT_MODEL_PATH=/path/to/GLM-5-W8A8/GLM-5-W8A8-MTP/
#Glm-5 导出的mtp权重
MASTER_NODE_ADDR="11.87.49.110:10015"
LOCAL_HOST="11.87.49.110"
# Service Port
START_PORT=18994
START_DEVICE=0
LOG_DIR="logs"
NNODES=16
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
nohup numactl -C $((DEVICE*40))-$((DEVICE*40+39)) $XLLM_PATH \
--model $MODEL_PATH \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--node_rank=$i \
--max_memory_utilization=0.85 \
--max_tokens_per_batch=8192 \
--max_seqs_per_batch=32 \
--block_size=128 \
--enable_prefix_cache=false \
--enable_chunked_prefill=true \
--communication_backend="hccl" \
--enable_schedule_overlap=true \
--enable_graph=true \
--enable_graph_mode_decode_no_padding=true \
--draft_model=$DRAFT_MODEL_PATH \
--draft_devices="npu:$DEVICE" \
--num_speculative_tokens=1 \
--ep_size=8 \
--dp_size=1 \
> $LOG_FILE 2>&1 &
done
# numactl -C xxxxx 亲和性绑核(NUMA亲和性查询命令： npu-smi info -t topo)
#--max_memory_utilization 单卡最大显存占用比例
#--max_tokens_per_batch 单batch最大token数 （主要限制prefill）
#--max_seqs_per_batch 单batch最大请求数 （主要限制decoe）
#--communication_backend 通信backend 可选(hccl / lccl) 此处建议hccl
#--enable_schedule_overlap 开启异步调度
#--enable_prefix_cache 开启prefix_cache
#--enable_chunked_prefill 开启chunked_prefill
#--enable_graph 开启aclgraph
#--draft_model mtp - mtp权重路径
#--draft_devices mtp - mtp推理设备(与主模型同一)
#--num_speculative_tokens mtp - 预测token数
```
日志出现"Brpc Server Started"表示服务成功拉起。
## 其他可选环境变量
```bash
#开启确定性计算
export LCCL_DETERMINISTIC=1
export HCCL_DETERMINISTIC=true
export ATB_MATMUL_SHUFFLE_K_ENABLE=0
# #开启动态profiling模式
# export PROFILING_MODE=dynamic
# \rm -rf ~/dynamic_profiling_socket_*
```
## 启动命令 - 双机拉起样例
### Node0 (master)
```bash
MASTER_NODE_ADDR="11.87.49.110:19990"
LOCAL_HOST="11.87.49.110"
START_PORT=15890
START_DEVICE=0
LOG_DIR="logs"
NNODES=32
LOCAL_NODES=16
export HCCL_IF_BASE_PORT=48439
unset HCCL_OP_EXPANSION_MODE
for (( i=0; i<$LOCAL_NODES; i++ ))do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i)); LOG_FILE="$LOG_DIR/node_$i.log"
nohup numactl -C $((DEVICE*40))-$((DEVICE*40+39)) $XLLM_PATH \ --model $MODEL_PATH \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--node_rank=$i \
--max_memory_utilization=0.85 \
--max_tokens_per_batch=8192 \
--max_seqs_per_batch=4 \
--block_size=128 \
--enable_prefix_cache=false \
--enable_chunked_prefill=true \
--communication_backend="hccl" \
--enable_schedule_overlap=true \
--enable_graph=true \
--enable_graph_mode_decode_no_padding=true \
--ep_size=16 \
--dp_size=1 \
--rank_tablefile=/yourPath/ranktable.json \
> $LOG_FILE 2>&1 &
done
```
#### Node1 (worker)
```bash
MASTER_NODE_ADDR="11.87.49.110:19990"
LOCAL_HOST="11.87.49.111"
START_PORT=15890
START_DEVICE=0
LOG_DIR="logs"
NNODES=32
LOCAL_NODES=16
export HCCL_IF_BASE_PORT=48439
unset HCCL_OP_EXPANSION_MODE
for (( i=0; i<$LOCAL_NODES; i++ ))do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i)); LOG_FILE="$LOG_DIR/node_$i.log"
nohup numactl -C $((DEVICE*40))-$((DEVICE*40+39)) $XLLM_PATH \ --model $MODEL_PATH \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--node_rank=$((i + LOCAL_NODES)) \
--max_memory_utilization=0.85 \
--max_tokens_per_batch=8192 \
--max_seqs_per_batch=4 \
--block_size=128 \
--enable_prefix_cache=false \
--enable_chunked_prefill=true \
--communication_backend="hccl" \
--enable_schedule_overlap=true \
--enable_graph=true \
--enable_graph_mode_decode_no_padding=true \
--ep_size=16 \
--dp_size=1 \
--rank_tablefile=/yourPath/ranktable.json \
> $LOG_FILE 2>&1 &
done
```
#### ranktable样例
ranktable配置指导：https://www.hiascend.com/document/detail/zh/canncommercial/83RC1/hccl/hcclug/hcclug_000014.html
```json
{
"version": "1.0",
"server_count": "2",
"server_list": [
{
"server_id": "11.87.49.110",
"device": [
{
"device_id": "0",
"device_ip": "11.86.23.210",
"rank_id": "0"
},
...
{
"device_id": "7",
"device_ip": "11.86.23.217",
"rank_id": "7"
}
],
"host_nic_ip": "reserve"
},
{
"server_id": "11.87.49.111",
"device": [
{
"device_id": "0",
"device_ip": "11.87.63.202",
"rank_id": "8"
},
...
{
"device_id": "7",
"device_ip": "11.87.63.209",
"rank_id": "15"
}
],
"host_nic_ip": "reserve"
}
],
"status": "completed"
}
```
## device NUMA亲和性查看
命令：
```bash
npu-smi info -t topo
```
前述命令中
```bash
numactl -C $((DEVICE*12))-$((DEVICE*12+11))
```
表示该进程绑在对应亲和的核上，可根据机器具体情况修改绑定的核id
## EX3.Glm-5 权重量化
### 安装msmodelslim
```bash
git clone https://gitcode.com/shenxiaolong/msmodelslim.git
cd msmodelslim
bash install.sh
```
### 修改tokenizer_config.json
```bash
"extra_special_tokens"
改成 "additional_special_tokens"
"tokenizer_class": "TokenizersBackend"
改成 "tokenizer_class": "PreTrainedTokenizer"
```
### 基于GLM-5-BF16 权重量化W8A8权重
```bash
### 预处理mtp相关权重
python example/GLM5/extract_mtp.py --model-dir ${model_path}
#指定transformers版本
pip install transformers==4.48.2
#量化执行（生成量化权重）
msmodelslim quant --model_path ${model_path} --save_path ${save_path} --model_type DeepSeek-V3.2 --quant_type w8a8 --trust_remote_code True
#拷贝chat_template文件
cp ${model_path}/chat_template.jinja ${save_path}
#量化mtp权重导出（用于xllm推理）
python example/GLM5/export_mtp.py --input-dir ${int8_save_path} --output-dir ${mtp_save_path}
```
## PD分离
### etcd\xllm-service 安装
#### PD分离部署
`xllm`支持PD分离部署，这需要与另一个开源库[xllm service](https://github.com/jd-opensource/xllm-service)配套使用。
##### xLLM Service依赖
首先，我们下载安装`xllm service`，与安装编译`xllm`类似：
```bash
git clone https://github.com/jd-opensource/xllm-service
cd xllm_service
git submodule init
git submodule update
```
##### etcd安装
`xllm_service`依赖[etcd](https://github.com/etcd-io/etcd)，使用etcd官方提供的[安装脚本](https://github.com/etcd-io/etcd/releases)进行安装，其脚本提供的默认安装路径是`/tmp/etcd-download-test/etcd`，我们可以手动修改其脚本中的安装路径，也可以运行完脚本之后手动迁移：
```bash
mv /tmp/etcd-download-test/etcd /path/to/your/etcd
```
##### xLLM Service编译
先应用patch:
```bash
sh prepare.sh
```
再执行编译:
```bash
mkdir -p build
cd build
cmake ..
make -j 8
cd ..
```
!!! warning "可能的错误"
这里能会遇到关于`boost-locale`和`boost-interprocess`的安装错误：`vcpkg-src/packages/boost-locale_x64-linux/include: No such file or directory`,`/vcpkg-src/packages/boost-interprocess_x64-linux/include: No such file or directory`
我们使用`vcpkg`重新安装这些包:
```bash
/path/to/vcpkg remove boost-locale boost-interprocess
/path/to/vcpkg install boost-locale:x64-linux
/path/to/vcpkg install boost-interprocess:x64-linux
```
### PD分离运行
启动etcd:
```bash
./etcd-download-test/etcd --listen-peer-urls 'http://localhost:2390' --listen-client-urls 'http://localhost:2389' --advertise-client-urls 'http://localhost:2391'
```
跨机配置时，etcd参考如下：
```bash
/tmp/etcd-download-test/etcd --listen-peer-urls 'http://0.0.0.0:3390' --listen-client-urls 'http://0.0.0.0:3389' --advertise-client-urls 'http://11.87.191.82:3389'
```
启动xllm service:
```bash
ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/export/home/models/GLM-5-W8A8/
```
跨机配置时，启动xllm service:
```bash
ENABLE_DECODE_RESPONSE_TO_SERVICE=true ../xllm-service/build/xllm_service/xllm_master_serving --etcd_addr="11.87.191.82:3389" --http_server_port 38888 --rpc_server_port 38889 --tokenizer_path=/export/home/models/GLM-5-W8A8/
```
- 启动Prefill实例
```bash
BATCH_SIZE=256
#推理最大batch数量
XLLM_PATH="./myxllm/xllm/build/xllm/core/server/xllm"
#推理入口文件路径（上一步中编译产物）
MODEL_PATH=/export/home/models/GLM-5-w8a8/
#模型路径（此处为int量化的Glm-5）
DRAFT_MODEL_PATH=/export/home/models/GLM-5-MTP/
MASTER_NODE_ADDR="11.87.49.110:10015"
LOCAL_HOST="11.87.49.110"
# Service Port
START_PORT=18994
START_DEVICE=0
LOG_DIR="logs"
NNODES=16
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
nohup numactl -C $((i*40))-$((i*40+39)) $XLLM_PATH \
--model $MODEL_PATH --model_id glmmoe \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--node_rank=$i \
--max_memory_utilization=0.86 \
--max_tokens_per_batch=5000 \
--max_seqs_per_batch=$BATCH_SIZE \
--communication_backend=hccl \
--enable_schedule_overlap=true \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_graph=true \
--draft_model $DRAFT_MODEL_PATH \
--draft_devices="npu:$DEVICE" \
--num_speculative_tokens 1 \
--enable_disagg_pd=true \
--instance_role=PREFILL \
--etcd_addr=$LOCAL_HOST:3389 \
--transfer_listen_port=$((36100 + i)) \
--disagg_pd_port=8877 \
> $LOG_FILE 2>&1 &
done
#--etcd_addr=$LOCAL_HOST:3389 参考etcd中advertise-client-urls的配置
#--instance_role=DECODE PD配置，DECODE\PREFILL
```
- 启动Decode实例
```bash
BATCH_SIZE=256
#推理最大batch数量
XLLM_PATH="./myxllm/xllm/build/xllm/core/server/xllm"
#推理入口文件路径（上一步中编译产物）
MODEL_PATH=/export/home/models/GLM-5-w8a8/
#模型路径（此处为int量化的Glm-5）
DRAFT_MODEL_PATH=/export/home/models/GLM-5-MTP/
MASTER_NODE_ADDR="11.87.49.110:10015"
LOCAL_HOST="11.87.49.110"
# Service Port
START_PORT=18994
START_DEVICE=0
LOG_DIR="logs"
NNODES=16
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
nohup numactl -C $((i*40))-$((i*40+39)) $XLLM_PATH \
--model $MODEL_PATH --model_id glmmoe \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--node_rank=$i \
--max_memory_utilization=0.86 \
--max_tokens_per_batch=5000 \
--max_seqs_per_batch=$BATCH_SIZE \
--communication_backend=hccl \
--enable_schedule_overlap=true \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_graph=true \
--draft_model $DRAFT_MODEL_PATH \
--draft_devices="npu:$DEVICE" \
--num_speculative_tokens 1 \
--enable_disagg_pd=true \
--instance_role=DECODE \
--etcd_addr=$LOCAL_HOST:3389 \
--transfer_listen_port=$((36100 + i)) \
--disagg_pd_port=8877 \
> $LOG_FILE 2>&1 &
done
#--etcd_addr=$LOCAL_HOST:3389 参考etcd中advertise-client-urls的配置
#--instance_role=DECODE PD配置，DECODE\PREFILL
```
需要注意：
- PD分离需要读取`/etc/hccn.conf`文件，确保将物理机上的该文件映射到了容器中
- `etcd_addr`需与`xllm_service`的`etcd_addr`相同
测试命令和上面类似，注意`curl http://localhost:{PORT}/v1/chat/completions ...`的`PORT`选择为启动xLLM service的`http_server_port`。
- 多机部署P或者Q时(例如部署两个P)，需要增加--rank_tablefile来完成通信。

66
upstream_ref/xllm/docs/zh/index.md Normal file
View File

@@ -0,0 +1,66 @@
---
hide:
- navigation
---
<style>
.md-content h1:first-of-type {
display: none;
}
</style>
<div style="text-align:center">
<img src="../assets/logo_with_llm.png" alt="xLLM" style="width:50%; height:auto;">
</div>
## 简介
**xLLM** 是一个高效且易用的开源智能推理框架，为模型在国产芯片上的推理提供企业级服务保障与高性能引擎计算能力。
#### 背景
当前，百亿至万亿参数规模的大语言模型正快速部署于智能客服、实时推荐、内容生成等核心业务场景，对国产计算硬件的高效支持已成为低成本推理部署的核心需求。现有推理引擎难以有效适配国产芯片等专用加速器的架构特性，硬件计算单元利用率低、MoE 架构下的负载不均衡与通信开销瓶颈、kv 缓存管理困难等问题，制约了请求的高效推理与系统的可扩展性。xLLM 推理引擎提升了 “通信 - 计算 - 存储” 全链路的资源利用效率，为大语言模型在实际业务中的规模化落地提供了关键技术支撑。
---
## 核心特性
xLLM 提供了强大的智能计算能力，通过硬件系统的算力优化与算法驱动的决策控制，联合加速推理过程，实现高吞吐、低延迟的分布式推理服务。
### 全图化/多层流水线执行编排
- 框架调度层的异步解耦调度，减少计算空泡；
- 模型图层的计算和通信异步并行，重叠计算与通信；
- 算子内核层的异构计算单元深度流水，重叠计算与访存。
### 动态shape的图执行优化
- 基于参数化与多图缓存方法的动态尺寸适配，提升静态图灵活性；
- 受管控的显存池，保证地址安全可复用；
- 集成适配性能关键的自定义算子（如 *PageAttention*, *AllReduce*）。
### MoE算子优化
- *GroupMatmul* 优化，提升计算效率；
- *Chunked Prefill* 优化，支撑长序列输入。
### 高效显存优化
- 离散物理内存与连续虚拟内存的映射管理；
- 按需分配内存空间，减少内存碎片与浪费；
- 智能调度内存空间，增加内存页复用，减小分配延迟；
- 国产芯片相应算子适配。
### 全局多级KV Cache管理
- 多级缓存的kv智能卸载与预取；
- 以kv cache为中心的分布式存储架构；
- 多节点间kv的智能传输路由。
### 算法优化
- 投机推理优化，多核并行提升效率；
- MoE专家的动态负载均衡，实现专家分布的高效调整。
## 设计文档
- [Graph Mode 设计文档](design/graph_mode_design.md)
- [生成式推荐设计文档](design/generative_recommendation_design.md)

49
upstream_ref/xllm/docs/zh/supported_models.md Normal file
View File

@@ -0,0 +1,49 @@
# 模型支持列表
## LLM
| | NPU | MLU | ILU |
| ------------------------ | :---: | :---: | :---: |
| DeepSeek-V3/R1/V3.1 | ✅ | ✅ | ❌ |
| DeepSeek-V3.2 | ✅ | ✅ | ❌ |
| DeepSeek-R1-Distill-Qwen | ✅ | ❌ | ❌ |
| Qwen2/2.5/QwQ | ✅ | ✅ | ✅ |
| Qwen3 | ✅ | ✅ | ✅ |
| Qwen3 Moe | ✅ | ✅ | ✅ |
| Kimi-k2 | ✅ | ❌ | ❌ |
| Llama2/3 | ✅ | ❌ | ✅ |
| GLM4.5 | ✅ | ❌ | ❌ |
| GLM4.6 | ✅ | ❌ | ❌ |
| GLM-4.7 | ✅ | ❌ | ❌ |
| GLM-5 | ✅ | ❌ | ❌ |
## VLM
| | NPU | MLU | ILU |
| ------------ | :---: | :---: | :---: |
| MiniCPM-V | ✅ | ❌ | ❌ |
| MiMo-VL | ✅ | ❌ | ❌ |
| Qwen2.5-VL | ✅ | ✅ | ❌ |
| Qwen3-VL | ✅ | ✅ | ❌ |
| Qwen3-VL-MoE | ✅ | ✅ | ❌ |
| GLM-4.6V | ✅ | ❌ | ❌ |
| VLM-R1 | ✅ | ❌ | ❌ |
## Rerank
| | NPU | MLU | ILU |
| -------------- | :---: | :---: | :---: |
| Qwen3-Reranker | ✅ | ❌ | ❌ |
## DiT
| | NPU | MLU | ILU |
| ---- | :---: | :---: | :---: |
| Flux | ✅ | ❌ | ❌ |
## Rec
| | NPU | MLU | ILU |
| --- | :---: | :---: | :---: |
| OneRec | ✅ | ❌ | ❌ |
| Qwen2 | ✅ | ❌ | ❌ |
| Qwen2.5 | ✅ | ❌ | ❌ |
| Qwen3 | ✅ | ❌ | ❌ |