dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

ref(upstream): FULL TREE — Deep-Spark xllm (1470) + ds_vllm csrc/models (703)

Browse Source 
Replaces cherry-picked upstream_ref with complete source trees.

xllm/ — Iluvatar official C++ inference engine (15MB, 1470 files)
  Complete: kernels → layers → models → runtime → scheduler → api
  Excluded: .git, binary images, third_party submodule checkouts

ds_vllm/ — Iluvatar official vllm fork (8MB, 703 files)
  Included: csrc/ (ALL CUDA kernels), fused_moe/, qwen3_5 model, _custom_ops
  Excluded: tests, benchmarks, docs, examples (not needed for reference)

Critical call chains now fully traceable:
  MoE: moe_topk_softmax_kernels.cuh → ixformer.h → fused_moe.cpp → layer
  GDN: qwen3_gated_delta_net_base.cpp → qwen3_5_gated_delta_net.cpp
  Attention: ixformer.h → xllm_paged_attention → attention.cpp
This commit is contained in:
EX Engine
2026-08-10 02:53:54 +00:00
parent 9e4fb3712f
commit 002f9879b2
2179 changed files with 494021 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
upstream_ref/xllm/docs/en/.readthedocs.yaml Normal file
View File

@@ -0,0 +1,28 @@
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the OS, Python version, and other tools you might need
build:
os: ubuntu-24.04
tools:
python: "3.13"
jobs:
pre_build:
# - cp -r docs/en/* docs/
- mv docs/en/* docs/
- rm -rf docs/zh/
- find docs/ -name "*.md" -exec sed -i 's#../assets/#assets/#g' {} \;
# Build documentation with Mkdocs
mkdocs:
configuration: mkdocs_en.yml
# Optionally, but recommended,
# declare the Python requirements required to build your documentation
# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: docs/requirements.txt

59
upstream_ref/xllm/docs/en/accuracy_test.md Normal file
View File

@@ -0,0 +1,59 @@
# 1. LLM Accuracy Test
## 1.1 Setup ais_bench
```bash
# Create a virtual environment for ais_bench using conda or uv
conda create --name ais_bench python=3.10 -y
conda activate ais_bench
# Clone ais_bench and install dependencies
git clone https://gitee.com/aisbench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517
# Download the dataset and copy it to the ais_bench directory
cp -r /path/to/dataset /path/to/benchmark/ais_bench/datasets
```
## 1.2 Modify Configuration
Modify the accuracy test configuration file according to your actual situation: `/path/to/benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py`. It is recommended to set the sampling parameters as follows:
```python
models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr='vllm-api-general-chat',
path="/path/to/model/Qwen3-8B", # Model path
model="Qwen3-8B", # Model name
request_rate = 0,
retry = 2,
host_ip = "127.0.0.1",
host_port = 19000, # xllm server port
max_out_len = 32768, # Limit maximum model length
batch_size=32,
trust_remote_code=False,
generation_kwargs = dict(
temperature = 0.6,
# top_k = -1,
top_p = 0.95,
# seed = None,
# repetition_penalty = 1,
),
pred_postprocessor=dict(type=extract_non_reasoning_content)
)
]
```
## 1.3 Launch ais_bench
Before using ais_bench, you need to start the xllm server first. Use `ais_bench -h` to get parameter descriptions. The launch commands for gsm8k and ceval datasets are as follows:
```bash
# Using gsm8k dataset
ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt --dump-eval-details
# Using ceval dataset
ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt --merge-ds --dump-eval-details
```
We will integrate ais_bench and datasets (ceval and gsm8k) into the development image in the future. The ais_bench documentation and datasets are as follows:
* [ais_bench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/index.html)
* [Datasets](https://ais-bench-benchmark.readthedocs.io/en/latest/base_tutorials/all_params/datasets.html)

83
upstream_ref/xllm/docs/en/cli_reference.md Normal file
View File

@@ -0,0 +1,83 @@
---
hide:
- navigation
---
# Service Startup Parameters
xLLM uses gflags to manage service startup parameters. The specific parameter meanings are as follows:
## Common Parameters
| Parameter Name | Data Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `master_node_addr` | `string` | "127.0.0.1:19888" | ip:port | The listening address of the master node's rpc server | [Details](./features/basics.md) |
| `host` | `string` | "" | The machine IP where the current device is located | The host IP used by the current device for communication. An rpc server is started on each device for multi-device communication. | |
| `port` | `int32` | 8010 | Any available port | Used in conjunction with the `host` parameter. The combination is used for rpc communication between devices. | |
| `model` | `string` | "" | | Path to the model. | |
| `devices` | `string` | "npu:0" | | Specifies the NPU devices used by the current process. | |
| `nnodes` | `int32` | 1 | | The total number of devices used by the current service. | |
| `node_rank` | `int32` | 0 | 0 ~ (total devices - 1) | The rank id of each device. | |
| `max_memory_utilization` | `double` | 0.8 | Between 0-1 | The maximum proportion of device memory available for model weights and KV Cache combined. | |
| `max_tokens_per_batch` | `int32` | 10240 | | The maximum number of tokens that can be computed per step. | |
| `max_seqs_per_batch` | `int32` | 1024 | | The maximum number of sequences that can be computed per step. | |
| `enable_chunked_prefill` | `bool` | true | false | Whether to enable chunked prefill. | |
| `enable_prefill_sp` | `bool` | false | true | Whether to enable prefill-only sequence parallel. | `enable_chunked_prefill=true` is supported only for prefill-only batches (`PREFILL` / `CHUNKED_PREFILL`); `MIXED` and `DECODE` batches do not run with sequence parallel. |
| `enable_schedule_overlap` | `bool` | false | true | Whether to enable asynchronous scheduling. | [Details](./features/async_schedule.md) |
| `enable_prefix_cache` | `bool` | true | false | Whether to enable prefix cache (not supported by DeepSeek currently). | |
| `communication_backend` | `string` | "hccl" | "lccl" | The backend used for communication operations. | |
| `block_size` | `int32` | 128 | | The block size for KV Cache storage. | |
| `task` | `string` | "generate" | "embed", "mm_embed" | Service type: generation, embedding, or multimodal embedding. | |
| `max_cache_size` | `int64` | 0 | | The usable KV Cache size in bytes. | |
| `kv_cache_dtype` | `string` | "auto" | "int8" | KV Cache data type. "auto" aligns with model dtype (no quantization), "int8" enables INT8 quantization to save ~50% memory. MLU backend only. | |
## MoE Model Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `dp_size` | `int32` | 1 | Power of 2 | The dp scale size for the Attention part. | |
| `ep_size` | `int32` | 1 | Power of 2 | The ep scale size for the MoE part. | |
| `expert_parallel_degree` | `int32` | 0 | 1,2 | Parameter related to ep parallelism. Defaults to 0 when ep is not used, and to 1 when ep is enabled. Can be set to 2 when `ep_size` equals the total number of devices (uses all2all communication). | |
## P-D Separation Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `enable_disagg_pd` | `bool` | false | true | Whether to enable P-D separation. | [Details](./features/disagg_pd.md) |
| `disagg_pd_port` | `int32` | 7777 | Any available port | Configuration when P-D separation is enabled. Corresponds to the listening port number of the pd separation rpc server started on each card. | |
| `instance_role` | `string` | DEFAULT | PREFILL, DECODE, MIX | Defaults to DEFAULT. Must be configured as PREFILL, DECODE, or MIX when P-D separation is enabled. | |
| `kv_cache_transfer_mode` | `string` | "PUSH" | "PULL" | The mode for transferring KV Cache in P-D separation. PUSH mode: Prefill transmits layer by layer to Decode; PULL mode: Decode pulls the KV Cache from Prefill in one go. | |
| `transfer_listen_port` | `int32` | 26000 | Any available port | Configuration when P-D separation is enabled. Corresponds to the listening port for KV Cache Transfer on each card. | |
## MTP Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `draft_model` | `string` | "" | | Path to the MTP model. | [Details](./features/mtp.md) |
| `draft_devices` | `string` | "npu:0" | Same format as `devices`, e.g. `npu:0` or `npu:0,npu:1` | Should be set consistently with the `devices` parameter. | |
| `num_speculative_tokens` | `int32` | 0 | Any integer, suggestion 1 or 2 | The number of tokens output by the MTP model per step. | |
## Graph Execution Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `enable_graph` | `bool` | false | true | Whether to enable graph execution mode to optimize decode phase performance. Only applied during decode phase and does not take effect during prefill phase. Supports ACL Graph (NPU), and MLU Graph. | [Details](./features/graph_mode.md) |
| `enable_graph_mode_decode_no_padding` | `bool` | false | true | Builds decode graphs with the actual `num_tokens` instead of the padded shape. | |
| `enable_prefill_piecewise_graph` | `bool` | false | true | Whether to enable piecewise graph for prefill phase. Attention runs eagerly while other ops are captured into CUDA graphs. | |
| `max_tokens_for_graph_mode` | `int32` | 2048 | Any integer greater than or equal to 0 | Maximum number of tokens for graph execution. If 0, no limit is applied. | |
## Parameters for Use with xLLM-service
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `etcd_addr` | `string` | "" | ip:port | The listening address of the etcd's rpc server. | |
| `enable_service_routing` | `bool` | false | true | Whether the request from the xllm service, use this when enable the xllm service. | |
## Other Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `max_concurrent_requests` | `int32` | 200 | Any integer greater than or equal to 0 | For rate limiting, restricts the total number of requests being processed in the instance. Set to 0 for no limit. | |
| `model_id` | `string` | "" | | Model name, not a path. | |
| `num_request_handling_threads` | `int32` | 4 | Any integer greater than 0 | The thread pool size for handling input requests. | |
| `prefill_scheduling_memory_usage_threshold` | `double` | 0.95 | Value between 0-1 | When kv cache usage reaches this threshold, scheduling of prefill requests is paused. | |
| `num_response_handling_threads` | `int32` | 4 | Any integer greater than 0 | The thread pool size for handling outputs. | |
| `rank_tablefile` | `string` | "" | | Configuration file for creating the communication domain. Required for multi-node scenarios. | |

1062
upstream_ref/xllm/docs/en/design/generative_recommendation_design.md Normal file
View File

File diff suppressed because it is too large Load Diff

474
upstream_ref/xllm/docs/en/design/graph_mode_design.md Normal file
View File

@@ -0,0 +1,474 @@
# Graph Mode Design Document
## Overview
xLLM's Graph Mode supports multiple graph execution backends. Its goal is to turn the original Host-driven stream of fine-grained kernel launches into a capture-then-replay execution flow in inference serving, thereby reducing Host scheduling overhead, reducing device-side bubbles, and improving throughput and latency stability.
This document is intended for developers who need to understand the implementation principles and key design choices. It focuses on:
- the basic Graph Mode mechanism and how xLLM applies it
- dynamic dimension parameterization
- Piecewise Graph
- a multi-shape reusable memory pool, including input tensor reuse
This document focuses on the unified Graph Mode design in xLLM and does not expand on backend-specific platform differences.
The design goals of this document are:
- provide a unified Graph Mode abstraction across xLLM backends
- explain the three key designs: dynamic dimension parameterization, Piecewise Graph, and multi-shape memory reuse
- clarify what problem each design solves, what assumptions it depends on, and where its boundary lies
The non-goals of this document are:
- full adaptation details for every operator or every model
- replacing feature documentation for flags and usage examples
Related design documents:
- for a recommendation-oriented case study that focuses on fixed scheduling, multi-step execution, and custom operators, see: [Generative Recommendation Design Document](generative_recommendation_design.md)
## 1. Graph Mode Fundamentals
### 1.1 Capture / Replay Basics
In traditional eager execution, one forward pass launches many kernels, memory copies, and synchronization operations from the Host. For decode-like workloads, where each step is small but requests are frequent, Host scheduling overhead becomes significant and device-side bubbles become more visible.
The core idea of Graph Mode is:
1. **Capture**: when a shape bucket is seen for the first time, run one forward pass on a dedicated stream and record the kernel launches, memory operations, and dependencies into a graph.
2. **Replay**: for later requests that hit the same bucket, replay the recorded graph instead of launching kernels one by one from the Host.
![Graph Mode](../../assets/graph_mode.png)
This mechanism usually requires:
- **Stable execution path**
- graph capture records one concrete execution path; once capture completes, the control flow, launch shape, and dependencies on that path are fixed
- therefore, the capture path cannot switch to a different dynamic branch during replay
- **Stable addresses for key tensors**
- key tensor addresses written into the graph during capture must still be valid at replay time
- **Stable operator behavior and result semantics**
- replay correctness depends on operators along the path being Graph Mode compatible and not changing semantics because of runtime condition changes
### 1.2 xLLM GraphMode Runtime Foundations
Starting from the three basic requirements in 1.1, the first two are mainly handled by the xLLM runtime: turning dynamic requests into execution units that can be replayed stably, and ensuring replay still accesses the fixed addresses recorded during capture. For that reason, xLLM introduces a unified Graph Executor in Graph Mode to centralize bucketing, persistent buffers, graph cache management, and the capture / replay lifecycle.
xLLM's runtime-side foundation work mainly includes:
- **Graph selection and execution scheduling around execution-path stability**
- requests must first be grouped into buckets by `num_tokens` or a nearby shape
- graph cache is maintained per bucket: replay directly on a hit, otherwise capture first and cache the graph
- use full graphs for paths that are fully capturable, and switch to Piecewise Graph when part of the path breaks graph capture
- **Persistent buffer and graph-instance management around address stability**
- dynamic inputs such as tokens, positions, seq_lens, and block_tables cannot be reallocated to arbitrary new addresses before replay; xLLM must write them into persistent buffers first and then update content at fixed addresses
- tensors allocated temporarily during model graph construction also need to be retained with the corresponding graph instance instead of being reclaimed when their local scope ends
- when the shared memory pool is enabled, captures of different shapes can reuse the same underlying physical memory, while the virtual addresses seen by the graph still remain stable
- **A unified Graph Executor abstraction**
- capture, cache, replay, graph-instance management, and backend abstraction all need to be handled centrally by the runtime
- the point of this layer is to pull Graph Mode runtime orchestration out of model code rather than scattering it across layers or operators
In concrete execution, the Graph Executor runtime flow can be summarized as:
1. **Bucket the request**: group the request into a bucket by `num_tokens` or a nearby shape.
2. **Prepare inputs**: write tokens, positions, seq_lens, and block tables into persistent buffers, and update runtime metadata such as `attn_metadata` and `plan_info`.
3. **Choose capture or replay**: replay directly if the bucket already has a cached graph; otherwise capture, cache, and then enter the replay path.
4. **Execute the graph**: run a full graph or a piecewise graph depending on the scenario.
The third requirement in 1.1, "stable operator behavior and result semantics", falls more on model-side and operator-side adaptation, which is covered in 1.3.
### 1.3 Model Adaptation Required for Graph Mode
After the xLLM runtime takes care of bucketing, buffers, and graph-instance management, the model side still needs to satisfy the third requirement from 1.1: operator behavior and result semantics must remain stable across capture / replay and must not break because of host participation, dispatch changes, or launch-shape changes.
The required model adaptations mainly include:
- **Remove operations that are not capture-safe**
- General requirement: the capture path must not contain operations that trigger host-side decisions, implicit host-device synchronization, or extra control-flow changes, such as stream synchronization, logic branches driven by host tensors, or implicit host-device interactions
- Typical example 1: operators may depend on host tensor contents to decide task partitioning or execution mode. For example, ATB [PA](https://www.hiascend.com/document/detail/zh/canncommercial/83RC1/API/ascendtbapi/ascendtb_01_0197.html) and [MLA](https://www.hiascend.com/document/detail/zh/canncommercial/83RC1/API/ascendtbapi/ascendtb_01_0314.html) in Ascend CANN commercial 8.3 can choose different task layouts based on host-side `kv_seq_lens` and `q_seq_lens`
- Typical example 2: a host scalar participates in tensor computation, or the host directly reads tensor data, for example:
```cpp
Tensor a;
Tensor b = a * 0.5; // the scalar is implicitly transferred to the tensor's device, causing a synchronized H2D
torch::Tensor max_of_seq = torch::max(input_params.kv_seq_lens);
max_seq_len_ = std::max(max_of_seq.item<int>(), max_seq_len_); // host reads tensor data
```
- **Adapt operators**
- General requirement: the kernel execution path must remain stable for the same input shape. For one input shape, graph execution should choose the same kernel every time, without mixing prefill, chunked prefill, and decode behavior. At the same time, `grid_dim`, `block_dim`, task count, workspace shape, and tiling content must also remain stable
- Typical example: in Ascend CANN commercial 8.3, ATB PA may decide at runtime whether to enable flash-decoding long-sequence mode based on sequence length and batch size. The long-sequence and short-sequence modes use different tiling keys, and the actual dispatched `kernel_name` is also different. This kind of behavior, where the same business path triggers different kernels under different runtime conditions, must be converged before entering Graph Mode
## 2. Dynamic Dimension Parameterization
### 2.1 Problem
Graph capture records a fixed kernel sequence and the launch parameters of every kernel. At replay time, the runtime can only repeat that recorded launch shape. It cannot re-decide `grid_dim`, `block_dim`, task count, or the tiling path inside the same graph.
As a result, not every dynamic request dimension can continue to vary once the request enters Graph Mode. For attention, the real dynamic factors usually include more than `num_tokens`: they also include `batch_size`, `q_seq_lens`, `kv_seq_lens`, and `block_tables_size`. If these dimensions further affect task partitioning, workspace layout, `tiling_params`, or kernel path, replay may reuse the stale capture-time configuration and become incorrect.
The core question of this chapter is therefore: which dynamic information can still be updated after the graph has been selected, and which dimensions must instead be handled by finer-grained bucketing or by building another graph.
### 2.2 Solution
The goal of dynamic dimension parameterization is not to make one graph accept arbitrary `num_tokens`. Instead, once the graph has already been selected by `bucket_num_tokens`, the goal is to let the same bucket safely cover more real dynamic requests.
#### 2.2.1 Parameterization Boundary
Whether a dimension can still vary at replay time does not depend on whether it is dynamic in the request. It depends on whether that dimension has already been folded into the execution shape during capture.
Dynamic factors can be divided into two categories:
1. **Dimensions that determine graph shape**
- once a dimension enters `grid_dim`, `block_dim`, task count, workspace layout, tiling key, or `tiling_params`, it becomes part of the capture result
- after that, it usually cannot be changed inside the same graph
2. **Dimensions that can still be updated after graph selection**
- such as `batch_size`, `q_seq_lens`, `kv_seq_lens`, `block_tables`, `new_cache_slots`, and `plan_info`
- they are suitable parameterization targets only if they do not further rewrite execution shape
A simplified norm-kernel example is:
```cpp
int grid_dim = num_tokens;
NormKernel<<<grid_dim, block_dim>>>(x, y, ...);
```
Here, `grid_dim` and `block_dim` are both launch parameters. Once `num_tokens` participates in launch configuration, it becomes part of graph shape; after capture, replay can no longer change `grid_dim = 128` into `grid_dim = 256` within the same graph.
So the boundary is straightforward: any dynamic factor that changes launch shape, workspace, or tiling path should not be handled only by updating parameters before replay.
#### 2.2.2 How xLLM Handles It
xLLM handles this in two layers.
The first layer handles `num_tokens`: xLLM uses bucketing and padding to normalize the raw request into a fixed `bucket_num_tokens`, and then selects the graph accordingly.
The second layer handles the remaining dynamic information inside that bucket: adapt operators that depend on host tensors, host planning, or host-side tiling so that request-varying information is stored in device-side persistent buffers, or otherwise externalized into data that can be refreshed before replay, instead of letting replay stay bound to the old host-side planning result from capture.
Take NPU attention as an example. A common pre-adaptation problem is that `q_seq_lens` and `kv_seq_lens` first participate in host-side planning, which produces task, workspace, or tiling-related data before launching the kernel:
```cpp
// Host side
auto plan_info = PlanAttention(q_seq_lens_host, kv_seq_lens_host, ...);
WritePlanToWorkspace(attention_workspace, plan_info);
AttentionKernel<<<grid_dim, block_dim>>>(
q, k, v, out,
attention_workspace,
tiling_params);
```
In this kind of implementation, graph replay does not automatically redo planning, so it can only repeat the launch shape and workspace layout from capture time. The adapted direction is to write request-varying information such as `q_seq_lens`, `kv_seq_lens`, `block_tables_size`, and `plan_info` into device-side persistent buffers under a fixed launch shape, and let the kernel read them directly. For example, a parameterized `kv_seq_lens` path may look like:
```cpp
struct AttentionLaunchArgs {
void* q;
void* k;
void* v;
void* out;
int32_t batch_size;
int32_t* kv_seq_lens; // device buffer
void* attention_workspace;
void* tiling_params;
};
LaunchAttentionKernel(args);
```
If a dynamic factor changes task count, array length, workspace shape, or tiling path as soon as it varies, xLLM does not force it into "parameterization updates". Instead, it uses one of two approaches:
1. expand the bucketing dimensions and include those factors in the graph-selection key, for example a combination of `num_tokens` and `num_comm_tokens`
2. keep that dynamic-shape-sensitive part outside the full graph and handle it with Piecewise Graph
### 2.3 Result
Dynamic dimension parameterization directly brings the following result: one `num_tokens` bucket can safely cover more real requests, and under a stable execution shape, operators such as attention can still see the correct dynamic information at replay time, including `batch_size`, `seq_lens`, `block_tables_size`, and `plan_info`.
One thing to keep in mind is that parameterization does not eliminate bucketing. The dynamic nature of `num_tokens` is still mainly absorbed by graph selection; if a bucket still contains other factors that rewrite tiling keys, task count, communication scale, or kernel path, those factors must still enter the bucketing key or move to Piecewise Graph. Communication-heavy scenarios are a typical example. In Attention DP + MoE EP, communication scale may depend on the maximum DP data size rather than strictly matching local `num_tokens` on one device.
## 3. Piecewise Graph
### 3.1 Problem
In prefill and chunked prefill scenarios, the full forward path is usually more complex than decode. Some operators may:
- fail to be captured stably
- be highly sensitive to dynamic metadata
- rely on extra runtime preparation during capture
If the requirement is that the entire path must be capturable as one full graph, Graph Mode coverage becomes very limited. Once one critical operator breaks graph capture, the whole path loses graph execution.
### 3.2 Solution
#### 3.2.1 Underlying Logic
One common question in implementation is: why can decode use a full graph while chunked prefill usually needs Piecewise Graph?
The key difference is attention.
Under current decode semantics, one step usually means one generated token per sequence, so `num_tokens` and `batch_size` are almost one-to-one. For attention, that means:
- when `num_tokens` grows, `batch_size` grows with it
- the pattern of `q_seq_lens` stays relatively stable
- task partitioning, workspace requirements, and launch arguments can mostly be bucketed together with `num_tokens`
Therefore, when building buckets by `num_tokens`, decode satisfies the reuse conditions for a full graph.
In chunked prefill, however, `num_tokens` and `batch_size` are no longer tied together. The same `num_tokens` may correspond to very different request combinations, for example:
- `num_tokens = 128, batch_size = 2, q_seq_lens = [64, 64]`
- `num_tokens = 128, batch_size = 4, q_seq_lens = [32, 32, 32, 32]`
- `num_tokens = 128, batch_size = 8, q_seq_lens = [16, 16, 16, 16, 16, 16, 16, 16]`
For attention, these requests share the same `num_tokens` but they are not the same shape. Attention task partitioning, local indexing, the lengths of `q_seq_lens` and `kv_seq_lens`, and the corresponding `plan_info` all directly depend on `batch_size`. If the system reuses a previously captured graph only by `num_tokens`, then replay may keep using the old batch partitioning and old task partitioning from capture, which can make the result incorrect.
`batch_size` is hard to parameterize here because it changes more than just one scalar value. It changes the structure of the launch arguments themselves:
- the lengths of `q_seq_lens` and `kv_seq_lens` change
- the task count and task boundaries of attention change
- `plan_info` and workspace layout change
- the indexing and partition logic inside the kernel change
In other words, in chunked prefill, `batch_size` is not a dimension that can be handled by updating a few parameters. It changes the execution shape of the attention part itself. Unless `num_tokens × batch_size`, or even more fine-grained `seq_lens` combinations, are added into bucketing, selecting graphs only by `num_tokens` is not safe.
That is why the current design is better suited to:
- **decode**: attention can be captured together with the full graph
- **chunked prefill**: attention stays outside the graph while the more stable surrounding parts enter Piecewise Graph
#### 3.2.2 Core Idea
The goal of Piecewise Graph is not to force full-graph capture. It is to split one full execution path into several pieces:
- graph-friendly parts: captured into subgraphs
- parts that are not suitable for graph capture: kept in eager mode
At replay time, those pieces are executed in the same order recorded during capture.
This preserves as much Graph Mode benefit as possible even when full-graph capture is not feasible.
The following diagram uses a three-layer Qwen3 decoder as an example. Attention stays as an independent runner, while the non-attention operators between attention calls are packed as tightly as possible into continuous graph pieces.
![Piecewise Graph Diagram](../../assets/piecewise_graph_diagram.svg)
#### 3.2.3 How xLLM Implements It
In xLLM, Piecewise Graph is mainly used for prefill-like scenarios and is controlled by `enable_prefill_piecewise_graph`.
Its core implementation is:
1. use `PiecewiseGraphs` to maintain the replay instruction sequence
2. store capturable segments as graph objects
3. record non-graph attention segments as runners through `AttentionRunner`
4. replay in the original order as `graph -> runner -> graph -> ...`
The key point is how attention is handled:
- during piecewise capture, attention temporarily ends the current graph capture
- attention itself is not executed during capture; instead, the tensors, workspace, and parameters required for attention are recorded
- at replay time, attention runs with the latest `plan_info`, `q_cu_seq_lens`, and `kv_cu_seq_lens`
The direct reason is that, in chunked prefill, attention has dynamic dimensions beyond `num_tokens`, while current graph bucketing is mainly organized by `num_tokens`. Keeping attention outside the graph avoids replay errors where the graph hits by `num_tokens`, but `batch_size` and `seq_lens` have already changed.
So the essence of Piecewise Graph is not to support all dynamic attention logic inside the graph. It is to:
- put the stably capturable parts into graphs
- keep the more dynamic-shape-sensitive attention as a separate execution unit
- preserve the original execution order through one unified instruction sequence
### 3.3 Result
The results and boundaries of this design are reflected in the following aspects.
#### 3.3.1 Applicable Scenarios
- prefill
- chunked prefill
- scenarios with local graph breaks where graph execution benefit is still desired for the overall path
#### 3.3.2 Benefits
- more flexible than all-or-nothing full-graph capture
- significantly expands Graph Mode coverage
- fits model structures where attention is highly dynamic while parts such as MLP remain relatively stable
#### 3.3.3 Limitations
- currently focused mainly on prefill-like paths
- replay depends on correctly updated `attn_metadata.plan_info`
## 4. Multi-Shape Reusable Memory Pool
### 4.1 Problem
In multi-shape scenarios, Graph Mode faces two problems at the same time:
1. **capture-time memory grows linearly with the number of shapes**
- each new shape entering capture usually gets its own independent memory buffer pool
- because Graph Mode requires stable addresses between capture and replay, these buffers often cannot be released in time
- memory usage grows from the expected `max(shape)` to `sum(shape)`
2. **input-side memory is also repeatedly duplicated**
- if each shape keeps its own tokens, positions, seq_lens, block_tables, and related tensors
- then even if the graph itself can be reused, input-side memory still accumulates by shape
So the reusable memory pool is not just about saving memory. It is about supporting dynamic new shapes while making memory usage converge from `sum(shape)` to `max(shape)`.
The concrete goals include:
1. **capture-time memory reuse**: different graphs share the same underlying physical memory
2. **input-side memory reuse**: replays of different shapes share one set of persistent input buffers
3. **no address conflicts**: every captured graph still keeps its own stable virtual address view
4. **on-demand support for new shapes**: a new shape can still be captured without invalidating existing graphs
5. **no extra replay overhead**: most reuse happens during capture and input preparation, while replay stays as unchanged as possible
#### 4.1.1 Root Cause
- on the CUDA path, the allocator typically allocates an independent memory buffer pool for every graph capture
- Graph Mode then requires stable addresses between capture and replay, so buffers from old shapes cannot be reclaimed like ordinary eager temporary memory
- as more shapes enter capture, memory usage grows from the expected `max(shape)` to `sum(shape)`
#### 4.1.2 Constraints
- under the current assumption, graphs of different shapes do not replay simultaneously
- dynamically appearing new shapes must still be supported through on-demand capture
- existing graph caches should not be torn down just because a new shape appears
- replay should not become more complex because of memory reuse
#### 4.1.3 Failed Approach and Technical Challenge
A more direct idea is to reset the allocation pointer before each capture so that different shapes reuse the same virtual address space.
But this is not safe in practice. Allocators usually track allocated blocks by address. If a new capture reuses an old address, it may overwrite the previous address record. Then, when tensors belonging to an old graph are later destructed or freed, the address record may already be gone, which can trigger errors such as `invalid device pointer`.
This reveals the real challenge: what Graph Mode truly needs to keep stable is the address view seen by each graph, while what really needs to be reused is the underlying physical memory. So the problem is not just memory saving. It is how to satisfy all of the following at the same time:
- the virtual address spaces of captured graphs do not conflict
- the underlying physical memory can still be shared across shapes
- adding a new shape into capture does not break replayability of already captured graphs
### 4.2 Solution
The core idea is not to let different shapes share one virtual address range. Instead, it is to let them share one set of physical memory while each captured graph keeps its own virtual address view.
The reason is simple: if different shapes are forced to reuse the same virtual address space, allocator address tracking will conflict, and old graphs may no longer find their original address records when they are released or destroyed. In other words, the thing that must be reused is the underlying physical memory, not the virtual address seen by the graph itself.
To show why xLLM chooses "shared physical memory + independent virtual address spaces", the common solutions can be compared as follows:
| Solution | Memory reuse mechanism | Physical memory outcome | Address-conflict risk / mitigation | Main limitation |
|----------|------------------------|-------------------------|------------------------------------|-----------------|
| vLLM | Shared graph memory pool + capture larger shapes first | may still approach `sum(shape)` | relies on a shared memory pool and free-block management, usually without explicit address conflicts | reuse efficiency depends on capture order and allocator behavior |
| SGLang | Global graph memory pool + capture larger shapes first | may still approach `sum(shape)` | relies on global pool or graph-private pool management to avoid conflicts | memory still grows when there are many distinct shapes |
| xLLM | Map multiple virtual address spaces to one set of physical memory | converges to `max(shape)` | every graph keeps an independent address view, avoiding address-record conflicts at the root | depends on VMM and allocator integration support from the runtime |
xLLM does not reuse the same virtual address space. Instead, each graph keeps its own address view, while the truly shareable part is pushed down into underlying physical memory.
#### 4.2.1 Multi-Shape Input Tensor Reuse
xLLM first solves repeated input-side memory usage. Instead of allocating independent input tensors for each shape, it pre-allocates one set of persistent buffers for the maximum shape and lets different shapes share them.
These buffers usually include:
- `tokens`
- `positions`
- `q_seq_lens`
- `kv_seq_lens`
- `block_tables`
- `new_cache_slots`
- `hidden_states`
Replay-time handling is straightforward:
1. write the actual input of the current request into the prefix of the buffer
2. zero-fill padding regions when needed
3. construct slice views for the actual shape
As a result, input-side memory no longer accumulates by shape and stays at one `max(shape)` allocation.
#### 4.2.2 Capture-Time Memory Reuse
After solving input duplication, xLLM still needs to solve the memory accumulation caused by graph capture itself.
The core design is:
- every capture uses a **new virtual address space**
- different virtual address spaces map to the **same physical memory**
This satisfies two requirements at once:
1. **stable addresses from the graph's point of view**: each captured graph sees its own fixed virtual addresses
2. **only one physical-memory footprint**: the underlying physical memory grows by the maximum demand instead of accumulating by shape
Compared with reusing one virtual address range, this design avoids address-tracking conflicts and therefore works better with existing allocator mechanisms.
This design relies on a unified virtual memory management capability, including:
- reserving a new virtual address space
- creating or expanding underlying physical memory
- mapping the same physical memory into multiple virtual address spaces
- performing unmap, address-space release, and access-permission operations when needed
The key property is that the same physical memory can be mapped into multiple virtual address spaces. That is exactly what makes "different graphs have independent addresses while still sharing the same physical memory" possible.
#### 4.2.2.1 Address Mapping Relation
The core mapping relation for capture-time memory reuse is shown below:
![Shared virtual-to-physical mapping](../../assets/shared_vmm_mapping_diagram.svg)
This diagram shows that, as the shape grows, each new capture switches to a new virtual address space, while all three virtual spaces reuse the same physical memory pool. Different shapes only use different portions of that pool, and the total physical memory converges to `max(shape)`.
### 4.3 Results
#### 4.3.1 Memory Efficiency Metrics
| Metric | Without memory reuse | With memory reuse |
|--------|----------------------|-------------------|
| Physical memory size | `sum(shape)` | `max(shape)` ✅ |
| Virtual memory | multiple copies | multiple copies |
| Stability | address-conflict risk | stable ✅ |
The key change is that physical memory converges to `max(shape)`, while virtual address space still grows with the number of captured shapes.
#### 4.3.2 Performance Impact
- **capture stage**: switching virtual address spaces and extending mappings introduces extra overhead
- **replay stage**: no extra graph execution overhead, and behavior is essentially the same as normal Graph Mode replay
- **input preparation stage**: one extra write into persistent buffers plus slice-view construction, which is still typically cheaper than repeated allocation
#### 4.3.3 Virtual Address Space Overhead
One cost of this design is that virtual address space grows as more shapes are captured. But this growth is address-space usage, not physical memory usage.
On 64-bit systems, virtual address space is usually much larger than single-device memory capacity, so it is typically not the primary bottleneck for this kind of graph capture scenario. In other words, this design uses abundant address space to bring physical memory down from `sum(shape)` to `max(shape)`.
#### 4.3.4 Result Validation
- captures of different shapes can share the same underlying physical memory
- every capture still keeps an independent virtual address space instead of reusing the same address view
- address-conflict issues are no longer exposed through the failed "reuse one virtual address space" path
- memory usage converges from `sum(shape)` to `max(shape)`
#### 4.3.5 Practical Effects
- in multi-shape scenarios, memory behavior changes from "grows linearly with captures" to "physical memory converges to the maximum shape"
- graph reuse becomes more stable because direct reuse of one virtual address space is avoided
- the design supports on-demand capture for new shapes without tearing down existing graph caches
- replay stays unchanged or nearly unchanged, while most added complexity is concentrated in capture and input preparation
#### 4.3.6 Assumptions and Boundaries
- virtual address space still grows with the number of captured shapes, even though physical memory only grows with the maximum demand
- the design depends on runtime support for virtual address management, physical memory mapping, and allocator integration
- it currently assumes that graphs of different shapes do not replay simultaneously; that assumption is what makes physical memory sharing safe
- the current implementation still keeps physical memory and virtual spaces for the full lifecycle instead of doing fine-grained dynamic release
## 5. References
- [ACLGraph Capture / Replay mechanism](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1alpha002/appdevg/acldevg/aclcppdevg_000519.html)
- [CudaGraph capture mechanism](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs)
- [vLLM graph execution reference](https://github.com/vllm-project/vllm/blob/main/vllm/v1/worker/gpu/cudagraph_utils.py)
- [SGLang graph execution reference](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/cuda_graph_runner.py)

27
upstream_ref/xllm/docs/en/dev_guide/code_arch.md Normal file
View File

@@ -0,0 +1,27 @@
# Code Architecture
```
├── xllm/
| : main source folder
│ ├── api_service/ # code for api services
│ ├── core/
│ │ : xllm core features folder
│ │ ├── common/
│ │ ├── distributed_runtime/ # code for distributed and pd serving
│ │ ├── framework/ # code for execution orchestration
│ │ ├── kernels/ # adaption for npu kernels adaption
│ │ ├── layers/ # model layers impl
│ │ ├── platform/ # adaption for various platform
│ │ ├── runtime/ # code for worker and executor
│ │ ├── scheduler/ # code for batch and pd scheduler
│ │ └── util/
│ ├── function_call # code for tool call parser
│ ├── models/ # models impl
│ ├── processors/ # code for vlm pre-processing
│ ├── proto/ # communication protocol
│ ├── pybind/ # code for python bind
| └── server/ # xLLM server
├── examples/ # examples of calling xLLM
├── tools/ # code for npu time generations
└── xllm.cpp # entrypoint of xLLM
```

396
upstream_ref/xllm/docs/en/dev_guide/tilelang_ascend_kernel_dev.md Normal file
View File

@@ -0,0 +1,396 @@
# xLLM Ascend TileLang Kernel Development Guide
This document explains how to add or modify an Ascend TileLang kernel in xLLM. The examples use the current `rope` kernel throughout.
Relevant directories:
- Python kernel definitions: `xllm/xllm/compiler/tilelang/targets/ascend/kernels`
- NPU runtime wrappers: `xllm/xllm/core/kernels/npu/tilelang`
Builds and tests should be run inside the NPU container.
## 1. First Decide What Kind of Change You Are Making
- Add a `specialization`
- Add one more compiled parameter combination to an existing kernel
- Reuse the same wrapper, the same runtime dispatch fields, and the same C ABI
- Typical changes are updates to `DISPATCH_SCHEMA` or `SPECIALIZATIONS`
- Add a `kernel`
- Add a new logical operator
- Typical changes are a new Python kernel file, a new wrapper C++ file, and one CMake registration
For `rope`:
- Adding one more item like `{"variant_key": "...", "head_dim": ..., "rope_dim": ..., "dtype": ...}` to `SPECIALIZATIONS` means adding a new `specialization`
- Adding a new external interface such as `xxx_wrapper.cpp` means adding a new `kernel`
## 2. Development Order
The recommended order is:
1. Implement the TileLang kernel in a Python file such as `rope.py`
2. Implement `generate_source(...)` to lower the kernel into Ascend-C source
3. Declare `DISPATCH_SCHEMA` and `SPECIALIZATIONS`
4. Generate `registry.inc` once and inspect it
5. Then write or update the runtime specialization construction logic in the wrapper
6. Wire it into CMake and run tests
The key idea behind this order is:
- implement the kernel itself first
- then fix the runtime dispatch schema
- then write the wrapper against the generated `registry.inc`
## 3. Write the Python Kernel
Using `rope.py` as the example, the Python side can be understood in three layers:
- `build_rope_kernel(...)`: kernel implementation
- `generate_source(...)`: AOT export
- `RopeKernel`: kernel registration plus dispatch schema and compiled instance declaration
### 3.1 Implement `build_rope_kernel(...)`
`build_rope_kernel(...)` is the actual TileLang kernel implementation. This is where you write:
- `@T.prim_func`
- input and output tensor shapes
- parallel task organization under `with T.Kernel(...)`
- UB allocation and the actual compute logic
The simplified structure in `rope.py` looks like this:
```python
def build_rope_kernel(
head_dim: int,
rope_dim: int,
vec_core_num: int,
ub_buffer_bytes: int,
):
task_num = vec_core_num
m_num = vec_core_num // 2
@T.prim_func
def rope_in_place_kernel(...):
with T.Kernel(m_num, is_npu=True) as (cid, vid):
task_id = cid * 2 + vid
...
return rope_in_place_kernel
```
Here, `head_dim` and `rope_dim` are the compile-time parameters that this implementation actually depends on.
Vector kernels such as `rope` must also follow the fixed-task convention used by the current AOT path. In the current AOT flow, the kernel launch `block_num` is fixed at compile time, which means:
- runtime input shapes do not change the kernel launch `block_num`
- runtime input shapes only change workload splitting across the fixed tasks
The convention in the current `rope.py` is:
```python
task_num = vec_core_num
m_num = vec_core_num // 2
with T.Kernel(m_num, is_npu=True) as (cid, vid):
task_id = cid * 2 + vid
```
This means:
- `cid` ranges over `[0, vec_core_num // 2)`
- `vid` ranges over `[0, 2)`
- the total task count is fixed as `task_num = vec_core_num`
As a result, `rope.py` also derives the compile-time token count for one specialization using the fixed task count:
```python
max_rows_num_in_ub = _derive_max_rows_num_in_ub(...)
compile_num_tokens = task_num * max_rows_num_in_ub
```
### 3.2 Implement `generate_source(...)`
`generate_source(...)` lowers the TileLang kernel above into the final source code. The export layer takes one specialization and turns it into compilable Ascend-C source.
For `rope`, the core logic is:
```python
@staticmethod
def generate_source(head_dim: int, rope_dim: int, dtype: str) -> str:
vec_core_num = detect_vec_core_num()
tilelang_kernel = build_rope_kernel(
head_dim=head_dim,
rope_dim=rope_dim,
vec_core_num=vec_core_num,
ub_buffer_bytes=FIXED_UB_BUFFER_BYTES,
)
with tilelang.tvm.transform.PassContext(...):
kernel = tilelang.engine.lower(tilelang_kernel)
return kernel.kernel_source
```
The rules here are:
- the inputs to `generate_source(...)` come from the current `SPECIALIZATIONS` entry
- `generate_source(...)` calls `build_rope_kernel(...)`
- the return value is the lowered source string
### 3.3 Declare `DISPATCH_SCHEMA` and `SPECIALIZATIONS`
After the kernel implementation and export layer are done, use an `@register_kernel` class to attach the kernel to the framework.
The current minimal template in `rope.py` is:
```python
from ....common.spec import DispatchField, TilelangKernel, register_kernel
@register_kernel
class RopeKernel(TilelangKernel):
DISPATCH_SCHEMA = [
DispatchField("head_dim", "int32"),
DispatchField("rope_dim", "int32"),
DispatchField("dtype", "dtype"),
]
SPECIALIZATIONS = [
{
"variant_key": "hd128_rd128_bf16",
"head_dim": 128,
"rope_dim": 128,
"dtype": "bf16",
},
{
"variant_key": "hd576_rd64_bf16",
"head_dim": 576,
"rope_dim": 64,
"dtype": "bf16",
},
]
@staticmethod
def generate_source(head_dim: int, rope_dim: int, dtype: str) -> str:
...
```
There are two concepts to distinguish here:
- `DISPATCH_SCHEMA`
- defines the field names, order, and types of the runtime specialization
- is the single source of truth for the C++ specialization struct, builder, and lookup interface
- `SPECIALIZATIONS`
- represents the set of instances that will actually be compiled
- each item corresponds to one variant
The rules are:
- every field in `DISPATCH_SCHEMA` must appear in every `SPECIALIZATIONS` item
- `SPECIALIZATIONS` may contain extra fields; those fields are passed into `generate_source(...)`, but do not enter the runtime dispatch schema
- `variant_key` is the unique identifier for that specialization
- `DISPATCH_SCHEMA` and `SPECIALIZATIONS` must match the runtime specialization one-to-one
For `rope`, the runtime dispatch dimensions are:
- `head_dim`
- `rope_dim`
- `dtype`
So these three fields must appear in both:
- `DISPATCH_SCHEMA`
- every `SPECIALIZATIONS` item
At build time, Ascend build resolves the actual `bisheng_arch` from the `--device a2|a3` value passed by the main build path.
### 3.4 Inspect the Generated Ascend-C Source
When debugging the implementation details in `build_rope_kernel(...)`, or comparing how different kernel styles affect the final code generation, use the common `compile-kernels` entry to regenerate artifacts and inspect the Ascend-C source for the specialization you care about.
For `rope`, you can fix:
- `head_dim=576`
- `rope_dim=64`
- `dtype=bf16`
Then regenerate the `rope` artifacts:
```bash
python xllm/compiler/tilelang_launcher.py compile-kernels \
--target ascend \
--device a3 \
--output-root /tmp/tilelang_debug \
--kernels rope \
--force
```
It is recommended to keep `--force` so the source and object files are regenerated from the current code instead of reusing an old cache hit.
This command uses an isolated debug output directory, `/tmp/tilelang_debug`, so only the debug artifacts for `rope` are generated there and they do not get mixed with artifacts from other kernels in the main build directory.
After that, you can directly inspect the generated source for the specialization, including the entry function, UB allocation, and vector compute logic:
```bash
sed -n '1,200p' \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
rg -n 'extern "C"|__global__|alloc_ub|alloc_shared|g_tilingKey' \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
```
To compare two kernel implementations, keep the specialization fixed, run `compile-kernels --force` before and after the change, then diff the generated `.cpp` file:
```bash
cp /tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp \
/tmp/rope_before.cpp
diff -u /tmp/rope_before.cpp \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
```
This helps isolate specialization changes from kernel implementation changes.
After generation, the main files to inspect are:
- `/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp`
- `/tmp/tilelang_debug/targets/ascend/rope/registry.inc`
- `/tmp/tilelang_debug/targets/ascend/rope/manifest.json`
These correspond to:
- the final Ascend-C source for one specialization
- the runtime dispatch interface directly included by the wrapper
- the full compiled artifact record for the current kernel
The recommended debugging sequence is:
1. run `compile-kernels --force` to regenerate the current kernel artifacts
2. inspect the `.cpp` for the specialization and analyze the code generation result
3. inspect `registry.inc` and `manifest.json` to confirm they match expectations
4. finally run `rope_wrapper_test` to check end-to-end behavior and performance
## 4. Update the Wrapper
When adding a new `kernel`, you need a new wrapper. When adding a new `specialization`, the wrapper only needs an update if the runtime specialization semantics change.
For `rope_wrapper.cpp`, the manually written parts should remain:
- tensor shape, dtype, and layout validation
- reshaping inputs into `x_rows / sin_rows / cos_rows`
- constructing the runtime specialization from tensors
- assembling launch arguments and calling `entry->fn(...)`
### 4.1 What `registry.inc` Generates Automatically
`registry.inc` is generated automatically from the Python-side `DISPATCH_SCHEMA`, `SPECIALIZATIONS`, and the exported Ascend-C ABI.
For `rope`, the generated content includes:
- `RopeSpecialization`
- `RopeHeadDim`
- `RopeRopeDim`
- `RopeDType`
- `RopeKernelFn`
- `make_rope_specialization(...)`
- `find_rope_kernel_entry(...)`
- `available_rope_variant_keys()`
For `rope_wrapper.cpp`, `registry.inc` directly provides dispatch-related definitions such as `RopeSpecialization`, `operator==(...)`, and `RopeKernelFn`. Dtype conversion uses the shared helper `to_tilelang_dtype(...)`.
### 4.2 What the Wrapper Actually Needs to Write
The most important handwritten logic in `rope_wrapper.cpp` is constructing the runtime specialization from the tensors. The current code looks like this:
```cpp
RopeSpecialization build_runtime_specialization(const torch::Tensor& x_rows) {
return make_rope_specialization(
RopeHeadDim{static_cast<int32_t>(x_rows.stride(0))},
RopeRopeDim{static_cast<int32_t>(x_rows.size(1))},
RopeDType{to_tilelang_dtype(x_rows.scalar_type())});
}
```
For `rope`:
- `head_dim` maps to `x_rows.stride(0)`, which is the `x_stride` used by the kernel
- `rope_dim` maps to `x_rows.size(1)`
- `dtype` maps to `x_rows.scalar_type()`
The runtime path is:
1. the wrapper reshapes the inputs into `x_rows / sin_rows / cos_rows`
2. `build_runtime_specialization(...)` constructs a specialization from `x_rows`
3. `find_rope_kernel_entry(...)` performs an exact match in the static registry
4. after a match, `entry->fn(...)` calls the actual compiled symbol
The current lookup strategy is a linear scan with exact matching. If any of `head_dim`, `rope_dim`, or `dtype` differs, the lookup will miss.
So when you add a new `specialization`, the main things to cross-check are:
- the field semantics in Python-side `DISPATCH_SCHEMA`
- the field values in Python-side `SPECIALIZATIONS`
- the field values constructed by `build_runtime_specialization(...)` in the wrapper
All three must match exactly.
### 4.3 Generate and Inspect `registry.inc` First
Before writing or modifying wrapper code, generate `registry.inc` once and inspect it. Focus on:
- whether the generated field order in `RopeSpecialization` matches expectations
- whether the generated wrapped field type names match expectations
- the parameter order of `make_rope_specialization(...)`
- the generated entry symbol names
`registry.inc` is the direct contract for the wrapper. Inspect it first, then write the wrapper against it.
## 5. Update CMake
When adding a new `kernel`, register it in `xllm/xllm/core/kernels/npu/tilelang/CMakeLists.txt`.
CMake registration is unified through the high-level helper:
- `tilelang_register_runtime_kernel(NAME <kernel> WRAPPER_SRCS <srcs...>)`
Using `rope` as the example, the minimal template is:
```cmake
tilelang_register_runtime_kernel(
NAME rope
WRAPPER_SRCS rope_wrapper.cpp
)
```
This helper will:
- derive the manifest path as `TILELANG_GENERATED_ROOT/targets/ascend/<kernel>/manifest.json`
- import the manifest
- add the wrapper source and compiled objects into `tilelang_kernels`
- append the `XLLM_TL_<KERNEL>_REGISTRY_INC=...` compile definition automatically
So when adding a new runtime kernel, the CMake-side work mainly consists of two things:
1. make sure the Python side can already generate the manifest for that kernel
2. add one `tilelang_register_runtime_kernel(...)` entry in the TileLang CMakeLists
For day-to-day kernel additions, add one `tilelang_register_runtime_kernel(...)` line directly in CMake. `tilelang_import_kernel_manifest(...)` stays underneath as the implementation base for that higher-level helper.
## 6. Validate
The recommended validation order is:
1. compile the TileLang kernel and inspect the generated `registry.inc`
2. then run the full wrapper test
Common commands:
```bash
python xllm/compiler/tilelang_launcher.py compile-kernels \
--target ascend \
--device a3 \
--output-root build/cmake.linux-aarch64-cpython-311/xllm/compiler/tilelang \
--kernels rope
python setup.py test --test-name rope_wrapper_test --device a3
```
The first command generates `manifest.json`, `registry.inc`, and the object files. The second command validates the full integration path.

29
upstream_ref/xllm/docs/en/features/async_schedule.md Normal file
View File

@@ -0,0 +1,29 @@
# Async schedule
## Background
The inference process of large language models can be divided into three sequential stages:1) CPU-side scheduling (preparing model inputs), 2) Device computation (GPU/TPU execution), 3) CPU-side post-processing (output handling).
Due to the sequential nature of decoding operations, the input for step-i+1 depends on the output of step-i. This forces strict serial execution of all three stages, creating device idle periods (“bubbles”) during CPU-bound stages 1 and 3, leading to suboptimal resource utilization.
## Introduction
xLLM addresses this at the framework level by supporting asynchronous scheduling, where the CPU proactively executes scheduling operations for step-i+1 while the device is computing step-i. This allows the device to immediately begin computing step-i+1 upon completing step-i, thereby eliminating bubbles. Specifically, after initiating the computation call for step-i, the CPU does not wait for the device to finish computing. Instead, it constructs fake tokens for the step-i request, uses these fake tokens to perform scheduling operations for step-i+1 (such as allocating KV Cache), and replaces them with the true tokens computed in step-i when launching step-i+1 computation to ensure correctness. Meanwhile, the CPU processes the results of step-i in a separate thread and returns them to the client.
In the overall architecture, stages 1 and 3 on the CPU side are handled by different thread pools, and RPC function calls employ non-blocking C++ future and promise mechanisms to achieve a fully asynchronous runtime.
![Async schedule](../../assets/async_schedule_architecture.jpg)
## Usage
xLLM provides the gflags parameter `enable_schedule_overlap`, which defaults to false. To enable this feature, set it to true in xLLM's service startup script:
```shell
--enable_schedule_overlap=true
```
## Performance
- With asynchronous scheduling enabled, the device idle time between two steps is approximately 200us - comparable to a single kernel launch duration.
- On the DeepSeek-R1-Distill-Qwen-1.5B model with TPOT constrained to 50ms, this achieves 17% throughput improvement.
## Notice
The asynchronous scheduling feature requires the server to compute one additional step. For use cases involving limited output tokens (e.g., few-token generation) or single-output scenarios like embedding models, enabling this feature is not recommended as it may reduce server-side throughput, thus hard-disabled internally.
The VLM model is currently being adapted, will be temporarily disabled.

5
upstream_ref/xllm/docs/en/features/basics.md Normal file
View File

@@ -0,0 +1,5 @@
# Basics
- xLLM uses a one-device-per-process architecture. Across multiple devices, RPC is used for function calls, and data communication during model computation uses device collective communication libraries.
- HCCL/LCCL are high-performance collective communication frameworks that provide data-parallel and model-parallel collective communication for both single-node multi-device and multi-node multi-device scenarios.

19
upstream_ref/xllm/docs/en/features/chunked_scheduler.md Normal file
View File

@@ -0,0 +1,19 @@
# ChunkedPrefill Scheduler
## Feature Introduction
xLLM supports the chunked prefill scheduling strategy. Chunked prefill is a technique that optimizes large language model inference by splitting long prompts into smaller chunks for batch processing, rather than processing the entire prompt at once.
This method can effectively reduce peak GPU memory usage, improve device utilization, and better schedule and mix processing with requests from the decode stage.
## Usage
The aforementioned strategy has been implemented in xLLM and is exposed through gflags parameters to control the feature's on/off state.
- Enable chunked prefill and set the chunked size, if not set chunked size, its default value is equal to max_tokens_per_batch.
```bash
--enable_chunked_prefill=true
--max_tokens_per_chunk_for_prefill=20480 # optional
```
## Performance Impact
After enabling chunked prefill, on the Qwen3-8B model with a TPOT constraint of 50ms, the TTFT latency **decreased by 46%**.

7
upstream_ref/xllm/docs/en/features/continuous_scheduler.md Normal file
View File

@@ -0,0 +1,7 @@
# Continuous Scheduler
## Feature Introduction
xLLM implements a scheduling strategy that supports continuous batching. Continuous batching is a dynamic batching strategy that does not wait for a batch to be filled. Instead, it starts processing as soon as requests are available, while continuously accepting new requests and adding them to the currently executing batch. This approach significantly reduces latency while maintaining high throughput.
## Usage
xLLM implements the continuous batching scheduling strategy. And, the default scheduler is chunked prefill, `enable_chunked_prefill=true` by default.

83
upstream_ref/xllm/docs/en/features/disagg_pd.md Normal file
View File

@@ -0,0 +1,83 @@
# Disaggregated PD
## Background
LLM online inference services typically need to meet two performance metrics: TTFT and TPOT. Traditional Contiguous Batching scheduling strategies mix Prefill and Decode requests during scheduling, causing Prefill and Decode phases to compete for computational resources. This prevents maximized utilization of computing resources and impacts performance metrics. To resolve this conflict, the Prefill and Decode phases are split to run on independent computational resources, enabling parallel execution. This simultaneously reduces TTFT and TPOT while improving throughput.
## Introduction
The xLLM PD Separation feature is primarily implemented through the following three modules:
- **etcd**: Stores metadata such as instance information.
- **xLLM Service**: Schedules requests and manages all computing instances.
- **xLLM**: Handles request computation instances.
The overall architecture is shown below:
![xLLM PD Separation Architecture](../../assets/pd_architecture.jpg)
## Usage
### Preparation
#### Install Dependencies
- **xLLM**: Refer to [Installation && Compilation](../getting_started/quick_start.md)
- **xLLM Service**: Refer to [PD disaggregation](../getting_started/disagg_pd.md)
#### Obtain Environment Information
Deploying Disaggregated PD Service requires obtaining the Device IP of the machine to create communication resources. Execute the command `cat /etc/hccn.conf | grep address` on the current AI Server to get the Device IP, for example:
```
address_0=xx.xx.xx.xx
address_1=xx.xx.xx.xx
```
`address_xx` represents the Device IP.
### Start Disaggregated PD Service
1. Start etcd
```bash
./etcd
```
2. Start xLLM Service
```bash
ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
3. Start xLLM
- Taking Qwen2-7B as an example
- Start Prefill Instance
```bash
/path/to/xllm --model=Qwen2-7B-Instruct \
--port=8010 \
--devices="npu:0" \
--master_node_addr="127.0.0.1:18888" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=PREFILL \
--etcd_addr="127.0.0.1:12389" \
--transfer_listen_port=26000 \
--disagg_pd_port=7777 \
--node_rank=0 \
--nnodes=1
```
- Start Decode Instance
```bash
/path/to/xllm --model=Qwen2-7B-Instruct \
--port=8020 \
--devices="npu:1" \
--master_node_addr="127.0.0.1:18898" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=DECODE \
--etcd_addr="127.0.0.1:12389" \
--transfer_listen_port=26100 \
--disagg_pd_port=7787 \
--node_rank=0 \
--nnodes=1
```
Important notes:
- PD disaggregation requires reading the `/etc/hccn.conf` file. Make sure this file on the physical machine is mapped into the container.
- `etcd_addr` must match the `etcd_addr` of `xllm_service`
## Notice
Disaggregated PD **does not support** enabling prefix cache or chunked prefill. These features must be disabled using the following parameters:
```shell
--enable_prefix_cache=false
--enable_chunked_prefill=false
```

30
upstream_ref/xllm/docs/en/features/eplb.md Normal file
View File

@@ -0,0 +1,30 @@
# MoE Load Balancing (EPLB)
## Background
MoE models rely on dynamic token routing to distribute tokens among experts. However, in real-world deployments, uneven data distribution leads to expert load imbalance (some overloaded while others idle). Expert redundancy adjustment (e.g., adding/removing replicas) consumes additional GPU memory and may impact inference latency due to weight migration, posing significant implementation challenges. To address this, we employ an expert redundancy strategy (replicating hot experts) combined with hierarchical and global dynamic load balancing to achieve dynamic MoE load balancing.
## Features
The xLLM MoE Load Balancing (EPLB) functionality is implemented through three main modules:
- **EPLB Manager**: Responsible for monitoring expert loads, collecting and managing expert distribution updates. It uses a layer-by-layer update mechanism, determining whether to update each layer based on expert load changes.
- **EPLB Executor**: The actual executor for expert distribution updates.
- **EPLB Policy**: Strategy for generating new expert load tables.
Overall architecture diagram:
![xLLM eplb](../../assets/eplb_architecture.png)
## Usage
Simply add the following gflag parameters when launching xLLM:
(Replace with actual number of devices. `ep_size` must match the number of devices)
- xLLM provides the gflag parameter `enable_eplb` (default: false). Set to true in the xLLM service startup script to enable dynamic expert load balancing.
- `expert_parallel_degree` and `ep_size` are MoE-related parameters. `expert_parallel_degree` should be set to `2`, and `ep_size` must match the actual number of NPU/GPU devices. See [moe_params](./moe_params.md)
- `eplb_update_interval` sets the expert distribution update interval in seconds (default: 1000).
- The expert distribution update uses a layer-by-layer mechanism based on expert load. When the similarity between consecutive loads for a layer is below `eplb_update_threshold`, that layer is updated (default: 1, range: 0-1).
```bash
--enable_eplb=true --expert_parallel_degree=2 --ep_size=16 --eplb_update_interval=2000 --eplb_update_threshold=0.9

42
upstream_ref/xllm/docs/en/features/global_kvcache.md Normal file
View File

@@ -0,0 +1,42 @@
# Global Multi-Level KV Cache
## Background
In the decoding phase of large language models (LLMs), frequent access to historical KV cache due to autoregressive generation creates a bottleneck in memory bandwidth. As model sizes and context windows expand (e.g., 128K Tokens consuming over 40GB of memory), the pressure on single-device memory increases dramatically. Existing solutions (such as vLLM) exhibit significant limitations in long-context scenarios: prefill time surges, severe memory bandwidth contention during decoding, and the need for excessive resource reservation to meet SLO requirements (TTFT < 2s, TBT < 100ms). This often results in GPU utilization below 40% and difficulties in leveraging cross-server resources. To address this, we propose a distributed global multi-level KV cache management system, adopting a memory-compute integrated architecture to break through single-machine resource constraints.
## Feature Introduction
The xLLM Global KV Cache feature is primarily implemented through the following three modules:
- **etcd**: For cluster service registration, load information synchronization, and global cache state management.
- **xLLM Service**: For scheduling requests and managing all compute instances.
- **xLLM**: The compute instances handling requests.
The overall architecture is shown in the diagram below:
![xLLM Global Multi-Level KV Cache](../../assets/globalkvcache_architecture.png)
## Usage Example
### Preparation
#### Install Dependencies
- **xLLM**: Refer to [Quick Start](../getting_started/quick_start.md)
- **xLLM Service**: Refer to [PD disaggregation](../getting_started/disagg_pd.md)
### Usage Instructions
1. **etcd Startup Configuration:**
```bash
./etcd --listen-peer-urls=http://0.0.0.0:10999 --listen-client-urls=http://0.0.0.0:10998
```
2. **xLLM Service Startup Configuration:**
```bash
./xllm_master_serving --etcd_addr="127.0.0.1:10998" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
3. **xLLM Startup Configuration:**
Add the following gflag parameters when starting xLLM:
```bash
--enable_service_routing=true
--enable_cache_upload=true
# PD separation currently does not support Global KVCache Management
--enable_disagg_pd=false
```

79
upstream_ref/xllm/docs/en/features/graph_mode.md Normal file
View File

@@ -0,0 +1,79 @@
# Graph Mode
## Overview
xLLM supports Graph Mode: computation graphs are pre-captured and replayed in subsequent runs to reduce CPU overhead and improve inference performance. Graph Mode has corresponding implementations on different hardware platforms.
## Feature Description
To optimize Host-side scheduling, graph mode submits a large task from the CPU once and then executes small kernels in a streaming manner on the device, significantly reducing startup time and device bubbles.
In the xLLM engine, Graph Mode provides the following:
### Dynamic Shape Parameterization
- Key dynamic dimensions other than num_tokens are treated as whole-graph input parameters, including batch_size, kv_seq_lens, q_seq_lens, block_table_size, and the like, for flexibility. During memory allocation and kernel configuration. At graph launch, the actual values of these parameters are passed so kernels use the correct strides to access data.
### Piecewise Graph
- When some operators do not support graph capture and thus break the full graph, each segment (piece) after the break is captured as a separate graph. This maximizes graph-mode benefits even when the full graph cannot be captured, and is commonly used for prefill and chunked prefill.
### Multi-Shape Reusable Memory Pool
- To avoid waste from separate memory buffers (input, output, and intermediate tensors) per shape, we use an expandable memory pool. Multiple shapes share the pool base address, with different shapes using different offsets from that base.
## Usage
These capabilities are implemented inside the xLLM engine and are generally controlled through gflags.
The minimal configuration only needs `enable_graph` to turn on Graph Mode for the decode phase:
```shell
--enable_graph=true
```
Common companion flags include:
- `enable_graph`: enables the base Graph Mode capability for the decode phase
- `enable_prefill_piecewise_graph`: enables Piecewise Graph for the prefill phase
- `enable_graph_mode_decode_no_padding`: builds decode graphs with the actual `num_tokens` instead of the padded shape
- `max_tokens_for_graph_mode`: limits the maximum number of tokens covered by Graph Mode; `0` means no limit
If you want to enable both decode Graph and prefill Piecewise Graph, use:
```shell
--enable_graph=true \
--enable_prefill_piecewise_graph=true \
--max_tokens_for_graph_mode=2048
```
If you need decode graph capture without padding, add:
```shell
--enable_graph=true \
--enable_graph_mode_decode_no_padding=true
```
For a more complete description of the flags, see [CLI Reference](../cli_reference.md).
## Performance Impact
- With Graph Mode enabled, decode-phase throughput **improves by about 8%–10%** on models such as Qwen3-0.6B and Qwen3-1.7B.
## Model Support
The following table lists each model’s support on ACLGraph, CudaGraph, and MLUGraph.
| Model | ACLGraph | CudaGraph | MLUGraph |
|------|----------|-----------|----------|
| Qwen3/Qwen3-MoE | ✅ | ✅ | ✅ |
| DeepseekV3.2 | ✅ | | ✅ |
| GLM4.5/4.6/4.7 | ✅ | | |
| Qwen2.5-VL | | | ✅ |
| Qwen3-VL/Qwen3-VL-MoE | ✅ | | |
| GLM4V | ✅ | | |
| GLM4V-MoE | ✅ | | |
!!! warning "Adding Graph Mode support for new models"
Ensure that the kernels used in the computation implement dynamic dimension parameterization; otherwise the graph may break and kernels may need to be re-implemented.
## Related Documentation
- For more detailed Graph Mode design and implementation notes, including ACL Graph / CUDA Graph fundamentals, dynamic dimension parameterization, Piecewise Graph, and multi-shape memory reuse, see: [Graph Mode Design Document](../design/graph_mode_design.md)

40
upstream_ref/xllm/docs/en/features/groupgemm.md Normal file
View File

@@ -0,0 +1,40 @@
# GroupGEMM Operator Optimization
## Background
The Mixture of Experts (MoE) architecture has become an important paradigm for scaling large language models. Its core idea is to dynamically route input tokens to different expert sub-networks for processing. During inference, the GroupGEMM operator is a key computational unit in the MoE architecture, responsible for efficiently executing parallel computations of multiple expert matrix multiplications, which dominate the overall inference time.
## Function Introduction
Considering the current performance bottleneck of GroupGEMM being I/O constrained, an optimization scheme is proposed that replaces data copying with index reordering, eliminating multiple copies of the token vector and instead maintaining an index table for expert allocation. Through this line number index, tokens are directly mapped to the corresponding expert computation units, and the allocation scheduling of tokens is fused with matrix multiplication into a single kernel.
## User Interface
### Operator Direct Call API
```c++
aclnnStatus aclnnIndexGroupMatmulGetWorkspaceSize(
const aclTensorList *x,
const aclTensorList *weight,
const aclTensorList *scale,
const aclTensorList *perTokenScale,
const aclTensor *groupList,
const aclTensorList *out,
uint64_t *workspaceSize,
aclOpExecutor **executor);
aclnnStatus aclnnIndexGroupMatmul(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream);
```
- `x`: The input tensor list containing the data to be processed.
- `weight`: The weight tensor containing the model parameters.
- `scale`: The scaling factor used to adjust the values of the input tensors.
- `perTokenScale`: The scaling factor for each token, used for dynamic adjustment.
- `groupList`: The list of expert groups indicating which experts participate in the computation.
- `out`: The output tensor list that stores the computation results.
## Performance Effects
![groupmatmul](../../assets/groupmatmul_performance.png)
* The optimized GroupMatmul operator shows significant advantages in computation time, especially when \( k = 128 \) and \( m = 64 \). As shown in the figure, the computation delay of the optimized operator is **reduced by 50%**.

24
upstream_ref/xllm/docs/en/features/moe_params.md Normal file
View File

@@ -0,0 +1,24 @@
# EP Parallelism
## Background
When deploying the DeepSeek-R1 671B parameter-scale model, traditional distributed deployment faces core bottlenecks including low GPU memory utilization, high communication overhead, and expensive hardware costs. Therefore, Expert Parallelism (EP) is introduced.
Key advantages:
+ With the same resources, fewer Experts per GPU means more memory available for KV Cache, allowing more tokens to be cached.
+ Due to MLA characteristics, smaller TP Size with the same resources means less redundant KV Cache, enabling more tokens to be cached.
+ Large-scale EP deployment can concentrate token computations for the same expert on the same device, improving hardware utilization.
## Parameter Configuration
+ **dp_size**: Sets the data parallelism scale for the Attention part. Default: 1, can be set to powers of 2. When dp_size doesn't equal the total number of devices, TP parallelism is used within the DP group.
+ **ep_size**: Sets the expert parallelism scale for the MoE part. Default: 1, can be set to powers of 2. When ep_size doesn't equal the total number of devices, TP parallelism is used within the DP group.
+ **expert_parallel_degree**: EP-related parameter. Default is 0 when EP is disabled. When EP is enabled, default is 1 (EP Level 1). When ep_size equals the total number of devices, can be set to 2 to enable EP Level 2.
## Solution Design
+ When EP is enabled (default EP Level 1), after computing both Attention and MoE parts, data is sent to the next stage via All Gather communication across all devices. Example for 64 devices (Attention: dp32tp2, MoE: ep32tp2):
![Alt text](../../assets/moe_eplevel1.jpg)
+ When ep_size equals the total number of devices, EP Level 2 can be enabled. Communication between Attention and MoE parts changes to ALL2ALL, sending data only to required devices to reduce communication volume and overhead. Example for 64 devices:
![Alt text](../../assets/moe_eplevel2.jpg)

149
upstream_ref/xllm/docs/en/features/mtp.md Normal file
View File

@@ -0,0 +1,149 @@
# MTP Speculative Inference
## Background
MTP (Multi-Token Prediction) is an innovative inference acceleration technique that addresses efficiency bottlenecks in large language model generation. By incorporating specialized pre-training designs, MTP provides efficient draft token prediction capabilities during inference, significantly improving generation speed. Its core value lies in balancing inference efficiency with output quality, offering an optimal solution for long-sequence generation problems in LLMs, ultimately optimizing inference performance.
## Key Features
MTP offers the following core acceleration capabilities:
- **Efficient Draft Generation**: Uses a lightweight MTP architecture to rapidly generate draft tokens that serve as input for the main model's verification, dramatically reducing computation overhead compared to traditional autoregressive generation.
- **Batch Verification Mechanism**: The main model can simultaneously verify multiple MTP-generated draft tokens in batch, rather than processing them sequentially, significantly boosting inference speed.
- **High Sampling Accuracy**: MTP solves the critical pain point of low token acceptance rates in post-training draft modules (like Eagle and Medusa). By optimizing draft generation during pre-training, MTP produces tokens with higher accuracy, reducing the verification burden on the main model.
- **Reduced Inference Latency**: By pre-generating multiple potential subsequent tokens, MTP effectively decreases cumulative latency during long-text generation, creating a smoother user experience.
- **Optimized Resource Consumption**: Compared to other inference acceleration techniques, MTP maintains acceleration effects while requiring fewer additional computational resources, making it suitable for deployment in resource-constrained environments.
MTP technology provides a novel efficiency optimization solution for LLM inference, particularly well-suited for real-time applications requiring rapid responses, representing an important direction in language model inference optimization.
!!! note "Model Support"
Currently supports MTP structure export for the following models:
- DeepSeek-V3 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v3_mtp)
- DeepSeek-V3.2 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v32_mtp)
- DeepSeek-R1 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v3_mtp)
- GLM4 MoE (e.g., GLM-4.5-Air, exported MTP model_type: glm4_moe_mtp)
Note:
- DeepSeek V3 and R1 both have input model_type "deepseek_v3", and the exported MTP model will have model_type "deepseek_v3_mtp"
- DeepSeek V3.2 has input model_type "deepseek_v3" (but can be auto-detected by index_head_dim fields), and the exported MTP model will have model_type "deepseek_v32_mtp"
## Usage Example
### Export Model
The script will automatically detect the model type, or you can manually specify it.
#### DeepSeek-V3
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-V3 \
--output-dir /path/to/DeepSeek-V3-mtp
```
#### DeepSeek-V3.2
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-V3.2 \
--output-dir /path/to/DeepSeek-V3.2-mtp
```
#### DeepSeek-R1
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-R1 \
--output-dir /path/to/DeepSeek-R1-mtp
```
#### GLM4 MoE
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/GLM-4.5-Air \
--output-dir /path/to/GLM-4.5-Air-mtp
```
#### Manually Specify Model Type
If auto-detection fails, you can manually specify the model type:
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/model \
--output-dir /path/to/model-mtp \
--model-type deepseek_v3 # Options: deepseek_v3 (for V3/R1), deepseek_v32 (for V3.2), glm4_moe
```
Input model references:
- [DeepSeek-V3](https://huggingface.co/deepseek-ai/DeepSeek-V3)
- [DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)
- [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1)
- [GLM-4.5-Air](https://huggingface.co/zai-org/GLM-4.5-Air)
### Launch Script
When using MTP for inference, you need to specify both the main model and the draft model (MTP model).
#### DeepSeek-V3/V3.2/R1 Launch Example
```bash
MODEL_PATH="/models/DeepSeek-V3"
DRAFT_MODEL_PATH="/models/DeepSeek-V3-mtp"
MASTER_NODE_ADDR="127.0.0.1:42123"
START_PORT=13222
START_DEVICE=0
LOG_DIR="log"
NNODES=16
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
nohup ./xllm \
--model $MODEL_PATH \
--devices="npu:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--draft_model $DRAFT_MODEL_PATH \
--draft_devices="npu:$DEVICE" \
--num_speculative_tokens 1 \
--max_memory_utilization=0.90 \
--max_tokens_per_batch=10000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--ep_size=1 \
--dp_size=1 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--node_rank=$i > $LOG_FILE 2>&1 &
sleep 0.5
done
```
#### GLM4 MoE Launch Example
```bash
MODEL_PATH="/models/GLM-4.5-Air"
DRAFT_MODEL_PATH="/models/GLM-4.5-Air-mtp"
# ... same other configurations
```
# Performance Data
Based on ShareGPT dataset with input length=2500, output length=1500, total requests=80.
| method | Concurrency | Mean TPOT(ms) | Mean TTFT(ms) | Output Tokens/s | Total Tokens/s |
|:---------:|:-----------:|:-------------:|:-------------:|:---------------:|:--------------:|
| baseline | 1 | 40.61 | 141.80 | 24.20 | 65.77 |
| mtp | 1 | 28.33 | 142.35 | 35.19 | 95.52 |
| baseline | 2 | 42.69 | 178.59 | 45.16 | 122.74 |
| mtp | 2 | 29.81 | 187.97 | 64.75 | 175.78 |
| baseline | 4 | 46.18 | 172.34 | 79.83 | 216.96 |
| mtp | 4 | 33.54 | 194.22 | 111.18 | 301.81 |
| baseline | 8 | 53.16 | 181.49 | 110.68 | 300.81 |
| mtp | 8 | 40.99 | 203.37 | 154.46 | 419.34 |
| baseline | 16 | 68.50 | 213.89 | 143.81 | 390.84 |
| mtp | 16 | 57.04 | 254.99 | 201.89 | 548.04 |
| baseline | 20 | 74.72 | 228.80 | 154.77 | 420.65 |
| mtp | 20 | 61.73 | 264.34 | 206.24 | 559.84 |
| baseline | 40 | 119.68 | 559.32 | 180.22 | 489.80 |
| mtp | 40 | 105.70 | 544.54 | 252.91 | 686.74 |
| baseline | 80 | 180.89 | 2996.21 | 192.09 | 522.06 |
| mtp | 80 | 152.19 | 2163.72 | 278.07 | 755.12 |

31
upstream_ref/xllm/docs/en/features/multi_streams.md Normal file
View File

@@ -0,0 +1,31 @@
# Multi-stream parallel
## Background
In distributed inference scenarios for large-scale models, additional communication operations are required to aggregate computation results from different devices. Taking large-scale MoE models like Deepseek as an example, the distributed scale is typically substantial, leading to increased communication overhead.
If both computation and communication are performed on the same stream, the device’s computing resources will remain idle while waiting for communication to complete, resulting in wasted computational capacity before subsequent calculations can begin.
## Introduction
xLLM implements multi-stream parallelism at the model layer, where the input batch is split into 2 micro-batches. One stream handles computation for the first micro-batch, another concurrently executes communication for the second micro-batch.
This overlap of computation and communication effectively hides the communication latency.![multi_streams_parallel](../../assets/multi_streams_architecture.jpg)
## Usage
xLLM provides the gflags parameter `enable_multi_stream_parallel`, which defaults to false. To enable this feature, set it to true in xLLM’s service startup script, as:
```shell
--enable_multi_stream_parallel=true
```
## Performance
With prefill dual-stream parallelism enabled, it can effectively mask over 75% of communication overhead.
On the DeepSeek-R1 model, when generating just 1 token, this achieves:
- **7%** reduction in TTFT.
- **7%** throughput improvement.
## Notice
The dual-stream parallelism currently only supports the prefill phase, with greater performance benefits observed for longer input requests.
Only Support DeepSeek, Qwen3 dense(non-MoE) models.

51
upstream_ref/xllm/docs/en/features/overview.md Normal file
View File

@@ -0,0 +1,51 @@
# Overall Architecture
## Backgroud
In recent years, with the groundbreaking progress of large language models (LLMs) ranging from tens of billions to trillions of parameters (such as GPT, Claude, DeepSeek, LLaMA, etc.) in the fields of natural language processing and multimodal interaction, the industry has an urgent need for efficient inference engines and service systems. How to reduce cluster inference costs and improve inference efficiency has become a key challenge for achieving large-scale commercial deployment.
Although a number of optimization engines for large model inference have emerged, several technical bottlenecks remain in practical deployment:
* **Hardware Adaptability Challenges:** Existing inference engines lack sufficient support for the architectural characteristics of specialized accelerators like domestic chips, making it difficult to fully exploit the performance potential of heterogeneous computing hardware, leading to low computational resource utilization.
* **MoE Architecture Optimization Difficulties:** The token distribution process in expert parallelism mechanisms generates significant All-to-All communication overhead, while dynamic routing strategies cause expert load imbalance, severely constraining system scalability.
* **Long Context Management Bottlenecks:** As model context windows continue to expand, the efficiency of optimizations in KV cache handling—such as memory fragmentation management and cross-node synchronization—directly impacts overall inference throughput performance.
* **Hybrid Deployment Efficiency Limitations:** Existing inference clusters struggle to simultaneously guarantee service quality (SLO) and optimize resource utilization when handling both online services and offline tasks.
* **Insufficient Dynamic PD Adaptation:** When input/output sequence lengths fluctuate drastically, static PD resource partitioning lacks the ability to adjust PD resource configurations in real-time. This can lead to idle GPU resources and poses a risk of SLO violations.
To address these challenges, we present xLLM—an efficient and user-friendly open-source intelligent inference framework that provides enterprise-grade service guarantees and high-performance engine computing capabilities for model inference on domestic chips.
## Feature Introduction
xLLM provides intelligent computing capabilities, implementing joint inference acceleration across multiple computational system layers and algorithm-driven layers:
### Computational System Layer
#### Multi-Level Pipeline Execution Orchestration
Asynchronizes CPU scheduling at the framework layer to pipeline it with chip inference computation, reducing computation bubbles; at the model graph layer, splits single batches to create pipelines between micro-batches, overlapping computation and communication; at the operator kernel layer, pipelines different computing units, overlapping computation and memory access.
#### Dynamic Shape Graph Execution Optimization
Addressing the static graph adaptation problem for large language models processing dynamic inputs (e.g., variable sequence lengths and batch sizes), xLLM achieves dynamic adaptation through parametric design capturing input dimensions. It combines a multi-graph caching scheme to reduce compilation overhead and uses a managed memory pool instead of absolute addresses to ensure safe reuse, ultimately achieving high execution efficiency while maintaining high flexibility.
#### Operator Optimization
xLLM implements specific optimizations for key operators in LLMs on domestic hardware chips, including GroupMatmul, Chunked Prefill, and others.
#### xTensor Memory Management
The xTensor memory management framework employs a method of *pre-allocated physical memory page pools + contiguous virtual address mapping*. It achieves efficient dynamic memory management through dynamic on-demand mapping of physical pages, reuse of reusable memory pages (Reusable), and optimized scheduling with asynchronous pre-mapping. Combined with NPU operator adaptation (such as virtual address FlashMLA), it results in improved memory utilization and reduced latency.
### Algorithm-Driven Layer
#### PD Separation
xLLM fully supports PD separation scenarios, enabling efficient management of PD instances, communication between PD instances, and KV cache transfer.
#### Global Scheduling
xLLM provides intelligent, full-lifecycle resource scheduling management for requests and instances.
##### Instance Scheduling
We have implemented various instance scheduling strategies to select how to assign instances to more suitable ones. These include a simple Round Robin strategy, a prefix cache-aware strategy based on the prefix cache hit rate of requests on each instance, and a KV cache-aware strategy based on the free memory level of instances. Furthermore, for PD separation scenarios, where static PD ratios often struggle with traffic fluctuations and sudden changes in request input/output lengths, we implemented an adaptive PD dynamic scheduler responsible for global instance allocation for online requests and runtime PD dynamic adjustment.
##### Request Scheduling
We have implemented various request scheduling strategies supporting continuous batching, including chunked prefill, prefill priority, decode priority, and other batch strategies, all while fully supporting PD separation scenarios.
#### Global KV Cache Management
Utilizes ETCD as a metadata service middleware at the global level for cluster service registration, load information synchronization, and global cache state management. Each compute instance maintains a local multi-level cache pool. Regarding scheduling strategy, the system adopts a dynamic decision-making mechanism based on KV cache: it first performs prefix matching detection, calculates the KV cache reuse rate of candidate nodes, and finally selects the node with the optimal comprehensive performance for processing, achieving dynamic offloading and migration of KV cache.
#### Speculative Inference
xLLM incorporates an optimized speculative inference algorithm that generates multiple tokens at once to boost throughput. xLLM reduces communication costs by sinking the speculative module and optimizes speculative inference computation through methods like overlapping scheduling and computation timelines and reducing operator data movement in speculative scenarios.
#### MoE Load Balancing
xLLM implements expert weight updates based on historical expert load statistics for MoE models. During inference, it achieves effective dynamic load balancing through efficient expert load statistics and double-buffered, seamless expert weight updates.
### Multimodal Support
xLLM provides comprehensive support for various multimodal models, including Qwen2-VL and MiniCPMV.

36
upstream_ref/xllm/docs/en/features/ppmatmul.md Normal file
View File

@@ -0,0 +1,36 @@
# PpMatmul Operator Optimization
## Background
In the inference of large models, matrix multiplication accounts for a high proportion and takes a long time. We have optimized the implementation of the matrix multiplication operator.
## Feature Introduction
The PpMatmul operator uses a Tiling strategy to decompose matrix multiplication into multiple smaller matrix multiplication tasks. However, when the number of tiles is small, tasks cannot be evenly distributed across all NPU cores, leading to the tail effect problem, which affects computational efficiency. We optimize the performance of the PpMatmul operator by prefetching memory or redistributing tasks.
## User Interface
### Operator Direct Call API
```cpp
aclnnStatus aclnnPpMatmulOptGetWorkspaceSize(
const aclTensor *a,
const aclTensor *b,
const aclTensor *out,
uint64_t *workspaceSize,
aclOpExecutor **executor);
aclnnStatus aclnnPpMatmulOpt(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream);
```
- `a`: Input matrix A.
- `b`: Input matrix B.
- `out`: Output matrix, storing the computation result.
## Performance Effect
For cases with a small number of tiles (e.g., when M is small, corresponding to a small batch size), there is an **18%** performance improvement of the operator compared to before optimization when (TP=4).

19
upstream_ref/xllm/docs/en/features/prefix_cache.md Normal file
View File

@@ -0,0 +1,19 @@
# Prefix Cache Optimization
## Feature Introduction
xLLM supports prefix cache matching. The prefix cache is based on `murmur_hash` and uses an LRU eviction policy, delivering superior matching efficiency and increased prefix cache hit rates.
Additionally, the prefix cache has been optimized to support the `continuous_scheduler`, `chunked_scheduler`, and `zero_evict_scheduler`. The cache is updated immediately after prefill operations, enhancing matching timeliness. For the `chunked_scheduler`, multi-stage chunked prefill matching is supported, reducing computational overhead and minimizing KV cache usage as much as possible.
## Usage
The prefix cache is implemented in xLLM and exposed through gflags parameters to control its functionality.
- Enable prefix cache with specific policy and settings:
```
--enable_prefix_cache=true
```
## Performance Impact
After enabling prefix cache, on the Qwen3-8B model with a TPOT constraint of 50ms, the E2E latency **decreased by 10%**.
!!! warning "Note"
PD separation scheduler is not currently supported.

27
upstream_ref/xllm/docs/en/features/topk_topp.md Normal file
View File

@@ -0,0 +1,27 @@
# Topk & Topp Operator Optimization
## Background
In natural language generation tasks, the topK and topP sampling strategies are widely used to control the diversity and quality of generated text. However, in small models, the computation time for these two strategies is relatively long. This is mainly due to the fewer parameters in small models, which leads to reduced efficiency in sorting and filtering when processing probability distributions, thereby affecting generation speed. Therefore, optimizing the implementation of topK and topP in small models can enhance their sampling efficiency.
## Feature Introduction
The implementation of the topKtopP operator merges multiple small operators, such as sorting, topK, softmax, and topP, into a single large operator, thereby improving computational efficiency and performance.
## User Interface
### Operator Call API
```c++
void top_k_top_p(torch::Tensor& logits,
const torch::Tensor& topK,
const torch::Tensor& topP);
```
- `logits`: The input logits tensor containing the model's output scores.
- `topK`: The threshold tensor for selecting the top K probabilities.
- `topP`: The threshold tensor for selecting the cumulative probabilities.
## Performance Effect
* After using the topKtopP fused operator, in the qwen2-0.5B model, TTOT **decreased by 37%**, and TTFT **increased by 10%**.

54
upstream_ref/xllm/docs/en/features/xllm_service_overview.md Normal file
View File

@@ -0,0 +1,54 @@
# xLLM Service
<p align="right">[:simple-github: xLLM Service](https://github.com/jd-opensource/xllm-service)</p>
## Project Overview
**xLLM-service** is a service-layer framework developed based on the **xLLM** inference engine, providing efficient, fault-tolerant, and flexible LLM inference services for clustered deployment.
xLLM-service targets to address key challenges in enterprise-level service scenarios:
- How to ensure the SLA of online services and improve resource utilization of offline tasks in a hybrid online-offline deployment environment.
- How to react to changing request loads in actual businesses, such as fluctuations in input/output lengths.
- Resolving performance bottlenecks of multimodal model requests.
- Ensuring high reliability of computing instances.
#### Background
LLM with parameter scales ranging from tens of billions to trillions are being rapidly deployed in core business scenarios such as intelligent customer service, real-time recommendation, and content generation. Efficient support for domestic computing hardware has become a core requirement for low-cost inference deployment. Existing inference engines struggle to effectively adapt to the architectural characteristics of dedicated accelerators like domestic chips. Performance issues such as low utilization of computing units, load imbalance and communication overhead bottlenecks under the MoE architecture, and difficulties in kv cache management have restricted the efficient inference of requests and the scalability of the system. The xLLM-service + xLLM inference engine improves the efficiency of the entire performance link and it provides crucial technical support for the large-scale implementation of LLM in real-world business scenarios.
---
## Overall Architecture
The overall architecture of xLLM-service is shown in the figure below:
![1](../../assets/service_arch.png)
## Core Components
### ETCD Cluster
It is used for metadata management, including the storage and management of metadata such as models, xllm instances, and requests. It also provides xllm node registration and discovery services.
### Fault Tolerance
xLLM-service provides fault tolerance management to ensure service quality and stability.
### Global Scheduler
It implements globally aware scheduling. Based on the current system status, it accurately dispatches requests to the optimal instances for execution, effectively improving the overall service response efficiency and resource utilization.
### Global KV Cache Manager
It is responsible for global KV Cache management. Its core capabilities include distributed KV cache awareness, Prefix matching, and dynamic migration of KV Cache, which optimize the efficiency of cache resource usage.
### Instance Manager
It focuses on the full-lifecycle management of instances. All xllm instances must register to service after startup. Based on preset policies, the module provides support for instances such as scheduling adaptation and fault tolerance handling.
### Event Plane
As the metrics and event hub, it receives Metrics data reported by various instances, uniformly collects and organizes statistical indicators, and provides data support for decisions such as service scheduling, fault tolerance, and scaling.
### Planner
It undertakes the functions of strategy analysis and decision-making. Based on the Metrics data reported by the Event Plane (including instance runtime indicators, machine load indicators, etc.), it analyzes the service scaling needs and the necessity of expanding hot instances, and outputs resource adjustment and instance optimization strategies.

17
upstream_ref/xllm/docs/en/features/zero_evict_scheduler.md Normal file
View File

@@ -0,0 +1,17 @@
# Zero Evict Scheduler
## Feature Introduction
xLLM supports the zero evict scheduling strategy. The zero evict scheduling strategy is an algorithm designed to minimize request eviction rates, reducing the need for prefill computation on evicted requests and consequently improving TPOT (Time Per Output Token).
This scheduling algorithm employs simulation rounds to detect whether a request can be scheduled without causing the eviction of other requests.
## Usage
The aforementioned strategy has been implemented in xLLM and is exposed through gflags parameters to control the feature's on/off state.
- Enable the zero evict strategy and set the maximum decode tokens per sequence.
```
--use_zero_evict=true
--max_decode_token_per_sequence=256
```
## Performance Impact
After enabling zero evict, on the Qwen3-8B model with an E2E latency constraint, the TPOT latency **decreased by 27%**.

99
upstream_ref/xllm/docs/en/getting_started/disagg_pd.md Normal file
View File

@@ -0,0 +1,99 @@
# PD disaggregation
`xllm` supports PD disaggregation deployment, which requires integration with our other open-source library [xllm service](https://github.com/jd-opensource/xllm-service).
## xLLM Service Dependencies
First, download and install `xllm service`, similar to installing and compiling `xllm`:
```bash
git clone https://github.com/jd-opensource/xllm-service
cd xllm_service
git submodule init
git submodule update
```
### etcd Installation
`xllm_service` compilation and operation depend on [etcd](https://github.com/etcd-io/etcd).Use the [installation script](https://github.com/etcd-io/etcd/releases) provided by etcd for installation. The default installation path provided by the script is `/tmp/etcd-download-test/etcd`. You can either manually modify the installation path in the script or manually migrate after running the script:
```bash
mv /tmp/etcd-download-test/etcd /path/to/your/etcd
```
### xLLM Service Compilation
Apply patch:
```bash
sh prepare.sh
```
Then execute the compilation:
```bash
mkdir -p build
cd build
cmake ..
make -j 8
cd ..
```
!!! warning "Potential Errors"
You may encounter installation errors related to `boost-locale` and `boost-interprocess`: `vcpkg-src/packages/boost-locale_x64-linux/include: No such file or directory`, `/vcpkg-src/packages/boost-interprocess_x64-linux/include: No such file or directory`
Reinstall these packages using `vcpkg`:
```bash
/path/to/vcpkg remove boost-locale boost-interprocess
/path/to/vcpkg install boost-locale:x64-linux
/path/to/vcpkg install boost-interprocess:x64-linux
```
## PD Disaggregation Execution
Start etcd:
```bash
./etcd-download-test/etcd --listen-peer-urls 'http://localhost:2390' --listen-client-urls 'http://localhost:2389' --advertise-client-urls 'http://localhost:2391'
```
Start xllm service:
```bash
ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
Taking Qwen2-7B as an example:
- Start Prefill Instance
```bash
/path/to/xllm --model=path/to/Qwen2-7B-Instruct \
--port=8010 \
--devices="npu:0" \
--master_node_addr="127.0.0.1:18888" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=PREFILL \
--etcd_addr=127.0.0.1:12389 \
--transfer_listen_port=26000 \
--disagg_pd_port=7777 \
--node_rank=0 \
--nnodes=1
```
- Start Decode Instance
```bash
/path/to/xllm --model=path/to/Qwen2-7B-Instruct \
--port=8020 \
--devices="npu:1" \
--master_node_addr="127.0.0.1:18898" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=DECODE \
--etcd_addr=127.0.0.1:12389 \
--transfer_listen_port=26100 \
--disagg_pd_port=7787 \
--node_rank=0 \
--nnodes=1
```
Important notes:
- PD disaggregation requires reading the `/etc/hccn.conf` file. Make sure this file on the physical machine is mapped into the container.
- `etcd_addr` must match the `etcd_addr` of `xllm_service`
The test command is similar to above. Note that the `PORT` in `curl http://localhost:{PORT}/v1/chat/completions ...` should be the `port` of the `http_server_port` of xLLM service.

125
upstream_ref/xllm/docs/en/getting_started/launch_xllm.md Normal file
View File

@@ -0,0 +1,125 @@
# Launch xllm
Taking Qwen3 as an example, the script for launching xllm is as follows. The provided script is suitable for both single-node single-device and single-node multi-device scenarios. When using multiple devices on a single node, you need to modify `NNODES` (one device represents one node), as well as environment variables such as `ASCEND_RT_VISIBLE_DEVICES`, `CUDA_VISIBLE_DEVICES`, or `MLU_VISIBLE_DEVICES`.
## NPU
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_RT_VISIBLE_DEVICES=0
export HCCL_IF_BASE_PORT=43432 # HCCL communication base port
MODEL_PATH="/path/to/model/Qwen3-8B" # Model path
MASTER_NODE_ADDR="127.0.0.1:9748" # Master node address (must be globally consistent)
START_PORT=18000 # Service starting port
START_DEVICE=0 # Starting logical device number
LOG_DIR="log" # Log directory
NNODES=1 # Number of nodes (current script launches 1 process)
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="npu:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--block_size=128 \
--communication_backend="hccl" \
--enable_prefix_cache=false \
--enable_chunked_prefill=true \
--enable_schedule_overlap=true \
--enable_shm=true \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
## NVIDIA GPU
```bash
#!/bin/bash
set -e
rm -rf core.*
export CUDA_VISIBLE_DEVICES=0
# for debug
# export CUDA_LAUNCH_BLOCKING=1
MODEL_PATH="/path/to/model/Qwen3-8B"
MASTER_NODE_ADDR="127.0.0.1:9748"
START_PORT=18000
START_DEVICE=0
LOG_DIR="log"
NNODES=1
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="cuda:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--block_size=32 \
--max_memory_utilization=0.8 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_schedule_overlap=true \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
## MLU
```bash
#!/bin/bash
set -e
rm -rf core.*
export MLU_VISIBLE_DEVICES=0
MODEL_PATH="/path/to/model/Qwen3-8B"
MASTER_NODE_ADDR="127.0.0.1:9748"
START_PORT=18000
START_DEVICE=0
LOG_DIR="log"
NNODES=1
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="mlu:$DEVICE" \
--port $PORT \
--nnodes=$NNODES \
--master_node_addr=$MASTER_NODE_ADDR \
--block_size=16 \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```

109
upstream_ref/xllm/docs/en/getting_started/multi_machine.md Normal file
View File

@@ -0,0 +1,109 @@
# Multi-Node Deployment
This example demonstrates how to launch a 32-GPU (NPU) deployment across 2 machines.
Launching Services on the First Machine:
```shell
bash start_deepseek_machine_1.sh
```
The start_deepseek_machine_1.sh script is as follows:
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export HCCL_IF_BASE_PORT=43432 # HCCL communication base port
# 4. Start distributed service
MODEL_PATH="/path/to/your/DeepSeek-R1" # Model path
MASTER_NODE_ADDR="123.123.123.123:9748" # Master node address (must be globally consistent)
LOCAL_HOST=123.123.123.123 # Local IP for service launch
START_PORT=18000 # Service starting port
START_DEVICE=0 # Starting NPU logical device number
LOCAL_NODES=16 # Number of local processes (this script launches 16 processes)
LOG_DIR="log" # Log directory
NNODES=32 # Total number of GPUs/NPUs (32 in this 2-machine example)
mkdir -p $LOG_DIR
for (( i=0; i<$LOCAL_NODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--max_tokens_per_batch=40000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--communication_backend="hccl" \
--enable_schedule_overlap=true \
--rank_tablefile=./ranktable_2s_32p.json \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
Launching Services on the Second Machine:
```shell
bash start_deepseek_machine_2.sh
```
The start_deepseek_machine_2.sh script is as follows:
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export HCCL_IF_BASE_PORT=43432 # HCCL communication base port
MODEL_PATH="/path/to/your/DeepSeek-R1" # Model path
MASTER_NODE_ADDR="123.123.123.123:9748" # Master node address (must be globally consistent)
LOCAL_HOST=456.456.456.456 # Local IP for service launch
START_PORT=18000 # Service starting port
START_DEVICE=0 # Starting NPU logical device number
LOCAL_NODES=16 # Number of local processes (this script launches 16 processes)
LOG_DIR="log" # Log directory
NNODES=32 # Total number of GPUs/NPUs (32 in this 2-machine example)
mkdir -p $LOG_DIR
for (( i=0; i<$LOCAL_NODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--max_tokens_per_batch=40000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--communication_backend="hccl" \
--enable_schedule_overlap=true \
--rank_tablefile=./ranktable_2s_32p.json \
--node_rank=$((i + LOCAL_NODES)) \ > $LOG_FILE 2>&1 &
done
```
This example uses 2 machines. You can set the total number of GPUs/NPUs via `--nnodes`, where `--node_rank` specifies the global rank ID for each node.
The `--rank_tablefile=./ranktable_2s_32p.json`parameter points to the configuration file required for establishing the NPU communication domain. For instructions on generating this file, refer to [Ranktable Generation](https://gitee.com/mindspore/models/blob/master/utils/hccl_tools/README.md).

15
upstream_ref/xllm/docs/en/getting_started/offline_service.md Normal file
View File

@@ -0,0 +1,15 @@
# Offline Inference
To facilitate users in quickly using xLLM for offline inference, we provide Python script examples for launching offline inference.
## LLM
LLM inference example: [:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate.py)
## Embedding
Generate embedding example: [:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate_embedding.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate_embedding.py)
## VLM
VLM inference example: [:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py)

225
upstream_ref/xllm/docs/en/getting_started/online_service.md Normal file
View File

@@ -0,0 +1,225 @@
# Online Service
First, start the xllm service according to the [xllm launch documentation](launch_xllm.md). Below are examples of client calls for LLM and VLM. Please modify the parameters according to your actual situation.
## LLM Client Calls
### HTTP Call
Chat mode:
```bash
curl http://localhost:9977/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"max_tokens": 10,
"temperature": 0,
"stream": true,
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "hello xllm"
}
]
}'
```
Completions mode:
```bash
curl http://127.0.0.1:9977/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"prompt": "hello xllm",
"max_tokens": 10,
"temperature": 0,
"stream": true
}'
```
Sample mode:
```bash
curl http://127.0.0.1:9977/v1/sample \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"prompt": "Question: <emb_0> matched or not. Conclusion: <emb_0>",
"selector": {
"type": "literal",
"value": "<emb_0>"
},
"logprobs": 5,
"request_id": "sample-demo-001"
}'
```
Typical response:
```json
{
"id": "sample-demo-001",
"object": "sample_completion",
"created": 1773369600,
"model": "Qwen2-7B-Instruct",
"choices": [
{
"index": 0,
"text": "True",
"logprobs": {
"tokens": ["True", "False"],
"token_ids": [3456, 7890],
"token_logprobs": [-0.12, -2.31]
},
"finish_reason": "selector_match"
},
{
"index": 1,
"text": "",
"logprobs": {
"tokens": [],
"token_ids": [],
"token_logprobs": []
},
"finish_reason": "empty_logprobs"
}
],
"usage": {
"prompt_tokens": 20,
"completion_tokens": 2,
"total_tokens": 22
}
}
```
`/v1/sample` notes:
- Only `--backend=llm` is supported. VLM/DiT/Rec are not supported yet.
- `selector.type` is currently fixed to `literal`. `selector.value` is matched against prompt text in full and in order.
- `logprobs` defaults to `5`, with an allowed range of `[1, 5]`.
- `choices[i].index` is the matched `sample_id`, corresponding one-to-one with the matched order in prompt.
- If no selector match is found, the service returns `200` with `choices=[]`. If a matched position has no available logprobs, it returns `finish_reason="empty_logprobs"`.
- Service logs only summary fields such as `request_id`, `sample_id`, `match_count`, and `model`, and do not log the full prompt.
`/v1/sample` common error semantics:
- Missing `model/prompt/selector/selector.value`, `selector.type != literal`, or out-of-range `logprobs` returns `INVALID_ARGUMENT`.
- If the model does not exist or the backend is not `llm`, it returns `UNKNOWN`.
- When concurrency reaches the upper limit, it returns `RESOURCE_EXHAUSTED`.
- When the model is in sleep state, it returns `UNAVAILABLE`.
### Python Call
```python
import requests
import json
url = f"http://localhost:9977/v1/chat/completions"
messages = [
{'role': 'user', 'content': "List three countries and their capitals."}
]
request_data = {
"model": "Qwen2-7B-Instruct",
"messages": messages,
"stream": False,
"temperature": 0.6,
"max_tokens": 2048,
}
response = requests.post(url, json=request_data)
if response.status_code != 200:
print(response.status_code, response.text)
else:
ans = json.loads(response.text)["choices"]
print(ans[0]['message'])
```
## VLM Client Calls
### HTTP API
```python
import base64
import requests
api_url = "http://localhost:12345/v1/chat/completions"
image_url = ""
def encode_image(url: str) -> str:
with requests.get(url) as response:
response.raise_for_status()
result = base64.b64encode(response.content).decode("utf-8")
return result
image_base64 = encode_image(image_url)
payload = {
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{image_base64}"},
},
],
}
],
"model": "Qwen2.5-VL-7B-Instruct",
"max_completion_tokens": 128,
}
response = requests.post(
api_url,
json=payload,
headers={"Content-Type": "application/json"}
)
print(response.json())
```
### OpenAI API
```python
from openai import OpenAI
import base64
import requests
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:12345/v1"
image_url = ""
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
def encode_image(url: str) -> str:
with requests.get(url) as response:
response.raise_for_status()
result = base64.b64encode(response.content).decode("utf-8")
return result
image_base64 = encode_image(image_url)
chat_completion = client.chat.completions.create(
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{image_base64}"},
},
],
}
],
model="Qwen2.5-VL-7B-Instruct",
max_completion_tokens=128,
)
result = chat_completion.choices[0].message.content
print("Chat completion output:", result)
```

107
upstream_ref/xllm/docs/en/getting_started/quick_start.md Normal file
View File

@@ -0,0 +1,107 @@
# Quick Start
## Environment Setup
All images are stored [here](https://quay.io/repository/jd_xllm/xllm-ai?tab=tags). The docker startup command below uses the dev image as an example.
### NPU
Below are our pre-built dev image.
```bash
# A2 x86
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a2-x86-20260429
# A2 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a2-arm-20260429
# A3 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a3-arm-20260429
```
Container startup command:
```bash
docker run -it \
--ipc=host \
-u 0 \
--name xllm-npu \
--privileged \
--network=host \
--device=/dev/davinci0 \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/local/sbin/:/usr/local/sbin/ \
-v /var/log/npu/conf/slog/slog.conf:/var/log/npu/conf/slog/slog.conf \
-v /var/log/npu/slog/:/var/log/npu/slog \
-v /var/log/npu/profiling/:/var/log/npu/profiling \
-v /var/log/npu/dump/:/var/log/npu/dump \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
### NVIDIA GPU
We provide a [Dockerfile](../../../docker/Dockerfile.cuda) for NVIDIA GPU usage, which can be used to build custom image. Of course, you can also use dev image we built based on the default Dockerfile:
```bash
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-cuda-x86
```
Container startup command:
```bash
sudo docker run -it \
--privileged \
--shm-size '128gb' \
--ipc=host \
--net=host \
--pid=host \
--name=xllm-cuda \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
### MLU
We cannot provide MLU image. If you already have the dev image, you can start the container with the following command:
```bash
sudo docker run -it \
--privileged \
--shm-size '128gb' \
--ipc=host \
--net=host \
--pid=host \
--name xllm-mlu \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
## Build xllm
If you download a release image, i.e., an image with a version number in the tag, you can skip this step because the release image comes with a pre-compiled xllm binary, located at `/usr/local/bin/xllm`.
Download xllm and dependencies:
```bash
git clone https://github.com/jd-opensource/xllm
cd xllm
# Install pre-commit for the first time
pip install pre-commit
pre-commit install
git submodule update --init --recursive
```
The compiled binary file is located at `/path/to/xllm/build/xllm/core/server/xllm`. In a new image, the first compilation of xllm takes a long time because all dependencies in vcpkg need to be compiled, but subsequent compilations will be much faster.
```bash
python setup.py build
```
## Launch xllm
Please refer to [How to Launch xllm](launch_xllm.md).

57
upstream_ref/xllm/docs/en/index.md Normal file
View File

@@ -0,0 +1,57 @@
---
hide:
- navigation
---
<style>
.md-content h1:first-of-type {
display: none;
}
</style>
<div style="text-align:center">
<img src="../assets/logo_with_llm.png" alt="xLLM" style="width:50%; height:auto;">
</div>
## Project Overview
**xLLM** is an efficient and user-friendly LLM intelligent inference framework that provides enterprise-level service guarantees and high-performance engine computing capabilities for model inference on domestic AI accelerators.
#### Background
LLM with parameter scales ranging from tens of billions to trillions are being rapidly deployed in core business scenarios such as intelligent customer service, real-time recommendation, and content generation. Efficient support for domestic computing hardware has become a core requirement for low-cost inference deployment. Existing inference engines struggle to effectively adapt to the architectural characteristics of dedicated accelerators like domestic chips. Performance issues such as low utilization of computing units, load imbalance and communication overhead bottlenecks under the MoE architecture, and difficulties in kv cache management have restricted the efficient inference of requests and the scalability of the system. The xLLM inference engine improves the resource efficiency of the entire "communication-computation-storage" performance link and it provides crucial technical support for the large-scale implementation of LLM in real-world business scenarios.
## Core Features
**xLLM** delivers robust intelligent computing capabilities. By leveraging hardware system optimization and algorithm-driven decision control, it jointly accelerates the inference process, enabling high-throughput, low-latency distributed inference services.
**Full Graph Pipeline Execution Orchestration**
- Asynchronous decoupled scheduling at the requests scheduling layer, to reduce computational bubbles.
- Asynchronous parallelism of computation and communication at the model graph layer, overlapping computation and communication.
- Pipelining of heterogeneous computing units at the operator kernel layer, overlapping computation and memory access.
**Graph Optimization for Dynamic Shapes**
- Dynamic shape adaptation based on parameterization and multi-graph caching methods to enhance the flexibility of static graph.
- Controlled tensor memory pool to ensure address security and reusability.
- Integration and adaptation of performance-critical custom operators (e.g., *PageAttention*, *AllReduce*).
**MoE Kernel Optimization**
- *GroupMatmul* optimization to improve computational efficiency.
- Chunked Prefill optimization to support long-sequence inputs.
**Efficient Memory Optimization**
- Mapping management between discrete physical memory and continuous virtual memory.
- On-demand memory allocation to reduce memory fragmentation.
- Intelligent scheduling of memory pages to increase memory reusability.
- Adaptation of corresponding operators for domestic accelerators.
**Global KV Cache Management**
- Intelligent offloading and prefetching of KV in hierarchical caches.
- KV cache-centric distributed storage architecture.
- Intelligent KV routing among computing nodes.
**Algorithm-driven Acceleration**
- Speculative decoding optimization to improve efficiency through multi-core parallelism.
- Dynamic load balancing of MoE experts to achieve efficient adjustment of expert distribution.

49
upstream_ref/xllm/docs/en/supported_models.md Normal file
View File

@@ -0,0 +1,49 @@
# Model Support List
## LLM
| | NPU | MLU | ILU |
| ------------------------ | :---: | :---: | :---: |
| DeepSeek-V3/R1/V3.1 | ✅ | ✅ | ❌ |
| DeepSeek-V3.2 | ✅ | ✅ | ❌ |
| DeepSeek-R1-Distill-Qwen | ✅ | ❌ | ❌ |
| Qwen2/2.5/QwQ | ✅ | ✅ | ✅ |
| Qwen3 | ✅ | ✅ | ✅ |
| Qwen3 Moe | ✅ | ✅ | ✅ |
| Kimi-k2 | ✅ | ❌ | ❌ |
| Llama2/3 | ✅ | ❌ | ✅ |
| GLM4.5 | ✅ | ❌ | ❌ |
| GLM4.6 | ✅ | ❌ | ❌ |
| GLM-4.7 | ✅ | ❌ | ❌ |
| GLM-5 | ✅ | ❌ | ❌ |
## VLM
| | NPU | MLU | ILU |
| ------------ | :---: | :---: | :---: |
| MiniCPM-V | ✅ | ❌ | ❌ |
| MiMo-VL | ✅ | ❌ | ❌ |
| Qwen2.5-VL | ✅ | ✅ | ❌ |
| Qwen3-VL | ✅ | ✅ | ❌ |
| Qwen3-VL-MoE | ✅ | ✅ | ❌ |
| GLM-4.6V | ✅ | ❌ | ❌ |
| VLM-R1 | ✅ | ❌ | ❌ |
## Rerank
| | NPU | MLU | ILU |
| -------------- | :---: | :---: | :---: |
| Qwen3-Reranker | ✅ | ❌ | ❌ |
## DiT
| | NPU | MLU | ILU |
| ---- | :---: | :---: | :---: |
| Flux | ✅ | ❌ | ❌ |
## Rec
| | NPU | MLU | ILU |
| --- | :---: | :---: | :---: |
| OneRec | ✅ | ❌ | ❌ |
| Qwen2 | ✅ | ❌ | ❌ |
| Qwen2.5 | ✅ | ❌ | ❌ |
| Qwen3 | ✅ | ❌ | ❌ |