ref(upstream): FULL TREE — Deep-Spark xllm (1470) + ds_vllm csrc/models (703)

Replaces cherry-picked upstream_ref with complete source trees.

xllm/ — Iluvatar official C++ inference engine (15MB, 1470 files)
  Complete: kernels → layers → models → runtime → scheduler → api
  Excluded: .git, binary images, third_party submodule checkouts

ds_vllm/ — Iluvatar official vllm fork (8MB, 703 files)
  Included: csrc/ (ALL CUDA kernels), fused_moe/, qwen3_5 model, _custom_ops
  Excluded: tests, benchmarks, docs, examples (not needed for reference)

Critical call chains now fully traceable:
  MoE: moe_topk_softmax_kernels.cuh → ixformer.h → fused_moe.cpp → layer
  GDN: qwen3_gated_delta_net_base.cpp → qwen3_5_gated_delta_net.cpp
  Attention: ixformer.h → xllm_paged_attention → attention.cpp

upstream_ref/xllm/docs/en/features/async_schedule.md

@@ -0,0 +1,29 @@
+# Async schedule
+
+## Background
+The inference process of large language models can be divided into three sequential stages:1) CPU-side scheduling (preparing model inputs), 2) Device computation (GPU/TPU execution), 3) CPU-side post-processing (output handling).
+Due to the sequential nature of decoding operations, the input for step-i+1 depends on the output of step-i. This forces strict serial execution of all three stages, creating device idle periods (“bubbles”) during CPU-bound stages 1 and 3, leading to suboptimal resource utilization.
+
+
+## Introduction
+xLLM addresses this at the framework level by supporting asynchronous scheduling, where the CPU proactively executes scheduling operations for step-i+1 while the device is computing step-i. This allows the device to immediately begin computing step-i+1 upon completing step-i, thereby eliminating bubbles. Specifically, after initiating the computation call for step-i, the CPU does not wait for the device to finish computing. Instead, it constructs fake tokens for the step-i request, uses these fake tokens to perform scheduling operations for step-i+1 (such as allocating KV Cache), and replaces them with the true tokens computed in step-i when launching step-i+1 computation to ensure correctness. Meanwhile, the CPU processes the results of step-i in a separate thread and returns them to the client. 
+In the overall architecture, stages 1 and 3 on the CPU side are handled by different thread pools, and RPC function calls employ non-blocking C++ future and promise mechanisms to achieve a fully asynchronous runtime.
+![Async schedule](../../assets/async_schedule_architecture.jpg)
+
+
+## Usage
+
+xLLM provides the gflags parameter `enable_schedule_overlap`, which defaults to false. To enable this feature, set it to true in xLLM's service startup script:
+```shell
+--enable_schedule_overlap=true
+```
+
+## Performance
+
+- With asynchronous scheduling enabled, the device idle time between two steps is approximately 200us - comparable to a single kernel launch duration.
+- On the DeepSeek-R1-Distill-Qwen-1.5B model with TPOT constrained to 50ms, this achieves 17% throughput improvement.
+
+
+## Notice
+The asynchronous scheduling feature requires the server to compute one additional step. For use cases involving limited output tokens (e.g., few-token generation) or single-output scenarios like embedding models, enabling this feature is not recommended as it may reduce server-side throughput, thus hard-disabled internally.
+The VLM model is currently being adapted, will be temporarily disabled.

upstream_ref/xllm/docs/en/features/basics.md

@@ -0,0 +1,5 @@
+# Basics
+
+- xLLM uses a one-device-per-process architecture. Across multiple devices, RPC is used for function calls, and data communication during model computation uses device collective communication libraries.
+
+- HCCL/LCCL are high-performance collective communication frameworks that provide data-parallel and model-parallel collective communication for both single-node multi-device and multi-node multi-device scenarios.

upstream_ref/xllm/docs/en/features/chunked_scheduler.md

@@ -0,0 +1,19 @@
+# ChunkedPrefill Scheduler
+
+## Feature Introduction
+xLLM supports the chunked prefill scheduling strategy. Chunked prefill is a technique that optimizes large language model inference by splitting long prompts into smaller chunks for batch processing, rather than processing the entire prompt at once.
+This method can effectively reduce peak GPU memory usage, improve device utilization, and better schedule and mix processing with requests from the decode stage.
+
+## Usage
+The aforementioned strategy has been implemented in xLLM and is exposed through gflags parameters to control the feature's on/off state.
+
+- Enable chunked prefill and set the chunked size, if not set chunked size, its default value is equal to max_tokens_per_batch.
+```bash
+--enable_chunked_prefill=true
+--max_tokens_per_chunk_for_prefill=20480 # optional
+```
+
+
+
+## Performance Impact
+After enabling chunked prefill, on the Qwen3-8B model with a TPOT constraint of 50ms, the TTFT latency **decreased by 46%**.

upstream_ref/xllm/docs/en/features/continuous_scheduler.md

@@ -0,0 +1,7 @@
+# Continuous Scheduler
+
+## Feature Introduction
+xLLM implements a scheduling strategy that supports continuous batching. Continuous batching is a dynamic batching strategy that does not wait for a batch to be filled. Instead, it starts processing as soon as requests are available, while continuously accepting new requests and adding them to the currently executing batch. This approach significantly reduces latency while maintaining high throughput.
+
+## Usage
+xLLM implements the continuous batching scheduling strategy. And, the default scheduler is chunked prefill, `enable_chunked_prefill=true` by default.

upstream_ref/xllm/docs/en/features/disagg_pd.md

@@ -0,0 +1,83 @@
+# Disaggregated PD
+## Background
+LLM online inference services typically need to meet two performance metrics: TTFT and TPOT. Traditional Contiguous Batching scheduling strategies mix Prefill and Decode requests during scheduling, causing Prefill and Decode phases to compete for computational resources. This prevents maximized utilization of computing resources and impacts performance metrics. To resolve this conflict, the Prefill and Decode phases are split to run on independent computational resources, enabling parallel execution. This simultaneously reduces TTFT and TPOT while improving throughput.
+
+## Introduction
+The xLLM PD Separation feature is primarily implemented through the following three modules:  
+
+- **etcd**: Stores metadata such as instance information.  
+- **xLLM Service**: Schedules requests and manages all computing instances.  
+- **xLLM**: Handles request computation instances.  
+
+The overall architecture is shown below: 
+![xLLM PD Separation Architecture](../../assets/pd_architecture.jpg)
+
+## Usage
+### Preparation
+#### Install Dependencies
+- **xLLM**: Refer to [Installation && Compilation](../getting_started/quick_start.md)
+- **xLLM Service**: Refer to [PD disaggregation](../getting_started/disagg_pd.md)
+
+#### Obtain Environment Information  
+Deploying Disaggregated PD Service requires obtaining the Device IP of the machine to create communication resources. Execute the command `cat /etc/hccn.conf | grep address` on the current AI Server to get the Device IP, for example:
+```
+address_0=xx.xx.xx.xx
+address_1=xx.xx.xx.xx
+```
+`address_xx` represents the Device IP.
+
+### Start Disaggregated PD Service
+1. Start etcd
+```bash
+./etcd
+```
+2. Start xLLM Service
+```bash
+ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
+```
+3. Start xLLM  
+- Taking Qwen2-7B as an example  
+    - Start Prefill Instance
+        ```bash
+        /path/to/xllm --model=Qwen2-7B-Instruct \
+               --port=8010 \
+               --devices="npu:0" \
+               --master_node_addr="127.0.0.1:18888" \
+               --enable_prefix_cache=false \
+               --enable_chunked_prefill=false \
+               --enable_disagg_pd=true \
+               --instance_role=PREFILL \
+               --etcd_addr="127.0.0.1:12389" \
+               --transfer_listen_port=26000 \
+               --disagg_pd_port=7777 \
+               --node_rank=0 \
+               --nnodes=1
+        ```
+    - Start Decode Instance 
+        ```bash 
+        /path/to/xllm --model=Qwen2-7B-Instruct \
+               --port=8020 \
+               --devices="npu:1" \
+               --master_node_addr="127.0.0.1:18898" \
+               --enable_prefix_cache=false \
+               --enable_chunked_prefill=false \
+               --enable_disagg_pd=true \
+               --instance_role=DECODE \
+               --etcd_addr="127.0.0.1:12389" \  
+               --transfer_listen_port=26100 \
+               --disagg_pd_port=7787 \
+               --node_rank=0 \
+               --nnodes=1
+        ```
+    Important notes:
+    
+    - PD disaggregation requires reading the `/etc/hccn.conf` file. Make sure this file on the physical machine is mapped into the container.
+
+    - `etcd_addr` must match the `etcd_addr` of `xllm_service`
+
+## Notice
+Disaggregated PD **does not support** enabling prefix cache or chunked prefill. These features must be disabled using the following parameters:
+```shell
+--enable_prefix_cache=false  
+--enable_chunked_prefill=false  
+```

upstream_ref/xllm/docs/en/features/eplb.md

@@ -0,0 +1,30 @@
+# MoE Load Balancing (EPLB)
+
+## Background
+
+MoE models rely on dynamic token routing to distribute tokens among experts. However, in real-world deployments, uneven data distribution leads to expert load imbalance (some overloaded while others idle). Expert redundancy adjustment (e.g., adding/removing replicas) consumes additional GPU memory and may impact inference latency due to weight migration, posing significant implementation challenges. To address this, we employ an expert redundancy strategy (replicating hot experts) combined with hierarchical and global dynamic load balancing to achieve dynamic MoE load balancing.
+
+## Features
+
+The xLLM MoE Load Balancing (EPLB) functionality is implemented through three main modules:
+
+- **EPLB Manager**: Responsible for monitoring expert loads, collecting and managing expert distribution updates. It uses a layer-by-layer update mechanism, determining whether to update each layer based on expert load changes.
+- **EPLB Executor**: The actual executor for expert distribution updates.
+- **EPLB Policy**: Strategy for generating new expert load tables.
+
+Overall architecture diagram:
+![xLLM eplb](../../assets/eplb_architecture.png)
+
+## Usage
+
+Simply add the following gflag parameters when launching xLLM:
+
+(Replace with actual number of devices. `ep_size` must match the number of devices)
+
+- xLLM provides the gflag parameter `enable_eplb` (default: false). Set to true in the xLLM service startup script to enable dynamic expert load balancing.
+- `expert_parallel_degree` and `ep_size` are MoE-related parameters. `expert_parallel_degree` should be set to `2`, and `ep_size` must match the actual number of NPU/GPU devices. See [moe_params](./moe_params.md)
+- `eplb_update_interval` sets the expert distribution update interval in seconds (default: 1000).
+- The expert distribution update uses a layer-by-layer mechanism based on expert load. When the similarity between consecutive loads for a layer is below `eplb_update_threshold`, that layer is updated (default: 1, range: 0-1).
+
+```bash
+--enable_eplb=true --expert_parallel_degree=2 --ep_size=16 --eplb_update_interval=2000 --eplb_update_threshold=0.9

upstream_ref/xllm/docs/en/features/global_kvcache.md

@@ -0,0 +1,42 @@
+# Global Multi-Level KV Cache
+
+## Background
+In the decoding phase of large language models (LLMs), frequent access to historical KV cache due to autoregressive generation creates a bottleneck in memory bandwidth. As model sizes and context windows expand (e.g., 128K Tokens consuming over 40GB of memory), the pressure on single-device memory increases dramatically. Existing solutions (such as vLLM) exhibit significant limitations in long-context scenarios: prefill time surges, severe memory bandwidth contention during decoding, and the need for excessive resource reservation to meet SLO requirements (TTFT < 2s, TBT < 100ms). This often results in GPU utilization below 40% and difficulties in leveraging cross-server resources. To address this, we propose a distributed global multi-level KV cache management system, adopting a memory-compute integrated architecture to break through single-machine resource constraints.
+
+## Feature Introduction
+The xLLM Global KV Cache feature is primarily implemented through the following three modules:
+-   **etcd**: For cluster service registration, load information synchronization, and global cache state management.
+-   **xLLM Service**: For scheduling requests and managing all compute instances.
+-   **xLLM**: The compute instances handling requests.
+
+The overall architecture is shown in the diagram below:
+![xLLM Global Multi-Level KV Cache](../../assets/globalkvcache_architecture.png)
+
+## Usage Example
+
+### Preparation
+
+#### Install Dependencies
+- **xLLM**: Refer to [Quick Start](../getting_started/quick_start.md)
+- **xLLM Service**: Refer to [PD disaggregation](../getting_started/disagg_pd.md)
+
+### Usage Instructions
+
+1.  **etcd Startup Configuration:**
+    ```bash
+    ./etcd --listen-peer-urls=http://0.0.0.0:10999 --listen-client-urls=http://0.0.0.0:10998
+    ```
+
+2.  **xLLM Service Startup Configuration:**
+    ```bash
+    ./xllm_master_serving --etcd_addr="127.0.0.1:10998" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
+    ```
+
+3.  **xLLM Startup Configuration:**
+    Add the following gflag parameters when starting xLLM:
+    ```bash
+    --enable_service_routing=true
+    --enable_cache_upload=true
+    # PD separation currently does not support Global KVCache Management 
+    --enable_disagg_pd=false
+    ```

upstream_ref/xllm/docs/en/features/graph_mode.md

@@ -0,0 +1,79 @@
+# Graph Mode
+
+## Overview
+
+xLLM supports Graph Mode: computation graphs are pre-captured and replayed in subsequent runs to reduce CPU overhead and improve inference performance. Graph Mode has corresponding implementations on different hardware platforms.
+
+## Feature Description
+
+To optimize Host-side scheduling, graph mode submits a large task from the CPU once and then executes small kernels in a streaming manner on the device, significantly reducing startup time and device bubbles.
+
+In the xLLM engine, Graph Mode provides the following:
+
+### Dynamic Shape Parameterization
+  - Key dynamic dimensions other than num_tokens are treated as whole-graph input parameters, including batch_size, kv_seq_lens, q_seq_lens, block_table_size, and the like, for flexibility. During memory allocation and kernel configuration. At graph launch, the actual values of these parameters are passed so kernels use the correct strides to access data.
+
+### Piecewise Graph
+  - When some operators do not support graph capture and thus break the full graph, each segment (piece) after the break is captured as a separate graph. This maximizes graph-mode benefits even when the full graph cannot be captured, and is commonly used for prefill and chunked prefill.
+
+### Multi-Shape Reusable Memory Pool
+  - To avoid waste from separate memory buffers (input, output, and intermediate tensors) per shape, we use an expandable memory pool. Multiple shapes share the pool base address, with different shapes using different offsets from that base.
+
+## Usage
+
+These capabilities are implemented inside the xLLM engine and are generally controlled through gflags.
+
+The minimal configuration only needs `enable_graph` to turn on Graph Mode for the decode phase:
+
+```shell
+--enable_graph=true
+```
+
+Common companion flags include:
+
+- `enable_graph`: enables the base Graph Mode capability for the decode phase
+- `enable_prefill_piecewise_graph`: enables Piecewise Graph for the prefill phase
+- `enable_graph_mode_decode_no_padding`: builds decode graphs with the actual `num_tokens` instead of the padded shape
+- `max_tokens_for_graph_mode`: limits the maximum number of tokens covered by Graph Mode; `0` means no limit
+
+If you want to enable both decode Graph and prefill Piecewise Graph, use:
+
+```shell
+--enable_graph=true \
+--enable_prefill_piecewise_graph=true \
+--max_tokens_for_graph_mode=2048
+```
+
+If you need decode graph capture without padding, add:
+
+```shell
+--enable_graph=true \
+--enable_graph_mode_decode_no_padding=true
+```
+
+For a more complete description of the flags, see [CLI Reference](../cli_reference.md).
+
+## Performance Impact
+
+- With Graph Mode enabled, decode-phase throughput **improves by about 8%–10%** on models such as Qwen3-0.6B and Qwen3-1.7B.
+
+## Model Support
+
+The following table lists each model’s support on ACLGraph, CudaGraph, and MLUGraph.
+
+| Model | ACLGraph | CudaGraph | MLUGraph |
+|------|----------|-----------|----------|
+| Qwen3/Qwen3-MoE | ✅ | ✅ | ✅ |
+| DeepseekV3.2 | ✅ | | ✅ |
+| GLM4.5/4.6/4.7 | ✅ | | |
+| Qwen2.5-VL | | | ✅ |
+| Qwen3-VL/Qwen3-VL-MoE | ✅ | | |
+| GLM4V | ✅ | | |
+| GLM4V-MoE | ✅ | | |
+
+!!! warning "Adding Graph Mode support for new models"
+    Ensure that the kernels used in the computation implement dynamic dimension parameterization; otherwise the graph may break and kernels may need to be re-implemented.
+
+## Related Documentation
+
+- For more detailed Graph Mode design and implementation notes, including ACL Graph / CUDA Graph fundamentals, dynamic dimension parameterization, Piecewise Graph, and multi-shape memory reuse, see: [Graph Mode Design Document](../design/graph_mode_design.md)

upstream_ref/xllm/docs/en/features/groupgemm.md

@@ -0,0 +1,40 @@
+# GroupGEMM Operator Optimization
+
+## Background
+The Mixture of Experts (MoE) architecture has become an important paradigm for scaling large language models. Its core idea is to dynamically route input tokens to different expert sub-networks for processing. During inference, the GroupGEMM operator is a key computational unit in the MoE architecture, responsible for efficiently executing parallel computations of multiple expert matrix multiplications, which dominate the overall inference time.
+
+## Function Introduction
+Considering the current performance bottleneck of GroupGEMM being I/O constrained, an optimization scheme is proposed that replaces data copying with index reordering, eliminating multiple copies of the token vector and instead maintaining an index table for expert allocation. Through this line number index, tokens are directly mapped to the corresponding expert computation units, and the allocation scheduling of tokens is fused with matrix multiplication into a single kernel.
+
+## User Interface
+
+### Operator Direct Call API
+```c++
+aclnnStatus aclnnIndexGroupMatmulGetWorkspaceSize(
+    const aclTensorList *x,
+    const aclTensorList *weight,
+    const aclTensorList *scale,
+    const aclTensorList *perTokenScale,
+    const aclTensor *groupList,
+    const aclTensorList *out,
+    uint64_t *workspaceSize,
+    aclOpExecutor **executor);
+
+aclnnStatus aclnnIndexGroupMatmul(
+    void *workspace,
+    uint64_t workspaceSize,
+    aclOpExecutor *executor,
+    aclrtStream stream);
+```
+
+- `x`: The input tensor list containing the data to be processed.
+- `weight`: The weight tensor containing the model parameters.
+- `scale`: The scaling factor used to adjust the values of the input tensors.
+- `perTokenScale`: The scaling factor for each token, used for dynamic adjustment.
+- `groupList`: The list of expert groups indicating which experts participate in the computation.
+- `out`: The output tensor list that stores the computation results.
+
+## Performance Effects
+![groupmatmul](../../assets/groupmatmul_performance.png)
+
+* The optimized GroupMatmul operator shows significant advantages in computation time, especially when \( k = 128 \) and \( m = 64 \). As shown in the figure, the computation delay of the optimized operator is **reduced by 50%**.

upstream_ref/xllm/docs/en/features/moe_params.md

@@ -0,0 +1,24 @@
+# EP Parallelism
+
+## Background
+
+When deploying the DeepSeek-R1 671B parameter-scale model, traditional distributed deployment faces core bottlenecks including low GPU memory utilization, high communication overhead, and expensive hardware costs. Therefore, Expert Parallelism (EP) is introduced.
+
+Key advantages:
++ With the same resources, fewer Experts per GPU means more memory available for KV Cache, allowing more tokens to be cached.
++ Due to MLA characteristics, smaller TP Size with the same resources means less redundant KV Cache, enabling more tokens to be cached.
++ Large-scale EP deployment can concentrate token computations for the same expert on the same device, improving hardware utilization.
+
+## Parameter Configuration
+
++ **dp_size**: Sets the data parallelism scale for the Attention part. Default: 1, can be set to powers of 2. When dp_size doesn't equal the total number of devices, TP parallelism is used within the DP group.
++ **ep_size**: Sets the expert parallelism scale for the MoE part. Default: 1, can be set to powers of 2. When ep_size doesn't equal the total number of devices, TP parallelism is used within the DP group.
++ **expert_parallel_degree**: EP-related parameter. Default is 0 when EP is disabled. When EP is enabled, default is 1 (EP Level 1). When ep_size equals the total number of devices, can be set to 2 to enable EP Level 2.
+
+## Solution Design
+
++ When EP is enabled (default EP Level 1), after computing both Attention and MoE parts, data is sent to the next stage via All Gather communication across all devices. Example for 64 devices (Attention: dp32tp2, MoE: ep32tp2):
+![Alt text](../../assets/moe_eplevel1.jpg)
+
++ When ep_size equals the total number of devices, EP Level 2 can be enabled. Communication between Attention and MoE parts changes to ALL2ALL, sending data only to required devices to reduce communication volume and overhead. Example for 64 devices:
+![Alt text](../../assets/moe_eplevel2.jpg)

upstream_ref/xllm/docs/en/features/mtp.md

@@ -0,0 +1,149 @@
+# MTP Speculative Inference
+
+## Background
+MTP (Multi-Token Prediction) is an innovative inference acceleration technique that addresses efficiency bottlenecks in large language model generation. By incorporating specialized pre-training designs, MTP provides efficient draft token prediction capabilities during inference, significantly improving generation speed. Its core value lies in balancing inference efficiency with output quality, offering an optimal solution for long-sequence generation problems in LLMs, ultimately optimizing inference performance.
+
+## Key Features
+MTP offers the following core acceleration capabilities:
+
+- **Efficient Draft Generation**: Uses a lightweight MTP architecture to rapidly generate draft tokens that serve as input for the main model's verification, dramatically reducing computation overhead compared to traditional autoregressive generation.
+
+- **Batch Verification Mechanism**: The main model can simultaneously verify multiple MTP-generated draft tokens in batch, rather than processing them sequentially, significantly boosting inference speed.
+
+- **High Sampling Accuracy**: MTP solves the critical pain point of low token acceptance rates in post-training draft modules (like Eagle and Medusa). By optimizing draft generation during pre-training, MTP produces tokens with higher accuracy, reducing the verification burden on the main model.
+
+- **Reduced Inference Latency**: By pre-generating multiple potential subsequent tokens, MTP effectively decreases cumulative latency during long-text generation, creating a smoother user experience.
+
+- **Optimized Resource Consumption**: Compared to other inference acceleration techniques, MTP maintains acceleration effects while requiring fewer additional computational resources, making it suitable for deployment in resource-constrained environments.
+
+MTP technology provides a novel efficiency optimization solution for LLM inference, particularly well-suited for real-time applications requiring rapid responses, representing an important direction in language model inference optimization.
+
+!!! note "Model Support"
+    Currently supports MTP structure export for the following models:
+    - DeepSeek-V3 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v3_mtp)
+    - DeepSeek-V3.2 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v32_mtp)
+    - DeepSeek-R1 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v3_mtp)
+    - GLM4 MoE (e.g., GLM-4.5-Air, exported MTP model_type: glm4_moe_mtp)
+    
+    Note:
+    - DeepSeek V3 and R1 both have input model_type "deepseek_v3", and the exported MTP model will have model_type "deepseek_v3_mtp"
+    - DeepSeek V3.2 has input model_type "deepseek_v3" (but can be auto-detected by index_head_dim fields), and the exported MTP model will have model_type "deepseek_v32_mtp"
+
+## Usage Example
+
+### Export Model
+
+The script will automatically detect the model type, or you can manually specify it.
+
+#### DeepSeek-V3
+```bash
+python3 tools/export_mtp.py \
+    --input-dir /path/to/DeepSeek-V3 \
+    --output-dir /path/to/DeepSeek-V3-mtp
+```
+
+#### DeepSeek-V3.2
+```bash
+python3 tools/export_mtp.py \
+    --input-dir /path/to/DeepSeek-V3.2 \
+    --output-dir /path/to/DeepSeek-V3.2-mtp
+```
+
+#### DeepSeek-R1
+```bash
+python3 tools/export_mtp.py \
+    --input-dir /path/to/DeepSeek-R1 \
+    --output-dir /path/to/DeepSeek-R1-mtp
+```
+
+#### GLM4 MoE
+```bash
+python3 tools/export_mtp.py \
+    --input-dir /path/to/GLM-4.5-Air \
+    --output-dir /path/to/GLM-4.5-Air-mtp
+```
+
+#### Manually Specify Model Type
+If auto-detection fails, you can manually specify the model type:
+```bash
+python3 tools/export_mtp.py \
+    --input-dir /path/to/model \
+    --output-dir /path/to/model-mtp \
+    --model-type deepseek_v3  # Options: deepseek_v3 (for V3/R1), deepseek_v32 (for V3.2), glm4_moe
+```
+
+Input model references:
+- [DeepSeek-V3](https://huggingface.co/deepseek-ai/DeepSeek-V3)
+- [DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)
+- [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1)
+- [GLM-4.5-Air](https://huggingface.co/zai-org/GLM-4.5-Air)
+
+### Launch Script
+
+When using MTP for inference, you need to specify both the main model and the draft model (MTP model).
+
+#### DeepSeek-V3/V3.2/R1 Launch Example
+```bash
+MODEL_PATH="/models/DeepSeek-V3"
+DRAFT_MODEL_PATH="/models/DeepSeek-V3-mtp"
+MASTER_NODE_ADDR="127.0.0.1:42123"
+START_PORT=13222
+START_DEVICE=0
+LOG_DIR="log"
+NNODES=16
+
+for (( i=0; i<$NNODES; i++ ))
+do
+  PORT=$((START_PORT + i))
+  DEVICE=$((START_DEVICE + i))
+  LOG_FILE="$LOG_DIR/node_$i.log"
+  nohup ./xllm \
+    --model $MODEL_PATH \
+    --devices="npu:$DEVICE" \
+    --port $PORT \
+    --master_node_addr=$MASTER_NODE_ADDR \
+    --nnodes=$NNODES \
+    --draft_model $DRAFT_MODEL_PATH \
+    --draft_devices="npu:$DEVICE" \
+    --num_speculative_tokens 1 \
+    --max_memory_utilization=0.90 \
+    --max_tokens_per_batch=10000 \
+    --max_seqs_per_batch=256 \
+    --block_size=128 \
+    --ep_size=1 \
+    --dp_size=1 \
+    --enable_prefix_cache=false \
+    --enable_chunked_prefill=false \
+    --node_rank=$i > $LOG_FILE 2>&1 &
+  sleep 0.5
+done
+```
+
+#### GLM4 MoE Launch Example
+```bash
+MODEL_PATH="/models/GLM-4.5-Air"
+DRAFT_MODEL_PATH="/models/GLM-4.5-Air-mtp"
+# ... same other configurations
+```
+
+# Performance Data
+Based on ShareGPT dataset with input length=2500, output length=1500, total requests=80.
+
+| method    | Concurrency | Mean TPOT(ms) | Mean TTFT(ms) | Output Tokens/s | Total Tokens/s |
+|:---------:|:-----------:|:-------------:|:-------------:|:---------------:|:--------------:|
+| baseline  |      1      |     40.61     |    141.80     |      24.20      |     65.77      |
+| mtp       |      1      |     28.33     |    142.35     |      35.19      |     95.52      |
+| baseline  |      2      |     42.69     |    178.59     |      45.16      |    122.74      |
+| mtp       |      2      |     29.81     |    187.97     |      64.75      |    175.78      |
+| baseline  |      4      |     46.18     |    172.34     |      79.83      |    216.96      |
+| mtp       |      4      |     33.54     |    194.22     |     111.18      |    301.81      |
+| baseline  |      8      |     53.16     |    181.49     |     110.68      |    300.81      |
+| mtp       |      8      |     40.99     |    203.37     |     154.46      |    419.34      |
+| baseline  |     16      |     68.50     |    213.89     |     143.81      |    390.84      |
+| mtp       |     16      |     57.04     |    254.99     |     201.89      |    548.04      |
+| baseline  |     20      |     74.72     |    228.80     |     154.77      |    420.65      |
+| mtp       |     20      |     61.73     |    264.34     |     206.24      |    559.84      |
+| baseline  |     40      |    119.68     |    559.32     |     180.22      |    489.80      |
+| mtp       |     40      |    105.70     |    544.54     |     252.91      |    686.74      |
+| baseline  |     80      |    180.89     |   2996.21     |     192.09      |    522.06      |
+| mtp       |     80      |    152.19     |   2163.72     |     278.07      |    755.12      |

upstream_ref/xllm/docs/en/features/multi_streams.md

@@ -0,0 +1,31 @@
+# Multi-stream parallel
+
+## Background
+In distributed inference scenarios for large-scale models, additional communication operations are required to aggregate computation results from different devices. Taking large-scale MoE models like Deepseek as an example, the distributed scale is typically substantial, leading to increased communication overhead.
+
+If both computation and communication are performed on the same stream, the device’s computing resources will remain idle while waiting for communication to complete, resulting in wasted computational capacity before subsequent calculations can begin.
+
+
+## Introduction
+xLLM implements multi-stream parallelism at the model layer, where the input batch is split into 2 micro-batches. One stream handles computation for the first micro-batch, another concurrently executes communication for the second micro-batch.
+This overlap of computation and communication effectively hides the communication latency.![multi_streams_parallel](../../assets/multi_streams_architecture.jpg)
+
+
+## Usage
+
+xLLM provides the gflags parameter `enable_multi_stream_parallel`, which defaults to false. To enable this feature, set it to true in xLLM’s service startup script, as:
+```shell
+--enable_multi_stream_parallel=true
+```
+
+
+## Performance
+With prefill dual-stream parallelism enabled, it can effectively mask over 75% of communication overhead.
+On the DeepSeek-R1 model, when generating just 1 token, this achieves:
+-  **7%** reduction in TTFT.
+-  **7%** throughput improvement.
+
+
+## Notice
+The dual-stream parallelism currently only supports the prefill phase, with greater performance benefits observed for longer input requests.
+Only Support DeepSeek, Qwen3 dense(non-MoE) models.

upstream_ref/xllm/docs/en/features/overview.md

@@ -0,0 +1,51 @@
+# Overall Architecture
+
+## Backgroud
+
+In recent years, with the groundbreaking progress of large language models (LLMs) ranging from tens of billions to trillions of parameters (such as GPT, Claude, DeepSeek, LLaMA, etc.) in the fields of natural language processing and multimodal interaction, the industry has an urgent need for efficient inference engines and service systems. How to reduce cluster inference costs and improve inference efficiency has become a key challenge for achieving large-scale commercial deployment.
+
+Although a number of optimization engines for large model inference have emerged, several technical bottlenecks remain in practical deployment:
+
+*   **Hardware Adaptability Challenges:** Existing inference engines lack sufficient support for the architectural characteristics of specialized accelerators like domestic chips, making it difficult to fully exploit the performance potential of heterogeneous computing hardware, leading to low computational resource utilization.
+*   **MoE Architecture Optimization Difficulties:** The token distribution process in expert parallelism mechanisms generates significant All-to-All communication overhead, while dynamic routing strategies cause expert load imbalance, severely constraining system scalability.
+*   **Long Context Management Bottlenecks:** As model context windows continue to expand, the efficiency of optimizations in KV cache handling—such as memory fragmentation management and cross-node synchronization—directly impacts overall inference throughput performance.
+*   **Hybrid Deployment Efficiency Limitations:** Existing inference clusters struggle to simultaneously guarantee service quality (SLO) and optimize resource utilization when handling both online services and offline tasks.
+*   **Insufficient Dynamic PD Adaptation:** When input/output sequence lengths fluctuate drastically, static PD resource partitioning lacks the ability to adjust PD resource configurations in real-time. This can lead to idle GPU resources and poses a risk of SLO violations.
+
+To address these challenges, we present xLLM—an efficient and user-friendly open-source intelligent inference framework that provides enterprise-grade service guarantees and high-performance engine computing capabilities for model inference on domestic chips.
+
+## Feature Introduction
+
+xLLM provides intelligent computing capabilities, implementing joint inference acceleration across multiple computational system layers and algorithm-driven layers:
+
+### Computational System Layer
+
+#### Multi-Level Pipeline Execution Orchestration
+Asynchronizes CPU scheduling at the framework layer to pipeline it with chip inference computation, reducing computation bubbles; at the model graph layer, splits single batches to create pipelines between micro-batches, overlapping computation and communication; at the operator kernel layer, pipelines different computing units, overlapping computation and memory access.
+#### Dynamic Shape Graph Execution Optimization
+Addressing the static graph adaptation problem for large language models processing dynamic inputs (e.g., variable sequence lengths and batch sizes), xLLM achieves dynamic adaptation through parametric design capturing input dimensions. It combines a multi-graph caching scheme to reduce compilation overhead and uses a managed memory pool instead of absolute addresses to ensure safe reuse, ultimately achieving high execution efficiency while maintaining high flexibility.
+#### Operator Optimization
+xLLM implements specific optimizations for key operators in LLMs on domestic hardware chips, including GroupMatmul, Chunked Prefill, and others.
+#### xTensor Memory Management
+The xTensor memory management framework employs a method of *pre-allocated physical memory page pools + contiguous virtual address mapping*. It achieves efficient dynamic memory management through dynamic on-demand mapping of physical pages, reuse of reusable memory pages (Reusable), and optimized scheduling with asynchronous pre-mapping. Combined with NPU operator adaptation (such as virtual address FlashMLA), it results in improved memory utilization and reduced latency.
+
+### Algorithm-Driven Layer
+
+#### PD Separation
+xLLM fully supports PD separation scenarios, enabling efficient management of PD instances, communication between PD instances, and KV cache transfer.
+#### Global Scheduling
+xLLM provides intelligent, full-lifecycle resource scheduling management for requests and instances.
+##### Instance Scheduling
+We have implemented various instance scheduling strategies to select how to assign instances to more suitable ones. These include a simple Round Robin strategy, a prefix cache-aware strategy based on the prefix cache hit rate of requests on each instance, and a KV cache-aware strategy based on the free memory level of instances. Furthermore, for PD separation scenarios, where static PD ratios often struggle with traffic fluctuations and sudden changes in request input/output lengths, we implemented an adaptive PD dynamic scheduler responsible for global instance allocation for online requests and runtime PD dynamic adjustment.
+##### Request Scheduling
+We have implemented various request scheduling strategies supporting continuous batching, including chunked prefill, prefill priority, decode priority, and other batch strategies, all while fully supporting PD separation scenarios.
+#### Global KV Cache Management
+Utilizes ETCD as a metadata service middleware at the global level for cluster service registration, load information synchronization, and global cache state management. Each compute instance maintains a local multi-level cache pool. Regarding scheduling strategy, the system adopts a dynamic decision-making mechanism based on KV cache: it first performs prefix matching detection, calculates the KV cache reuse rate of candidate nodes, and finally selects the node with the optimal comprehensive performance for processing, achieving dynamic offloading and migration of KV cache.
+#### Speculative Inference
+xLLM incorporates an optimized speculative inference algorithm that generates multiple tokens at once to boost throughput. xLLM reduces communication costs by sinking the speculative module and optimizes speculative inference computation through methods like overlapping scheduling and computation timelines and reducing operator data movement in speculative scenarios.
+#### MoE Load Balancing
+xLLM implements expert weight updates based on historical expert load statistics for MoE models. During inference, it achieves effective dynamic load balancing through efficient expert load statistics and double-buffered, seamless expert weight updates.
+
+### Multimodal Support
+
+xLLM provides comprehensive support for various multimodal models, including Qwen2-VL and MiniCPMV.

upstream_ref/xllm/docs/en/features/ppmatmul.md

@@ -0,0 +1,36 @@
+# PpMatmul Operator Optimization
+
+## Background
+
+In the inference of large models, matrix multiplication accounts for a high proportion and takes a long time. We have optimized the implementation of the matrix multiplication operator.
+
+## Feature Introduction
+
+The PpMatmul operator uses a Tiling strategy to decompose matrix multiplication into multiple smaller matrix multiplication tasks. However, when the number of tiles is small, tasks cannot be evenly distributed across all NPU cores, leading to the tail effect problem, which affects computational efficiency. We optimize the performance of the PpMatmul operator by prefetching memory or redistributing tasks.
+
+## User Interface
+
+### Operator Direct Call API
+
+```cpp
+aclnnStatus aclnnPpMatmulOptGetWorkspaceSize(
+    const aclTensor *a,
+    const aclTensor *b,
+    const aclTensor *out,
+    uint64_t *workspaceSize,
+    aclOpExecutor **executor);
+
+aclnnStatus aclnnPpMatmulOpt(
+    void *workspace,
+    uint64_t workspaceSize,
+    aclOpExecutor *executor,
+    aclrtStream stream);
+```
+
+- `a`: Input matrix A.
+- `b`: Input matrix B.
+- `out`: Output matrix, storing the computation result.
+
+## Performance Effect
+
+For cases with a small number of tiles (e.g., when M is small, corresponding to a small batch size), there is an **18%** performance improvement of the operator compared to before optimization when (TP=4).

upstream_ref/xllm/docs/en/features/prefix_cache.md

@@ -0,0 +1,19 @@
+# Prefix Cache Optimization
+
+## Feature Introduction
+xLLM supports prefix cache matching. The prefix cache is based on `murmur_hash` and uses an LRU eviction policy, delivering superior matching efficiency and increased prefix cache hit rates.
+Additionally, the prefix cache has been optimized to support the `continuous_scheduler`, `chunked_scheduler`, and `zero_evict_scheduler`. The cache is updated immediately after prefill operations, enhancing matching timeliness. For the `chunked_scheduler`, multi-stage chunked prefill matching is supported, reducing computational overhead and minimizing KV cache usage as much as possible.
+
+## Usage
+The prefix cache is implemented in xLLM and exposed through gflags parameters to control its functionality.
+
+- Enable prefix cache with specific policy and settings:
+```
+--enable_prefix_cache=true
+```
+
+## Performance Impact
+After enabling prefix cache, on the Qwen3-8B model with a TPOT constraint of 50ms, the E2E latency **decreased by 10%**.
+
+!!! warning "Note"
+    PD separation scheduler is not currently supported.

upstream_ref/xllm/docs/en/features/topk_topp.md

@@ -0,0 +1,27 @@
+# Topk & Topp Operator Optimization
+
+## Background
+
+In natural language generation tasks, the topK and topP sampling strategies are widely used to control the diversity and quality of generated text. However, in small models, the computation time for these two strategies is relatively long. This is mainly due to the fewer parameters in small models, which leads to reduced efficiency in sorting and filtering when processing probability distributions, thereby affecting generation speed. Therefore, optimizing the implementation of topK and topP in small models can enhance their sampling efficiency.
+
+## Feature Introduction
+
+The implementation of the topKtopP operator merges multiple small operators, such as sorting, topK, softmax, and topP, into a single large operator, thereby improving computational efficiency and performance.
+
+## User Interface
+
+### Operator Call API
+
+```c++
+void top_k_top_p(torch::Tensor& logits,
+                 const torch::Tensor& topK,
+                 const torch::Tensor& topP);
+```
+
+- `logits`: The input logits tensor containing the model's output scores.
+- `topK`: The threshold tensor for selecting the top K probabilities.
+- `topP`: The threshold tensor for selecting the cumulative probabilities.
+
+## Performance Effect
+
+* After using the topKtopP fused operator, in the qwen2-0.5B model, TTOT **decreased by 37%**, and TTFT **increased by 10%**.

upstream_ref/xllm/docs/en/features/xllm_service_overview.md

@@ -0,0 +1,54 @@
+
+
+# xLLM Service
+
+<p align="right">[:simple-github: xLLM Service](https://github.com/jd-opensource/xllm-service)</p>
+
+
+## Project Overview
+
+**xLLM-service** is a service-layer framework developed based on the **xLLM** inference engine, providing efficient, fault-tolerant, and flexible LLM inference services for clustered deployment.
+
+xLLM-service targets to address key challenges in enterprise-level service scenarios:
+
+- How to ensure the SLA of online services and improve resource utilization of offline tasks in a hybrid online-offline deployment environment.
+
+- How to react to changing request loads in actual businesses, such as fluctuations in input/output lengths.
+
+- Resolving performance bottlenecks of multimodal model requests.
+
+- Ensuring high reliability of computing instances.
+
+#### Background
+
+LLM with parameter scales ranging from tens of billions to trillions are being rapidly deployed in core business scenarios such as intelligent customer service, real-time recommendation, and content generation. Efficient support for domestic computing hardware has become a core requirement for low-cost inference deployment. Existing inference engines struggle to effectively adapt to the architectural characteristics of dedicated accelerators like domestic chips. Performance issues such as low utilization of computing units, load imbalance and communication overhead bottlenecks under the MoE architecture, and difficulties in kv cache management have restricted the efficient inference of requests and the scalability of the system. The xLLM-service + xLLM inference engine improves the efficiency of the entire performance link and it provides crucial technical support for the large-scale implementation of LLM in real-world business scenarios.
+
+--- 
+
+## Overall Architecture
+The overall architecture of xLLM-service is shown in the figure below:
+
+![1](../../assets/service_arch.png)
+
+## Core Components
+
+### ETCD Cluster
+It is used for metadata management, including the storage and management of metadata such as models, xllm instances, and requests. It also provides xllm node registration and discovery services.
+
+### Fault Tolerance
+xLLM-service provides fault tolerance management to ensure service quality and stability.
+
+### Global Scheduler
+It implements globally aware scheduling. Based on the current system status, it accurately dispatches requests to the optimal instances for execution, effectively improving the overall service response efficiency and resource utilization.
+
+### Global KV Cache Manager
+It is responsible for global KV Cache management. Its core capabilities include distributed KV cache awareness, Prefix matching, and dynamic migration of KV Cache, which optimize the efficiency of cache resource usage.
+
+### Instance Manager
+It focuses on the full-lifecycle management of instances. All xllm instances must register to service after startup. Based on preset policies, the module provides support for instances such as scheduling adaptation and fault tolerance handling.
+
+### Event Plane
+As the metrics and event hub, it receives Metrics data reported by various instances, uniformly collects and organizes statistical indicators, and provides data support for decisions such as service scheduling, fault tolerance, and scaling.
+
+### Planner
+It undertakes the functions of strategy analysis and decision-making. Based on the Metrics data reported by the Event Plane (including instance runtime indicators, machine load indicators, etc.), it analyzes the service scaling needs and the necessity of expanding hot instances, and outputs resource adjustment and instance optimization strategies.

upstream_ref/xllm/docs/en/features/zero_evict_scheduler.md

@@ -0,0 +1,17 @@
+# Zero Evict Scheduler
+
+## Feature Introduction
+xLLM supports the zero evict scheduling strategy. The zero evict scheduling strategy is an algorithm designed to minimize request eviction rates, reducing the need for prefill computation on evicted requests and consequently improving TPOT (Time Per Output Token).
+This scheduling algorithm employs simulation rounds to detect whether a request can be scheduled without causing the eviction of other requests.
+
+## Usage
+The aforementioned strategy has been implemented in xLLM and is exposed through gflags parameters to control the feature's on/off state.
+
+- Enable the zero evict strategy and set the maximum decode tokens per sequence.
+```
+--use_zero_evict=true
+--max_decode_token_per_sequence=256
+```
+
+## Performance Impact
+After enabling zero evict, on the Qwen3-8B model with an E2E latency constraint, the TPOT latency **decreased by 27%**.