dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

ref(upstream): FULL TREE — Deep-Spark xllm (1470) + ds_vllm csrc/models (703)

Browse Source 
Replaces cherry-picked upstream_ref with complete source trees.

xllm/ — Iluvatar official C++ inference engine (15MB, 1470 files)
  Complete: kernels → layers → models → runtime → scheduler → api
  Excluded: .git, binary images, third_party submodule checkouts

ds_vllm/ — Iluvatar official vllm fork (8MB, 703 files)
  Included: csrc/ (ALL CUDA kernels), fused_moe/, qwen3_5 model, _custom_ops
  Excluded: tests, benchmarks, docs, examples (not needed for reference)

Critical call chains now fully traceable:
  MoE: moe_topk_softmax_kernels.cuh → ixformer.h → fused_moe.cpp → layer
  GDN: qwen3_gated_delta_net_base.cpp → qwen3_5_gated_delta_net.cpp
  Attention: ixformer.h → xllm_paged_attention → attention.cpp
This commit is contained in:
EX Engine
2026-08-10 02:53:54 +00:00
parent 9e4fb3712f
commit 002f9879b2
2179 changed files with 494021 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
upstream_ref/xllm/.agents/skills/add-unit-test/SKILL.md Normal file
View File

@@ -0,0 +1,48 @@
---
name: add-unit-test
description: Add or update xLLM unit tests in the repository. Use when Codex needs to create a new C++/CUDA/NPU/MLU unit test, place a test under tests/, wire it into CMake with cc_test, update an existing test target, choose platform gates, or validate test naming and dependencies against current xLLM test conventions.
---
# Add Unit Test
## Workflow
1. Inspect the production code and the nearest existing tests before writing a new test.
- Match the production path under `xllm/` to `tests/` where possible.
- Prefer extending an existing nearby `*_test.cpp` and `cc_test` target when the behavior belongs to the same domain.
- Create a new test source only when it improves isolation, keeps platform setup separate, or follows an existing directory pattern.
2. Read the project style guide before editing production files under `xllm/`, and apply the same C++ style discipline to new test code:
`.agents/skills/code-review/references/custom-code-style.md`.
3. Follow the current test layout and CMake conventions.
- Read [xllm-test-patterns.md](references/xllm-test-patterns.md) when adding a new test file, new `cc_test`, platform-specific test, or test directory.
- Use `*_test.cpp` for C++ test files and `*_test.cu` for CUDA source tests.
- Do not create nested `test/` or `tests/` directories for new unit tests unless the surrounding tree already requires that structure.
4. Wire tests through CMake with `include(cc_test)` and `cc_test(...)`.
- Keep source names relative to the current test directory unless an existing target already uses an absolute source path for a production `.cpp`.
- Use target names ending in `_test`.
- Put platform-directory gates in the parent `CMakeLists.txt` when the whole child directory is platform-specific.
- Use target-level `if(USE_NPU)`, `if(USE_MLU)`, `if(USE_CUDA)`, or generator expressions only when a mixed directory contains both generic and platform-specific tests.
5. Write tests for observable behavior, not implementation trivia.
- Cover success, edge, and error paths touched by the change.
- Prefer deterministic inputs, fixed seeds, and small tensors/data structures.
- Keep helpers file-local in an anonymous namespace unless shared by multiple test files.
- Use `TEST`/`TEST_F` names that describe behavior clearly.
6. Validate narrowly before finishing.
- Always run `git diff --check` for the changed test paths.
- Search for stale filenames after moving or renaming tests.
- Run the narrowest build/test command available locally; if not feasible, state the exact reason and what was checked instead.
## Common Commands
```bash
rg --files tests/<area>
rg "old_test_name|old_file_name" tests xllm CMakeLists.txt
git diff --check -- tests/<area>
```
For full remote validation on the development machine, use the repository AGENTS instructions for SSH, container, build, and test commands.

124
upstream_ref/xllm/.agents/skills/add-unit-test/references/xllm-test-patterns.md Normal file
View File

@@ -0,0 +1,124 @@
# xLLM Unit Test Patterns
Use this reference when creating or changing unit tests under `tests/`.
## Layout
- Mirror production structure where practical:
- `xllm/core/framework/tokenizer` -> `tests/core/framework/tokenizer`
- `xllm/core/layers/mlu` -> `tests/core/layers/mlu`
- `xllm/function_call/...` -> `tests/function_call/...`
- Keep tests directly in the relevant leaf directory.
- Avoid new nested `test/` or `tests/` directories. Recent cleanup moved those tests into their parent directories.
- Shared helpers may live beside tests, such as `tests/core/layers/mlu/tests_utils.cpp`.
## Naming
- Test source files use singular suffixes:
- C++: `thing_test.cpp`
- CUDA source: `thing_test.cu`
- Test CMake target names also end in `_test`.
- If a target aggregates several source files, keep the target name at the domain level, for example `layer_test`, `moe_layer_test`, or `sampler_test`.
- Keep helper files out of the `*_test.cpp` suffix unless they define tests.
## CMake Basics
Use `cc_test` for C++/CUDA unit test binaries:
```cmake
include(cc_test)
cc_test(
NAME
feature_test
SRCS
feature_test.cpp
DEPS
:feature
GTest::gtest_main
glog::glog
)
```
Rules:
- Add `include(cc_test)` in each CMake file that declares `cc_test`.
- Use dependencies that match nearby tests first.
- Prefer `GTest::gtest_main`; add `GTest::gtest` only when nearby tests need it or the target explicitly uses it.
- Add `target_link_libraries(...)` and `add_dependencies(...)` after `cc_test` when needed for `brpc`, `OpenSSL`, `protobuf`, platform runtime libraries, or link-group handling.
- Use `:target_name` for local production CMake targets where existing tests do so.
## Platform Gates
Gate entire platform-only directories in the parent `CMakeLists.txt`.
Current examples:
```cmake
if(USE_CUDA)
add_subdirectory(cuda)
endif()
if(USE_NPU)
add_subdirectory(npu)
endif()
```
```cmake
if(USE_CUDA)
add_subdirectory(cuda)
endif()
if(USE_MLU)
add_subdirectory(mlu)
endif()
```
Do not repeat the same platform `if(...)` inside every child CMake file when the parent already gates the directory.
Use target-level platform gates only for mixed directories where generic and platform-specific tests coexist, such as `tests/core/runtime` or framework directories with both generic and NPU-only targets.
Use generator expressions for platform-specific optional link libraries when the target exists across platforms:
```cmake
target_link_libraries(example_test
PUBLIC
Python::Python
$<$<BOOL:${USE_NPU}>:ascendcl>
$<$<BOOL:${USE_NPU}>:hccl>
$<$<BOOL:${USE_NPU}>:c_sec>)
```
## Source Style
- Add the xLLM copyright header to new files, using the current year.
- Include `<gtest/gtest.h>` in every test source.
- Use project-root-relative includes; avoid `../` includes.
- Put file-local helpers in an anonymous namespace.
- Prefer fixed-width integers (`int32_t`, `int64_t`) unless an API requires plain `int`.
- Use `static_cast`, `nullptr`, braces on all control statements, and concise comments only where they clarify test setup.
- Keep deterministic random or tensor tests seeded with stable labels or fixed seeds.
## Test Design
- Test behavior through public or stable internal interfaces used by nearby tests.
- Cover the regression or edge case that motivated the test.
- For parser and pure logic tests, keep inputs small and assert exact outputs/errors.
- For tensor/device tests, keep tensor shapes small, check dtype/device expectations, and compare against a simple reference implementation.
- For forked-process or device-init-sensitive tests, follow nearby standalone target patterns and leave a short CMake comment explaining why the target is isolated.
## Validation Checklist
Before finishing:
```bash
rg --files tests/<area>
rg "old_file_name|old_target_name" tests xllm CMakeLists.txt
git diff --check -- tests/<area>
```
Run the narrowest feasible validation:
- Local CMake/build target if available.
- `python setup.py test` in the project container when full validation is requested or risk is high.
- For development-machine validation, follow the repo AGENTS instructions for `ssh gpu-h800-195`, `/export/home/zhangxu709/xllm`, container `zx-xllm-cuda`, and the build/test commands.

115
upstream_ref/xllm/.agents/skills/code-review/SKILL.md Normal file
View File

@@ -0,0 +1,115 @@
---
name: code-review
description: Review code changes for quality, security, performance, and correctness following project-specific standards. Use when reviewing pull requests, examining git diffs, or when the user asks for a code review. This skill should be used proactively — when the user asks for a review without specifying commits, automatically detect the current branch and diff against the main branch.
---
# Code Review
## Workflow
### Step 1: Determine the diff
If the user provides explicit SHAs or a PR link, use those. Otherwise, **auto-detect**:
```bash
# Fetch latest remote state
git fetch origin main --quiet
# Detect current branch
CURRENT_BRANCH=$(git branch --show-current)
# Find the merge base with origin/main
MERGE_BASE=$(git merge-base origin/main HEAD)
# Show what changed
git diff --stat $MERGE_BASE..HEAD
git diff $MERGE_BASE..HEAD
```
If `CURRENT_BRANCH` is `main`, warn the user and ask which commits to review.
### Step 2: Read project standards
Read [custom-code-style.md](references/custom-code-style.md) for project-specific coding style.
### Step 3: Review against the checklist
**Correctness:**
- Logic handles edge cases and boundary conditions
- Error handling is comprehensive (no silent failures)
- Type safety maintained (no unsafe casts, proper use of `std::optional`)
- Resource lifecycle correct (RAII, no leaks, proper cleanup order)
**Architecture:**
- Clean separation of concerns, no layer violations
- Dependencies flow in the correct direction
- Changes align with existing patterns in the codebase
- No unnecessary coupling introduced
**Performance & Concurrency:**
- No performance regressions on hot paths
- Thread safety: proper locking, no data races
- CUDA/NPU kernels: memory coalescing, occupancy, sync correctness
- No unnecessary copies of large objects (tensors, vectors)
**Testing:**
- Tests verify actual logic, not just mock wiring
- Edge cases and error paths covered
- Integration tests for cross-component changes
**Production Readiness:**
- Backward compatibility maintained (or breaking changes documented)
- Migration strategy for schema/config changes
- No hardcoded values that should be configurable
### Step 4: Output findings
Use the format below.
## Output Format
### Strengths
[Specific things done well, with file:line references]
### Issues
#### Critical (Must Fix)
[Bugs, security holes, data loss risks, broken functionality]
#### Important (Should Fix)
[Architecture problems, missing error handling, test gaps, performance issues]
#### Minor (Nice to Have)
[Style, optimization opportunities, documentation improvements]
**Each issue must include:**
- **File:line** reference
- **What** is wrong
- **Why** it matters
- **How** to fix (if not obvious)
### Recommendations
[Broader improvements for code quality, architecture, or process]
### Assessment
**Ready to merge?** [Yes / No / With fixes]
**Reasoning:** [1-2 sentence technical assessment]
## Rules
**DO:**
- Apply project-specific style from [custom-code-style.md](references/custom-code-style.md)
- Follow DDD (Domain Driven Design) principles, and keep the codebase clean and maintainable
- Categorize by actual severity (not everything is Critical)
- Be specific with file:line references
- Explain WHY issues matter
- Acknowledge strengths
- Give a clear verdict
**DON'T:**
- Approve without thorough review
- Mark nitpicks as Critical
- Give feedback on code not in the diff
- Be vague (e.g., "improve error handling" without specifics)

316
upstream_ref/xllm/.agents/skills/code-review/references/custom-code-style.md Normal file
View File

@@ -0,0 +1,316 @@
# Custom Code Style
Project-specific coding style for xllm. The reviewer **MUST** enforce these style.
---
## 1. Naming Conventions
### C++
| Element | Style | Example |
|------------------|------------------------------------|--------------------------------------|
| Namespace | `snake_case` | `xllm`, `xllm::detail` |
| Class / Struct | `PascalCase` | `LlmModelImplBase`, `KVCache` |
| Function | `snake_case` | `get_input_embeddings`, `forward` |
| Member variable | `snake_case_` (trailing underscore)| `model_type_`, `embed_tokens_` |
| Local variable | `snake_case` | `inputs_embeds`, `kv_caches` |
| Constant | `k` + `PascalCase` | `kContentLength`, `kMaxBatchSize` |
| Enum type | `PascalCase` | `EngineType`, `DeviceType` |
| Enum value | `ALL_CAPS` | `LLM`, `VLM`, `INVALID` |
| Template param | `PascalCase` | `DecoderLayerType` |
| Macro | `ALL_CAPS` | `XLLM_CHECK`, `LOG_EVERY_N` |
| File name | `snake_case` | `llm_model_base.h`, `types.h` |
| Header guard | `#pragma once` | - |
### Python
| Element | Style | Example |
|------------------|------------------------|--------------------------------------|
| Module / file | `snake_case` | `model_loader.py` |
| Class | `PascalCase` | `TokenizerConfig` |
| Function | `snake_case` | `load_model` |
| Variable | `snake_case` | `batch_size` |
| Constant | `ALL_CAPS` | `MAX_SEQ_LEN` |
| Private member | `_leading_underscore` | `_internal_state` |
---
## 2. File & Header Rules
- **Copyright header required** on all new files. Use the correct year matching the file creation date.
```cpp
/* Copyright 2025 The xLLM Authors. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
...
==============================================================================*/
```
- **No relative paths in `#include`**. Always use project-root-relative paths.
```cpp
// Good
#include "core/common/types.h"
// Bad
#include "../common/types.h"
#include "./types.h"
```
- **Remove redundant and duplicate includes**. Each header should be included exactly once, and unused includes must be cleaned up.
---
## 3. Type System & Declarations
- **Use fixed-width integers** (`int32_t`, `int64_t`) instead of plain `int`, unless the API you are calling explicitly requires `int`.
```cpp
// Good
int32_t batch_size = 16;
int64_t total_tokens = 0;
// Bad
int batch_size = 16;
```
- **Use `static_cast`** for all type conversions. Never use C-style casts.
```cpp
// Good
auto len = static_cast<int32_t>(vec.size());
// Bad
auto len = (int32_t)vec.size();
```
- **Do not use `auto` for simple/primitive types**. `auto` is acceptable for complex types (iterators, lambdas, template-deduced types) but not for `int32_t`, `float`, `bool`, `std::string`, etc.
```cpp
// Good
int32_t count = 0;
auto it = map.find(key); // complex iterator type, auto is fine
// Bad
auto count = 0;
auto name = std::string("model");
```
- **Use `using` instead of `typedef`** for type aliases. Prefer aliases for complex types to improve readability.
```cpp
// Good
using TensorVec = std::vector<torch::Tensor>;
using CallbackFn = std::function<void(int32_t)>;
// Bad
typedef std::vector<torch::Tensor> TensorVec;
```
- **Use `enum class`** instead of plain `enum` to provide type safety and prevent implicit conversions.
```cpp
// Good
enum class DeviceType : int8_t { CPU = 0, CUDA = 1, NPU = 2 };
// Bad
enum DeviceType { CPU = 0, CUDA = 1, NPU = 2 };
```
- **Use `nullptr`** instead of `NULL` or `0` for null pointers.
- **Choose the right container**: use `std::unordered_map` / `std::unordered_set` when key ordering is irrelevant (O(1) average lookup). Use `std::map` / `std::set` only when sorted iteration or key ordering is required.
---
## 4. Class Design
- **Mark classes `final`** if they are not designed to be inherited from.
```cpp
// Good
class TokenizerConfig final { ... };
// Bad – class has no virtual functions and is not intended as a base
class TokenizerConfig { ... };
```
- **Use `explicit`** on any constructor that can be invoked with a single argument. This includes multi-parameter constructors where all parameters except the first have default values.
```cpp
// Good
explicit ModelArgs(const std::string& path, int32_t num_layers = 12);
// Bad – allows implicit conversion from std::string
ModelArgs(const std::string& path, int32_t num_layers = 12);
```
- **Use `override`** when overriding virtual functions in derived classes. Never repeat the `virtual` keyword on overrides.
```cpp
// Good
ModelOutput forward(torch::Tensor tokens, ...) override;
// Bad
virtual ModelOutput forward(torch::Tensor tokens, ...);
```
- **Structs must not have member functions**. If you need methods, use a `class`. Structs are for plain data aggregation only.
---
## 5. Memory & Resource Management
- **Avoid raw pointers**. Prefer smart pointers (`std::unique_ptr`, `std::shared_ptr`) for ownership semantics.
- Use `std::unique_ptr` by default (sole ownership).
- Use `std::shared_ptr` only when shared ownership is genuinely needed.
- Raw pointers are acceptable only for non-owning references where the lifetime is clearly managed elsewhere.
---
## 6. Scoping & Visibility
### C++
- **File-local functions and variables** (used only within a single `.cpp` file) must be placed in an **anonymous namespace**.
```cpp
namespace {
int32_t compute_padding(int32_t seq_len, int32_t alignment) {
return (alignment - seq_len % alignment) % alignment;
}
} // namespace
```
### Python
- **File-local helper functions** (not part of the public API) must be prefixed with `_`.
- **Non-public member functions** of a class must be prefixed with `_`.
```python
def _validate_config(config: dict) -> bool:
...
class ModelLoader:
def load(self, path: str) -> Model:
self._check_path(path)
...
def _check_path(self, path: str) -> None:
...
```
---
## 7. Torch & Framework API Usage
- **Use `torch::` namespace** instead of `at::` or `c10::` wherever possible. Prefer the highest-level PyTorch C++ API.
```cpp
// Good
torch::Tensor output = torch::zeros({batch_size, hidden_dim});
// Bad
at::Tensor output = at::zeros({batch_size, hidden_dim});
c10::optional<torch::Tensor> mask = c10::nullopt; // use std::optional
```
- **Use `CHECK`** (glog) instead of `TORCH_CHECK` for assertions.
```cpp
// Good
CHECK(tensor.is_contiguous()) << "Input tensor must be contiguous";
// Bad
TORCH_CHECK(tensor.is_contiguous(), "Input tensor must be contiguous");
```
- **Use `LOG(FATAL)`** for unrecoverable errors instead of throwing `std::runtime_error`.
```cpp
// Good
LOG(FATAL) << "Unsupported model type: " << model_type;
// Bad
throw std::runtime_error("Unsupported model type: " + model_type);
```
---
## 8. Code Style & Control Flow
- **Always use braces `{}`** with `if`, `while`, `for`, even for single-line bodies.
```cpp
// Good
if (x > 0) {
return x;
}
// Bad
if (x > 0) return x;
```
- **Avoid `if` inside `for` loops** when possible. Prefer filtering the data beforehand or restructuring the logic (e.g., early `continue`, separate loops, `std::copy_if`).
- **Define variables close to first use**. Do not declare all variables at the top of a function.
- **Annotate constant arguments** with a comment indicating the parameter name when calling functions or constructors.
```cpp
// Good
auto layer = DecoderLayer(/*hidden_size=*/4096, /*num_heads=*/32);
// Bad
auto layer = DecoderLayer(4096, 32);
```
---
## 9. STL Best Practices
- **Always `reserve()` before filling a `std::vector`** when the size is known or can be estimated.
```cpp
// Good
std::vector<torch::Tensor> outputs;
outputs.reserve(num_layers);
for (int32_t i = 0; i < num_layers; ++i) {
outputs.emplace_back(compute_layer(i));
}
// Bad – causes multiple reallocations
std::vector<torch::Tensor> outputs;
for (int32_t i = 0; i < num_layers; ++i) {
outputs.push_back(compute_layer(i));
}
```
- **Prefer `emplace_back`** over `push_back` to construct elements in-place and avoid unnecessary copies.
---
## 10. Global Flags
- **Do not overuse `FLAGS_` global variables**. Prefer passing configuration through constructor parameters or config structs. Only use global flags for top-level, process-wide settings.
- **Register new flags in `help_formatter.h`**. When adding a new global flag, always add a corresponding entry in `help_formatter.h` so it appears in `--help` output.
---
## 11. Python-Specific Rules
- **Type annotations are required** on all function signatures (parameters and return types). Use `typing` module types where needed.
```python
# Good
def load_model(path: str, device: str = "cuda") -> nn.Module:
...
# Bad
def load_model(path, device="cuda"):
...
```
- **Private helpers**: prefix with `_` (see Section 6).

49
upstream_ref/xllm/.agents/skills/git-workflow/SKILL.md Normal file
View File

@@ -0,0 +1,49 @@
---
name: git-workflow
description: Use when the task involves Git operations for the public xLLM repository, including choosing branch or tag names, preparing commits and pull requests, backporting fixes, checking repo-specific review expectations, or drafting commit messages from actual diffs.
---
# Git Workflow
Use xLLM repo reality, not generic Git habits.
## Reference Map
Load only the file that matches the user's immediate Git task.
| File | What it is for | When to load it |
| --- | --- | --- |
| `references/source-of-truth.md` | Repo-specific source priority and canonical files to consult | Load first when repo docs, local state, and user wording may disagree |
| `references/branch-naming.md` | Branch naming patterns and default branch conventions | Load when the user asks how to name a branch or which branch to branch from |
| `references/development-flow.md` | Day-to-day fork, sync, branch, validate, and push flow | Load when the user asks for normal development steps from local change to push |
| `references/pr-review.md` | PR targeting, PR scope, and review expectations | Load when the task is about opening a PR, choosing the target branch, or deciding who should review |
| `references/release-layout.md` | Release branch and tag shapes used by xLLM | Load when the task mentions release branches, release tags, or patch version naming |
| `references/backport-flow.md` | Preferred backport and hotfix flow for released lines | Load when the task mentions cherry-picks, hotfixes, or fixing an already released branch |
| `references/commit-format.md` | Commit title/body conventions and xLLM-style examples | Load when the user asks for a commit message, commit style guidance, or message cleanup |
## Workflow
1. Decide which subtask the user actually needs.
2. Read `references/source-of-truth.md` when you need repo-specific confirmation.
3. Then load only the most relevant task file from the table above.
4. For commit message drafting, run `bash scripts/collect_git_context.sh [--staged|--all|--unstaged]` before writing the final message.
5. Draft commit messages from the actual diff, not from filenames alone.
6. If one diff mixes unrelated concerns, recommend splitting the commit instead of forcing one vague summary.
7. Default PR targets to `main` unless the task is clearly a release or backport flow.
8. For released lines, prefer landing on `main` first and then backporting unless the user explicitly wants a direct hotfix flow.
## Output
Return the smallest useful answer for the user's Git task:
- workflow questions: concrete branch, tag, PR, or backport steps
- commit message requests: `<type>: <subject>` plus an optional short bullet body
- repo-convention questions: current xLLM-specific guidance, not generic Git advice
## Quick Checks
- branch names match xLLM style such as `feat/<topic>`, `bugfix/<topic>`, or `release/vX.Y.Z`
- PR target is `main` unless this is a release or backport task
- commit title matches the dominant change in the diff
- release tags use semantic versions such as `v0.9.0` or `v0.9.1`
- owner-review expectations come from `.github/CODEOWNERS` when relevant

30
upstream_ref/xllm/.agents/skills/git-workflow/references/backport-flow.md Normal file
View File

@@ -0,0 +1,30 @@
# Backport Flow
When a released line needs a bugfix, prefer this flow:
1. land the fix on `main` first unless the user explicitly needs a direct hotfix flow
2. cherry-pick or backport the fix to the matching `release/vX.Y.Z` branch
3. update release content as needed on that release branch
4. create the next patch tag for that release line
Example shape:
```bash
# land on main first
git checkout main
git pull --rebase upstream main
git checkout -b bugfix/<topic>
git commit -m "bugfix: fix <summary>."
# then backport
git checkout release/v0.9.0
git pull --rebase upstream release/v0.9.0
git cherry-pick <bugfix_commit>
git tag v0.9.1
```
## Quick Checklist
- backport from a commit already landed on `main` when possible
- cherry-pick onto the matching `release/vX.Y.Z` branch
- use the next semantic patch tag for the release line

54
upstream_ref/xllm/.agents/skills/git-workflow/references/branch-naming.md Normal file
View File

@@ -0,0 +1,54 @@
# Branch Naming
Use one of these branch shapes:
```text
<type>/<topic>
<namespace>/<type>/<topic>
preview/<topic>
release/vX.Y.Z
```
Recommended lowercase branch types:
- `feat`: new user-visible capability or feature work
- `bugfix`: incorrect behavior, regressions, or hot fixes
- `refactor`: structural changes without intended behavior changes
- `docs`: documentation-only work when a dedicated branch is useful
- `test`: test-only work when separated from product changes
- `perf`: runtime or memory improvements
- `chore`: repo maintenance that does not fit the other categories
- `build`: dependency, CI, packaging, or release tooling changes
Topic guidelines:
- use lowercase letters, numbers, and hyphens by default
- keep the topic short, specific, and review-friendly
- prefer nouns or short noun phrases like `scheduler`, `lm-head-new`, `npu-template`
- avoid spaces, uppercase letters, and vague names like `misc`, `temp`, `test-branch`
- avoid repeating the type in the topic, such as `feat/feature-x`
- use slash-separated namespace prefixes only when the work clearly belongs to a scoped stream
Scoped branch guidelines:
- use `<namespace>/<type>/<topic>` for team-, model-, or project-scoped work such as `dsv4/feat/rope-dsv4`
- keep the namespace stable and meaningful, not personal or temporary
- use `preview/<topic>` only for preview-track work that intentionally aligns with preview branches upstream
- reserve `release/vX.Y.Z` for release preparation or release-only changes
- avoid direct development on `main` and `release/*` unless the user explicitly asks for it
Examples:
- `feat/skills`
- `feat/lm_head_new`
- `bugfix/scheduler`
- `refactor/npu_template`
- `preview/glm-5`
- `release/v0.9.0`
Notes for xLLM:
- `main` is the default development branch
- `feat/*`, `bugfix/*`, and `refactor/*` appear in current branch usage and are safe defaults
- `preview/*` and `release/*` are long-lived integration branches, not ordinary personal topic branches
- both hyphen and underscore appear in existing history, but prefer hyphens for new branch topics unless matching an established naming family

56
upstream_ref/xllm/.agents/skills/git-workflow/references/commit-format.md Normal file
View File

@@ -0,0 +1,56 @@
# Commit Format
Use this exact first-line format:
```text
<type>: <subject>
```
Allowed lowercase types:
- `feat`: add user-visible behavior or a new capability
- `bugfix`: correct incorrect behavior or a regression
- `docs`: change documentation only
- `test`: add or update tests only
- `refactor`: improve structure without changing intended behavior
- `chore`: repository maintenance that does not fit the other types
- `style`: formatting or style-only changes without logic changes
- `revert`: revert an earlier commit
- `perf`: improve runtime or memory behavior
- `model`: change model definitions, checkpoints, prompts, or inference behavior
- `build`: change build, release, or dependency wiring
- `release`: change release versioning, release notes, or release-only metadata
Subject guidelines:
- use lowercase letters by default
- include at least 4 words
- end with a period
- start with a verb like `add`, `fix`, `remove`, `refactor`, `document`
- keep it specific enough that a reviewer understands the main change
- avoid filler like `update`, `misc`, `stuff`, `changes`
- describe the effect or intent, not a mechanical file list
Body guidelines:
- add a body only when the title alone is not enough
- use short bullets for secondary details or important context
- mention follow-up work, migration steps, or compatibility impact when relevant
- if confidence is low because the diff is partial or noisy, say that explicitly
Observed xLLM-style examples:
- `feat: add rope_in_place tilelang kernel for npu device. (#964)`
- `bugfix: align rec initialization flags with options. (#1142)`
- `docs: update the document to align them with the latest code. (#1113)`
- `refactor: extract multi-modal input processors to processors dir. (#1022)`
- `perf: reserve vector capacity before batch push_back. (#1089)`
- `release: update xllm release version to v0.9.0. (#1124)`
- `feat: support qwen3.5/qwen3.5-moe mtp draft model for speculative decoding[3/N]. (#1119)`
Notes for xLLM:
- `bugfix:` appears more often than `fix:` in current history and should be the default bug-repair prefix
- `fix:` exists in a few historical commits but is less consistent than `bugfix:`
- PR-number suffixes like `(#1142)` are common in merged history but are optional unless the user explicitly wants them
- staged series markers like `[1/N]` or `[3/N]` appear when a change is intentionally split across multiple commits

33
upstream_ref/xllm/.agents/skills/git-workflow/references/development-flow.md Normal file
View File

@@ -0,0 +1,33 @@
# Development Flow
Follow this sequence unless the user asks for a different workflow:
1. fork the upstream repository
2. sync local `main` with `upstream/main`
3. create a focused topic branch from `main`
4. implement the change
5. run formatting and the narrowest relevant validation
6. commit in clear English
7. push to the fork
8. open a PR to upstream `main`
Example commands:
```bash
git fetch upstream
git checkout main
git pull --rebase upstream main
git checkout -b feat/<topic>
# after development
git add <files>
git commit -m "feat: add <change summary>."
git push origin feat/<topic>
```
## Quick Checklist
- branch from `main` unless this is a release or backport task
- keep the branch focused on one change
- run the narrowest relevant validation before commit
- push to your fork before opening the PR

28
upstream_ref/xllm/.agents/skills/git-workflow/references/pr-review.md Normal file
View File

@@ -0,0 +1,28 @@
# PR And Review
## Pull Request Guidance
The public repo guidance is lightweight:
- `README.md` and `CONTRIBUTING.md` ask contributors to fork, create a branch, and send a pull request
- keep PRs focused and easy to review, even though the public docs do not publish a hard line-count limit
- write commit messages and PR descriptions in clear English
- avoid unnecessary merge noise in branch history; prefer a clean linear history when practical
## Target Branch
- use `main` unless this is explicitly a release or backport task
## Review Expectations
Use `.github/CODEOWNERS` as the visible review signal:
- changes under `/xllm/` have listed code owners
- expect owner review or owner attention for those paths
- if the user asks who should review a change under `/xllm/`, check `CODEOWNERS`
## Quick Checklist
- PR target is `main` unless this is a backport or release task
- PR is focused and clearly described
- review expectations are checked through `CODEOWNERS`

15
upstream_ref/xllm/.agents/skills/git-workflow/references/release-layout.md Normal file
View File

@@ -0,0 +1,15 @@
# Release Layout
Observed public release layout:
- release branches use `release/vX.Y.Z`
- observed release branches include `release/v0.6.0`, `release/v0.7.0`, `release/v0.8.0`, and `release/v0.9.0`
- release notes are tracked in `RELEASE.md`
- public tags use semantic version tags such as `v0.9.0`
- patch tags use normal patch versions such as `v0.7.1` and `v0.7.2`, not `-rcN`
## Naming Guardrails
- do not switch to `release_0.1.0` branches unless the user explicitly wants an older internal workflow
- do not assume `v0.1.0-rc0` style tags unless the user explicitly asks for them
- prefer the next semantic patch tag for bugfix releases

29
upstream_ref/xllm/.agents/skills/git-workflow/references/source-of-truth.md Normal file
View File

@@ -0,0 +1,29 @@
# Source Of Truth
Use these repo files first when the user asks for xLLM-specific Git guidance:
- `README.md`
- `CONTRIBUTING.md`
- `RELEASE.md`
- `.github/workflows/check_format.yml`
- `.pre-commit-config.yaml`
- `.github/CODEOWNERS`
Guidance priority:
1. direct user request
2. current repo files and branch reality
3. visible remote repo conventions
4. older internal notes or remembered habits
When repo docs, local repo state, and user wording disagree:
- say the conflict explicitly
- prefer the most concrete source available
- avoid inventing rules that are not visible in the public repo
Common examples of rules you should not assume without evidence:
- rebase-only or squash-only merge requirements
- mandatory reviewer counts beyond what `CODEOWNERS` implies
- old naming patterns such as `features/*`, `release_0.1.0`, or `v0.1.0-rc0`

104
upstream_ref/xllm/.agents/skills/git-workflow/scripts/collect_git_context.sh Normal file
View File

@@ -0,0 +1,104 @@
#!/usr/bin/env bash
set -euo pipefail
mode="auto"
if [[ $# -gt 1 ]]; then
echo "usage: $0 [--staged|--all|--unstaged]" >&2
exit 1
fi
if [[ $# -eq 1 ]]; then
case "$1" in
--staged)
mode="staged"
;;
--all)
mode="all"
;;
--unstaged)
mode="unstaged"
;;
*)
echo "unknown option: $1" >&2
echo "usage: $0 [--staged|--all|--unstaged]" >&2
exit 1
;;
esac
fi
if ! git rev-parse --show-toplevel >/dev/null 2>&1; then
echo "not inside a git repository" >&2
exit 1
fi
repo_root="$(git rev-parse --show-toplevel)"
cd "$repo_root"
staged_count="$(git diff --cached --name-only | wc -l | tr -d ' ')"
untracked_files="$(git ls-files --others --exclude-standard)"
if [[ "$mode" == "auto" ]]; then
if [[ "$staged_count" != "0" ]]; then
mode="staged"
else
mode="all"
fi
fi
case "$mode" in
staged)
status_cmd=(git diff --cached --name-status)
stat_cmd=(git diff --cached --stat)
patch_cmd=(git diff --cached --unified=1 --no-color)
headline="staged changes"
;;
unstaged)
status_cmd=(git diff --name-status)
stat_cmd=(git diff --stat)
patch_cmd=(git diff --unified=1 --no-color)
headline="unstaged changes"
;;
all)
headline="all local changes"
;;
*)
echo "invalid mode: $mode" >&2
exit 1
;;
esac
echo "repo: $repo_root"
echo "branch: $(git branch --show-current)"
echo "scope: $headline"
echo
echo "status:"
git status --short
echo
if [[ -n "$untracked_files" ]]; then
echo "untracked files:"
printf '%s\n' "$untracked_files"
echo
fi
if [[ "$mode" == "all" ]]; then
echo "changed files:"
git diff HEAD --name-status
echo
echo "diff stat:"
git diff HEAD --stat
echo
echo "patch excerpt:"
git diff HEAD --unified=1 --no-color | sed -n '1,400p'
else
echo "changed files:"
"${status_cmd[@]}"
echo
echo "diff stat:"
"${stat_cmd[@]}"
echo
echo "patch excerpt:"
"${patch_cmd[@]}" | sed -n '1,400p'
fi

1
upstream_ref/xllm/.agents/skills/tilelang-api-best-practices Symbolic link
View File

@@ -0,0 +1 @@
../../third_party/tilelang-ascend/.agents/skills/tilelang-custom-skill/tilelang-api-best-practices

140
upstream_ref/xllm/.agents/skills/tilelang-ascend-kernel/SKILL.md Normal file
View File

@@ -0,0 +1,140 @@
---
name: tilelang-ascend-kernel
description: Use when the user wants to add, modify, debug, or review an xLLM TileLang Ascend kernel or specialization, including Python kernel definitions, generated Ascend-C source, runtime wrapper dispatch, TileLang CMake wiring, and NPU tests.
---
# TileLang Ascend Kernel
## When to use
Use this skill when the task involves any of the following in the xLLM repo:
- `xllm/compiler/tilelang/targets/ascend/kernels/*.py`
- `xllm/core/kernels/npu/tilelang/*_wrapper.cpp`
- `xllm/core/kernels/npu/tilelang/CMakeLists.txt`
- generated TileLang artifacts such as `manifest.json`, `registry.inc`, or specialization `.cpp`
Run build and test commands inside the NPU container.
Run TileLang commands from the xLLM repo root, not from an installed-package environment.
## Entry points and TL_ROOT
- Prefer `python xllm/compiler/tilelang_launcher.py ...` for end-to-end TileLang compile flows.
- From the xLLM repo root, use `export TL_ROOT=$PWD/third_party/tilelang-ascend` for xLLM TileLang tooling and verify `test -f "$TL_ROOT/tilelang/__init__.py"`.
- Before any raw script does `import tilelang`, run `export TL_ROOT=$PWD/third_party/tilelang-ascend && source third_party/tilelang-ascend/set_env.sh`, then execute the script.
- Do not run kernel files directly with `python rope.py`; use module execution because these files rely on relative imports.
- For direct kernel-script debugging, run them as modules and pass required CLI args:
```bash
cd xllm
python -m compiler.tilelang.targets.ascend.kernels.rope \
--output ../.tmp/rope.cpp
# Expected: [INFO] RoPE output matches torch reference
```
- The same module-style rule applies to other kernel files under `xllm/compiler/tilelang/targets/ascend/kernels/`.
## Primary Reference And Mode Preference
Primary reference:
- `third_party/tilelang-ascend/docs/TileLang-Ascend Programming Guide.md`
Use `third_party/tilelang-ascend/.agents/skills/tilelang-custom-skill/tilelang-api-best-practices/references/api-tile-ops.md` when the task depends on `T.tile.xxx` semantics such as `compare`, `select`, `cast`, or other vector intrinsics.
Default to Expert mode for xLLM Ascend kernels:
- prefer `T.tile.xxx`, explicit UB/shared allocation, and explicit `T.copy`
- prefer explicit `T.serial` control for row/block traversal
- do not introduce Developer mode `T.Parallel` unless the kernel is a clearly tile-local element-wise expression and the change does not reduce control over UB usage, temporary buffers, or exact runtime semantics
- when translating Triton kernels, preserve the Triton runtime semantics first, then choose the smallest Expert-mode lowering that matches them
## Common Triton To TileLang-Ascend Semantics
Use this table as the quick semantic mapping when translating Triton kernels:
| Triton pattern | TileLang-Ascend pattern | Notes |
| --- | --- | --- |
| `x + y`, `x - y`, `x * y`, `x / y` | `T.tile.add/sub/mul/div` | Prefer tile ops in Expert-style vector code instead of hand-written scalar loops. |
| `tl.exp(x)`, `tl.log(x)`, `tl.abs(x)` | `T.tile.exp`, `T.tile.ln`, `T.tile.abs` | TileLang uses `ln`, not `log`. |
| `x.to(tl.float32)` or `tl.cast(...)` | `T.tile.cast(dst, src, "CAST_NONE", count)` | Pick a non-default cast mode only when rounding semantics are required. |
| `x <= y`, `x < y`, `x >= y`, `x == y` | `T.tile.compare(mask, x, y, "LE"/"LT"/"GE"/"EQ")` | `T.tile.compare` produces a bit mask, not a float tensor. |
| `tl.where(cond, a, b)` | `T.tile.select(dst, selMask, a, b, selMode)` | API-level match. If `cond` is a comparison expression such as `x <= y`, materialize `selMask` with `T.tile.compare(...)` first; use the matching `VSEL_*` mode for tensor-tensor or tensor-scalar selection. |
| `tl.full(shape, value, dtype)` | allocate buffer + `T.tile.fill(dst, value)` | Separate allocation from initialization. |
| `tl.arange(0, N)` | `T.tile.createvecindex(dst, 0)` or explicit loop indices | Prefer `createvecindex` only when the kernel truly needs a vector index tensor. |
Rules for semantic-preserving lowering:
- Preserve Triton control-flow, masking, and parameter semantics. Do not substitute a numerically similar formula unless the runtime-visible behavior is unchanged for the supported input domain.
- Keep the kernel ABI aligned with the lowering. Every runtime parameter must either participate in the TileLang implementation or be removed from the interface.
- Add targeted tests for branch, mask, and boundary behavior. Do not rely only on random inputs if some paths are hit only under specific values.
- Choose the correct `VSEL_*` mode based on the source operands. `VSEL_CMPMASK_SPR` is the natural match for a mask produced by `T.tile.compare`; `VSEL_TENSOR_SCALAR_MODE` and `VSEL_TENSOR_TENSOR_MODE` are for explicit tensor/scalar or tensor/tensor selection modes.
## New kernel
Follow this order:
1. Implement `build_<kernel>_kernel(...)`
2. Implement `generate_source(...)`
3. Declare `DISPATCH_SCHEMA` and `SPECIALIZATIONS`
4. Run TileLang compilation once and inspect `registry.inc`
5. Add or update `<kernel>_wrapper.cpp`
6. Register the kernel in `xllm/core/kernels/npu/tilelang/CMakeLists.txt` with:
- `tilelang_register_runtime_kernel(NAME <kernel> WRAPPER_SRCS <srcs...>)`
For wrapper work:
- do kernel precision alignment on the Python side first (`build_<kernel>_kernel(...)` / `generate_source(...)`), not in the C++ wrapper
- handwrite tensor checks, layout transforms, and `build_runtime_specialization(...)`
- use generated `make_<kernel>_specialization(...)` and `find_<kernel>_kernel_entry(...)`
- do not handwrite kernel-specific specialization structs or kernel fn typedefs
- **never use `permute`, `contiguous`, `transpose`, `reshape` (when non-trivial), `clone`, or any operation that triggers a device memory copy in the wrapper**. The wrapper must only pass pointers to existing tensors. If the kernel needs a different layout, handle it inside the kernel itself or require the caller to provide the correct layout.
## New specialization
Use this path when the kernel logic and wrapper ABI stay the same.
1. Update the existing kernel's `SPECIALIZATIONS`
2. Confirm every runtime dispatch field still matches `DISPATCH_SCHEMA`
3. Re-run TileLang compilation
4. Check that `registry.inc` contains the new entry
5. Check that the wrapper's `build_runtime_specialization(...)` still constructs matching values
## Debug generated Ascend-C
When the task is to inspect codegen or compare two kernel implementations, use an isolated output root:
```bash
python xllm/compiler/tilelang_launcher.py compile-kernels \
--target ascend \
--device a3 \
--output-root .tmp/tilelang_debug \
--kernels <kernel> \
--force
```
Then inspect:
- `.tmp/tilelang_debug/targets/ascend/<kernel>/<variant_key>/<kernel>_<variant_key>_kernel.cpp`
- `.tmp/tilelang_debug/targets/ascend/<kernel>/registry.inc`
- `.tmp/tilelang_debug/targets/ascend/<kernel>/manifest.json`
Use this path before changing wrapper code when you need to understand generated symbols, field order, or ABI.
## Validate
Prefer the narrowest command first:
- `python -m py_compile xllm/compiler/tilelang/targets/ascend/kernels/<kernel>.py`
- `cd xllm && python -m compiler.tilelang.targets.ascend.kernels.<kernel> --output ../.tmp/<kernel>.cpp`
- `python xllm/compiler/tilelang_launcher.py prepare-ascend`
- `python setup.py test --test-name <wrapper_test_target> --device npu`
## References
Read `docs/en/dev_guide/tilelang_ascend_kernel_dev.md` for mechanism details.
Use `rope` as the concrete template:
- `xllm/compiler/tilelang/targets/ascend/kernels/rope.py`
- `xllm/core/kernels/npu/tilelang/rope_wrapper.cpp`
- `xllm/core/kernels/npu/tilelang/CMakeLists.txt`

1
upstream_ref/xllm/.agents/skills/tilelang-debug-helper Symbolic link
View File

@@ -0,0 +1 @@
../../third_party/tilelang-ascend/.agents/skills/tilelang-custom-skill/tilelang-debug-helper

1
upstream_ref/xllm/.agents/skills/tilelang-expert-to-developer Symbolic link
View File

@@ -0,0 +1 @@
../../third_party/tilelang-ascend/.agents/skills/tilelang-custom-skill/tilelang-expert-to-developer

13
upstream_ref/xllm/.clang-format Normal file
View File

@@ -0,0 +1,13 @@
Language: Cpp
BasedOnStyle: Google
UseTab: Never
IndentWidth: 2
ColumnLimit: 80
BinPackParameters: false
BinPackArguments: false
ExperimentalAutoDetectBinPacking: false
AllowAllParametersOfDeclarationOnNextLine: false
DerivePointerAlignment: false
PointerAlignment: Left
...

1
upstream_ref/xllm/.claude/skills Symbolic link
View File

@@ -0,0 +1 @@
../.agents/skills/

6
upstream_ref/xllm/.gemini/config.yaml Normal file
View File

@@ -0,0 +1,6 @@
# https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github
have_fun: false # Just review the code
code_review:
comment_severity_threshold: HIGH # Reduce quantity of comments
pull_request_opened:
summary: false # Don't summarize the PR in a separate comment

1
upstream_ref/xllm/.gemini/styleguide.md Symbolic link
View File

@@ -0,0 +1 @@
../.agents/skills/code-review/references/custom-code-style.md

1
upstream_ref/xllm/.github/CODEOWNERS vendored Normal file
View File

@@ -0,0 +1 @@
/xllm/ @yq33victor @liutongxuan @walsonyang @RobbieLeung @JimHsiung @DongheJin @XuZhang99

45
upstream_ref/xllm/.github/CONTRIBUTING.md vendored Normal file
View File

@@ -0,0 +1,45 @@
<!-- Copyright 2025 JD.com
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this project except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. -->
[English](./CONTRIBUTING.md) | [中文](./CONTRIBUTING_zh.md)
# Contribute to xLLM
+ Write / translate / fix our documentation
+ Raise questions / Answer questions
+ Provide demos, examples or test cases
+ Give suggestions or other comments
+ Paticipate in [issues](https://github.com/xxx/xLLM/issues) or [discussions](https://github.com/xxx/xLLM/discussions)
+ Pull requests
+ Sharing related research / application
+ Any other ways to improve xLLM
For developers who want to contribute to our code, here is the guidance:
## 1. Choose an issue to contribute
+ Issues with label `PR welcome`, which means:
+ A reproducible bug
+ A function in plan
## 2. Install environment for development
+ We strongly suggest you to read our **[Document](http://xxx/docs/)** before developing
+ For setting environment, please check our **[Readme file](/README.md)**
## 3. Build our project
+ You could run our demo to check whether the requirements are successfully installed:
## 4. Test
After the PR is submitted, we will format and test the code.
Our tests are still far from perfect, so you are welcomed to add tests to our project!

48
upstream_ref/xllm/.github/CONTRIBUTING_zh.md vendored Normal file
View File

@@ -0,0 +1,48 @@
<!-- Copyright 2025 JD.com
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this project except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. -->
[English](./CONTRIBUTING.md) | [中文](./CONTRIBUTING_zh.md)
# xLLM 贡献指南
xLLM致力于为每一位用户和开发者提供开放的XX，因此无论您是XX开发者还是专注于XX用户，我们都欢迎您参与我们的项目。
您可以通过以下方法为项目作出贡献：
+ 撰写/翻译/修改文档
+ 提出或回答问题
+ 提供使用或测试样例
+ 提供建议或其他评论
+ 参与[issues](https://github.com/xxx/xLLM/issues) 或[discussions](https://github.com/xxx/xLLM/discussions)
+ 提交Pull request
+ 分享相关研究或应用场景
+ 其他任何对xLLM的帮助
如果您希望参与xLLM的开发，请参考以下提示：
## 1. 选择参与贡献的issue
+ 您可以选择带有`PR welcome`标签的issue，包括:
+ 可复现的bug
+ 计划实现的功能
## 2. 配置开发环境
+ 在开发之前，可以参考我们的 **[文档](http://xxx/docs/)**
+ 关于环境配置，参见 **[Readme file](/README.md)**
## 3. 项目构建和运行
+ 您可以运行如下样例：
## 4. 测试
在pr提交之后，我们会对代码进行格式化及进一步测试。
我们的测试目前还很不完善，因此欢迎开发者为测试作出贡献！

28
upstream_ref/xllm/.github/ISSUE_TEMPLATE/bug-report.yaml vendored Normal file
View File

@@ -0,0 +1,28 @@
name: 🐛 Bug report
description: Raise an issue here if you find a bug.
title: "[Bug]: "
labels: ["bug"]
body:
- type: markdown
attributes:
value: >
#### Before submitting an issue, please make sure the issue hasn't been already addressed. please search: [existing issues](https://github.com/jd-opensource/xllm/issues).
- type: textarea
attributes:
label: Your environment
description: |
Please provide what the environment you are running.
validations:
required: true
- type: textarea
attributes:
label: 🐛 Describe the bug
description: |
Please provide a clear and concise description of what the bug is.
validations:
required: true
- type: markdown
attributes:
value: >
Thanks for report the bug!

31
upstream_ref/xllm/.github/ISSUE_TEMPLATE/feature-request.yml vendored Normal file
View File

@@ -0,0 +1,31 @@
name: 🚀 Feature request
description: Submit a request for a new feature
title: "[Feature]: "
labels: ["feature"]
body:
- type: markdown
attributes:
value: >
#### Before submitting an issue, please make sure the issue hasn't been already addressed. please search: [existing issues](https://github.com/jd-opensource/xllm/issues).
- type: textarea
attributes:
label: 🚀 The motivation and feature
description: >
A clear and concise description of the feature proposal. Please outline the motivation for the proposal.
validations:
required: true
- type: textarea
attributes:
label: Alternatives
description: >
A description of any alternative solutions or features you've considered, if any.
- type: textarea
attributes:
label: Additional context
description: >
Add any other context or screenshots about the feature request.
- type: markdown
attributes:
value: >
Thanks for contributing 🎉!

21
upstream_ref/xllm/.github/ISSUE_TEMPLATE/question.yaml vendored Normal file
View File

@@ -0,0 +1,21 @@
name: ❓ Question
description: Submit a question
title: "[Question]: "
labels: ["question"]
body:
- type: markdown
attributes:
value: >
#### Before submitting an issue, please make sure the issue hasn't been already addressed. please search: [existing issues](https://github.com/jd-opensource/xllm/issues).
- type: textarea
attributes:
label: ❓ Describe the question
description: |
Please provide a clear and concise description of your question.
validations:
required: true
- type: markdown
attributes:
value: >
Thanks for contributing 🎉!

122
upstream_ref/xllm/.github/workflows/build_x86_64_cuda.yaml vendored Normal file
View File

@@ -0,0 +1,122 @@
name: xLLM Build x86_64 CUDA
on:
workflow_dispatch:
push:
branches: [main]
paths-ignore:
- '.github/**'
- 'cibuild/**'
- 'cmake/**'
- 'docs/**'
- 'third_party/**'
- 'tools/**'
- '*.md'
- '*.txt'
- '*.yml'
pull_request:
branches: [main]
types: [opened, synchronize, reopened]
paths-ignore:
- 'cmake/**'
- 'docs/**'
- 'third_party/**'
- 'tools/**'
- '*.md'
- '*.txt'
- '*.yml'
pull_request_review:
types: [submitted]
paths:
- '.github/**.yaml'
- 'cibuild/**.sh'
- 'setup.py'
- 'examples/generate.py'
env:
JOBNAME: xllm-x86_64-cuda-cibuild-${{ github.run_id }}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
jobs:
# need to review code first when sensitive files are modified.
check-sensitive:
runs-on: [self-hosted]
outputs:
requires_approval: ${{ steps.check_sensitive.outputs.requires_approval }}
do_build: ${{ steps.decide.outputs.do_build }}
steps:
- name: Checkout Code
uses: actions/checkout@v4
with:
fetch-depth: 0 # Ensure we can compare commits
- name: Install jq
run: yum install -y jq
- name: Check if sensitive files were changed
id: check_sensitive
run: |
sensitive_files=(
".github/**.yaml"
"cibuild/**.sh"
"setup.py"
)
changed_files=$(git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.sha }})
requires_approval="false"
while IFS= read -r changed_file; do
[[ -z "$changed_file" ]] && continue
for pattern in "${sensitive_files[@]}"; do
if [[ "$changed_file" == $pattern ]]; then
requires_approval="true"
break 2
fi
done
done < <(git diff --name-only "${{ github.event.pull_request.base.sha }}" "${{ github.sha }}")
echo "requires_approval=$requires_approval" >> $GITHUB_OUTPUT
- name: Decide whether to check build
id: decide
run: |
event="${{ github.event_name }}"
if [[ "$event" == "workflow_dispatch" || "$event" == "push" ]]; then
echo "do_build=true" >> $GITHUB_OUTPUT
elif [[ "$event" == "pull_request" ]]; then
if [ "${{ steps.check_sensitive.outputs.requires_approval }}" == "true" ]; then
echo "do_build=false" >> $GITHUB_OUTPUT
else
echo "do_build=true" >> $GITHUB_OUTPUT
fi
elif [[ "$event" == "pull_request_review" ]]; then
# Since pull_request_review now only triggers when sensitive files are modified,
# we only need to check if the review is approved
if [[ "${{ github.event.review.state }}" == "approved" ]]; then
echo "do_build=true" >> $GITHUB_OUTPUT
else
echo "do_build=false" >> $GITHUB_OUTPUT
fi
else
echo "do_build=false" >> $GITHUB_OUTPUT
fi
build:
needs: check-sensitive
if: >
(github.event_name == 'workflow_dispatch' || github.event_name == 'push') ||
needs.check-sensitive.outputs.do_build == 'true'
runs-on: [self-hosted]
steps:
- name: Checkout Code
timeout-minutes: 5
uses: actions/checkout@v4
with:
submodules: true
- name: Build
if: ${{ success() }}
timeout-minutes: 60
run: |
chmod +x ./cibuild/build_cuda.sh
bash cibuild/build_cuda.sh 'pip install pre-commit -i https://pypi.tuna.tsinghua.edu.cn/simple; python setup.py build --device cuda'

122
upstream_ref/xllm/.github/workflows/build_x86_64_ilu.yaml vendored Normal file
View File

@@ -0,0 +1,122 @@
name: xLLM Build x86_64 ILU
on:
workflow_dispatch:
push:
branches: [main]
paths-ignore:
- '.github/**'
- 'cibuild/**'
- 'cmake/**'
- 'docs/**'
- 'third_party/**'
- 'tools/**'
- '*.md'
- '*.txt'
- '*.yml'
pull_request:
branches: [main]
types: [opened, synchronize, reopened]
paths-ignore:
- 'cmake/**'
- 'docs/**'
- 'third_party/**'
- 'tools/**'
- '*.md'
- '*.txt'
- '*.yml'
pull_request_review:
types: [submitted]
paths:
- '.github/**.yaml'
- 'cibuild/**.sh'
- 'setup.py'
- 'examples/generate.py'
env:
JOBNAME: xllm-x86_64-ilu-cibuild-${{ github.run_id }}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
jobs:
# need to review code first when sensitive files are modified.
check-sensitive:
runs-on: [self-hosted]
outputs:
requires_approval: ${{ steps.check_sensitive.outputs.requires_approval }}
do_build: ${{ steps.decide.outputs.do_build }}
steps:
- name: Checkout Code
uses: actions/checkout@v4
with:
fetch-depth: 0 # Ensure we can compare commits
- name: Install jq
run: yum install -y jq
- name: Check if sensitive files were changed
id: check_sensitive
run: |
sensitive_files=(
".github/**.yaml"
"cibuild/**.sh"
"setup.py"
)
changed_files=$(git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.sha }})
requires_approval="false"
while IFS= read -r changed_file; do
[[ -z "$changed_file" ]] && continue
for pattern in "${sensitive_files[@]}"; do
if [[ "$changed_file" == $pattern ]]; then
requires_approval="true"
break 2
fi
done
done < <(git diff --name-only "${{ github.event.pull_request.base.sha }}" "${{ github.sha }}")
echo "requires_approval=$requires_approval" >> $GITHUB_OUTPUT
- name: Decide whether to check build
id: decide
run: |
event="${{ github.event_name }}"
if [[ "$event" == "workflow_dispatch" || "$event" == "push" ]]; then
echo "do_build=true" >> $GITHUB_OUTPUT
elif [[ "$event" == "pull_request" ]]; then
if [ "${{ steps.check_sensitive.outputs.requires_approval }}" == "true" ]; then
echo "do_build=false" >> $GITHUB_OUTPUT
else
echo "do_build=true" >> $GITHUB_OUTPUT
fi
elif [[ "$event" == "pull_request_review" ]]; then
# Since pull_request_review now only triggers when sensitive files are modified,
# we only need to check if the review is approved
if [[ "${{ github.event.review.state }}" == "approved" ]]; then
echo "do_build=true" >> $GITHUB_OUTPUT
else
echo "do_build=false" >> $GITHUB_OUTPUT
fi
else
echo "do_build=false" >> $GITHUB_OUTPUT
fi
build:
needs: check-sensitive
if: >
(github.event_name == 'workflow_dispatch' || github.event_name == 'push') ||
needs.check-sensitive.outputs.do_build == 'true'
runs-on: [self-hosted]
steps:
- name: Checkout Code
timeout-minutes: 5
uses: actions/checkout@v4
with:
submodules: true
- name: Build
if: ${{ success() }}
timeout-minutes: 60
run: |
chmod +x ./cibuild/build_ilu.sh
bash cibuild/build_ilu.sh 'pip3 install pre-commit -i https://pypi.tuna.tsinghua.edu.cn/simple && python setup.py build --device ilu'

122
upstream_ref/xllm/.github/workflows/build_x86_64_mlu.yaml vendored Normal file
View File

@@ -0,0 +1,122 @@
name: xLLM Build x86_64 MLU
on:
workflow_dispatch:
push:
branches: [main]
paths-ignore:
- '.github/**'
- 'cibuild/**'
- 'cmake/**'
- 'docs/**'
- 'third_party/**'
- 'tools/**'
- '*.md'
- '*.txt'
- '*.yml'
pull_request:
branches: [main]
types: [opened, synchronize, reopened]
paths-ignore:
- 'cmake/**'
- 'docs/**'
- 'third_party/**'
- 'tools/**'
- '*.md'
- '*.txt'
- '*.yml'
pull_request_review:
types: [submitted]
paths:
- '.github/**.yaml'
- 'cibuild/**.sh'
- 'setup.py'
- 'examples/generate.py'
env:
JOBNAME: xllm-x86_64-mlu-cibuild-${{ github.run_id }}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
jobs:
# need to review code first when sensitive files are modified.
check-sensitive:
runs-on: [self-hosted]
outputs:
requires_approval: ${{ steps.check_sensitive.outputs.requires_approval }}
do_build: ${{ steps.decide.outputs.do_build }}
steps:
- name: Checkout Code
uses: actions/checkout@v4
with:
fetch-depth: 0 # Ensure we can compare commits
- name: Install jq
run: yum install -y jq
- name: Check if sensitive files were changed
id: check_sensitive
run: |
sensitive_files=(
".github/**.yaml"
"cibuild/**.sh"
"setup.py"
)
changed_files=$(git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.sha }})
requires_approval="false"
while IFS= read -r changed_file; do
[[ -z "$changed_file" ]] && continue
for pattern in "${sensitive_files[@]}"; do
if [[ "$changed_file" == $pattern ]]; then
requires_approval="true"
break 2
fi
done
done < <(git diff --name-only "${{ github.event.pull_request.base.sha }}" "${{ github.sha }}")
echo "requires_approval=$requires_approval" >> $GITHUB_OUTPUT
- name: Decide whether to check build
id: decide
run: |
event="${{ github.event_name }}"
if [[ "$event" == "workflow_dispatch" || "$event" == "push" ]]; then
echo "do_build=true" >> $GITHUB_OUTPUT
elif [[ "$event" == "pull_request" ]]; then
if [ "${{ steps.check_sensitive.outputs.requires_approval }}" == "true" ]; then
echo "do_build=false" >> $GITHUB_OUTPUT
else
echo "do_build=true" >> $GITHUB_OUTPUT
fi
elif [[ "$event" == "pull_request_review" ]]; then
# Since pull_request_review now only triggers when sensitive files are modified,
# we only need to check if the review is approved
if [[ "${{ github.event.review.state }}" == "approved" ]]; then
echo "do_build=true" >> $GITHUB_OUTPUT
else
echo "do_build=false" >> $GITHUB_OUTPUT
fi
else
echo "do_build=false" >> $GITHUB_OUTPUT
fi
build:
needs: check-sensitive
if: >
(github.event_name == 'workflow_dispatch' || github.event_name == 'push') ||
needs.check-sensitive.outputs.do_build == 'true'
runs-on: [self-hosted]
steps:
- name: Checkout Code
timeout-minutes: 5
uses: actions/checkout@v4
with:
submodules: true
- name: Build
if: ${{ success() }}
timeout-minutes: 60
run: |
chmod +x ./cibuild/build_mlu.sh
bash cibuild/build_mlu.sh 'pip install pre-commit -i https://pypi.tuna.tsinghua.edu.cn/simple; python setup.py build --device mlu'

136
upstream_ref/xllm/.github/workflows/build_x86_64_npu.yaml vendored Normal file
View File

@@ -0,0 +1,136 @@
name: xLLM Build x86_64 NPU
on:
workflow_dispatch:
push:
branches: [main]
paths-ignore:
- '.github/**'
- 'cibuild/**'
- 'cmake/**'
- 'docs/**'
- 'third_party/**'
- 'tools/**'
- '*.md'
- '*.txt'
- '*.yml'
pull_request:
branches: [main]
types: [opened, synchronize, reopened]
paths-ignore:
- 'cmake/**'
- 'docs/**'
- 'third_party/**'
- 'tools/**'
- '*.md'
- '*.txt'
- '*.yml'
pull_request_review:
types: [submitted]
paths:
- '.github/**.yaml'
- 'cibuild/**.sh'
- 'setup.py'
- 'examples/generate.py'
env:
JOBNAME: xllm-x86_64-npu-cibuild-${{ github.run_id }}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
jobs:
# need to review code first when sensitive files are modified.
check-sensitive:
runs-on: [self-hosted]
outputs:
requires_approval: ${{ steps.check_sensitive.outputs.requires_approval }}
do_build: ${{ steps.decide.outputs.do_build }}
steps:
- name: Checkout Code
uses: actions/checkout@v4
with:
fetch-depth: 0 # Ensure we can compare commits
- name: Install jq
run: yum install -y jq
- name: Check if sensitive files were changed
id: check_sensitive
run: |
sensitive_files=(
".github/**.yaml"
"cibuild/**.sh"
"setup.py"
"examples/generate.py"
)
changed_files=$(git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.sha }})
requires_approval="false"
while IFS= read -r changed_file; do
[[ -z "$changed_file" ]] && continue
for pattern in "${sensitive_files[@]}"; do
if [[ "$changed_file" == $pattern ]]; then
requires_approval="true"
break 2
fi
done
done < <(git diff --name-only "${{ github.event.pull_request.base.sha }}" "${{ github.sha }}")
echo "requires_approval=$requires_approval" >> $GITHUB_OUTPUT
- name: Decide whether to check build
id: decide
run: |
event="${{ github.event_name }}"
if [[ "$event" == "workflow_dispatch" || "$event" == "push" ]]; then
echo "do_build=true" >> $GITHUB_OUTPUT
elif [[ "$event" == "pull_request" ]]; then
if [ "${{ steps.check_sensitive.outputs.requires_approval }}" == "true" ]; then
echo "do_build=false" >> $GITHUB_OUTPUT
else
echo "do_build=true" >> $GITHUB_OUTPUT
fi
elif [[ "$event" == "pull_request_review" ]]; then
# Since pull_request_review now only triggers when sensitive files are modified,
# we only need to check if the review is approved
if [[ "${{ github.event.review.state }}" == "approved" ]]; then
echo "do_build=true" >> $GITHUB_OUTPUT
else
echo "do_build=false" >> $GITHUB_OUTPUT
fi
else
echo "do_build=false" >> $GITHUB_OUTPUT
fi
build:
needs: check-sensitive
if: >
(github.event_name == 'workflow_dispatch' || github.event_name == 'push') ||
needs.check-sensitive.outputs.do_build == 'true'
runs-on: [self-hosted]
steps:
- name: Prepare submodule checkout
run: |
git config --global url."https://gitcode.com/xLLM-AI/tvm".insteadOf "https://github.com/TileLang/tvm"
git config --global url."https://gitcode.com/xLLM-AI/composable_kernel".insteadOf "https://github.com/ROCm/composable_kernel"
git config --global url."https://gitcode.com/xLLM-AI/cutlass".insteadOf "https://github.com/NVIDIA/cutlass"
if [ -d .git/modules ]; then
find .git/modules -type f -name '*.lock' -print -delete || true
fi
rm -rf .git/modules/third_party/tilelang-ascend
rm -rf third_party/tilelang-ascend
- name: Checkout Code
timeout-minutes: 5
uses: actions/checkout@v4
with:
clean: true
submodules: recursive
- name: Build
if: ${{ success() }}
timeout-minutes: 60
run: |
chmod +x ./cibuild/build_npu.sh
bash cibuild/build_npu.sh 'pip install pre-commit -i https://pypi.tuna.tsinghua.edu.cn/simple; python setup.py bdist_wheel; pip install dist/* --force-reinstall; python examples/generate.py --model="/export/home/models/Qwen2-7B-Instruct" --devices="npu:7"'

104
upstream_ref/xllm/.github/workflows/check_format.yml vendored Normal file
View File

@@ -0,0 +1,104 @@
name: CheckFormat
on:
workflow_dispatch:
push:
branches: [main]
paths-ignore:
- '.github/**'
- 'cibuild/**'
- 'cmake/**'
- 'docs/**'
- 'third_party/**'
- 'tools/**'
pull_request:
types: [opened, synchronize, reopened]
branches: [main]
paths-ignore:
- '.github/**'
- 'cibuild/**'
- 'cmake/**'
- 'docs/**'
- 'third_party/**'
- 'tools/**'
jobs:
format-check:
runs-on: [self-hosted]
steps:
- name: Install clang-format
run: |
pip install clang-format==20.1.6
clang-format --version
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Determine base commit for comparison
id: get_base_commit
run: |
# pull_request action
if [ "${{ github.event_name }}" = "pull_request" ]; then
echo "base_commit=${{ github.event.pull_request.base.sha }}" >> $GITHUB_OUTPUT
else
# push action
echo "base_commit=${{ github.sha }}~1" >> $GITHUB_OUTPUT
fi
- name: Verify clang-format configuration
run: |
if [ ! -f ".clang-format" ]; then
echo "❌ .clang-format file not found in repository root"
exit 1
fi
clang-format --style=file --dump-config > /dev/null || {
echo "❌ .clang-format file has invalid format"
exit 1
}
- name: Check code format
shell: /usr/bin/bash {0}
run: |
BASE_COMMIT="${{ steps.get_base_commit.outputs.base_commit }}"
CLANG_FORMAT_FILE="$(pwd)/.clang-format"
# ignore path
IGNORED_PATHS=(
"^.github/.*"
"^cibuild/.*"
"^cmake/.*"
"^docs/.*"
"^third_party/.*"
"^tools/.*"
)
# igonore files
FILES=$(git diff --name-only "$BASE_COMMIT" -- '*.c' '*.h' '*.cc' '*.cp' '*.cpp' '*.c++' '*.cxx' '*.hh' '*.hpp' '*.hxx' '*.inc' '*.cu' '*.cuh' | \
grep -v -E "$(IFS=\|; echo "${IGNORED_PATHS[*]}")")
if [ -z "$FILES" ]; then
echo "✅ No files to check (all excluded)"
exit 0
fi
# run clang-format
diff=$(git-clang-format \
--style=file:"$CLANG_FORMAT_FILE" \
--extensions="c,h,cc,cp,cpp,c++,cxx,hh,hpp,hxx,inc,cu,cuh" \
--commit "$BASE_COMMIT" \
--diff \
-- $FILES)
# check diff
if [ "$diff" = "no modified files to format" ] || [ "$diff" = "clang-format did not modify any files" ]; then
echo "✅ Code format is correct"
exit 0
fi
printf "\n❌ You have introduced coding style breakages.\n"
printf "\n\033[1mSuggested changes:\n\n"
echo "$diff"
exit 1

61
upstream_ref/xllm/.gitignore vendored Normal file
View File

@@ -0,0 +1,61 @@
# Visual Studio Code
/.vscode*
# Idea
/.idea
/cmake-build-debug/
/cmake-build-release/
# CMake
/build*
# vcpkg
/.vcpkg*
# cache
/.*cache
# deps
/.deps
# libtorch
/libtorch
# tests
/Testing*
# rust
Cargo.lock
# distribution / packaging
.Python
build/
dist/
eggs/
.eggs/
sdist/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# Python module builds
*.egg-info/
xllm/*.pyd
xllm/*.so
xllm/version.py
__pycache__/
.pkl_memoize_py3/
# compile_commands.json from nvbench
compile_commands.json
# ascend kernel meta files
/kernel_meta
# local files
/local
/logs
/log

57
upstream_ref/xllm/.gitmodules vendored Executable file
View File

@@ -0,0 +1,57 @@
[submodule "third_party/brpc"]
path = third_party/brpc
url = https://gitcode.com/xLLM-AI/brpc.git
fetchRecurseSubmodules = false
[submodule "third_party/cpprestsdk"]
path = third_party/cpprestsdk
url = https://gitcode.com/xLLM-AI/cpprestsdk.git
fetchRecurseSubmodules = false
[submodule "third_party/minja"]
path = third_party/minja
url = https://gitcode.com/xLLM-AI/minja.git
fetchRecurseSubmodules = false
[submodule "third_party/sentencepiece"]
path = third_party/sentencepiece
url = https://gitcode.com/xLLM-AI/sentencepiece.git
fetchRecurseSubmodules = false
[submodule "third_party/smhasher"]
path = third_party/smhasher
url = https://gitcode.com/xLLM-AI/smhasher.git
fetchRecurseSubmodules = false
[submodule "third_party/xllm_ops"]
path = third_party/xllm_ops
url = https://gitcode.com/xLLM-AI/xllm_ops.git
fetchRecurseSubmodules = true
[submodule "third_party/etcd_cpp_apiv3"]
path = third_party/etcd_cpp_apiv3
url = https://gitcode.com/xLLM-AI/etcd-cpp-apiv3.git
fetchRecurseSubmodules = false
[submodule "third_party/spdlog"]
path = third_party/spdlog
url = https://gitcode.com/xLLM-AI/spdlog.git
fetchRecurseSubmodules = false
[submodule "third_party/Mooncake"]
path = third_party/Mooncake
url = https://gitcode.com/xLLM-AI/Mooncake.git
fetchRecurseSubmodules = false
[submodule "third_party/torch_npu_ops"]
path = third_party/torch_npu_ops
url = https://gitcode.com/xLLM-AI/torch_npu_ops.git
fetchRecurseSubmodules = false
[submodule "third_party/cutlass"]
path = third_party/cutlass
url = https://gitcode.com/xLLM-AI/cutlass.git
fetchRecurseSubmodules = false
[submodule "third_party/xllm_atb_layers"]
path = third_party/xllm_atb_layers
url = https://gitcode.com/xLLM-AI/xllm_atb_layers.git
fetchRecurseSubmodules = false
[submodule "third_party/xxHash"]
path = third_party/xxHash
url = https://gitcode.com/xLLM-AI/xxHash.git
fetchRecurseSubmodules = false
[submodule "third_party/tilelang-ascend"]
path = third_party/tilelang-ascend
url = https://gitcode.com/xLLM-AI/tilelang-ascend.git
branch = ascendc_pto
fetchRecurseSubmodules = true

11
upstream_ref/xllm/.pre-commit-config.yaml Executable file
View File

@@ -0,0 +1,11 @@
# pre-commit install
# pre-commit run --all-files
repos:
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: v20.1.6
hooks:
- id: clang-format
types_or: [c++, c, cuda]
exclude: ^(cibuild/|tools/|third_party/|cmake/|build/|.*\.ptx\.h$)

51
upstream_ref/xllm/AGENTS.md Normal file
View File

@@ -0,0 +1,51 @@
# xLLM Coding Agent Instructions
## Directory Structure
```
├── xllm/
| : main source folder
│ ├── api_service/ # code for api services
│ ├── c_api/ # code for c api
│ ├── cc_api/ # code for cc api
│ ├── core/
│ │ : xllm core features folder
│ │ ├── common/
│ │ ├── distributed_runtime/ # code for distributed and pd serving
│ │ ├── framework/ # code for execution orchestration
│ │ ├── kernels/ # adaption for npu kernels adaption
│ │ ├── layers/ # model layers impl
│ │ ├── platform/ # adaption for various platform
│ │ ├── runtime/ # code for worker and executor
│ │ ├── scheduler/ # code for batch and pd scheduler
│ │ └── util/
│ ├── function_call # code for tool call parser
│ ├── models/ # models impl
│ ├── parser/ # parser reasoning
│ ├── processors/ # code for vlm pre-processing
│ ├── proto/ # communication protocol
│ ├── pybind/ # code for python bind
| └── server/ # xLLM server
├── examples/ # examples of calling xLLM
├── tools/ # code for npu time generations
└── xllm.cpp # entrypoint of xLLM
```
## Code Style Guide
* Before editing, creating, refactoring, or reviewing any file under `xllm/`, you **MUST** read [custom-code-style.md](.agents/skills/code-review/references/custom-code-style.md).
* The file above is a **required instruction file**, not an optional reference. Do not skip reading it.
* Apply the rules in [custom-code-style.md](.agents/skills/code-review/references/custom-code-style.md) to **both code generation and code review**.
* Follow DDD (Domain Driven Design) principles, and keep the codebase clean and maintainable.
* If [custom-code-style.md](.agents/skills/code-review/references/custom-code-style.md) specifies a rule, that rule takes precedence over the Google C++/Python Style Guide.
* Use the Google C++/Python Style Guide only for cases not specified in [custom-code-style.md](.agents/skills/code-review/references/custom-code-style.md).
## Review Instructions
* For code review tasks, you **MUST** first read [code-review/SKILL.md](.agents/skills/code-review/SKILL.md).
* Then read [custom-code-style.md](.agents/skills/code-review/references/custom-code-style.md) and apply it during the review.
* Review code changes for quality, security, performance, correctness, and maintainability following the project-specific standards.
* Review code changes for DDD (Domain Driven Design) principles, and keep the codebase clean and maintainable.
* Use the review workflow, checklist, severity rules, and output format defined in [code-review/SKILL.md](.agents/skills/code-review/SKILL.md).
* Apply the Google C++/Python Style Guide only when the project-specific style guide does not define the rule.
* Focus the review on the requested diff or changed files. Do not comment on unrelated code.

1
upstream_ref/xllm/CLAUDE.md Symbolic link
View File

@@ -0,0 +1 @@
AGENTS.md

605
upstream_ref/xllm/CMakeLists.txt Executable file
View File

@@ -0,0 +1,605 @@
cmake_minimum_required(VERSION 3.26)
set_property(GLOBAL PROPERTY USE_FOLDERS ON)
option(USE_NPU "Enable NPU support" OFF)
option(USE_MLU "Enable MLU support" OFF)
option(USE_ILU "Enable ILU support" OFF)
option(USE_CUDA "Enable CUDA support" OFF)
option(USE_MUSA "Enable MUSA support" OFF)
add_compile_definitions(YLT_ENABLE_IBV)
add_definitions(-DYLT_ENABLE_IBV)
set(YLT_ENABLE_IBV ON)
if(DEVICE_ARCH STREQUAL "ARM")
set(CMAKE_SYSTEM_PROCESSOR aarch64)
set(RUST_TARGET aarch64-unknown-linux-gnu)
endif()
if(USE_NPU)
# Override Mooncake option for mooncake transfer engine
# CANN 8.5+ migration: ascend_direct_transport replaces ascend_transport
set(USE_ASCEND_DIRECT ON CACHE BOOL "Enable ADXL engine for Ascend NPU" FORCE)
execute_process(
COMMAND git -c "safe.directory=${CMAKE_SOURCE_DIR}/third_party/xllm_ops" -C "${CMAKE_SOURCE_DIR}/third_party/xllm_ops" rev-parse HEAD
OUTPUT_VARIABLE XLLM_OPS_GIT_HEAD
OUTPUT_STRIP_TRAILING_WHITESPACE
ERROR_QUIET
)
if(DEFINED ENV{ASCEND_HOME_PATH} AND NOT "$ENV{ASCEND_HOME_PATH}" STREQUAL "")
set(XLLM_OPS_MARKER_PATH "$ENV{ASCEND_HOME_PATH}/opp/vendors/xllm/.xllm_ops_git_head")
else()
set(XLLM_OPS_MARKER_PATH "/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/xllm/.xllm_ops_git_head")
endif()
if(NOT DEFINED XLLM_OPS_GIT_HEAD_CACHED OR NOT XLLM_OPS_GIT_HEAD STREQUAL XLLM_OPS_GIT_HEAD_CACHED)
message(STATUS "xllm_ops git HEAD changed; running precompile via execute_process")
execute_process(
COMMAND bash ${CMAKE_SOURCE_DIR}/third_party/xllm_ops/build.sh
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/third_party/xllm_ops
RESULT_VARIABLE XLLM_OPS_RESULT
)
if(NOT XLLM_OPS_RESULT EQUAL 0)
message(FATAL_ERROR "Failed to precompile xllm ops, error code: ${XLLM_OPS_RESULT}")
endif()
set(XLLM_OPS_GIT_HEAD_CACHED "${XLLM_OPS_GIT_HEAD}" CACHE INTERNAL "" FORCE)
get_filename_component(XLLM_OPS_MARKER_DIR "${XLLM_OPS_MARKER_PATH}" DIRECTORY)
file(MAKE_DIRECTORY "${XLLM_OPS_MARKER_DIR}")
file(WRITE "${XLLM_OPS_MARKER_PATH}" "${XLLM_OPS_GIT_HEAD}\n")
message(STATUS "xllm ops precompiled and HEAD cache updated")
else()
message(STATUS "xllm_ops git HEAD unchanged; skipping precompile")
endif()
endif()
enable_testing()
if(NOT TARGET all_tests)
add_custom_target(all_tests)
endif()
if(NOT TARGET export_module)
add_custom_target(export_module)
endif()
if (CMAKE_BUILD_TYPE STREQUAL "Release")
add_compile_options(-O3)
endif()
option(USE_MSPTI "Enable MSPTI for NPU Profiling" OFF)
if(USE_MSPTI)
add_definitions(-DUSE_MSPTI)
endif()
option(USE_CCACHE "Attempt using CCache to wrap the compilation" ON)
option(USE_CXX11_ABI "Use the new C++-11 ABI, which is not backwards compatible." OFF)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS ON)
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fPIC")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fPIC")
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
set(CMAKE_COLOR_DIAGNOSTICS ON)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
set(CMAKE_MESSAGE_LOG_LEVEL STATUS)
set(CMAKE_VERBOSE_MAKEFILE ON)
if(POLICY CMP0135)
# CMP0135: ExternalProject ignores timestamps in archives by default for the URL download method.
cmake_policy(SET CMP0135 NEW)
endif()
function(parse_make_options options prefix)
foreach(option ${options})
string(REGEX REPLACE "(-D|-)" "" option ${option})
string(REPLACE "=" ";" option ${option})
list(GET option 0 option_name)
list(GET option 1 option_value)
set(${prefix}_${option_name}
${option_value}
PARENT_SCOPE)
endforeach()
endfunction()
# Set default build type
if(NOT CMAKE_BUILD_TYPE)
message(STATUS "Build type not set - defaulting to Release")
set(CMAKE_BUILD_TYPE "Release"
CACHE STRING "Choose the type of build from: Debug Release RelWithDebInfo MinSizeRel Coverage."
FORCE
)
endif()
# Convert the bool variable to integer.
if(USE_CXX11_ABI)
set(USE_CXX11_ABI 1)
message(STATUS "Using the C++-11 ABI.")
else()
set(USE_CXX11_ABI 0)
message(STATUS "Using the pre C++-11 ABI.")
endif()
if(USE_CCACHE)
find_program(CCACHE_PROGRAM ccache)
if(CCACHE_PROGRAM)
set(CMAKE_C_COMPILER_LAUNCHER "${CCACHE_PROGRAM}" CACHE STRING "C compiler launcher")
set(CMAKE_CXX_COMPILER_LAUNCHER "${CCACHE_PROGRAM}" CACHE STRING "CXX compiler launcher")
message(STATUS "Using ccache: ${CCACHE_PROGRAM}")
if (DEFINED ENV{CCACHE_DIR})
message(STATUS "Using CCACHE_DIR: $ENV{CCACHE_DIR}")
endif()
else()
message(WARNING "Could not find ccache. Consider installing ccache to speed up compilation.")
endif()
endif()
# if defined, create and use the default binary cache for vcpkg
if (DEFINED ENV{VCPKG_DEFAULT_BINARY_CACHE})
file(MAKE_DIRECTORY $ENV{VCPKG_DEFAULT_BINARY_CACHE})
message(STATUS "Using VCPKG_DEFAULT_BINARY_CACHE: $ENV{VCPKG_DEFAULT_BINARY_CACHE}")
endif()
if (DEFINED ENV{DEPENDENCES_ROOT})
message(STATUS "Using DEPENDENCES_ROOT: $ENV{DEPENDENCES_ROOT}")
endif()
if(USE_ILU)
set(CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}/cmake/Modules;${CMAKE_MODULE_PATH}")
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
set(CMAKE_CUDA_ARCHITECTURES "ivcore11")
set(WARNINGS_AS_ERRORS OFF)
if (CMAKE_CXX_COMPILER_ID MATCHES "Clang")
add_definitions(
-Wno-c++11-narrowing
-Wno-thread-safety-analysis
)
endif()
endif()
# configure vcpkg
# have to set CMAKE_TOOLCHAIN_FILE before first project call.
# if (DEFINED ENV{VCPKG_ROOT} AND NOT DEFINED CMAKE_TOOLCHAIN_FILE)
if (DEFINED ENV{VCPKG_ROOT})
set(CMAKE_TOOLCHAIN_FILE "$ENV{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake"
CACHE STRING "Vcpkg toolchain file")
message(STATUS "VCPKG_ROOT found, using vcpkg at $ENV{VCPKG_ROOT}")
else()
include(FetchContent)
if (DEFINED ENV{DEPENDENCES_ROOT})
set(VCPKG_SOURCE_DIR $ENV{DEPENDENCES_ROOT}/vcpkg-src)
else()
set(VCPKG_SOURCE_DIR ${FETCHCONTENT_BASE_DIR}/vcpkg-src)
endif()
if (USE_CXX11_ABI)
FetchContent_Declare(vcpkg
GIT_REPOSITORY "https://github.com/microsoft/vcpkg.git"
GIT_TAG "2024.02.14"
SOURCE_DIR ${VCPKG_SOURCE_DIR}
)
else()
FetchContent_Declare(vcpkg
GIT_REPOSITORY "https://gitcode.com/xLLM-AI/vcpkg.git"
GIT_TAG "ffc42e97c866ce9692f5c441394832b86548422c" #disable cxx11_abi
SOURCE_DIR ${VCPKG_SOURCE_DIR}
)
message(STATUS "Using custom vcpkg with cxx11_abi disabled")
endif()
FetchContent_MakeAvailable(vcpkg)
message(STATUS "Downloading and using vcpkg at ${vcpkg_SOURCE_DIR}")
set(CMAKE_TOOLCHAIN_FILE ${vcpkg_SOURCE_DIR}/scripts/buildsystems/vcpkg.cmake
CACHE STRING "Vcpkg toolchain file")
endif()
set(CPPREST_EXCLUDE_WEBSOCKETS ON CACHE BOOL "Exclude websockets functionality." FORCE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-format-truncation")
list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR}/cmake)
if(USE_CUDA)
project("xllm" LANGUAGES C CXX CUDA)
find_package(CUDAToolkit REQUIRED)
elseif(USE_MUSA)
project("xllm" LANGUAGES C CXX MUSA)
add_compile_options(
-Wno-c++11-narrowing
)
else()
project("xllm" LANGUAGES C CXX)
endif()
# ---------------------------------------------------------------------------
# ARM architecture optimization (must be after project() so CXX is enabled).
# -march=native lets the compiler use NEON, SVE, LSE atomics, crypto, etc.
# that are available on the local CPU. Fall back to armv8.2-a if the flag
# is unsupported (cross-compilation scenario).
# armv8.2-a+dotprod+crypto is good for Kunpeng 920, Neoverse N1, etc.
# ---------------------------------------------------------------------------
if(DEVICE_ARCH STREQUAL "ARM")
include(CheckCXXCompilerFlag)
check_cxx_compiler_flag("-march=armv8.2-a+dotprod+crypto" HAS_MARCH_CUSTOMIZATION)
if(HAS_MARCH_CUSTOMIZATION)
add_compile_options(-march=armv8.2-a+dotprod+crypto)
message(STATUS "ARM: using -march=armv8.2-a+dotprod+crypto for optimal ISA")
else()
add_compile_options(-march=native)
message(STATUS "ARM: using -march=native (cross-compile fallback)")
endif()
endif()
# find_package(CUDAToolkit REQUIRED)
# setup CMake module path, defines path for include() and find_package()
list(APPEND CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake)
enable_language(Rust)
find_package(Rust REQUIRED)
if(UNIX)
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -Og")
endif()
find_package(Boost REQUIRED)
find_package(Boost REQUIRED COMPONENTS serialization)
find_package(Threads REQUIRED)
# find all dependencies from vcpkg
find_package(folly CONFIG REQUIRED)
find_package(glog CONFIG REQUIRED)
find_package(gflags CONFIG REQUIRED)
find_package(leveldb CONFIG REQUIRED)
find_package(OpenSSL REQUIRED)
find_package(absl CONFIG REQUIRED)
find_package(Protobuf CONFIG REQUIRED)
# Ensure vcpkg protobuf headers are found before PyTorch's bundled older version.
# Use BEFORE SYSTEM so that:
# 1) vcpkg protobuf is first in -isystem list (before torch's -isystem)
# 2) project-local -I paths (e.g. Mooncake fake_include stubs) still win over -isystem
# Protobuf_INCLUDE_DIRS may be empty in CONFIG mode; fall back to the imported target.
if(NOT Protobuf_INCLUDE_DIRS AND TARGET protobuf::libprotobuf)
get_target_property(Protobuf_INCLUDE_DIRS protobuf::libprotobuf INTERFACE_INCLUDE_DIRECTORIES)
endif()
if(Protobuf_INCLUDE_DIRS)
include_directories(BEFORE SYSTEM ${Protobuf_INCLUDE_DIRS})
endif()
find_package(gRPC CONFIG REQUIRED)
find_package(GTest CONFIG REQUIRED)
find_package(benchmark CONFIG REQUIRED)
find_package(nlohmann_json CONFIG REQUIRED)
find_package(OpenCV CONFIG REQUIRED)
find_package(Python COMPONENTS Development REQUIRED)
find_package(pybind11 CONFIG REQUIRED)
if (USE_CXX11_ABI)
# only use jemalloc if using the new C++-11 ABI
find_package(Jemalloc)
if(Jemalloc_FOUND)
link_libraries(Jemalloc::jemalloc)
endif()
endif()
# Important Note: Always invoke find_package for other dependencies
# before including libtorch, as doing so afterwards may lead to
# unexpected linker errors.
if (DEFINED ENV{LIBTORCH_ROOT})
find_package(Torch REQUIRED HINTS "$ENV{LIBTORCH_ROOT}")
message(STATUS "Using libtorch at $ENV{LIBTORCH_ROOT}")
else()
include(FetchContent)
if (USE_CXX11_ABI)
set(LIBTORCH_URL "https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-2.1.0%2Bcpu.zip")
else()
set(LIBTORCH_URL "https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-2.1.0%2Bcpu.zip")
endif()
if (DEFINED ENV{DEPENDENCES_ROOT})
set(LIBTORCH_SOURCE_DIR $ENV{DEPENDENCES_ROOT}/libtorch-src)
else()
set(LIBTORCH_SOURCE_DIR ${FETCHCONTENT_BASE_DIR}/libtorch-src)
endif()
FetchContent_Declare(libtorch
URL ${LIBTORCH_URL}
SOURCE_DIR ${LIBTORCH_SOURCE_DIR}
)
FetchContent_MakeAvailable(libtorch)
find_package(Torch REQUIRED PATHS ${LIBTORCH_SOURCE_DIR} NO_DEFAULT_PATH)
message(STATUS "Downloading and using libtorch 2.1.0 for CPU at ${LIBTORCH_SOURCE_DIR}")
endif()
if(USE_NPU)
add_definitions(-DUSE_NPU)
add_definitions(-DBUILD_LIBTORCH)
add_definitions(-DTORCH_SETCUSTOMHANDLER=ON)
# set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,--no-undefined")
set(CMAKE_VERBOSE_MAKEFILE ON)
add_definitions(-DTORCH_HIGHER_THAN_PTA6)
# Use vcpkg header files as the first priority search directory,
#-> because the scope of third-party softwares managed by vcpkg is used throughout the entire xllm.
message(STATUS "VCPKG_INCLUDE_DIR = ${CMAKE_BINARY_DIR}/vcpkg_installed/${VCPKG_TARGET_TRIPLET}/include")
include_directories("${CMAKE_BINARY_DIR}/vcpkg_installed/${VCPKG_TARGET_TRIPLET}/include")
add_subdirectory(
"${XLLM_ATB_LAYERS_SOURCE_DIR}"
"${CMAKE_BINARY_DIR}/xllm_atb_layers"
)
message(STATUS "Configured xllm_atb_layers from source: ${XLLM_ATB_LAYERS_SOURCE_DIR}")
# torch npu_torch and npu ops is only used by npu related modules/classes, so system priority is reasonable.
include_directories(SYSTEM
$ENV{PYTHON_INCLUDE_PATH}
$ENV{PYTORCH_INSTALL_PATH}/include
$ENV{PYTORCH_INSTALL_PATH}/include/torch/csrc/api/include
$ENV{PYTORCH_NPU_INSTALL_PATH}/include
$ENV{PYTORCH_INSTALL_PATH}/include/torch/csrc/distributed
$ENV{NPU_HOME_PATH}/include
$ENV{ATB_HOME_PATH}/include
$ENV{NPU_HOME_PATH}/opp/vendors/xllm/op_api/include/
${CMAKE_CURRENT_SOURCE_DIR}/third_party/torch_npu_ops/
)
# Keep third-party warnings suppressed while providing headers expected by
# legacy includes like "atb_speed/log.h" from xllm_atb_layers.
include_directories(SYSTEM
${XLLM_ATB_LAYERS_SOURCE_DIR}
${XLLM_ATB_LAYERS_SOURCE_DIR}/core/include
)
link_directories(
$ENV{PYTHON_LIB_PATH}
$ENV{PYTORCH_INSTALL_PATH}/lib
$ENV{PYTORCH_NPU_INSTALL_PATH}/lib
$ENV{NPU_HOME_PATH}/lib64
$ENV{ATB_HOME_PATH}/lib
$ENV{NPU_TOOLKIT_HOME}/lib64
$ENV{NPU_HOME_PATH}/opp/vendors/xllm/op_api/lib/
)
link_libraries(cust_opapi)
# avoid conflicts with xllm libruntime.a
find_library(
NPU_RUNTIME_SO
NAMES libruntime.so
PATHS $ENV{ASCEND_TOOLKIT_HOME}/lib64
NO_DEFAULT_PATH
)
if(NOT NPU_RUNTIME_SO)
message(FATAL_ERROR "Failed to find npu libruntime.so in ${ASCEND_TOOLKIT_HOME}/lib64")
endif()
set(NPU_RUNTIME_SO ${NPU_RUNTIME_SO} CACHE INTERNAL "CACHED ASCEND libruntime.so path")
endif()
if(USE_MLU)
add_definitions(-DUSE_MLU)
set(CMAKE_VERBOSE_MAKEFILE ON)
include_directories(
$ENV{PYTHON_INCLUDE_PATH}
$ENV{PYTORCH_INSTALL_PATH}/include
$ENV{PYTORCH_INSTALL_PATH}/include/torch/csrc/api/include
$ENV{PYTORCH_MLU_INSTALL_PATH}
$ENV{PYTORCH_MLU_INSTALL_PATH}/../
$ENV{PYTORCH_MLU_INSTALL_PATH}/csrc
$ENV{PYTORCH_MLU_INSTALL_PATH}/csrc/include
$ENV{NEUWARE_HOME}/include
)
link_directories(
$ENV{PYTHON_LIB_PATH}
$ENV{PYTORCH_INSTALL_PATH}/lib
$ENV{PYTORCH_MLU_INSTALL_PATH}/csrc/lib
$ENV{PYTORCH_MLU_INSTALL_PATH}
$ENV{NEUWARE_HOME}/lib64
)
endif()
if(USE_MUSA)
add_definitions(-DUSE_MUSA)
add_compile_definitions(TORCH_CUDA=1)
if(NOT DEFINED MUSA_PATH)
set(MUSA_PATH /usr/local/musa)
endif()
list(APPEND CMAKE_MODULE_PATH "${MUSA_PATH}/cmake")
find_package(MUSA REQUIRED)
if(NOT DEFINED ENV{MTT_OPLIB_PATH})
set(ENV{MTT_OPLIB_PATH} $ENV{MUSA_HOME}/tools/MTTOplib)
endif()
message(STATUS "using MTT Oplib at: $ENV{MTT_OPLIB_PATH}")
set(MTTOplib_DIR $ENV{MTT_OPLIB_PATH}/cmake)
find_package(MTTOplib REQUIRED)
find_package(Python COMPONENTS Interpreter REQUIRED)
list(APPEND CMAKE_PREFIX_PATH $ENV{TORCH_MUSA_PYTHONPATH})
find_package(TorchMusa REQUIRED CONFIG)
list(POP_BACK CMAKE_PREFIX_PATH)
find_package(MKL REQUIRED)
execute_process(
COMMAND tvm-ffi-config --includedir
OUTPUT_VARIABLE tvm_ffi_INCLUDE_DIR
OUTPUT_STRIP_TRAILING_WHITESPACE
)
execute_process(
COMMAND tvm-ffi-config --libdir
OUTPUT_VARIABLE tvm_ffi_LIB_DIR
OUTPUT_STRIP_TRAILING_WHITESPACE
)
string(REGEX MATCH "[^\n]*/include[^\n]*" tvm_ffi_INCLUDE_DIR "${tvm_ffi_INCLUDE_DIR}")
string(REGEX MATCH "[^\n]*/lib[^\n]*" tvm_ffi_LIB_DIR "${tvm_ffi_LIB_DIR}")
include_directories(
${MUSA_INCLUDE_DIRS}
${TorchMusa_INCLUDE_DIRS}
${Python_SITELIB}/torch_musa_compiled/share/torch_musa_codegen
$ENV{PYTHON_INCLUDE_PATH}
$ENV{PYTORCH_INSTALL_PATH}/include
$ENV{PYTORCH_INSTALL_PATH}/include/torch/csrc/api/include
${MTTOplib_INCLUDE_DIRS}
${tvm_ffi_INCLUDE_DIR}
)
link_directories(
${MUSA_LIB_PATH}
$ENV{PYTHON_LIB_PATH}
$ENV{PYTORCH_INSTALL_PATH}/lib
${TorchMusa_LIB_PATH}
${MKL_LIB_PATH}
${tvm_ffi_LIB_DIR}
${TorchMusa_INSTALL_PREFIX}/lib
)
set(MUSA_NVCC_FLAGS ${CUDA_NVCC_FLAGS} -O3)
endif()
if(USE_CUDA)
message(STATUS "TORCH_CUDA_ARCH_LIST: ${TORCH_CUDA_ARCH_LIST}")
add_definitions(-DUSE_CUDA)
add_compile_definitions(TORCH_CUDA=1)
set(CMAKE_VERBOSE_MAKEFILE ON)
include_directories($ENV{PYTHON_INCLUDE_PATH})
include_directories(SYSTEM
$ENV{PYTORCH_INSTALL_PATH}/include
$ENV{PYTORCH_INSTALL_PATH}/include/torch/csrc/api/include
)
link_directories(
$ENV{PYTHON_LIB_PATH}
$ENV{PYTORCH_INSTALL_PATH}/lib
$ENV{CUDA_TOOLKIT_ROOT_DIR}/lib64
)
# To reduce compilation time during development, use fewer architectures:
# export TORCH_CUDA_ARCH_LIST="9.0"
option(CUDA_DEV_MODE "Use -O1 instead of -O3 for faster CUDA compilation during development" OFF)
if(CUDA_DEV_MODE)
set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -O1")
else()
set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -O3")
endif()
# The following definitions must be undefined since half-precision operation is required.
string(APPEND CMAKE_CUDA_FLAGS
" -U__CUDA_NO_HALF_OPERATORS__"
" -U__CUDA_NO_HALF_CONVERSIONS__"
" -U__CUDA_NO_HALF2_OPERATORS__"
" -U__CUDA_NO_BFLOAT16_CONVERSIONS__"
" --use_fast_math"
" -Xfatbin -compress-all")
# Parallel nvcc compilation: compile multiple GPU architectures simultaneously within each .cu
# file. --threads 0 = auto-detect CPU core count. Requires CUDA >= 11.2.
if(CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 11.2)
string(APPEND CMAKE_CUDA_FLAGS " --threads 0")
endif()
message(STATUS "CMAKE_CUDA_FLAGS: ${CMAKE_CUDA_FLAGS}")
# find_package(NCCL REQUIRED)
# find cudnn
execute_process(COMMAND python -c "import nvidia.cudnn; print(nvidia.cudnn.__file__)" OUTPUT_VARIABLE CUDNN_PYTHON_PATH)
get_filename_component(CUDNN_ROOT_DIR "${CUDNN_PYTHON_PATH}" DIRECTORY)
link_directories(
${CUDNN_ROOT_DIR}/lib64
${CUDNN_ROOT_DIR}/lib
)
endif()
if(USE_ILU)
add_definitions(-DUSE_ILU)
set(CMAKE_VERBOSE_MAKEFILE ON)
include_directories(
$ENV{PYTHON_INCLUDE_PATH}
$ENV{PYTORCH_INSTALL_PATH}/include
$ENV{PYTORCH_INSTALL_PATH}/include/torch/csrc/api/include
$ENV{IXFORMER_INSTALL_PATH}/csrc/include/ixformer
)
link_directories(
$ENV{PYTHON_LIB_PATH}
$ENV{PYTORCH_INSTALL_PATH}/lib
$ENV{IXFORMER_INSTALL_PATH}
)
endif()
# check if USE_CXX11_ABI is set correctly
# if (DEFINED USE_CXX11_ABI)
# parse_make_options(${TORCH_CXX_FLAGS} "TORCH_CXX_FLAGS")
# if(DEFINED TORCH_CXX_FLAGS__GLIBCXX_USE_CXX11_ABI
# AND NOT ${TORCH_CXX_FLAGS__GLIBCXX_USE_CXX11_ABI} EQUAL ${USE_CXX11_ABI})
# message(FATAL_ERROR
# "The libtorch compilation options _GLIBCXX_USE_CXX11_ABI=${TORCH_CXX_FLAGS__GLIBCXX_USE_CXX11_ABI} "
# "found by CMake conflict with the project setting USE_CXX11_ABI=${USE_CXX11_ABI}.")
# endif()
# endif()
# carry over torch flags to the rest of the project
message(STATUS "TORCH_CXX_FLAGS: ${TORCH_CXX_FLAGS}")
add_compile_options(${TORCH_CXX_FLAGS})
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DC10_USE_GLOG")
message(STATUS "CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
message(STATUS "CMAKE_CXX_FLAGS_DEBUG: ${CMAKE_CXX_FLAGS_DEBUG}")
# enable testing in this directory so we can do a top-level `make test`.
# this also includes the BUILD_TESTING option, which is on by default.
include(CTest)
include(GoogleTest)
option(BUILD_TESTING "Build the testing tree." ON)
set(XLLM_TESTS_DIR "${PROJECT_SOURCE_DIR}/tests")
# include current path
list(APPEND COMMON_INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR})
list(APPEND COMMON_INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/xllm/core)
# Note: third_party is not added to COMMON_INCLUDE_DIRS to avoid warnings
# Individual third_party directories are added as SYSTEM headers below
list(APPEND COMMON_INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/third_party/etcd_cpp_apiv3)
# brpc - mark as system headers to suppress warnings from third-party code
set(BRPC_OUTPUT_DIR ${CMAKE_BINARY_DIR}/third_party/brpc/output)
include_directories(SYSTEM ${BRPC_OUTPUT_DIR}/include)
link_directories(${BRPC_OUTPUT_DIR}/lib)
# Mooncake - mark as system headers to suppress warnings from third-party code
# Add third_party as SYSTEM to support #include <Mooncake/...> style includes
# Use absolute path to ensure it works correctly
include_directories(SYSTEM ${CMAKE_CURRENT_SOURCE_DIR}/third_party)
include_directories(SYSTEM ${CMAKE_CURRENT_SOURCE_DIR}/third_party/Mooncake/mooncake-common/include)
include_directories(SYSTEM ${CMAKE_CURRENT_SOURCE_DIR}/third_party/Mooncake/mooncake-transfer-engine/include)
include_directories(SYSTEM ${CMAKE_CURRENT_SOURCE_DIR}/third_party/Mooncake/mooncake-store/include)
# CUTLASS - CUDA kernel library for scaled matmul
if(USE_CUDA)
set(CUTLASS_DIR ${CMAKE_CURRENT_SOURCE_DIR}/third_party/cutlass)
set(CUTLASS_INCLUDE_DIR ${CUTLASS_DIR}/include)
set(CUTLASS_TOOLS_UTIL_INCLUDE_DIR ${CUTLASS_DIR}/tools/util/include)
include_directories(SYSTEM ${CUTLASS_INCLUDE_DIR})
include_directories(SYSTEM ${CUTLASS_TOOLS_UTIL_INCLUDE_DIR})
endif()
# sentencepiece - mark as system headers to suppress warnings from third-party code
# This must be added before add_subdirectory(third_party) to ensure it takes precedence
include_directories(SYSTEM ${CMAKE_CURRENT_SOURCE_DIR}/third_party/sentencepiece)
# Also add other commonly used third_party directories as system headers
include_directories(SYSTEM ${CMAKE_CURRENT_SOURCE_DIR}/third_party/minja)
if(USE_MUSA)
set(CMAKE_MUSA_FLAGS_DEBUG "-g -std=gnu++20 -fPIC --offload-arch=mp_31")
set(CMAKE_MUSA_FLAGS_RELEASE "-O2 -DNDEBUG -std=gnu++20 -fPIC --offload-arch=mp_31")
endif()
# add subdirectories
# Configure third_party first so xllm can link against real CMake targets.
add_subdirectory(third_party)
if(TARGET brpc-static AND NOT TARGET brpc)
# Expose a canonical target name used across xllm/CMakeLists.
add_library(brpc ALIAS brpc-static)
endif()
add_subdirectory(xllm)
add_subdirectory(tests)

201
upstream_ref/xllm/LICENSE Normal file
View File

@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

14
upstream_ref/xllm/MANIFEST.in Normal file
View File

@@ -0,0 +1,14 @@
include MANIFEST.in
include CMakeLists.txt
include LICENSE
include .gitmodules
recursive-include src *.*
recursive-include xllm *.py
recursive-include examples *.py
recursive-include third_party *
recursive-include docs *.*
recursive-include tools *.*
recursive-include scripts *.*
recursive-include proto *.*
prune */__pycache__
global-exclude *.o *.so *.dylib *.a .git *.pyc *.swp

1127
upstream_ref/xllm/NOTICE_Third_Party.md Normal file
View File

File diff suppressed because it is too large Load Diff

173
upstream_ref/xllm/README.md Executable file
View File

@@ -0,0 +1,173 @@
<!-- Copyright 2022 JD Co.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this project except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. -->
[English](./README.md) | [中文](./docs/project/README_zh.md)
<div align="center">
<img src="docs/assets/logo_with_llm.png" alt="xLLM" style="width:50%; height:auto;">
[![Document](https://img.shields.io/badge/Document-black?logo=html5&labelColor=grey&color=red)](https://xllm.readthedocs.io/zh-cn/latest/) [![Docker](https://img.shields.io/badge/Docker-black?logo=docker&labelColor=grey&color=%231E90FF)](https://hub.docker.com/r/xllm/xllm-ai) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen?labelColor=grey)](https://opensource.org/licenses/Apache-2.0) [![report](https://img.shields.io/badge/Technical%20Report-red?logo=arxiv&logoColor=%23B31B1B&labelColor=%23F0EBEB&color=%23D42626)](https://arxiv.org/abs/2510.14686) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/jd-opensource/xllm)
</div>
---------------------
<p align="center">
| <a href="https://xllm.readthedocs.io/zh-cn/latest/"><b>Documentation</b></a> | <a href="https://arxiv.org/abs/2510.14686"><b>Technical Report</b></a> |
</p>
### 📢 News
- 2026-04-24: 🎉 We day-0 support the [DeepSeek-V4](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) model, please refer to the [Deployment Document](https://github.com/jd-opensource/xllm/blob/preview/deepseek-v4-mlu/testspace/run_deepseek_v4.sh) for deployment.
- 2026-02-12: 🎉 We day-0 support high-performance inference for the [GLM-5](https://github.com/zai-org/GLM-5) model, please refer to the [Deployment Document](https://github.com/zai-org/GLM-5/blob/main/example/ascend.md) for deployment.
- 2025-12-21: 🎉 We day-0 support high-performance inference for the [GLM-4.7](https://github.com/zai-org) model.
- 2025-12-08: 🎉 We day-0 support high-performance inference for the [GLM-4.6V](https://github.com/zai-org/GLM-V) model.
- 2025-12-05: 🎉 We now support high-performance inference for the [GLM-4.5/GLM-4.6](https://github.com/zai-org/GLM-4.5/blob/main/README_zh.md) series models.
- 2025-12-05: 🎉 We now support high-performance inference for the [VLM-R1](https://github.com/om-ai-lab/VLM-R1) model.
- 2025-12-05: 🎉 We build hybrid KV cache management based on [Mooncake](https://github.com/kvcache-ai/Mooncake), supporting global KV cache management with intelligent offloading and prefetching.
- 2025-10-16: 🎉 We recently have released our [xLLM Technical Report](https://arxiv.org/abs/2510.14686) on arXiv, providing comprehensive technical blueprints and implementation insights.
## Project Overview
**xLLM** is an **efficient LLM inference framework**, specifically optimized for **Chinese AI accelerators**, enabling enterprise-grade deployment with enhanced efficiency and reduced cost. The framework adopts a **service-engine decoupled** inference architecture, achieving breakthrough efficiency through several technologies: at the service layer, including elastic scheduling of online/offline requests, dynamic PD disaggregation, a hybrid EPD mechanism for multimodal and high-availability fault tolerance; and at the engine layer, combined with technologies such as multi-stream parallel computing, graph fusion optimization, speculative inference, dynamic load balancing and global KV cache management. The overall architecture is shown below:
<div align="center">
<img src="docs/assets/xllm_arch.png" alt="xllm_arch" style="width:90%; height:auto;">
</div>
**xLLM** already supports efficient deployment of mainstream large models (such as *DeepSeek-V3.1*, *Qwen2/3*, etc.) on Chinese AI accelerators, empowering enterprises to implement high-performance, low-cost AI large model applications. xLLM has been fully deployed in JD.com’s real core retail businesses, covering a variety of scenarios including intelligent customer service, risk control, supply chain optimization, ad recommendation, and more.
## Core Features
**xLLM** delivers robust intelligent computing capabilities. By leveraging hardware system optimization and algorithm-driven decision control, it jointly accelerates the inference process, enabling high-throughput, low-latency distributed inference services.
**Full Graph Pipeline Execution Orchestration**
- Asynchronous decoupled scheduling at the requests scheduling layer, to reduce computational bubbles.
- Asynchronous parallelism of computation and communication at the model graph layer, overlapping computation and communication.
- Pipelining of heterogeneous computing units at the operator kernel layer, overlapping computation and memory access.
**Graph Optimization for Dynamic Shapes**
- Dynamic shape adaptation based on parameterization and multi-graph caching methods to enhance the flexibility of static graph.
- Controlled tensor memory pool to ensure address security and reusability.
- Integration and adaptation of performance-critical custom operators (e.g., *PageAttention*, *AllReduce*).
**Efficient Memory Optimization**
- Mapping management between discrete physical memory and continuous virtual memory.
- On-demand memory allocation to reduce memory fragmentation.
- Intelligent scheduling of memory pages to increase memory reusability.
- Adaptation of corresponding operators for domestic accelerators.
**Global KV Cache Management**
- Intelligent offloading and prefetching of KV in hierarchical caches.
- KV cache-centric distributed storage architecture.
- Intelligent KV routing among computing nodes.
**Algorithm-driven Acceleration**
- Speculative decoding optimization to improve efficiency through multi-core parallelism.
- Dynamic load balancing of MoE experts to achieve efficient adjustment of expert distribution.
---
## Hardware Support
| Hardware | Example | Remark |
| -------- | ------- | --------------- |
| NPU | A2, A3 | HDK Driver 25.2.0 + |
| MLU | | |
| ILU | BI150 | |
| MUSA | S5000 | |
Besides, please check the supported models on different hardwares at [Supported Models List](docs/en/supported_models.md).
---
## Quick Start
Please refer to [Quick Start](docs/en/getting_started/quick_start.md) for more details.
---
## Contributing
There are several ways you can contribute to xLLM:
1. Reporting Issues (Bugs & Errors)
2. Suggesting Enhancements
3. Improving Documentation
+ Fork the repository
+ Add your view in document
+ Send your pull request
4. Writing Code
+ Fork the repository
+ Create a new branch
+ Add your feature or improvement
+ Send your pull request
We appreciate all kinds of contributions! 🎉🎉🎉
If you have problems about development, please check our document: **[Document](https://xllm.readthedocs.io/zh-cn/latest)**
---
## Community & Support
If you encounter any issues along the way, you are welcomed to submit reproducible steps and log snippets in the project's Issues area, or contact the xLLM Core team directly via your internal Slack. In addition, we have established official WeChat groups. You can access the following QR code to join. Welcome to contact us!
<div align="center">
<img src="docs/assets/wechat_qrcode.png" alt="qrcode3" width="50%" />
</div>
## Acknowledgment
This project was made possible thanks to the following open-source projects:
- [ScaleLLM](https://github.com/vectorch-ai/ScaleLLM) - xLLM draws inspiration from ScaleLLM's graph construction method and references its runtime execution.
- [Mooncake](https://github.com/kvcache-ai/Mooncake) - Build xLLM hybrid KV cache management based on Mooncake.
- [brpc](https://github.com/apache/brpc) - Build high-performance http service based on brpc.
- [tokenizers-cpp](https://github.com/mlc-ai/tokenizers-cpp) - Build C++ tokenizer based on tokenizers-cpp.
- [safetensors](https://github.com/huggingface/safetensors) - xLLM relies on the C binding safetensors capability.
- [Partial JSON Parser](https://github.com/promplate/partial-json-parser) - Implement xLLM's C++ JSON parser with insights from Python and Go implementations.
- [concurrentqueue](https://github.com/cameron314/concurrentqueue) - A fast multi-producer, multi-consumer lock-free concurrent queue for C++11.
Thanks to the following collaborating university laboratories:
- [THU-MIG](https://ise.thss.tsinghua.edu.cn/mig/projects.html) (School of Software, BNRist, Tsinghua University)
- USTC-Cloudlab (Cloud Computing Lab, University of Science and Technology of China)
- [Beihang-HiPO](https://github.com/buaa-hipo) (Beihang HiPO research group)
- PKU-DS-LAB (Data Structure Laboratory, Peking University)
- PKU-NetSys-LAB (NetSys Lab, Peking University)
- [TJU-TANKLab](https://flashserve.org/) (TANK Lab, Tianjin University)
Thanks to all the following [developers](https://github.com/jd-opensource/xllm/graphs/contributors) who have contributed to xLLM.
<a href="https://github.com/jd-opensource/xllm/graphs/contributors">
<img src="https://contrib.rocks/image?repo=jd-opensource/xllm" />
</a>
---
## License
[Apache License](LICENSE)
#### xLLM is provided by JD.com
#### Thanks for your Contributions!
## Citation
If you think this repository is helpful to you, welcome to cite us:
```
@article{liu2025xllm,
title={xLLM Technical Report},
author={Liu, Tongxuan and Peng, Tao and Yang, Peijun and Zhao, Xiaoyang and Lu, Xiusheng and Huang, Weizhe and Liu, Zirui and Chen, Xiaoyu and Liang, Zhiwei and Xiong, Jun and others},
journal={arXiv preprint arXiv:2510.14686},
year={2025}
}
```

29
upstream_ref/xllm/cibuild/build_cuda.sh Normal file
View File

@@ -0,0 +1,29 @@
#!/bin/bash
set -e
function error() {
echo "Require build command, e.g. python setup.py build --device cuda"
exit 1
}
IMAGE="quay.io/jd_xllm/xllm-ai:xllm-dev-cuda-x86"
RUN_OPTS=(
--rm
-t
--privileged
--ipc=host
--network=host
--pid=host
--shm-size '128gb'
-v /export/home:/export/home
-v /export/home/cuda_vcpkg_cache:/root/.cache/vcpkg # cached vcpkg installed dir
-w /export/home
)
CMD="$*"
[[ -z "${CMD}" ]] && error
[[ ! -x $(command -v docker) ]] && echo "ERROR: 'docker' command is missing." && exit 1
docker run "${RUN_OPTS[@]}" "${IMAGE}" bash -c "set -euo pipefail; cd $(pwd); ${CMD}"

29
upstream_ref/xllm/cibuild/build_ilu.sh Normal file
View File

@@ -0,0 +1,29 @@
#!/bin/bash
set -ex
function error() {
echo "Require build command, e.g. python setup.py build --device ilu"
exit 1
}
REGISTRY="registry.iluvatar.com.cn:10443/infra"
COREX_VERSION="4.4.0.20251229"
IMAGE="${REGISTRY}/xllm-builder:${COREX_VERSION}-ubuntu22.04-py310-xllm-x86_64"
RUN_OPTS=(
--rm
-t
--privileged
--ipc=host
--network=host
-v /export/home:/export/home
-v /export/home/ilu_vcpkg_cache:/root/.cache/vcpkg # cached vcpkg installed dir
-w /export/home
)
CMD="$*"
[[ -z "${CMD}" ]] && error
[[ ! -x $(command -v docker) ]] && echo "ERROR: 'docker' command is missing." && exit 1
docker run "${RUN_OPTS[@]}" "${IMAGE}" bash -c "set -euo pipefail; cd $(pwd); ${CMD}"

30
upstream_ref/xllm/cibuild/build_mlu.sh Normal file
View File

@@ -0,0 +1,30 @@
#!/bin/bash
set -e
function error() {
echo "Require build command, e.g. python setup.py build --device mlu"
exit 1
}
IMAGE="cambricon-xllm:v25.12.2-torch2.9.1-torchmlu1.30.2-ubuntu22.04-py310-mlu-dev-x86"
RUN_OPTS=(
--rm
-t
--privileged
--ipc=host
--network=host
--pid=host
--shm-size '128gb'
-v /export/home:/export/home
-v /usr/bin/cnmon:/usr/bin/cnmon
-v /export/home/mlu_vcpkg_cache:/root/.cache/vcpkg # cached vcpkg installed dir
-w /export/home
)
CMD="$*"
[[ -z "${CMD}" ]] && error
[[ ! -x $(command -v docker) ]] && echo "ERROR: 'docker' command is missing." && exit 1
docker run "${RUN_OPTS[@]}" "${IMAGE}" bash -c "set -euo pipefail; cd $(pwd); ${CMD}"

36
upstream_ref/xllm/cibuild/build_npu.sh Normal file
View File

@@ -0,0 +1,36 @@
#!/bin/bash
set -e
function error() {
echo "Require build command, e.g. python setup.py build --device npu"
exit 1
}
IMAGE="quay.io/jd_xllm/xllm-ai:xllm-dev-a2-x86-20260429"
RUN_OPTS=(
--rm
-t
--privileged
--ipc=host
--network=host
--device=/dev/davinci0
--device=/dev/davinci_manager
--device=/dev/devmm_svm
--device=/dev/hisi_hdc
-v /var/queue_schedule:/var/queue_schedule
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi
-v /usr/local/sbin/:/usr/local/sbin/
-v /export/home:/export/home
-v /export/home/npu_vcpkg_cache_abi_1:/root/.cache/vcpkg # cached vcpkg installed dir
-v /etc/hccn.conf:/etc/hccn.conf
-w /export/home
)
CMD="$*"
[[ -z "${CMD}" ]] && error
[[ ! -x $(command -v docker) ]] && echo "ERROR: 'docker' command is missing." && exit 1
docker run "${RUN_OPTS[@]}" "${IMAGE}" bash -c "set -euo pipefail; cd $(pwd); ${CMD}"

17
upstream_ref/xllm/cmake/CMakeDetermineMUSACompiler.cmake Normal file
View File

@@ -0,0 +1,17 @@
# Try to find the compiler
find_program(CMAKE_MUSA_COMPILER
NAMES mcc
DOC "MUSA compiler"
)
set(CMAKE_MUSA_COMPILER_ENV_VAR "MUSA")
# Check if compiler was found
if(CMAKE_MUSA_COMPILER)
set(CMAKE_MUSA_COMPILER_LOADED 1)
message(STATUS "Found MUSA compiler: ${CMAKE_MUSA_COMPILER}")
else()
message(FATAL_ERROR "MUSA compiler not found")
endif()
configure_file(${CMAKE_CURRENT_LIST_DIR}/CMakeMUSACompiler.cmake.in
${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/${CMAKE_VERSION}/CMakeMUSACompiler.cmake IMMEDIATE @ONLY)

25
upstream_ref/xllm/cmake/CMakeDetermineRustCompiler.cmake Normal file
View File

@@ -0,0 +1,25 @@
# ported from https://github.com/Devolutions/CMakeRust
if(NOT CMAKE_Rust_COMPILER)
find_package(Rust)
if(RUST_FOUND)
set(CMAKE_Rust_COMPILER "${RUSTC_EXECUTABLE}")
set(CMAKE_Rust_COMPILER_ID "Rust")
set(CMAKE_Rust_COMPILER_VERSION "${RUST_VERSION}")
set(CMAKE_Rust_PLATFORM_ID "Rust")
endif()
endif()
message(STATUS "Cargo Home: ${CARGO_HOME}")
message(STATUS "Rust Compiler Version: ${RUSTC_VERSION}")
mark_as_advanced(CMAKE_Rust_COMPILER)
if(CMAKE_Rust_COMPILER)
set(CMAKE_Rust_COMPILER_LOADED 1)
endif(CMAKE_Rust_COMPILER)
configure_file(${CMAKE_CURRENT_LIST_DIR}/CMakeRustCompiler.cmake.in
${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/${CMAKE_VERSION}/CMakeRustCompiler.cmake IMMEDIATE @ONLY)
set(CMAKE_Rust_COMPILER_ENV_VAR "RUSTC")

0
upstream_ref/xllm/cmake/CMakeMUSACompiler.cmake.in Normal file
View File

18
upstream_ref/xllm/cmake/CMakeMUSAInformation.cmake Normal file
View File

@@ -0,0 +1,18 @@
set(CMAKE_MUSA_SOURCE_FILE_EXTENSIONS mu;cu)
if(UNIX)
set(CMAKE_MUSA_OUTPUT_EXTENSION .o)
else()
set(CMAKE_MUSA_OUTPUT_EXTENSION .obj)
endif()
set(CMAKE_INCLUDE_FLAG_MUSA "-I")
set(CMAKE_MUSA_COMPILE_OBJECT
"<CMAKE_MUSA_COMPILER> <DEFINES> <INCLUDES> <FLAGS> -mtgpu -c <SOURCE> -o <OBJECT>"
)
set(CMAKE_MUSA_LINK_EXECUTABLE
"<CMAKE_MUSA_COMPILER> <FLAGS> <LINK_FLAGS> <OBJECTS> -lmusa -lmusart -o <TARGET> <LINK_LIBRARIES>"
)
set(CMAKE_MUSA_INFORMATION_LOADED 1)

12
upstream_ref/xllm/cmake/CMakeRustCompiler.cmake.in Normal file
View File

@@ -0,0 +1,12 @@
# ported from https://github.com/Devolutions/CMakeRust
set(CMAKE_Rust_COMPILER "@CMAKE_Rust_COMPILER@")
set(CMAKE_Rust_COMPILER_ID "@CMAKE_Rust_COMPILER_ID@")
set(CMAKE_Rust_COMPILER_VERSION "@CMAKE_Rust_COMPILER_VERSION@")
set(CMAKE_Rust_COMPILER_LOADED @CMAKE_Rust_COMPILER_LOADED@)
set(CMAKE_Rust_PLATFORM_ID "@CMAKE_Rust_PLATFORM_ID@")
SET(CMAKE_Rust_SOURCE_FILE_EXTENSIONS rs)
SET(CMAKE_Rust_LINKER_PREFERENCE 40)
set(CMAKE_Rust_COMPILER_ENV_VAR "RUSTC")

106
upstream_ref/xllm/cmake/CMakeRustInformation.cmake Normal file
View File

@@ -0,0 +1,106 @@
# ported from https://github.com/Devolutions/CMakeRust
#
# Usage: rustc [OPTIONS] INPUT
#
# Options:
# -h --help Display this message
# --cfg SPEC Configure the compilation environment
# -L [KIND=]PATH Add a directory to the library search path. The
# optional KIND can be one of dependency, crate, native,
# framework or all (the default).
# -l [KIND=]NAME Link the generated crate(s) to the specified native
# library NAME. The optional KIND can be one of static,
# dylib, or framework. If omitted, dylib is assumed.
# --crate-type [bin|lib|rlib|dylib|cdylib|staticlib|metadata]
# Comma separated list of types of crates for the
# compiler to emit
# --crate-name NAME Specify the name of the crate being built
# --emit [asm|llvm-bc|llvm-ir|obj|link|dep-info]
# Comma separated list of types of output for the
# compiler to emit
# --print [crate-name|file-names|sysroot|cfg|target-list|target-cpus|target-features|relocation-models|code-models]
# Comma separated list of compiler information to print
# on stdout
# -g Equivalent to -C debuginfo=2
# -O Equivalent to -C opt-level=2
# -o FILENAME Write output to <filename>
# --out-dir DIR Write output to compiler-chosen filename in <dir>
# --explain OPT Provide a detailed explanation of an error message
# --test Build a test harness
# --target TARGET Target triple for which the code is compiled
# -W --warn OPT Set lint warnings
# -A --allow OPT Set lint allowed
# -D --deny OPT Set lint denied
# -F --forbid OPT Set lint forbidden
# --cap-lints LEVEL Set the most restrictive lint level. More restrictive
# lints are capped at this level
# -C --codegen OPT[=VALUE]
# Set a codegen option
# -V --version Print version info and exit
# -v --verbose Use verbose output
#
# Additional help:
# -C help Print codegen options
# -W help Print 'lint' options and default settings
# -Z help Print internal options for debugging rustc
# --help -v Print the full set of options rustc accepts
#
# <TARGET> <TARGET_BASE> <OBJECT> <OBJECTS> <LINK_LIBRARIES> <FLAGS> <LINK_FLAGS> <SOURCE> <SOURCES>
include(CMakeLanguageInformation)
if(UNIX)
set(CMAKE_Rust_OUTPUT_EXTENSION .o)
else()
set(CMAKE_Rust_OUTPUT_EXTENSION .obj)
endif()
set(CMAKE_Rust_ECHO_ALL "echo \"TARGET: <TARGET> TARGET_BASE: <TARGET_BASE> ")
set(CMAKE_Rust_ECHO_ALL "${CMAKE_Rust_ECHO_ALL} OBJECT: <OBJECT> OBJECTS: <OBJECTS> OBJECT_DIR: <OBJECT_DIR> SOURCE: <SOURCE> SOURCES: <SOURCES> ")
set(CMAKE_Rust_ECHO_ALL "${CMAKE_Rust_ECHO_ALL} LINK_LIBRARIES: <LINK_LIBRARIES> FLAGS: <FLAGS> LINK_FLAGS: <LINK_FLAGS> \"")
if(NOT CMAKE_Rust_CREATE_SHARED_LIBRARY)
set(CMAKE_Rust_CREATE_SHARED_LIBRARY
"echo \"CMAKE_Rust_CREATE_SHARED_LIBRARY\""
"${CMAKE_Rust_ECHO_ALL}"
)
endif()
if(NOT CMAKE_Rust_CREATE_SHARED_MODULE)
set(CMAKE_Rust_CREATE_SHARED_MODULE
"echo \"CMAKE_Rust_CREATE_SHARED_MODULE\""
"${CMAKE_Rust_ECHO_ALL}"
)
endif()
if(NOT CMAKE_Rust_CREATE_STATIC_LIBRARY)
set(CMAKE_Rust_CREATE_STATIC_LIBRARY
"echo \"CMAKE_Rust_CREATE_STATIC_LIBRARY\""
"${CMAKE_Rust_ECHO_ALL}"
)
endif()
if(NOT CMAKE_Rust_COMPILE_OBJECT)
set(CMAKE_Rust_COMPILE_OBJECT
"echo \"CMAKE_Rust_COMPILE_OBJECT\""
"${CMAKE_Rust_ECHO_ALL}"
"${CMAKE_Rust_COMPILER} --emit obj <SOURCE> -o <OBJECT>")
endif()
if(NOT CMAKE_Rust_LINK_EXECUTABLE)
set(CMAKE_Rust_LINK_EXECUTABLE
"echo \"CMAKE_Rust_LINK_EXECUTABLE\""
"${CMAKE_Rust_ECHO_ALL}"
)
endif()
mark_as_advanced(
CMAKE_Rust_FLAGS
CMAKE_Rust_FLAGS_DEBUG
CMAKE_Rust_FLAGS_MINSIZEREL
CMAKE_Rust_FLAGS_RELEASE
CMAKE_Rust_FLAGS_RELWITHDEBINFO)
set(CMAKE_Rust_INFORMATION_LOADED 1)

0
upstream_ref/xllm/cmake/CMakeTestMUSACompiler.cmake Normal file
View File

1
upstream_ref/xllm/cmake/CMakeTestRustCompiler.cmake Normal file
View File

@@ -0,0 +1 @@
set(CMAKE_Rust_COMPILER_WORKS 1 CACHE INTERNAL "")

73
upstream_ref/xllm/cmake/FindRust.cmake Normal file
View File

@@ -0,0 +1,73 @@
# ported from https://github.com/Devolutions/CMakeRust
set(_CMAKE_FIND_ROOT_PATH_MODE_PROGRAM ${CMAKE_FIND_ROOT_PATH_MODE_PROGRAM})
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM BOTH)
set(_CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ${CMAKE_FIND_ROOT_PATH_MODE_INCLUDE})
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE BOTH)
if(CMAKE_HOST_WIN32)
set(USER_HOME "$ENV{USERPROFILE}")
else()
set(USER_HOME "$ENV{HOME}")
endif()
if(NOT DEFINED CARGO_HOME)
if("$ENV{CARGO_HOME}" STREQUAL "")
set(CARGO_HOME "${USER_HOME}/.cargo")
else()
set(CARGO_HOME "$ENV{CARGO_HOME}")
endif()
endif()
# Find cargo executable
find_program(CARGO_EXECUTABLE cargo
HINTS "${CARGO_HOME}"
PATH_SUFFIXES "bin")
mark_as_advanced(CARGO_EXECUTABLE)
# Find rustc executable
find_program(RUSTC_EXECUTABLE rustc
HINTS "${CARGO_HOME}"
PATH_SUFFIXES "bin")
mark_as_advanced(RUSTC_EXECUTABLE)
# Find rustdoc executable
find_program(RUSTDOC_EXECUTABLE rustdoc
HINTS "${CARGO_HOME}"
PATH_SUFFIXES "bin")
mark_as_advanced(RUSTDOC_EXECUTABLE)
# Find rust-gdb executable
find_program(RUST_GDB_EXECUTABLE rust-gdb
HINTS "${CARGO_HOME}"
PATH_SUFFIXES "bin")
mark_as_advanced(RUST_GDB_EXECUTABLE)
# Find rust-lldb executable
find_program(RUST_LLDB_EXECUTABLE rust-lldb
HINTS "${CARGO_HOME}"
PATH_SUFFIXES "bin")
mark_as_advanced(RUST_LLDB_EXECUTABLE)
# Find rustup executable
find_program(RUSTUP_EXECUTABLE rustup
HINTS "${CARGO_HOME}"
PATH_SUFFIXES "bin")
mark_as_advanced(RUSTUP_EXECUTABLE)
set(RUST_FOUND FALSE CACHE INTERNAL "")
if(CARGO_EXECUTABLE AND RUSTC_EXECUTABLE AND RUSTDOC_EXECUTABLE)
set(RUST_FOUND TRUE CACHE INTERNAL "")
set(CARGO_HOME "${CARGO_HOME}" CACHE PATH "Rust Cargo Home")
execute_process(COMMAND ${RUSTC_EXECUTABLE} --version OUTPUT_VARIABLE RUSTC_VERSION OUTPUT_STRIP_TRAILING_WHITESPACE)
string(REGEX REPLACE "rustc ([^ ]+) .*" "\\1" RUSTC_VERSION "${RUSTC_VERSION}")
endif()
if(NOT RUST_FOUND)
message(FATAL_ERROR "Could not find Rust!")
endif()
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM ${_CMAKE_FIND_ROOT_PATH_MODE_PROGRAM})
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ${_CMAKE_FIND_ROOT_PATH_MODE_INCLUDE})

94
upstream_ref/xllm/cmake/cargo_library.cmake Normal file
View File

@@ -0,0 +1,94 @@
include(CMakeParseArguments)
# inspired by https://github.com/abseil/abseil-cpp
# cc_library()
# CMake function to imitate Bazel's cc_library rule.
function(cargo_library)
cmake_parse_arguments(
CARGO # prefix
"" # options
"NAME" # one value args
"HDRS" # multi value args
${ARGN}
)
string(REPLACE "-" "_" LIB_NAME ${CARGO_NAME})
# set(CARGO_TARGET_DIR ${CMAKE_CURRENT_BINARY_DIR})
# figure out the target triple
if(WIN32)
if(CMAKE_SIZEOF_VOID_P EQUAL 8)
set(LIB_TARGET "x86_64-pc-windows-msvc")
else()
set(LIB_TARGET "i686-pc-windows-msvc")
endif()
elseif(ANDROID)
if(ANDROID_SYSROOT_ABI STREQUAL "x86")
set(LIB_TARGET "i686-linux-android")
elseif(ANDROID_SYSROOT_ABI STREQUAL "x86_64")
set(LIB_TARGET "x86_64-linux-android")
elseif(ANDROID_SYSROOT_ABI STREQUAL "arm")
set(LIB_TARGET "arm-linux-androideabi")
elseif(ANDROID_SYSROOT_ABI STREQUAL "arm64")
set(LIB_TARGET "aarch64-linux-android")
endif()
elseif(IOS)
set(LIB_TARGET "universal")
elseif(CMAKE_SYSTEM_NAME STREQUAL Darwin)
set(LIB_TARGET "x86_64-apple-darwin")
else()
if(CMAKE_SYSTEM_PROCESSOR STREQUAL "aarch64")
set(LIB_TARGET "aarch64-unknown-linux-gnu")
elseif(CMAKE_SIZEOF_VOID_P EQUAL 8)
set(LIB_TARGET "x86_64-unknown-linux-gnu")
else()
set(LIB_TARGET "i686-unknown-linux-gnu")
endif()
endif()
if(CMAKE_BUILD_TYPE STREQUAL "Debug")
set(LIB_BUILD_TYPE "debug")
else()
set(LIB_BUILD_TYPE "release")
endif()
if(IOS)
set(CARGO_ARGS "lipo")
else()
set(CARGO_ARGS "build")
list(APPEND CARGO_ARGS "--target" ${LIB_TARGET})
endif()
if(${LIB_BUILD_TYPE} STREQUAL "release")
list(APPEND CARGO_ARGS "--release")
endif()
file(GLOB_RECURSE LIB_SOURCES "*.rs")
set(CARGO_ENV_COMMAND ${CMAKE_COMMAND} -E env "CARGO_TARGET_DIR=${CMAKE_CURRENT_BINARY_DIR}")
# build the library target with cargo
set(STATIC_LIB_NAME
"${CMAKE_STATIC_LIBRARY_PREFIX}${LIB_NAME}${CMAKE_STATIC_LIBRARY_SUFFIX}")
set(LIB_FILE
"${CMAKE_CURRENT_BINARY_DIR}/${LIB_TARGET}/${LIB_BUILD_TYPE}/${STATIC_LIB_NAME}")
message(STATUS "running: ${CARGO_ENV_COMMAND} ${CARGO_EXECUTABLE} ARGS ${CARGO_ARGS}")
add_custom_command(
OUTPUT ${LIB_FILE}
COMMAND ${CARGO_ENV_COMMAND} ${CARGO_EXECUTABLE} ARGS ${CARGO_ARGS}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
DEPENDS ${LIB_SOURCES}
COMMENT "Building cargo library ${LIB_FILE}"
)
add_custom_target(${CARGO_NAME}_target ALL DEPENDS ${LIB_FILE})
# add the library target
add_library(${CARGO_NAME} STATIC IMPORTED GLOBAL)
add_dependencies(${CARGO_NAME} ${CARGO_NAME}_target)
set_target_properties(${CARGO_NAME} PROPERTIES
IMPORTED_LOCATION ${LIB_FILE}
)
target_sources(${CARGO_NAME} INTERFACE ${CARGO_HDRS})
endfunction()

93
upstream_ref/xllm/cmake/cargo_shared_library.cmake Normal file
View File

@@ -0,0 +1,93 @@
include(CMakeParseArguments)
# inspired by https://github.com/abseil/abseil-cpp
# cc_library()
# CMake function to imitate Bazel's cc_library rule.
function(cargo_shared_library)
cmake_parse_arguments(
CARGO # prefix
"" # options
"NAME" # one value args
"HDRS" # multi value args
${ARGN}
)
string(REPLACE "-" "_" LIB_NAME ${CARGO_NAME})
# set(CARGO_TARGET_DIR ${CMAKE_CURRENT_BINARY_DIR})
# figure out the target triple
if(WIN32)
if(CMAKE_SIZEOF_VOID_P EQUAL 8)
set(LIB_TARGET "x86_64-pc-windows-msvc")
else()
set(LIB_TARGET "i686-pc-windows-msvc")
endif()
elseif(ANDROID)
if(ANDROID_SYSROOT_ABI STREQUAL "x86")
set(LIB_TARGET "i686-linux-android")
elseif(ANDROID_SYSROOT_ABI STREQUAL "x86_64")
set(LIB_TARGET "x86_64-linux-android")
elseif(ANDROID_SYSROOT_ABI STREQUAL "arm")
set(LIB_TARGET "arm-linux-androideabi")
elseif(ANDROID_SYSROOT_ABI STREQUAL "arm64")
set(LIB_TARGET "aarch64-linux-android")
endif()
elseif(IOS)
set(LIB_TARGET "universal")
elseif(CMAKE_SYSTEM_NAME STREQUAL Darwin)
set(LIB_TARGET "x86_64-apple-darwin")
else()
if(CMAKE_SYSTEM_PROCESSOR STREQUAL "aarch64")
set(LIB_TARGET "aarch64-unknown-linux-gnu")
elseif(CMAKE_SIZEOF_VOID_P EQUAL 8)
set(LIB_TARGET "x86_64-unknown-linux-gnu")
else()
set(LIB_TARGET "i686-unknown-linux-gnu")
endif()
endif()
if(CMAKE_BUILD_TYPE STREQUAL "Debug")
set(LIB_BUILD_TYPE "debug")
else()
set(LIB_BUILD_TYPE "release")
endif()
if(IOS)
set(CARGO_ARGS "lipo")
else()
set(CARGO_ARGS "build")
list(APPEND CARGO_ARGS "--target" ${LIB_TARGET})
endif()
if(${LIB_BUILD_TYPE} STREQUAL "release")
list(APPEND CARGO_ARGS "--release")
endif()
file(GLOB_RECURSE LIB_SOURCES "*.rs")
set(CARGO_ENV_COMMAND ${CMAKE_COMMAND} -E env "CARGO_TARGET_DIR=${CMAKE_CURRENT_BINARY_DIR}")
# build the library target with cargo
set(SHARED_LIB_NAME
"${CMAKE_SHARED_LIBRARY_PREFIX}${LIB_NAME}${CMAKE_SHARED_LIBRARY_SUFFIX}")
set(LIB_FILE
"${CMAKE_CURRENT_BINARY_DIR}/${LIB_TARGET}/${LIB_BUILD_TYPE}/${SHARED_LIB_NAME}")
add_custom_command(
OUTPUT ${LIB_FILE}
COMMAND ${CARGO_ENV_COMMAND} ${CARGO_EXECUTABLE} ARGS ${CARGO_ARGS}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
DEPENDS ${LIB_SOURCES}
COMMENT "Building cargo library ${LIB_FILE}"
)
add_custom_target(${CARGO_NAME}_target ALL DEPENDS ${LIB_FILE})
# add the library target
add_library(${CARGO_NAME} SHARED IMPORTED GLOBAL)
add_dependencies(${CARGO_NAME} ${CARGO_NAME}_target)
set_target_properties(${CARGO_NAME} PROPERTIES
IMPORTED_LOCATION ${LIB_FILE}
INTERFACE_INCLUDE_DIRECTORIES ${CMAKE_CURRENT_SOURCE_DIR}
IMPORTED_NO_SONAME TRUE
)
endfunction()

60
upstream_ref/xllm/cmake/cc_binary.cmake Normal file
View File

@@ -0,0 +1,60 @@
include(CMakeParseArguments)
# inspired by https://github.com/abseil/abseil-cpp
# cc_binary()
# CMake function to imitate Bazel's cc_binary rule.
#
# Parameters:
# NAME: name of target
# HDRS: List of public header files for the library
# SRCS: List of source files for the library
# COPTS: List of private compile options
# DEFINES: List of public defines
# LINKOPTS: List of link options
# DEPS: List of other libraries to be linked in to the binary targets
#
# cc_library(
# NAME
# awesome
# HDRS
# "a.h"
# SRCS
# "a.cc"
# )
# cc_binary(
# NAME
# fantastic
# SRCS
# "b.cc"
# DEPS
# :awesome
# )
#
function(cc_binary)
cmake_parse_arguments(
CC_BINARY # prefix
"" # options
"NAME" # one value args
"HDRS;SRCS;COPTS;DEFINES;LINKOPTS;DEPS" # multi value args
${ARGN}
)
add_executable(${CC_BINARY_NAME} "")
target_sources(${CC_BINARY_NAME}
PRIVATE ${CC_BINARY_SRCS} ${CC_BINARY_HDRS}
)
target_link_libraries(${CC_BINARY_NAME}
PUBLIC
${CC_BINARY_DEPS}
PRIVATE
${CC_BINARY_LINKOPTS}
)
target_include_directories(${CC_BINARY_NAME}
PUBLIC
"$<BUILD_INTERFACE:${COMMON_INCLUDE_DIRS}>"
)
target_compile_options(${CC_BINARY_NAME} PRIVATE ${CC_BINARY_COPTS})
target_compile_definitions(${CC_BINARY_NAME} PUBLIC ${CC_BINARY_DEFINES})
add_executable(:${CC_BINARY_NAME} ALIAS ${CC_BINARY_NAME})
endfunction()

92
upstream_ref/xllm/cmake/cc_library.cmake Normal file
View File

@@ -0,0 +1,92 @@
include(CMakeParseArguments)
# inspired by https://github.com/abseil/abseil-cpp
# cc_library()
# CMake function to imitate Bazel's cc_library rule.
#
# Parameters:
# NAME: name of target
# HDRS: List of public header files for the library
# SRCS: List of source files for the library
# DEPS: List of other libraries to be linked in to the binary targets
# COPTS: List of private compile options
# DEFINES: List of public defines
# LINKOPTS: List of link options
#
# cc_library(
# NAME
# awesome
# HDRS
# "a.h"
# SRCS
# "a.cc"
# )
# cc_library(
# NAME
# fantastic_lib
# SRCS
# "b.cc"
# DEPS
# :awesome
# )
#
function(cc_library)
cmake_parse_arguments(
CC_LIB # prefix
"TESTONLY" # options
"NAME" # one value args
"HDRS;SRCS;COPTS;DEFINES;LINKOPTS;DEPS;INCLUDES" # multi value args
${ARGN}
)
if(CC_LIB_TESTONLY AND (NOT BUILD_TESTING))
return()
endif()
# Check if this is a header only library
set(_CC_SRCS "${CC_LIB_SRCS}")
foreach(src_file IN LISTS _CC_SRCS)
if(${src_file} MATCHES ".*\\.(h|inc)")
list(REMOVE_ITEM _CC_SRCS "${src_file}")
endif()
endforeach()
if(_CC_SRCS STREQUAL "")
set(CC_LIB_IS_INTERFACE 1)
else()
set(CC_LIB_IS_INTERFACE 0)
endif()
if(NOT CC_LIB_IS_INTERFACE)
add_library(${CC_LIB_NAME} STATIC)
target_sources(${CC_LIB_NAME}
PRIVATE ${CC_LIB_SRCS} ${CC_LIB_HDRS})
target_link_libraries(${CC_LIB_NAME}
PUBLIC ${CC_LIB_DEPS}
PRIVATE ${CC_LIB_LINKOPTS}
)
target_include_directories(${CC_LIB_NAME}
PUBLIC
"$<BUILD_INTERFACE:${COMMON_INCLUDE_DIRS}>"
${CC_LIB_INCLUDES}
)
target_compile_options(${CC_LIB_NAME} PRIVATE ${CC_LIB_COPTS})
target_compile_definitions(${CC_LIB_NAME} PUBLIC ${CC_LIB_DEFINES})
else()
# Generating header only library
add_library(${CC_LIB_NAME} INTERFACE)
target_include_directories(${CC_LIB_NAME}
INTERFACE
"$<BUILD_INTERFACE:${COMMON_INCLUDE_DIRS}>"
${CC_LIB_INCLUDES}
)
target_link_libraries(${CC_LIB_NAME}
INTERFACE ${CC_LIB_DEPS} ${CC_LIB_LINKOPTS}
)
target_compile_definitions(${CC_LIB_NAME} INTERFACE ${CC_LIB_DEFINES})
endif()
# add alias for the library target
add_library(:${CC_LIB_NAME} ALIAS ${CC_LIB_NAME})
endfunction()

104
upstream_ref/xllm/cmake/cc_shared_library.cmake Normal file
View File

@@ -0,0 +1,104 @@
include(CMakeParseArguments)
# inspired by https://github.com/abseil/abseil-cpp
# cc_shared_library()
# CMake function to imitate Bazel's cc_shared_library rule.
#
# Parameters:
# NAME: name of target
# HDRS: List of public header files for the library
# SRCS: List of source files for the library
# DEPS: List of other libraries to be linked in to the binary targets
# COPTS: List of private compile options
# DEFINES: List of public defines
# LINKOPTS: List of link options
#
# cc_library(
# NAME
# awesome
# HDRS
# "a.h"
# SRCS
# "a.cc"
# )
# cc_shared_library(
# NAME
# fantastic_lib
# SRCS
# "b.cc"
# DEPS
# :awesome
# )
#
function(cc_shared_library)
cmake_parse_arguments(
CC_LIB # prefix
"TESTONLY" # options
"NAME" # one value args
"HDRS;SRCS;COPTS;DEFINES;LINKOPTS;DEPS;INCLUDES" # multi value args
${ARGN}
)
if(CC_LIB_TESTONLY AND (NOT BUILD_TESTING))
return()
endif()
# Check if this is a header only library
set(_CC_SRCS "${CC_LIB_SRCS}")
foreach(src_file IN LISTS _CC_SRCS)
if(${src_file} MATCHES ".*\\.(h|inc)")
list(REMOVE_ITEM _CC_SRCS "${src_file}")
endif()
endforeach()
if(_CC_SRCS STREQUAL "")
set(CC_LIB_IS_INTERFACE 1)
else()
set(CC_LIB_IS_INTERFACE 0)
endif()
if(NOT CC_LIB_IS_INTERFACE)
add_library(${CC_LIB_NAME} SHARED)
target_sources(${CC_LIB_NAME}
PRIVATE ${CC_LIB_SRCS} ${CC_LIB_HDRS})
target_compile_options(${CC_LIB_NAME} PRIVATE
-fvisibility=hidden
-fvisibility-inlines-hidden
)
target_link_options(${CC_LIB_NAME} PRIVATE
-Wl,--exclude-libs,ALL
-Wl,--no-undefined
)
target_link_options(${CC_LIB_NAME} PRIVATE -Wl,--whole-archive)
target_link_libraries(${CC_LIB_NAME}
PUBLIC ${CC_LIB_DEPS}
PRIVATE ${CC_LIB_LINKOPTS}
)
target_link_options(${CC_LIB_NAME} PRIVATE -Wl,--no-whole-archive)
target_include_directories(${CC_LIB_NAME}
PUBLIC
"$<BUILD_INTERFACE:${COMMON_INCLUDE_DIRS}>"
${CC_LIB_INCLUDES}
)
target_compile_options(${CC_LIB_NAME} PRIVATE ${CC_LIB_COPTS})
target_compile_definitions(${CC_LIB_NAME} PUBLIC ${CC_LIB_DEFINES})
else()
# Generating header only library
add_library(${CC_LIB_NAME} INTERFACE)
target_include_directories(${CC_LIB_NAME}
INTERFACE
"$<BUILD_INTERFACE:${COMMON_INCLUDE_DIRS}>"
${CC_LIB_INCLUDES}
)
target_link_libraries(${CC_LIB_NAME}
INTERFACE ${CC_LIB_DEPS} ${CC_LIB_LINKOPTS}
)
target_compile_definitions(${CC_LIB_NAME} INTERFACE ${CC_LIB_DEFINES})
endif()
# add alias for the library target
add_library(:${CC_LIB_NAME} ALIAS ${CC_LIB_NAME})
endfunction()

127
upstream_ref/xllm/cmake/cc_test.cmake Normal file
View File

@@ -0,0 +1,127 @@
include(CMakeParseArguments)
# inspired by https://github.com/abseil/abseil-cpp
# cc_test()
# CMake function to imitate Bazel's cc_test rule.
#
# Parameters:
# NAME: name of target (see Usage below)
# SRCS: List of source files for the binary
# DEPS: List of other libraries to be linked in to the binary targets
# COPTS: List of private compile options
# LINKOPTS: List of link options
# ARGS: Command line arguments to test case
#
# Usage:
# cc_library(
# NAME
# awesome
# HDRS
# "a.h"
# SRCS
# "a.cc"
# )
#
# cc_test(
# NAME
# awesome_test
# SRCS
# "awesome_test.cc"
# DEPS
# :awesome
# GTest::gmock
# )
#
function(cc_test)
if(NOT BUILD_TESTING)
return()
endif()
cmake_parse_arguments(
CC_TEST # prefix
"" # options
"NAME" # one value args
"SRCS;COPTS;LINKOPTS;DEPS;INCLUDES;ARGS;DATA" # multi value args
${ARGN}
)
# place test data in build directory
if(CC_TEST_DATA)
foreach(data ${CC_TEST_DATA})
configure_file(${data} ${CMAKE_CURRENT_BINARY_DIR}/${data} COPYONLY)
endforeach()
endif()
set(_CC_TEST_SRCS "")
set(_CC_TEST_INCLUDE_DIRS ${CC_TEST_INCLUDES})
list(APPEND _CC_TEST_INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR})
# xllm test sources live under tests and often include private headers
# from the mirrored production source directory.
if(DEFINED XLLM_TESTS_DIR)
list(APPEND _CC_TEST_INCLUDE_DIRS
${PROJECT_SOURCE_DIR}/xllm
${XLLM_TESTS_DIR}
${XLLM_TESTS_DIR}/core
)
endif()
foreach(src IN LISTS CC_TEST_SRCS)
if(IS_ABSOLUTE "${src}")
list(APPEND _CC_TEST_SRCS "${src}")
get_filename_component(src_dir "${src}" DIRECTORY)
list(APPEND _CC_TEST_INCLUDE_DIRS "${src_dir}")
continue()
endif()
set(src_path "${CMAKE_CURRENT_SOURCE_DIR}/${src}")
if(EXISTS "${src_path}")
list(APPEND _CC_TEST_SRCS "${src}")
get_filename_component(src_dir "${src_path}" DIRECTORY)
list(APPEND _CC_TEST_INCLUDE_DIRS "${src_dir}")
if(DEFINED XLLM_TESTS_DIR)
file(RELATIVE_PATH xllm_test_src_dir "${XLLM_TESTS_DIR}" "${src_dir}")
if(NOT xllm_test_src_dir MATCHES "^\\.\\.")
list(APPEND _CC_TEST_INCLUDE_DIRS
"${PROJECT_SOURCE_DIR}/xllm/${xllm_test_src_dir}"
)
endif()
endif()
continue()
endif()
list(APPEND _CC_TEST_SRCS "${src}")
endforeach()
list(REMOVE_DUPLICATES _CC_TEST_INCLUDE_DIRS)
add_executable(${CC_TEST_NAME})
target_sources(${CC_TEST_NAME} PRIVATE ${_CC_TEST_SRCS})
target_include_directories(${CC_TEST_NAME}
PUBLIC
"$<BUILD_INTERFACE:${COMMON_INCLUDE_DIRS}>"
${_CC_TEST_INCLUDE_DIRS}
)
target_compile_options(${CC_TEST_NAME}
PRIVATE ${CC_TEST_COPTS}
)
target_link_libraries(${CC_TEST_NAME}
PUBLIC ${CC_TEST_DEPS}
PRIVATE ${CC_TEST_LINKOPTS}
)
if(USE_NPU)
set(COMMON_LIBS Python::Python torch_npu torch_python)
target_link_libraries(${CC_TEST_NAME} PRIVATE ${COMMON_LIBS})
endif()
add_dependencies(all_tests ${CC_TEST_NAME})
gtest_add_tests(
TARGET ${CC_TEST_NAME}
EXTRA_ARGS ${CC_TEST_ARGS}
)
#add_test(NAME ${CC_TEST_NAME} COMMAND ${CC_TEST_NAME} ${CC_TEST_ARGS})
endfunction()

69
upstream_ref/xllm/cmake/proto_library.cmake Normal file
View File

@@ -0,0 +1,69 @@
include(CMakeParseArguments)
include(CMakePrintHelpers)
# inspired by https://github.com/abseil/abseil-cpp
# proto_library()
# CMake function to imitate Bazel's proto_library rule.
#
# Parameters:
# NAME: name of target
# SRCS: List of proto source files for the library
# DEPS: List of other libraries to be linked in to the binary targets
# COPTS: List of private compile options
# DEFINES: List of public defines
# LINKOPTS: List of link options
#
# cc_library(
# NAME
# awesome
# HDRS
# "a.h"
# SRCS
# "a.cc"
# )
# proto_library(
# NAME
# proto_lib
# SRCS
# "b.proto"
# DEPS
# :awesome
# )
#
function(proto_library)
# parse arguments and set variables
cmake_parse_arguments(
PROTO_LIB # prefix
"" # options
"NAME" # one value args
"SRCS;COPTS;DEFINES;LINKOPTS;DEPS" # multi value args
${ARGN}
)
# generate cpp and hpp files from proto files using protoc compiler
protobuf_generate_cpp(PROTO_SRCS PROTO_HDRS ${PROTO_LIB_SRCS})
add_library(${PROTO_LIB_NAME} STATIC)
target_sources(${PROTO_LIB_NAME}
PRIVATE ${PROTO_SRCS} ${PROTO_HDRS}
)
target_link_libraries(${PROTO_LIB_NAME}
PUBLIC protobuf::libprotobuf
)
target_include_directories(${PROTO_LIB_NAME}
PUBLIC
${Protobuf_INCLUDE_DIRS}
${CMAKE_CURRENT_BINARY_DIR}
)
target_compile_options(${PROTO_LIB_NAME}
PRIVATE
${PROTO_LIB_COPTS}
-Wno-unused-parameter
)
target_compile_definitions(${PROTO_LIB_NAME}
PUBLIC
${PROTO_LIB_DEFINES}
)
add_library(proto::${PROTO_LIB_NAME} ALIAS ${PROTO_LIB_NAME})
endfunction()

103
upstream_ref/xllm/cmake/pybind_extension.cmake Normal file
View File

@@ -0,0 +1,103 @@
include(CMakeParseArguments)
# pybind_extension()
#
# Parameters:
# NAME: name of module
# HDRS: List of public header files for the library
# SRCS: List of source files for the library
# DEPS: List of other libraries to be linked in to the binary targets
# COPTS: List of private compile options
# DEFINES: List of public defines
# LINKOPTS: List of link options
#
# pybind_extension(
# NAME
# awesome
# HDRS
# "a.h"
# SRCS
# "a.cc"
# )
#
if(NOT DEFINED PYTHON_MODULE_EXTENSION OR NOT DEFINED PYTHON_MODULE_DEBUG_POSTFIX)
execute_process(
COMMAND
"${Python_EXECUTABLE}" "-c"
"import sys, importlib; s = importlib.import_module('distutils.sysconfig' if sys.version_info < (3, 10) else 'sysconfig'); print(s.get_config_var('EXT_SUFFIX') or s.get_config_var('SO'))"
OUTPUT_VARIABLE _PYTHON_MODULE_EXT_SUFFIX
ERROR_VARIABLE _PYTHON_MODULE_EXT_SUFFIX_ERR
OUTPUT_STRIP_TRAILING_WHITESPACE)
if(_PYTHON_MODULE_EXT_SUFFIX STREQUAL "")
message(
FATAL_ERROR "pybind11 could not query the module file extension, likely the 'distutils'"
"package is not installed. Full error message:\n${_PYTHON_MODULE_EXT_SUFFIX_ERR}"
)
endif()
# This needs to be available for the pybind11_extension function
if(NOT DEFINED PYTHON_MODULE_DEBUG_POSTFIX)
get_filename_component(_PYTHON_MODULE_DEBUG_POSTFIX "${_PYTHON_MODULE_EXT_SUFFIX}" NAME_WE)
set(PYTHON_MODULE_DEBUG_POSTFIX
"${_PYTHON_MODULE_DEBUG_POSTFIX}"
CACHE INTERNAL "")
endif()
if(NOT DEFINED PYTHON_MODULE_EXTENSION)
get_filename_component(_PYTHON_MODULE_EXTENSION "${_PYTHON_MODULE_EXT_SUFFIX}" EXT)
set(PYTHON_MODULE_EXTENSION
"${_PYTHON_MODULE_EXTENSION}"
CACHE INTERNAL "")
endif()
endif()
function(pybind_extension)
cmake_parse_arguments(
PY # prefix
"TESTONLY" # options
"NAME" # one value args
"HDRS;SRCS;COPTS;DEFINES;LINKOPTS;LINKDIRS;DEPS" # multi value args
${ARGN}
)
if(PY_TESTONLY AND (NOT BUILD_TESTING))
return()
endif()
add_dependencies(export_module ${PY_NAME})
add_library(${PY_NAME} SHARED)
target_sources(${PY_NAME}
PRIVATE ${PY_SRCS} ${PY_HDRS}
)
target_link_libraries(${PY_NAME}
PUBLIC ${PY_DEPS}
PRIVATE ${PY_LINKOPTS}
)
# search directories for libraries
target_link_directories(${PY_NAME}
PUBLIC ${PY_LINKDIRS}
)
target_compile_options(${PY_NAME} PRIVATE ${PY_COPTS})
target_compile_definitions(${PY_NAME} PUBLIC ${PY_DEFINES})
# -fvisibility=hidden is required to allow multiple modules compiled against
# different pybind versions to work properly, and for some features (e.g.
# py::module_local).
if(NOT DEFINED CMAKE_CXX_VISIBILITY_PRESET)
set_target_properties(${PY_NAME} PROPERTIES CXX_VISIBILITY_PRESET "hidden")
endif()
if(NOT DEFINED CMAKE_CUDA_VISIBILITY_PRESET)
set_target_properties(${PY_NAME} PROPERTIES CUDA_VISIBILITY_PRESET "hidden")
endif()
set_target_properties(
${PY_NAME}
PROPERTIES PREFIX ""
DEBUG_POSTFIX "${PYTHON_MODULE_DEBUG_POSTFIX}"
SUFFIX "${PYTHON_MODULE_EXTENSION}")
endfunction()

169
upstream_ref/xllm/docker/Dockerfile.cuda Executable file
View File

@@ -0,0 +1,169 @@
ARG CUDA_VERSION=12.8.0
ARG BASE_IMAGE=nvidia/cuda:${CUDA_VERSION}-cudnn-devel-ubuntu24.04
FROM ${BASE_IMAGE}
ARG CUDA_VERSION
ARG PYTHON_VERSION=3.11
ARG CMAKE_VERSION=3.27.9
ARG TORCH_VERSION=2.7.1
ARG TORCH_INDEX_URL=https://download.pytorch.org/whl/cu128
ARG TORCH_CUDA_ARCH_LIST="8.0 8.9 9.0a 10.0a 12.0a"
ARG FLASHINFER_VERSION=0.6.2
ARG FLASHINFER_CUDA_ARCH_LIST="8.0 8.9 9.0a 10.0a 12.0a"
ARG GET_PIP_URL="https://bootstrap.pypa.io/get-pip.py"
# Optional: use proxy for network access
ARG http_proxy=
ARG https_proxy=
# Optional: use pypi mirror (e.g. https://pypi.tuna.tsinghua.edu.cn/simple)
ARG UV_DEFAULT_INDEX=
# Optional: accelerate github https clone (e.g. https://gh-proxy.com/)
ARG GIT_HTTPS_MIRROR_PREFIX=
# Optional: use rustup proxy (example: https://rsproxy.cn)
ARG RUSTUP_DIST_SERVER=
ARG RUSTUP_UPDATE_ROOT=
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
WORKDIR /workspace
ENV CUDA_VERSION=${CUDA_VERSION}
ENV PYTHON_VERSION=${PYTHON_VERSION}
ENV CMAKE_VERSION=${CMAKE_VERSION}
ENV TORCH_VERSION=${TORCH_VERSION}
ENV TORCH_INDEX_URL=${TORCH_INDEX_URL}
ENV TORCH_CUDA_ARCH_LIST=${TORCH_CUDA_ARCH_LIST}
ENV DEBIAN_FRONTEND=noninteractive
ENV http_proxy=${http_proxy}
ENV https_proxy=${https_proxy}
ENV PIP_DISABLE_PIP_VERSION_CHECK=1
ENV PYTHONDONTWRITEBYTECODE=1
# # Install system dependencies including build tools
RUN set -eux; \
echo "tzdata tzdata/Areas select Asia" | debconf-set-selections; \
echo "tzdata tzdata/Zones/Asia select Shanghai" | debconf-set-selections; \
apt-get update -y && apt-get install -y --no-install-recommends \
ca-certificates gdb wget curl vim git zip unzip bison build-essential \
libx11-dev libxft-dev libxext-dev tar \
autoconf automake libtool pkg-config \
libxi-dev libxtst-dev ccache \
libwayland-dev libxrandr-dev libxinerama-dev libxcursor-dev libepoxy-dev libgtk-3-dev \
libnuma-dev libibverbs-dev nasm software-properties-common; \
add-apt-repository ppa:deadsnakes/ppa -y; \
apt install -y --no-install-recommends python${PYTHON_VERSION}-full python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv; \
update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1; \
update-alternatives --install /usr/bin/python python /usr/bin/python${PYTHON_VERSION} 1; \
update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION}; \
update-alternatives --set python /usr/bin/python${PYTHON_VERSION}; \
ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config; \
rm -rf /var/lib/apt/lists/*; \
wget -q https://bootstrap.pypa.io/get-pip.py && python get-pip.py --break-system-packages && rm get-pip.py; \
# Allow pip to install packages globally (PEP 668 workaround for Ubuntu 24.04)
python -m pip config set global.break-system-packages true; \
python --version && pip --version; \
apt-get clean
# UV envs
# Enable bytecode compilation
ENV UV_COMPILE_BYTECODE=1
ENV UV_DEFAULT_INDEX=${UV_DEFAULT_INDEX}
# This timeout (in seconds) is necessary when installing some dependencies via uv since it's likely to time out
# Reference: https://github.com/astral-sh/uv/pull/1694
ENV UV_HTTP_TIMEOUT=500
ENV UV_INDEX_STRATEGY="unsafe-best-match"
# Use copy mode to avoid hardlink failures with Docker cache mounts
ENV UV_LINK_MODE=copy
# Omit development dependencies
ENV UV_NO_DEV=1
# Ensure installed tools can be executed out of the box
ENV UV_TOOL_BIN_DIR=/usr/local/bin
ENV PIP_INDEX_URL=${PIP_INDEX_URL}
# Install python dependencies
RUN set -eux; \
pip install "torch==${TORCH_VERSION}" --index-url "${TORCH_INDEX_URL}"; \
pip install uv ninja "numpy<2" nvshmem4py-cu12 nvidia-nvshmem-cu12 \
aiohttp requests tqdm transformers
ENV NVSHMEM_HOME=/usr/local
# Install cmake
RUN set -eux; \
url="https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-linux-x86_64.tar.gz"; \
curl -fL --retry 3 --retry-delay 2 -o /tmp/cmake.tar.gz "${url}"; \
mkdir -p /opt/cmake; \
tar -xzf /tmp/cmake.tar.gz -C /opt/cmake --strip-components=1; \
rm -f /tmp/cmake.tar.gz; \
ln -sf /opt/cmake/bin/cmake /usr/local/bin/cmake; \
cmake --version
# Install Rust (minimal profile)
ENV RUSTUP_DIST_SERVER=${RUSTUP_DIST_SERVER}
ENV RUSTUP_UPDATE_ROOT=${RUSTUP_UPDATE_ROOT}
ENV CARGO_HOME=/root/.cargo
ENV RUSTUP_HOME=/root/.rustup
ENV PATH=${CARGO_HOME}/bin:$PATH
RUN set -eux; \
curl -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain stable; \
rustup --version; cargo --version; rustc --version
# Optional: use https mirror for git clone if provided
ENV GIT_HTTPS_MIRROR_PREFIX=${GIT_HTTPS_MIRROR_PREFIX}
RUN if [[ -n "${GIT_HTTPS_MIRROR_PREFIX}" ]]; then \
git config --global url."${GIT_HTTPS_MIRROR_PREFIX}".insteadOf https://; \
fi
# Install tvm-ffi
RUN set -eux; \
git clone --depth 1 --recurse-submodules https://github.com/apache/tvm-ffi.git /workspace/tvm-ffi; \
git -C /workspace/tvm-ffi submodule update --init --recursive; \
cmake -S /workspace/tvm-ffi -B /workspace/tvm-ffi/build_cpp -DCMAKE_BUILD_TYPE=RelWithDebInfo; \
cmake --build /workspace/tvm-ffi/build_cpp --parallel --config RelWithDebInfo --target tvm_ffi_shared; \
cmake --install /workspace/tvm-ffi/build_cpp --config RelWithDebInfo; \
rm -rf /workspace/tvm-ffi
# Install flashinfer + AOT compile
ENV FLASHINFER_VERSION=${FLASHINFER_VERSION}
ENV FLASHINFER_CUDA_ARCH_LIST=${FLASHINFER_CUDA_ARCH_LIST}
RUN set -eux; \
git clone --depth 1 --branch "v${FLASHINFER_VERSION}" --recurse-submodules https://github.com/flashinfer-ai/flashinfer.git /workspace/flashinfer; \
cd /workspace/flashinfer; \
uv venv --python ${PYTHON_VERSION} && source .venv/bin/activate; \
uv pip install -v .; \
uv pip install "torch==${TORCH_VERSION}" --index-url "${TORCH_INDEX_URL}"; \
uv pip install "cuda-python==${CUDA_VERSION}"; \
python -m flashinfer.aot; \
deactivate; \
ln -sf /root/.cache/flashinfer/${FLASHINFER_VERSION}/*/cached_ops /flashinfer_ops; \
rm -rf /workspace/flashinfer
ENV FLASHINFER_OPS_PATH=/flashinfer_ops
ENV LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
# Install VCPKG
RUN set -eux; \
git clone https://github.com/microsoft/vcpkg.git /vcpkg
ENV VCPKG_ROOT=/vcpkg
# Cleanup
RUN if [[ -n "${GIT_HTTPS_MIRROR_PREFIX}" ]]; then \
git config --global --unset-all url."${GIT_HTTPS_MIRROR_PREFIX}".insteadOf 2>/dev/null || true; \
fi; \
pip cache purge; \
uv cache clean
USER root
CMD ["/bin/bash"]

28
upstream_ref/xllm/docs/en/.readthedocs.yaml Normal file
View File

@@ -0,0 +1,28 @@
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the OS, Python version, and other tools you might need
build:
os: ubuntu-24.04
tools:
python: "3.13"
jobs:
pre_build:
# - cp -r docs/en/* docs/
- mv docs/en/* docs/
- rm -rf docs/zh/
- find docs/ -name "*.md" -exec sed -i 's#../assets/#assets/#g' {} \;
# Build documentation with Mkdocs
mkdocs:
configuration: mkdocs_en.yml
# Optionally, but recommended,
# declare the Python requirements required to build your documentation
# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: docs/requirements.txt

59
upstream_ref/xllm/docs/en/accuracy_test.md Normal file
View File

@@ -0,0 +1,59 @@
# 1. LLM Accuracy Test
## 1.1 Setup ais_bench
```bash
# Create a virtual environment for ais_bench using conda or uv
conda create --name ais_bench python=3.10 -y
conda activate ais_bench
# Clone ais_bench and install dependencies
git clone https://gitee.com/aisbench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517
# Download the dataset and copy it to the ais_bench directory
cp -r /path/to/dataset /path/to/benchmark/ais_bench/datasets
```
## 1.2 Modify Configuration
Modify the accuracy test configuration file according to your actual situation: `/path/to/benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py`. It is recommended to set the sampling parameters as follows:
```python
models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr='vllm-api-general-chat',
path="/path/to/model/Qwen3-8B", # Model path
model="Qwen3-8B", # Model name
request_rate = 0,
retry = 2,
host_ip = "127.0.0.1",
host_port = 19000, # xllm server port
max_out_len = 32768, # Limit maximum model length
batch_size=32,
trust_remote_code=False,
generation_kwargs = dict(
temperature = 0.6,
# top_k = -1,
top_p = 0.95,
# seed = None,
# repetition_penalty = 1,
),
pred_postprocessor=dict(type=extract_non_reasoning_content)
)
]
```
## 1.3 Launch ais_bench
Before using ais_bench, you need to start the xllm server first. Use `ais_bench -h` to get parameter descriptions. The launch commands for gsm8k and ceval datasets are as follows:
```bash
# Using gsm8k dataset
ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt --dump-eval-details
# Using ceval dataset
ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt --merge-ds --dump-eval-details
```
We will integrate ais_bench and datasets (ceval and gsm8k) into the development image in the future. The ais_bench documentation and datasets are as follows:
* [ais_bench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/index.html)
* [Datasets](https://ais-bench-benchmark.readthedocs.io/en/latest/base_tutorials/all_params/datasets.html)

83
upstream_ref/xllm/docs/en/cli_reference.md Normal file
View File

@@ -0,0 +1,83 @@
---
hide:
- navigation
---
# Service Startup Parameters
xLLM uses gflags to manage service startup parameters. The specific parameter meanings are as follows:
## Common Parameters
| Parameter Name | Data Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `master_node_addr` | `string` | "127.0.0.1:19888" | ip:port | The listening address of the master node's rpc server | [Details](./features/basics.md) |
| `host` | `string` | "" | The machine IP where the current device is located | The host IP used by the current device for communication. An rpc server is started on each device for multi-device communication. | |
| `port` | `int32` | 8010 | Any available port | Used in conjunction with the `host` parameter. The combination is used for rpc communication between devices. | |
| `model` | `string` | "" | | Path to the model. | |
| `devices` | `string` | "npu:0" | | Specifies the NPU devices used by the current process. | |
| `nnodes` | `int32` | 1 | | The total number of devices used by the current service. | |
| `node_rank` | `int32` | 0 | 0 ~ (total devices - 1) | The rank id of each device. | |
| `max_memory_utilization` | `double` | 0.8 | Between 0-1 | The maximum proportion of device memory available for model weights and KV Cache combined. | |
| `max_tokens_per_batch` | `int32` | 10240 | | The maximum number of tokens that can be computed per step. | |
| `max_seqs_per_batch` | `int32` | 1024 | | The maximum number of sequences that can be computed per step. | |
| `enable_chunked_prefill` | `bool` | true | false | Whether to enable chunked prefill. | |
| `enable_prefill_sp` | `bool` | false | true | Whether to enable prefill-only sequence parallel. | `enable_chunked_prefill=true` is supported only for prefill-only batches (`PREFILL` / `CHUNKED_PREFILL`); `MIXED` and `DECODE` batches do not run with sequence parallel. |
| `enable_schedule_overlap` | `bool` | false | true | Whether to enable asynchronous scheduling. | [Details](./features/async_schedule.md) |
| `enable_prefix_cache` | `bool` | true | false | Whether to enable prefix cache (not supported by DeepSeek currently). | |
| `communication_backend` | `string` | "hccl" | "lccl" | The backend used for communication operations. | |
| `block_size` | `int32` | 128 | | The block size for KV Cache storage. | |
| `task` | `string` | "generate" | "embed", "mm_embed" | Service type: generation, embedding, or multimodal embedding. | |
| `max_cache_size` | `int64` | 0 | | The usable KV Cache size in bytes. | |
| `kv_cache_dtype` | `string` | "auto" | "int8" | KV Cache data type. "auto" aligns with model dtype (no quantization), "int8" enables INT8 quantization to save ~50% memory. MLU backend only. | |
## MoE Model Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `dp_size` | `int32` | 1 | Power of 2 | The dp scale size for the Attention part. | |
| `ep_size` | `int32` | 1 | Power of 2 | The ep scale size for the MoE part. | |
| `expert_parallel_degree` | `int32` | 0 | 1,2 | Parameter related to ep parallelism. Defaults to 0 when ep is not used, and to 1 when ep is enabled. Can be set to 2 when `ep_size` equals the total number of devices (uses all2all communication). | |
## P-D Separation Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `enable_disagg_pd` | `bool` | false | true | Whether to enable P-D separation. | [Details](./features/disagg_pd.md) |
| `disagg_pd_port` | `int32` | 7777 | Any available port | Configuration when P-D separation is enabled. Corresponds to the listening port number of the pd separation rpc server started on each card. | |
| `instance_role` | `string` | DEFAULT | PREFILL, DECODE, MIX | Defaults to DEFAULT. Must be configured as PREFILL, DECODE, or MIX when P-D separation is enabled. | |
| `kv_cache_transfer_mode` | `string` | "PUSH" | "PULL" | The mode for transferring KV Cache in P-D separation. PUSH mode: Prefill transmits layer by layer to Decode; PULL mode: Decode pulls the KV Cache from Prefill in one go. | |
| `transfer_listen_port` | `int32` | 26000 | Any available port | Configuration when P-D separation is enabled. Corresponds to the listening port for KV Cache Transfer on each card. | |
## MTP Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `draft_model` | `string` | "" | | Path to the MTP model. | [Details](./features/mtp.md) |
| `draft_devices` | `string` | "npu:0" | Same format as `devices`, e.g. `npu:0` or `npu:0,npu:1` | Should be set consistently with the `devices` parameter. | |
| `num_speculative_tokens` | `int32` | 0 | Any integer, suggestion 1 or 2 | The number of tokens output by the MTP model per step. | |
## Graph Execution Related Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `enable_graph` | `bool` | false | true | Whether to enable graph execution mode to optimize decode phase performance. Only applied during decode phase and does not take effect during prefill phase. Supports ACL Graph (NPU), and MLU Graph. | [Details](./features/graph_mode.md) |
| `enable_graph_mode_decode_no_padding` | `bool` | false | true | Builds decode graphs with the actual `num_tokens` instead of the padded shape. | |
| `enable_prefill_piecewise_graph` | `bool` | false | true | Whether to enable piecewise graph for prefill phase. Attention runs eagerly while other ops are captured into CUDA graphs. | |
| `max_tokens_for_graph_mode` | `int32` | 2048 | Any integer greater than or equal to 0 | Maximum number of tokens for graph execution. If 0, no limit is applied. | |
## Parameters for Use with xLLM-service
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `etcd_addr` | `string` | "" | ip:port | The listening address of the etcd's rpc server. | |
| `enable_service_routing` | `bool` | false | true | Whether the request from the xllm service, use this when enable the xllm service. | |
## Other Parameters
| Parameter Name | Type | Default Value | Other Values | Description | Notes |
|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| `max_concurrent_requests` | `int32` | 200 | Any integer greater than or equal to 0 | For rate limiting, restricts the total number of requests being processed in the instance. Set to 0 for no limit. | |
| `model_id` | `string` | "" | | Model name, not a path. | |
| `num_request_handling_threads` | `int32` | 4 | Any integer greater than 0 | The thread pool size for handling input requests. | |
| `prefill_scheduling_memory_usage_threshold` | `double` | 0.95 | Value between 0-1 | When kv cache usage reaches this threshold, scheduling of prefill requests is paused. | |
| `num_response_handling_threads` | `int32` | 4 | Any integer greater than 0 | The thread pool size for handling outputs. | |
| `rank_tablefile` | `string` | "" | | Configuration file for creating the communication domain. Required for multi-node scenarios. | |

1062
upstream_ref/xllm/docs/en/design/generative_recommendation_design.md Normal file
View File

File diff suppressed because it is too large Load Diff

474
upstream_ref/xllm/docs/en/design/graph_mode_design.md Normal file
View File

@@ -0,0 +1,474 @@
# Graph Mode Design Document
## Overview
xLLM's Graph Mode supports multiple graph execution backends. Its goal is to turn the original Host-driven stream of fine-grained kernel launches into a capture-then-replay execution flow in inference serving, thereby reducing Host scheduling overhead, reducing device-side bubbles, and improving throughput and latency stability.
This document is intended for developers who need to understand the implementation principles and key design choices. It focuses on:
- the basic Graph Mode mechanism and how xLLM applies it
- dynamic dimension parameterization
- Piecewise Graph
- a multi-shape reusable memory pool, including input tensor reuse
This document focuses on the unified Graph Mode design in xLLM and does not expand on backend-specific platform differences.
The design goals of this document are:
- provide a unified Graph Mode abstraction across xLLM backends
- explain the three key designs: dynamic dimension parameterization, Piecewise Graph, and multi-shape memory reuse
- clarify what problem each design solves, what assumptions it depends on, and where its boundary lies
The non-goals of this document are:
- full adaptation details for every operator or every model
- replacing feature documentation for flags and usage examples
Related design documents:
- for a recommendation-oriented case study that focuses on fixed scheduling, multi-step execution, and custom operators, see: [Generative Recommendation Design Document](generative_recommendation_design.md)
## 1. Graph Mode Fundamentals
### 1.1 Capture / Replay Basics
In traditional eager execution, one forward pass launches many kernels, memory copies, and synchronization operations from the Host. For decode-like workloads, where each step is small but requests are frequent, Host scheduling overhead becomes significant and device-side bubbles become more visible.
The core idea of Graph Mode is:
1. **Capture**: when a shape bucket is seen for the first time, run one forward pass on a dedicated stream and record the kernel launches, memory operations, and dependencies into a graph.
2. **Replay**: for later requests that hit the same bucket, replay the recorded graph instead of launching kernels one by one from the Host.
![Graph Mode](../../assets/graph_mode.png)
This mechanism usually requires:
- **Stable execution path**
- graph capture records one concrete execution path; once capture completes, the control flow, launch shape, and dependencies on that path are fixed
- therefore, the capture path cannot switch to a different dynamic branch during replay
- **Stable addresses for key tensors**
- key tensor addresses written into the graph during capture must still be valid at replay time
- **Stable operator behavior and result semantics**
- replay correctness depends on operators along the path being Graph Mode compatible and not changing semantics because of runtime condition changes
### 1.2 xLLM GraphMode Runtime Foundations
Starting from the three basic requirements in 1.1, the first two are mainly handled by the xLLM runtime: turning dynamic requests into execution units that can be replayed stably, and ensuring replay still accesses the fixed addresses recorded during capture. For that reason, xLLM introduces a unified Graph Executor in Graph Mode to centralize bucketing, persistent buffers, graph cache management, and the capture / replay lifecycle.
xLLM's runtime-side foundation work mainly includes:
- **Graph selection and execution scheduling around execution-path stability**
- requests must first be grouped into buckets by `num_tokens` or a nearby shape
- graph cache is maintained per bucket: replay directly on a hit, otherwise capture first and cache the graph
- use full graphs for paths that are fully capturable, and switch to Piecewise Graph when part of the path breaks graph capture
- **Persistent buffer and graph-instance management around address stability**
- dynamic inputs such as tokens, positions, seq_lens, and block_tables cannot be reallocated to arbitrary new addresses before replay; xLLM must write them into persistent buffers first and then update content at fixed addresses
- tensors allocated temporarily during model graph construction also need to be retained with the corresponding graph instance instead of being reclaimed when their local scope ends
- when the shared memory pool is enabled, captures of different shapes can reuse the same underlying physical memory, while the virtual addresses seen by the graph still remain stable
- **A unified Graph Executor abstraction**
- capture, cache, replay, graph-instance management, and backend abstraction all need to be handled centrally by the runtime
- the point of this layer is to pull Graph Mode runtime orchestration out of model code rather than scattering it across layers or operators
In concrete execution, the Graph Executor runtime flow can be summarized as:
1. **Bucket the request**: group the request into a bucket by `num_tokens` or a nearby shape.
2. **Prepare inputs**: write tokens, positions, seq_lens, and block tables into persistent buffers, and update runtime metadata such as `attn_metadata` and `plan_info`.
3. **Choose capture or replay**: replay directly if the bucket already has a cached graph; otherwise capture, cache, and then enter the replay path.
4. **Execute the graph**: run a full graph or a piecewise graph depending on the scenario.
The third requirement in 1.1, "stable operator behavior and result semantics", falls more on model-side and operator-side adaptation, which is covered in 1.3.
### 1.3 Model Adaptation Required for Graph Mode
After the xLLM runtime takes care of bucketing, buffers, and graph-instance management, the model side still needs to satisfy the third requirement from 1.1: operator behavior and result semantics must remain stable across capture / replay and must not break because of host participation, dispatch changes, or launch-shape changes.
The required model adaptations mainly include:
- **Remove operations that are not capture-safe**
- General requirement: the capture path must not contain operations that trigger host-side decisions, implicit host-device synchronization, or extra control-flow changes, such as stream synchronization, logic branches driven by host tensors, or implicit host-device interactions
- Typical example 1: operators may depend on host tensor contents to decide task partitioning or execution mode. For example, ATB [PA](https://www.hiascend.com/document/detail/zh/canncommercial/83RC1/API/ascendtbapi/ascendtb_01_0197.html) and [MLA](https://www.hiascend.com/document/detail/zh/canncommercial/83RC1/API/ascendtbapi/ascendtb_01_0314.html) in Ascend CANN commercial 8.3 can choose different task layouts based on host-side `kv_seq_lens` and `q_seq_lens`
- Typical example 2: a host scalar participates in tensor computation, or the host directly reads tensor data, for example:
```cpp
Tensor a;
Tensor b = a * 0.5; // the scalar is implicitly transferred to the tensor's device, causing a synchronized H2D
torch::Tensor max_of_seq = torch::max(input_params.kv_seq_lens);
max_seq_len_ = std::max(max_of_seq.item<int>(), max_seq_len_); // host reads tensor data
```
- **Adapt operators**
- General requirement: the kernel execution path must remain stable for the same input shape. For one input shape, graph execution should choose the same kernel every time, without mixing prefill, chunked prefill, and decode behavior. At the same time, `grid_dim`, `block_dim`, task count, workspace shape, and tiling content must also remain stable
- Typical example: in Ascend CANN commercial 8.3, ATB PA may decide at runtime whether to enable flash-decoding long-sequence mode based on sequence length and batch size. The long-sequence and short-sequence modes use different tiling keys, and the actual dispatched `kernel_name` is also different. This kind of behavior, where the same business path triggers different kernels under different runtime conditions, must be converged before entering Graph Mode
## 2. Dynamic Dimension Parameterization
### 2.1 Problem
Graph capture records a fixed kernel sequence and the launch parameters of every kernel. At replay time, the runtime can only repeat that recorded launch shape. It cannot re-decide `grid_dim`, `block_dim`, task count, or the tiling path inside the same graph.
As a result, not every dynamic request dimension can continue to vary once the request enters Graph Mode. For attention, the real dynamic factors usually include more than `num_tokens`: they also include `batch_size`, `q_seq_lens`, `kv_seq_lens`, and `block_tables_size`. If these dimensions further affect task partitioning, workspace layout, `tiling_params`, or kernel path, replay may reuse the stale capture-time configuration and become incorrect.
The core question of this chapter is therefore: which dynamic information can still be updated after the graph has been selected, and which dimensions must instead be handled by finer-grained bucketing or by building another graph.
### 2.2 Solution
The goal of dynamic dimension parameterization is not to make one graph accept arbitrary `num_tokens`. Instead, once the graph has already been selected by `bucket_num_tokens`, the goal is to let the same bucket safely cover more real dynamic requests.
#### 2.2.1 Parameterization Boundary
Whether a dimension can still vary at replay time does not depend on whether it is dynamic in the request. It depends on whether that dimension has already been folded into the execution shape during capture.
Dynamic factors can be divided into two categories:
1. **Dimensions that determine graph shape**
- once a dimension enters `grid_dim`, `block_dim`, task count, workspace layout, tiling key, or `tiling_params`, it becomes part of the capture result
- after that, it usually cannot be changed inside the same graph
2. **Dimensions that can still be updated after graph selection**
- such as `batch_size`, `q_seq_lens`, `kv_seq_lens`, `block_tables`, `new_cache_slots`, and `plan_info`
- they are suitable parameterization targets only if they do not further rewrite execution shape
A simplified norm-kernel example is:
```cpp
int grid_dim = num_tokens;
NormKernel<<<grid_dim, block_dim>>>(x, y, ...);
```
Here, `grid_dim` and `block_dim` are both launch parameters. Once `num_tokens` participates in launch configuration, it becomes part of graph shape; after capture, replay can no longer change `grid_dim = 128` into `grid_dim = 256` within the same graph.
So the boundary is straightforward: any dynamic factor that changes launch shape, workspace, or tiling path should not be handled only by updating parameters before replay.
#### 2.2.2 How xLLM Handles It
xLLM handles this in two layers.
The first layer handles `num_tokens`: xLLM uses bucketing and padding to normalize the raw request into a fixed `bucket_num_tokens`, and then selects the graph accordingly.
The second layer handles the remaining dynamic information inside that bucket: adapt operators that depend on host tensors, host planning, or host-side tiling so that request-varying information is stored in device-side persistent buffers, or otherwise externalized into data that can be refreshed before replay, instead of letting replay stay bound to the old host-side planning result from capture.
Take NPU attention as an example. A common pre-adaptation problem is that `q_seq_lens` and `kv_seq_lens` first participate in host-side planning, which produces task, workspace, or tiling-related data before launching the kernel:
```cpp
// Host side
auto plan_info = PlanAttention(q_seq_lens_host, kv_seq_lens_host, ...);
WritePlanToWorkspace(attention_workspace, plan_info);
AttentionKernel<<<grid_dim, block_dim>>>(
q, k, v, out,
attention_workspace,
tiling_params);
```
In this kind of implementation, graph replay does not automatically redo planning, so it can only repeat the launch shape and workspace layout from capture time. The adapted direction is to write request-varying information such as `q_seq_lens`, `kv_seq_lens`, `block_tables_size`, and `plan_info` into device-side persistent buffers under a fixed launch shape, and let the kernel read them directly. For example, a parameterized `kv_seq_lens` path may look like:
```cpp
struct AttentionLaunchArgs {
void* q;
void* k;
void* v;
void* out;
int32_t batch_size;
int32_t* kv_seq_lens; // device buffer
void* attention_workspace;
void* tiling_params;
};
LaunchAttentionKernel(args);
```
If a dynamic factor changes task count, array length, workspace shape, or tiling path as soon as it varies, xLLM does not force it into "parameterization updates". Instead, it uses one of two approaches:
1. expand the bucketing dimensions and include those factors in the graph-selection key, for example a combination of `num_tokens` and `num_comm_tokens`
2. keep that dynamic-shape-sensitive part outside the full graph and handle it with Piecewise Graph
### 2.3 Result
Dynamic dimension parameterization directly brings the following result: one `num_tokens` bucket can safely cover more real requests, and under a stable execution shape, operators such as attention can still see the correct dynamic information at replay time, including `batch_size`, `seq_lens`, `block_tables_size`, and `plan_info`.
One thing to keep in mind is that parameterization does not eliminate bucketing. The dynamic nature of `num_tokens` is still mainly absorbed by graph selection; if a bucket still contains other factors that rewrite tiling keys, task count, communication scale, or kernel path, those factors must still enter the bucketing key or move to Piecewise Graph. Communication-heavy scenarios are a typical example. In Attention DP + MoE EP, communication scale may depend on the maximum DP data size rather than strictly matching local `num_tokens` on one device.
## 3. Piecewise Graph
### 3.1 Problem
In prefill and chunked prefill scenarios, the full forward path is usually more complex than decode. Some operators may:
- fail to be captured stably
- be highly sensitive to dynamic metadata
- rely on extra runtime preparation during capture
If the requirement is that the entire path must be capturable as one full graph, Graph Mode coverage becomes very limited. Once one critical operator breaks graph capture, the whole path loses graph execution.
### 3.2 Solution
#### 3.2.1 Underlying Logic
One common question in implementation is: why can decode use a full graph while chunked prefill usually needs Piecewise Graph?
The key difference is attention.
Under current decode semantics, one step usually means one generated token per sequence, so `num_tokens` and `batch_size` are almost one-to-one. For attention, that means:
- when `num_tokens` grows, `batch_size` grows with it
- the pattern of `q_seq_lens` stays relatively stable
- task partitioning, workspace requirements, and launch arguments can mostly be bucketed together with `num_tokens`
Therefore, when building buckets by `num_tokens`, decode satisfies the reuse conditions for a full graph.
In chunked prefill, however, `num_tokens` and `batch_size` are no longer tied together. The same `num_tokens` may correspond to very different request combinations, for example:
- `num_tokens = 128, batch_size = 2, q_seq_lens = [64, 64]`
- `num_tokens = 128, batch_size = 4, q_seq_lens = [32, 32, 32, 32]`
- `num_tokens = 128, batch_size = 8, q_seq_lens = [16, 16, 16, 16, 16, 16, 16, 16]`
For attention, these requests share the same `num_tokens` but they are not the same shape. Attention task partitioning, local indexing, the lengths of `q_seq_lens` and `kv_seq_lens`, and the corresponding `plan_info` all directly depend on `batch_size`. If the system reuses a previously captured graph only by `num_tokens`, then replay may keep using the old batch partitioning and old task partitioning from capture, which can make the result incorrect.
`batch_size` is hard to parameterize here because it changes more than just one scalar value. It changes the structure of the launch arguments themselves:
- the lengths of `q_seq_lens` and `kv_seq_lens` change
- the task count and task boundaries of attention change
- `plan_info` and workspace layout change
- the indexing and partition logic inside the kernel change
In other words, in chunked prefill, `batch_size` is not a dimension that can be handled by updating a few parameters. It changes the execution shape of the attention part itself. Unless `num_tokens × batch_size`, or even more fine-grained `seq_lens` combinations, are added into bucketing, selecting graphs only by `num_tokens` is not safe.
That is why the current design is better suited to:
- **decode**: attention can be captured together with the full graph
- **chunked prefill**: attention stays outside the graph while the more stable surrounding parts enter Piecewise Graph
#### 3.2.2 Core Idea
The goal of Piecewise Graph is not to force full-graph capture. It is to split one full execution path into several pieces:
- graph-friendly parts: captured into subgraphs
- parts that are not suitable for graph capture: kept in eager mode
At replay time, those pieces are executed in the same order recorded during capture.
This preserves as much Graph Mode benefit as possible even when full-graph capture is not feasible.
The following diagram uses a three-layer Qwen3 decoder as an example. Attention stays as an independent runner, while the non-attention operators between attention calls are packed as tightly as possible into continuous graph pieces.
![Piecewise Graph Diagram](../../assets/piecewise_graph_diagram.svg)
#### 3.2.3 How xLLM Implements It
In xLLM, Piecewise Graph is mainly used for prefill-like scenarios and is controlled by `enable_prefill_piecewise_graph`.
Its core implementation is:
1. use `PiecewiseGraphs` to maintain the replay instruction sequence
2. store capturable segments as graph objects
3. record non-graph attention segments as runners through `AttentionRunner`
4. replay in the original order as `graph -> runner -> graph -> ...`
The key point is how attention is handled:
- during piecewise capture, attention temporarily ends the current graph capture
- attention itself is not executed during capture; instead, the tensors, workspace, and parameters required for attention are recorded
- at replay time, attention runs with the latest `plan_info`, `q_cu_seq_lens`, and `kv_cu_seq_lens`
The direct reason is that, in chunked prefill, attention has dynamic dimensions beyond `num_tokens`, while current graph bucketing is mainly organized by `num_tokens`. Keeping attention outside the graph avoids replay errors where the graph hits by `num_tokens`, but `batch_size` and `seq_lens` have already changed.
So the essence of Piecewise Graph is not to support all dynamic attention logic inside the graph. It is to:
- put the stably capturable parts into graphs
- keep the more dynamic-shape-sensitive attention as a separate execution unit
- preserve the original execution order through one unified instruction sequence
### 3.3 Result
The results and boundaries of this design are reflected in the following aspects.
#### 3.3.1 Applicable Scenarios
- prefill
- chunked prefill
- scenarios with local graph breaks where graph execution benefit is still desired for the overall path
#### 3.3.2 Benefits
- more flexible than all-or-nothing full-graph capture
- significantly expands Graph Mode coverage
- fits model structures where attention is highly dynamic while parts such as MLP remain relatively stable
#### 3.3.3 Limitations
- currently focused mainly on prefill-like paths
- replay depends on correctly updated `attn_metadata.plan_info`
## 4. Multi-Shape Reusable Memory Pool
### 4.1 Problem
In multi-shape scenarios, Graph Mode faces two problems at the same time:
1. **capture-time memory grows linearly with the number of shapes**
- each new shape entering capture usually gets its own independent memory buffer pool
- because Graph Mode requires stable addresses between capture and replay, these buffers often cannot be released in time
- memory usage grows from the expected `max(shape)` to `sum(shape)`
2. **input-side memory is also repeatedly duplicated**
- if each shape keeps its own tokens, positions, seq_lens, block_tables, and related tensors
- then even if the graph itself can be reused, input-side memory still accumulates by shape
So the reusable memory pool is not just about saving memory. It is about supporting dynamic new shapes while making memory usage converge from `sum(shape)` to `max(shape)`.
The concrete goals include:
1. **capture-time memory reuse**: different graphs share the same underlying physical memory
2. **input-side memory reuse**: replays of different shapes share one set of persistent input buffers
3. **no address conflicts**: every captured graph still keeps its own stable virtual address view
4. **on-demand support for new shapes**: a new shape can still be captured without invalidating existing graphs
5. **no extra replay overhead**: most reuse happens during capture and input preparation, while replay stays as unchanged as possible
#### 4.1.1 Root Cause
- on the CUDA path, the allocator typically allocates an independent memory buffer pool for every graph capture
- Graph Mode then requires stable addresses between capture and replay, so buffers from old shapes cannot be reclaimed like ordinary eager temporary memory
- as more shapes enter capture, memory usage grows from the expected `max(shape)` to `sum(shape)`
#### 4.1.2 Constraints
- under the current assumption, graphs of different shapes do not replay simultaneously
- dynamically appearing new shapes must still be supported through on-demand capture
- existing graph caches should not be torn down just because a new shape appears
- replay should not become more complex because of memory reuse
#### 4.1.3 Failed Approach and Technical Challenge
A more direct idea is to reset the allocation pointer before each capture so that different shapes reuse the same virtual address space.
But this is not safe in practice. Allocators usually track allocated blocks by address. If a new capture reuses an old address, it may overwrite the previous address record. Then, when tensors belonging to an old graph are later destructed or freed, the address record may already be gone, which can trigger errors such as `invalid device pointer`.
This reveals the real challenge: what Graph Mode truly needs to keep stable is the address view seen by each graph, while what really needs to be reused is the underlying physical memory. So the problem is not just memory saving. It is how to satisfy all of the following at the same time:
- the virtual address spaces of captured graphs do not conflict
- the underlying physical memory can still be shared across shapes
- adding a new shape into capture does not break replayability of already captured graphs
### 4.2 Solution
The core idea is not to let different shapes share one virtual address range. Instead, it is to let them share one set of physical memory while each captured graph keeps its own virtual address view.
The reason is simple: if different shapes are forced to reuse the same virtual address space, allocator address tracking will conflict, and old graphs may no longer find their original address records when they are released or destroyed. In other words, the thing that must be reused is the underlying physical memory, not the virtual address seen by the graph itself.
To show why xLLM chooses "shared physical memory + independent virtual address spaces", the common solutions can be compared as follows:
| Solution | Memory reuse mechanism | Physical memory outcome | Address-conflict risk / mitigation | Main limitation |
|----------|------------------------|-------------------------|------------------------------------|-----------------|
| vLLM | Shared graph memory pool + capture larger shapes first | may still approach `sum(shape)` | relies on a shared memory pool and free-block management, usually without explicit address conflicts | reuse efficiency depends on capture order and allocator behavior |
| SGLang | Global graph memory pool + capture larger shapes first | may still approach `sum(shape)` | relies on global pool or graph-private pool management to avoid conflicts | memory still grows when there are many distinct shapes |
| xLLM | Map multiple virtual address spaces to one set of physical memory | converges to `max(shape)` | every graph keeps an independent address view, avoiding address-record conflicts at the root | depends on VMM and allocator integration support from the runtime |
xLLM does not reuse the same virtual address space. Instead, each graph keeps its own address view, while the truly shareable part is pushed down into underlying physical memory.
#### 4.2.1 Multi-Shape Input Tensor Reuse
xLLM first solves repeated input-side memory usage. Instead of allocating independent input tensors for each shape, it pre-allocates one set of persistent buffers for the maximum shape and lets different shapes share them.
These buffers usually include:
- `tokens`
- `positions`
- `q_seq_lens`
- `kv_seq_lens`
- `block_tables`
- `new_cache_slots`
- `hidden_states`
Replay-time handling is straightforward:
1. write the actual input of the current request into the prefix of the buffer
2. zero-fill padding regions when needed
3. construct slice views for the actual shape
As a result, input-side memory no longer accumulates by shape and stays at one `max(shape)` allocation.
#### 4.2.2 Capture-Time Memory Reuse
After solving input duplication, xLLM still needs to solve the memory accumulation caused by graph capture itself.
The core design is:
- every capture uses a **new virtual address space**
- different virtual address spaces map to the **same physical memory**
This satisfies two requirements at once:
1. **stable addresses from the graph's point of view**: each captured graph sees its own fixed virtual addresses
2. **only one physical-memory footprint**: the underlying physical memory grows by the maximum demand instead of accumulating by shape
Compared with reusing one virtual address range, this design avoids address-tracking conflicts and therefore works better with existing allocator mechanisms.
This design relies on a unified virtual memory management capability, including:
- reserving a new virtual address space
- creating or expanding underlying physical memory
- mapping the same physical memory into multiple virtual address spaces
- performing unmap, address-space release, and access-permission operations when needed
The key property is that the same physical memory can be mapped into multiple virtual address spaces. That is exactly what makes "different graphs have independent addresses while still sharing the same physical memory" possible.
#### 4.2.2.1 Address Mapping Relation
The core mapping relation for capture-time memory reuse is shown below:
![Shared virtual-to-physical mapping](../../assets/shared_vmm_mapping_diagram.svg)
This diagram shows that, as the shape grows, each new capture switches to a new virtual address space, while all three virtual spaces reuse the same physical memory pool. Different shapes only use different portions of that pool, and the total physical memory converges to `max(shape)`.
### 4.3 Results
#### 4.3.1 Memory Efficiency Metrics
| Metric | Without memory reuse | With memory reuse |
|--------|----------------------|-------------------|
| Physical memory size | `sum(shape)` | `max(shape)` ✅ |
| Virtual memory | multiple copies | multiple copies |
| Stability | address-conflict risk | stable ✅ |
The key change is that physical memory converges to `max(shape)`, while virtual address space still grows with the number of captured shapes.
#### 4.3.2 Performance Impact
- **capture stage**: switching virtual address spaces and extending mappings introduces extra overhead
- **replay stage**: no extra graph execution overhead, and behavior is essentially the same as normal Graph Mode replay
- **input preparation stage**: one extra write into persistent buffers plus slice-view construction, which is still typically cheaper than repeated allocation
#### 4.3.3 Virtual Address Space Overhead
One cost of this design is that virtual address space grows as more shapes are captured. But this growth is address-space usage, not physical memory usage.
On 64-bit systems, virtual address space is usually much larger than single-device memory capacity, so it is typically not the primary bottleneck for this kind of graph capture scenario. In other words, this design uses abundant address space to bring physical memory down from `sum(shape)` to `max(shape)`.
#### 4.3.4 Result Validation
- captures of different shapes can share the same underlying physical memory
- every capture still keeps an independent virtual address space instead of reusing the same address view
- address-conflict issues are no longer exposed through the failed "reuse one virtual address space" path
- memory usage converges from `sum(shape)` to `max(shape)`
#### 4.3.5 Practical Effects
- in multi-shape scenarios, memory behavior changes from "grows linearly with captures" to "physical memory converges to the maximum shape"
- graph reuse becomes more stable because direct reuse of one virtual address space is avoided
- the design supports on-demand capture for new shapes without tearing down existing graph caches
- replay stays unchanged or nearly unchanged, while most added complexity is concentrated in capture and input preparation
#### 4.3.6 Assumptions and Boundaries
- virtual address space still grows with the number of captured shapes, even though physical memory only grows with the maximum demand
- the design depends on runtime support for virtual address management, physical memory mapping, and allocator integration
- it currently assumes that graphs of different shapes do not replay simultaneously; that assumption is what makes physical memory sharing safe
- the current implementation still keeps physical memory and virtual spaces for the full lifecycle instead of doing fine-grained dynamic release
## 5. References
- [ACLGraph Capture / Replay mechanism](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1alpha002/appdevg/acldevg/aclcppdevg_000519.html)
- [CudaGraph capture mechanism](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs)
- [vLLM graph execution reference](https://github.com/vllm-project/vllm/blob/main/vllm/v1/worker/gpu/cudagraph_utils.py)
- [SGLang graph execution reference](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/cuda_graph_runner.py)

27
upstream_ref/xllm/docs/en/dev_guide/code_arch.md Normal file
View File

@@ -0,0 +1,27 @@
# Code Architecture
```
├── xllm/
| : main source folder
│ ├── api_service/ # code for api services
│ ├── core/
│ │ : xllm core features folder
│ │ ├── common/
│ │ ├── distributed_runtime/ # code for distributed and pd serving
│ │ ├── framework/ # code for execution orchestration
│ │ ├── kernels/ # adaption for npu kernels adaption
│ │ ├── layers/ # model layers impl
│ │ ├── platform/ # adaption for various platform
│ │ ├── runtime/ # code for worker and executor
│ │ ├── scheduler/ # code for batch and pd scheduler
│ │ └── util/
│ ├── function_call # code for tool call parser
│ ├── models/ # models impl
│ ├── processors/ # code for vlm pre-processing
│ ├── proto/ # communication protocol
│ ├── pybind/ # code for python bind
| └── server/ # xLLM server
├── examples/ # examples of calling xLLM
├── tools/ # code for npu time generations
└── xllm.cpp # entrypoint of xLLM
```

396
upstream_ref/xllm/docs/en/dev_guide/tilelang_ascend_kernel_dev.md Normal file
View File

@@ -0,0 +1,396 @@
# xLLM Ascend TileLang Kernel Development Guide
This document explains how to add or modify an Ascend TileLang kernel in xLLM. The examples use the current `rope` kernel throughout.
Relevant directories:
- Python kernel definitions: `xllm/xllm/compiler/tilelang/targets/ascend/kernels`
- NPU runtime wrappers: `xllm/xllm/core/kernels/npu/tilelang`
Builds and tests should be run inside the NPU container.
## 1. First Decide What Kind of Change You Are Making
- Add a `specialization`
- Add one more compiled parameter combination to an existing kernel
- Reuse the same wrapper, the same runtime dispatch fields, and the same C ABI
- Typical changes are updates to `DISPATCH_SCHEMA` or `SPECIALIZATIONS`
- Add a `kernel`
- Add a new logical operator
- Typical changes are a new Python kernel file, a new wrapper C++ file, and one CMake registration
For `rope`:
- Adding one more item like `{"variant_key": "...", "head_dim": ..., "rope_dim": ..., "dtype": ...}` to `SPECIALIZATIONS` means adding a new `specialization`
- Adding a new external interface such as `xxx_wrapper.cpp` means adding a new `kernel`
## 2. Development Order
The recommended order is:
1. Implement the TileLang kernel in a Python file such as `rope.py`
2. Implement `generate_source(...)` to lower the kernel into Ascend-C source
3. Declare `DISPATCH_SCHEMA` and `SPECIALIZATIONS`
4. Generate `registry.inc` once and inspect it
5. Then write or update the runtime specialization construction logic in the wrapper
6. Wire it into CMake and run tests
The key idea behind this order is:
- implement the kernel itself first
- then fix the runtime dispatch schema
- then write the wrapper against the generated `registry.inc`
## 3. Write the Python Kernel
Using `rope.py` as the example, the Python side can be understood in three layers:
- `build_rope_kernel(...)`: kernel implementation
- `generate_source(...)`: AOT export
- `RopeKernel`: kernel registration plus dispatch schema and compiled instance declaration
### 3.1 Implement `build_rope_kernel(...)`
`build_rope_kernel(...)` is the actual TileLang kernel implementation. This is where you write:
- `@T.prim_func`
- input and output tensor shapes
- parallel task organization under `with T.Kernel(...)`
- UB allocation and the actual compute logic
The simplified structure in `rope.py` looks like this:
```python
def build_rope_kernel(
head_dim: int,
rope_dim: int,
vec_core_num: int,
ub_buffer_bytes: int,
):
task_num = vec_core_num
m_num = vec_core_num // 2
@T.prim_func
def rope_in_place_kernel(...):
with T.Kernel(m_num, is_npu=True) as (cid, vid):
task_id = cid * 2 + vid
...
return rope_in_place_kernel
```
Here, `head_dim` and `rope_dim` are the compile-time parameters that this implementation actually depends on.
Vector kernels such as `rope` must also follow the fixed-task convention used by the current AOT path. In the current AOT flow, the kernel launch `block_num` is fixed at compile time, which means:
- runtime input shapes do not change the kernel launch `block_num`
- runtime input shapes only change workload splitting across the fixed tasks
The convention in the current `rope.py` is:
```python
task_num = vec_core_num
m_num = vec_core_num // 2
with T.Kernel(m_num, is_npu=True) as (cid, vid):
task_id = cid * 2 + vid
```
This means:
- `cid` ranges over `[0, vec_core_num // 2)`
- `vid` ranges over `[0, 2)`
- the total task count is fixed as `task_num = vec_core_num`
As a result, `rope.py` also derives the compile-time token count for one specialization using the fixed task count:
```python
max_rows_num_in_ub = _derive_max_rows_num_in_ub(...)
compile_num_tokens = task_num * max_rows_num_in_ub
```
### 3.2 Implement `generate_source(...)`
`generate_source(...)` lowers the TileLang kernel above into the final source code. The export layer takes one specialization and turns it into compilable Ascend-C source.
For `rope`, the core logic is:
```python
@staticmethod
def generate_source(head_dim: int, rope_dim: int, dtype: str) -> str:
vec_core_num = detect_vec_core_num()
tilelang_kernel = build_rope_kernel(
head_dim=head_dim,
rope_dim=rope_dim,
vec_core_num=vec_core_num,
ub_buffer_bytes=FIXED_UB_BUFFER_BYTES,
)
with tilelang.tvm.transform.PassContext(...):
kernel = tilelang.engine.lower(tilelang_kernel)
return kernel.kernel_source
```
The rules here are:
- the inputs to `generate_source(...)` come from the current `SPECIALIZATIONS` entry
- `generate_source(...)` calls `build_rope_kernel(...)`
- the return value is the lowered source string
### 3.3 Declare `DISPATCH_SCHEMA` and `SPECIALIZATIONS`
After the kernel implementation and export layer are done, use an `@register_kernel` class to attach the kernel to the framework.
The current minimal template in `rope.py` is:
```python
from ....common.spec import DispatchField, TilelangKernel, register_kernel
@register_kernel
class RopeKernel(TilelangKernel):
DISPATCH_SCHEMA = [
DispatchField("head_dim", "int32"),
DispatchField("rope_dim", "int32"),
DispatchField("dtype", "dtype"),
]
SPECIALIZATIONS = [
{
"variant_key": "hd128_rd128_bf16",
"head_dim": 128,
"rope_dim": 128,
"dtype": "bf16",
},
{
"variant_key": "hd576_rd64_bf16",
"head_dim": 576,
"rope_dim": 64,
"dtype": "bf16",
},
]
@staticmethod
def generate_source(head_dim: int, rope_dim: int, dtype: str) -> str:
...
```
There are two concepts to distinguish here:
- `DISPATCH_SCHEMA`
- defines the field names, order, and types of the runtime specialization
- is the single source of truth for the C++ specialization struct, builder, and lookup interface
- `SPECIALIZATIONS`
- represents the set of instances that will actually be compiled
- each item corresponds to one variant
The rules are:
- every field in `DISPATCH_SCHEMA` must appear in every `SPECIALIZATIONS` item
- `SPECIALIZATIONS` may contain extra fields; those fields are passed into `generate_source(...)`, but do not enter the runtime dispatch schema
- `variant_key` is the unique identifier for that specialization
- `DISPATCH_SCHEMA` and `SPECIALIZATIONS` must match the runtime specialization one-to-one
For `rope`, the runtime dispatch dimensions are:
- `head_dim`
- `rope_dim`
- `dtype`
So these three fields must appear in both:
- `DISPATCH_SCHEMA`
- every `SPECIALIZATIONS` item
At build time, Ascend build resolves the actual `bisheng_arch` from the `--device a2|a3` value passed by the main build path.
### 3.4 Inspect the Generated Ascend-C Source
When debugging the implementation details in `build_rope_kernel(...)`, or comparing how different kernel styles affect the final code generation, use the common `compile-kernels` entry to regenerate artifacts and inspect the Ascend-C source for the specialization you care about.
For `rope`, you can fix:
- `head_dim=576`
- `rope_dim=64`
- `dtype=bf16`
Then regenerate the `rope` artifacts:
```bash
python xllm/compiler/tilelang_launcher.py compile-kernels \
--target ascend \
--device a3 \
--output-root /tmp/tilelang_debug \
--kernels rope \
--force
```
It is recommended to keep `--force` so the source and object files are regenerated from the current code instead of reusing an old cache hit.
This command uses an isolated debug output directory, `/tmp/tilelang_debug`, so only the debug artifacts for `rope` are generated there and they do not get mixed with artifacts from other kernels in the main build directory.
After that, you can directly inspect the generated source for the specialization, including the entry function, UB allocation, and vector compute logic:
```bash
sed -n '1,200p' \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
rg -n 'extern "C"|__global__|alloc_ub|alloc_shared|g_tilingKey' \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
```
To compare two kernel implementations, keep the specialization fixed, run `compile-kernels --force` before and after the change, then diff the generated `.cpp` file:
```bash
cp /tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp \
/tmp/rope_before.cpp
diff -u /tmp/rope_before.cpp \
/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp
```
This helps isolate specialization changes from kernel implementation changes.
After generation, the main files to inspect are:
- `/tmp/tilelang_debug/targets/ascend/rope/hd576_rd64_bf16/rope_hd576_rd64_bf16_kernel.cpp`
- `/tmp/tilelang_debug/targets/ascend/rope/registry.inc`
- `/tmp/tilelang_debug/targets/ascend/rope/manifest.json`
These correspond to:
- the final Ascend-C source for one specialization
- the runtime dispatch interface directly included by the wrapper
- the full compiled artifact record for the current kernel
The recommended debugging sequence is:
1. run `compile-kernels --force` to regenerate the current kernel artifacts
2. inspect the `.cpp` for the specialization and analyze the code generation result
3. inspect `registry.inc` and `manifest.json` to confirm they match expectations
4. finally run `rope_wrapper_test` to check end-to-end behavior and performance
## 4. Update the Wrapper
When adding a new `kernel`, you need a new wrapper. When adding a new `specialization`, the wrapper only needs an update if the runtime specialization semantics change.
For `rope_wrapper.cpp`, the manually written parts should remain:
- tensor shape, dtype, and layout validation
- reshaping inputs into `x_rows / sin_rows / cos_rows`
- constructing the runtime specialization from tensors
- assembling launch arguments and calling `entry->fn(...)`
### 4.1 What `registry.inc` Generates Automatically
`registry.inc` is generated automatically from the Python-side `DISPATCH_SCHEMA`, `SPECIALIZATIONS`, and the exported Ascend-C ABI.
For `rope`, the generated content includes:
- `RopeSpecialization`
- `RopeHeadDim`
- `RopeRopeDim`
- `RopeDType`
- `RopeKernelFn`
- `make_rope_specialization(...)`
- `find_rope_kernel_entry(...)`
- `available_rope_variant_keys()`
For `rope_wrapper.cpp`, `registry.inc` directly provides dispatch-related definitions such as `RopeSpecialization`, `operator==(...)`, and `RopeKernelFn`. Dtype conversion uses the shared helper `to_tilelang_dtype(...)`.
### 4.2 What the Wrapper Actually Needs to Write
The most important handwritten logic in `rope_wrapper.cpp` is constructing the runtime specialization from the tensors. The current code looks like this:
```cpp
RopeSpecialization build_runtime_specialization(const torch::Tensor& x_rows) {
return make_rope_specialization(
RopeHeadDim{static_cast<int32_t>(x_rows.stride(0))},
RopeRopeDim{static_cast<int32_t>(x_rows.size(1))},
RopeDType{to_tilelang_dtype(x_rows.scalar_type())});
}
```
For `rope`:
- `head_dim` maps to `x_rows.stride(0)`, which is the `x_stride` used by the kernel
- `rope_dim` maps to `x_rows.size(1)`
- `dtype` maps to `x_rows.scalar_type()`
The runtime path is:
1. the wrapper reshapes the inputs into `x_rows / sin_rows / cos_rows`
2. `build_runtime_specialization(...)` constructs a specialization from `x_rows`
3. `find_rope_kernel_entry(...)` performs an exact match in the static registry
4. after a match, `entry->fn(...)` calls the actual compiled symbol
The current lookup strategy is a linear scan with exact matching. If any of `head_dim`, `rope_dim`, or `dtype` differs, the lookup will miss.
So when you add a new `specialization`, the main things to cross-check are:
- the field semantics in Python-side `DISPATCH_SCHEMA`
- the field values in Python-side `SPECIALIZATIONS`
- the field values constructed by `build_runtime_specialization(...)` in the wrapper
All three must match exactly.
### 4.3 Generate and Inspect `registry.inc` First
Before writing or modifying wrapper code, generate `registry.inc` once and inspect it. Focus on:
- whether the generated field order in `RopeSpecialization` matches expectations
- whether the generated wrapped field type names match expectations
- the parameter order of `make_rope_specialization(...)`
- the generated entry symbol names
`registry.inc` is the direct contract for the wrapper. Inspect it first, then write the wrapper against it.
## 5. Update CMake
When adding a new `kernel`, register it in `xllm/xllm/core/kernels/npu/tilelang/CMakeLists.txt`.
CMake registration is unified through the high-level helper:
- `tilelang_register_runtime_kernel(NAME <kernel> WRAPPER_SRCS <srcs...>)`
Using `rope` as the example, the minimal template is:
```cmake
tilelang_register_runtime_kernel(
NAME rope
WRAPPER_SRCS rope_wrapper.cpp
)
```
This helper will:
- derive the manifest path as `TILELANG_GENERATED_ROOT/targets/ascend/<kernel>/manifest.json`
- import the manifest
- add the wrapper source and compiled objects into `tilelang_kernels`
- append the `XLLM_TL_<KERNEL>_REGISTRY_INC=...` compile definition automatically
So when adding a new runtime kernel, the CMake-side work mainly consists of two things:
1. make sure the Python side can already generate the manifest for that kernel
2. add one `tilelang_register_runtime_kernel(...)` entry in the TileLang CMakeLists
For day-to-day kernel additions, add one `tilelang_register_runtime_kernel(...)` line directly in CMake. `tilelang_import_kernel_manifest(...)` stays underneath as the implementation base for that higher-level helper.
## 6. Validate
The recommended validation order is:
1. compile the TileLang kernel and inspect the generated `registry.inc`
2. then run the full wrapper test
Common commands:
```bash
python xllm/compiler/tilelang_launcher.py compile-kernels \
--target ascend \
--device a3 \
--output-root build/cmake.linux-aarch64-cpython-311/xllm/compiler/tilelang \
--kernels rope
python setup.py test --test-name rope_wrapper_test --device a3
```
The first command generates `manifest.json`, `registry.inc`, and the object files. The second command validates the full integration path.

29
upstream_ref/xllm/docs/en/features/async_schedule.md Normal file
View File

@@ -0,0 +1,29 @@
# Async schedule
## Background
The inference process of large language models can be divided into three sequential stages:1) CPU-side scheduling (preparing model inputs), 2) Device computation (GPU/TPU execution), 3) CPU-side post-processing (output handling).
Due to the sequential nature of decoding operations, the input for step-i+1 depends on the output of step-i. This forces strict serial execution of all three stages, creating device idle periods (“bubbles”) during CPU-bound stages 1 and 3, leading to suboptimal resource utilization.
## Introduction
xLLM addresses this at the framework level by supporting asynchronous scheduling, where the CPU proactively executes scheduling operations for step-i+1 while the device is computing step-i. This allows the device to immediately begin computing step-i+1 upon completing step-i, thereby eliminating bubbles. Specifically, after initiating the computation call for step-i, the CPU does not wait for the device to finish computing. Instead, it constructs fake tokens for the step-i request, uses these fake tokens to perform scheduling operations for step-i+1 (such as allocating KV Cache), and replaces them with the true tokens computed in step-i when launching step-i+1 computation to ensure correctness. Meanwhile, the CPU processes the results of step-i in a separate thread and returns them to the client.
In the overall architecture, stages 1 and 3 on the CPU side are handled by different thread pools, and RPC function calls employ non-blocking C++ future and promise mechanisms to achieve a fully asynchronous runtime.
![Async schedule](../../assets/async_schedule_architecture.jpg)
## Usage
xLLM provides the gflags parameter `enable_schedule_overlap`, which defaults to false. To enable this feature, set it to true in xLLM's service startup script:
```shell
--enable_schedule_overlap=true
```
## Performance
- With asynchronous scheduling enabled, the device idle time between two steps is approximately 200us - comparable to a single kernel launch duration.
- On the DeepSeek-R1-Distill-Qwen-1.5B model with TPOT constrained to 50ms, this achieves 17% throughput improvement.
## Notice
The asynchronous scheduling feature requires the server to compute one additional step. For use cases involving limited output tokens (e.g., few-token generation) or single-output scenarios like embedding models, enabling this feature is not recommended as it may reduce server-side throughput, thus hard-disabled internally.
The VLM model is currently being adapted, will be temporarily disabled.

5
upstream_ref/xllm/docs/en/features/basics.md Normal file
View File

@@ -0,0 +1,5 @@
# Basics
- xLLM uses a one-device-per-process architecture. Across multiple devices, RPC is used for function calls, and data communication during model computation uses device collective communication libraries.
- HCCL/LCCL are high-performance collective communication frameworks that provide data-parallel and model-parallel collective communication for both single-node multi-device and multi-node multi-device scenarios.

19
upstream_ref/xllm/docs/en/features/chunked_scheduler.md Normal file
View File

@@ -0,0 +1,19 @@
# ChunkedPrefill Scheduler
## Feature Introduction
xLLM supports the chunked prefill scheduling strategy. Chunked prefill is a technique that optimizes large language model inference by splitting long prompts into smaller chunks for batch processing, rather than processing the entire prompt at once.
This method can effectively reduce peak GPU memory usage, improve device utilization, and better schedule and mix processing with requests from the decode stage.
## Usage
The aforementioned strategy has been implemented in xLLM and is exposed through gflags parameters to control the feature's on/off state.
- Enable chunked prefill and set the chunked size, if not set chunked size, its default value is equal to max_tokens_per_batch.
```bash
--enable_chunked_prefill=true
--max_tokens_per_chunk_for_prefill=20480 # optional
```
## Performance Impact
After enabling chunked prefill, on the Qwen3-8B model with a TPOT constraint of 50ms, the TTFT latency **decreased by 46%**.

7
upstream_ref/xllm/docs/en/features/continuous_scheduler.md Normal file
View File

@@ -0,0 +1,7 @@
# Continuous Scheduler
## Feature Introduction
xLLM implements a scheduling strategy that supports continuous batching. Continuous batching is a dynamic batching strategy that does not wait for a batch to be filled. Instead, it starts processing as soon as requests are available, while continuously accepting new requests and adding them to the currently executing batch. This approach significantly reduces latency while maintaining high throughput.
## Usage
xLLM implements the continuous batching scheduling strategy. And, the default scheduler is chunked prefill, `enable_chunked_prefill=true` by default.

83
upstream_ref/xllm/docs/en/features/disagg_pd.md Normal file
View File

@@ -0,0 +1,83 @@
# Disaggregated PD
## Background
LLM online inference services typically need to meet two performance metrics: TTFT and TPOT. Traditional Contiguous Batching scheduling strategies mix Prefill and Decode requests during scheduling, causing Prefill and Decode phases to compete for computational resources. This prevents maximized utilization of computing resources and impacts performance metrics. To resolve this conflict, the Prefill and Decode phases are split to run on independent computational resources, enabling parallel execution. This simultaneously reduces TTFT and TPOT while improving throughput.
## Introduction
The xLLM PD Separation feature is primarily implemented through the following three modules:
- **etcd**: Stores metadata such as instance information.
- **xLLM Service**: Schedules requests and manages all computing instances.
- **xLLM**: Handles request computation instances.
The overall architecture is shown below:
![xLLM PD Separation Architecture](../../assets/pd_architecture.jpg)
## Usage
### Preparation
#### Install Dependencies
- **xLLM**: Refer to [Installation && Compilation](../getting_started/quick_start.md)
- **xLLM Service**: Refer to [PD disaggregation](../getting_started/disagg_pd.md)
#### Obtain Environment Information
Deploying Disaggregated PD Service requires obtaining the Device IP of the machine to create communication resources. Execute the command `cat /etc/hccn.conf | grep address` on the current AI Server to get the Device IP, for example:
```
address_0=xx.xx.xx.xx
address_1=xx.xx.xx.xx
```
`address_xx` represents the Device IP.
### Start Disaggregated PD Service
1. Start etcd
```bash
./etcd
```
2. Start xLLM Service
```bash
ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
3. Start xLLM
- Taking Qwen2-7B as an example
- Start Prefill Instance
```bash
/path/to/xllm --model=Qwen2-7B-Instruct \
--port=8010 \
--devices="npu:0" \
--master_node_addr="127.0.0.1:18888" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=PREFILL \
--etcd_addr="127.0.0.1:12389" \
--transfer_listen_port=26000 \
--disagg_pd_port=7777 \
--node_rank=0 \
--nnodes=1
```
- Start Decode Instance
```bash
/path/to/xllm --model=Qwen2-7B-Instruct \
--port=8020 \
--devices="npu:1" \
--master_node_addr="127.0.0.1:18898" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=DECODE \
--etcd_addr="127.0.0.1:12389" \
--transfer_listen_port=26100 \
--disagg_pd_port=7787 \
--node_rank=0 \
--nnodes=1
```
Important notes:
- PD disaggregation requires reading the `/etc/hccn.conf` file. Make sure this file on the physical machine is mapped into the container.
- `etcd_addr` must match the `etcd_addr` of `xllm_service`
## Notice
Disaggregated PD **does not support** enabling prefix cache or chunked prefill. These features must be disabled using the following parameters:
```shell
--enable_prefix_cache=false
--enable_chunked_prefill=false
```

30
upstream_ref/xllm/docs/en/features/eplb.md Normal file
View File

@@ -0,0 +1,30 @@
# MoE Load Balancing (EPLB)
## Background
MoE models rely on dynamic token routing to distribute tokens among experts. However, in real-world deployments, uneven data distribution leads to expert load imbalance (some overloaded while others idle). Expert redundancy adjustment (e.g., adding/removing replicas) consumes additional GPU memory and may impact inference latency due to weight migration, posing significant implementation challenges. To address this, we employ an expert redundancy strategy (replicating hot experts) combined with hierarchical and global dynamic load balancing to achieve dynamic MoE load balancing.
## Features
The xLLM MoE Load Balancing (EPLB) functionality is implemented through three main modules:
- **EPLB Manager**: Responsible for monitoring expert loads, collecting and managing expert distribution updates. It uses a layer-by-layer update mechanism, determining whether to update each layer based on expert load changes.
- **EPLB Executor**: The actual executor for expert distribution updates.
- **EPLB Policy**: Strategy for generating new expert load tables.
Overall architecture diagram:
![xLLM eplb](../../assets/eplb_architecture.png)
## Usage
Simply add the following gflag parameters when launching xLLM:
(Replace with actual number of devices. `ep_size` must match the number of devices)
- xLLM provides the gflag parameter `enable_eplb` (default: false). Set to true in the xLLM service startup script to enable dynamic expert load balancing.
- `expert_parallel_degree` and `ep_size` are MoE-related parameters. `expert_parallel_degree` should be set to `2`, and `ep_size` must match the actual number of NPU/GPU devices. See [moe_params](./moe_params.md)
- `eplb_update_interval` sets the expert distribution update interval in seconds (default: 1000).
- The expert distribution update uses a layer-by-layer mechanism based on expert load. When the similarity between consecutive loads for a layer is below `eplb_update_threshold`, that layer is updated (default: 1, range: 0-1).
```bash
--enable_eplb=true --expert_parallel_degree=2 --ep_size=16 --eplb_update_interval=2000 --eplb_update_threshold=0.9

42
upstream_ref/xllm/docs/en/features/global_kvcache.md Normal file
View File

@@ -0,0 +1,42 @@
# Global Multi-Level KV Cache
## Background
In the decoding phase of large language models (LLMs), frequent access to historical KV cache due to autoregressive generation creates a bottleneck in memory bandwidth. As model sizes and context windows expand (e.g., 128K Tokens consuming over 40GB of memory), the pressure on single-device memory increases dramatically. Existing solutions (such as vLLM) exhibit significant limitations in long-context scenarios: prefill time surges, severe memory bandwidth contention during decoding, and the need for excessive resource reservation to meet SLO requirements (TTFT < 2s, TBT < 100ms). This often results in GPU utilization below 40% and difficulties in leveraging cross-server resources. To address this, we propose a distributed global multi-level KV cache management system, adopting a memory-compute integrated architecture to break through single-machine resource constraints.
## Feature Introduction
The xLLM Global KV Cache feature is primarily implemented through the following three modules:
- **etcd**: For cluster service registration, load information synchronization, and global cache state management.
- **xLLM Service**: For scheduling requests and managing all compute instances.
- **xLLM**: The compute instances handling requests.
The overall architecture is shown in the diagram below:
![xLLM Global Multi-Level KV Cache](../../assets/globalkvcache_architecture.png)
## Usage Example
### Preparation
#### Install Dependencies
- **xLLM**: Refer to [Quick Start](../getting_started/quick_start.md)
- **xLLM Service**: Refer to [PD disaggregation](../getting_started/disagg_pd.md)
### Usage Instructions
1. **etcd Startup Configuration:**
```bash
./etcd --listen-peer-urls=http://0.0.0.0:10999 --listen-client-urls=http://0.0.0.0:10998
```
2. **xLLM Service Startup Configuration:**
```bash
./xllm_master_serving --etcd_addr="127.0.0.1:10998" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
3. **xLLM Startup Configuration:**
Add the following gflag parameters when starting xLLM:
```bash
--enable_service_routing=true
--enable_cache_upload=true
# PD separation currently does not support Global KVCache Management
--enable_disagg_pd=false
```

79
upstream_ref/xllm/docs/en/features/graph_mode.md Normal file
View File

@@ -0,0 +1,79 @@
# Graph Mode
## Overview
xLLM supports Graph Mode: computation graphs are pre-captured and replayed in subsequent runs to reduce CPU overhead and improve inference performance. Graph Mode has corresponding implementations on different hardware platforms.
## Feature Description
To optimize Host-side scheduling, graph mode submits a large task from the CPU once and then executes small kernels in a streaming manner on the device, significantly reducing startup time and device bubbles.
In the xLLM engine, Graph Mode provides the following:
### Dynamic Shape Parameterization
- Key dynamic dimensions other than num_tokens are treated as whole-graph input parameters, including batch_size, kv_seq_lens, q_seq_lens, block_table_size, and the like, for flexibility. During memory allocation and kernel configuration. At graph launch, the actual values of these parameters are passed so kernels use the correct strides to access data.
### Piecewise Graph
- When some operators do not support graph capture and thus break the full graph, each segment (piece) after the break is captured as a separate graph. This maximizes graph-mode benefits even when the full graph cannot be captured, and is commonly used for prefill and chunked prefill.
### Multi-Shape Reusable Memory Pool
- To avoid waste from separate memory buffers (input, output, and intermediate tensors) per shape, we use an expandable memory pool. Multiple shapes share the pool base address, with different shapes using different offsets from that base.
## Usage
These capabilities are implemented inside the xLLM engine and are generally controlled through gflags.
The minimal configuration only needs `enable_graph` to turn on Graph Mode for the decode phase:
```shell
--enable_graph=true
```
Common companion flags include:
- `enable_graph`: enables the base Graph Mode capability for the decode phase
- `enable_prefill_piecewise_graph`: enables Piecewise Graph for the prefill phase
- `enable_graph_mode_decode_no_padding`: builds decode graphs with the actual `num_tokens` instead of the padded shape
- `max_tokens_for_graph_mode`: limits the maximum number of tokens covered by Graph Mode; `0` means no limit
If you want to enable both decode Graph and prefill Piecewise Graph, use:
```shell
--enable_graph=true \
--enable_prefill_piecewise_graph=true \
--max_tokens_for_graph_mode=2048
```
If you need decode graph capture without padding, add:
```shell
--enable_graph=true \
--enable_graph_mode_decode_no_padding=true
```
For a more complete description of the flags, see [CLI Reference](../cli_reference.md).
## Performance Impact
- With Graph Mode enabled, decode-phase throughput **improves by about 8%–10%** on models such as Qwen3-0.6B and Qwen3-1.7B.
## Model Support
The following table lists each model’s support on ACLGraph, CudaGraph, and MLUGraph.
| Model | ACLGraph | CudaGraph | MLUGraph |
|------|----------|-----------|----------|
| Qwen3/Qwen3-MoE | ✅ | ✅ | ✅ |
| DeepseekV3.2 | ✅ | | ✅ |
| GLM4.5/4.6/4.7 | ✅ | | |
| Qwen2.5-VL | | | ✅ |
| Qwen3-VL/Qwen3-VL-MoE | ✅ | | |
| GLM4V | ✅ | | |
| GLM4V-MoE | ✅ | | |
!!! warning "Adding Graph Mode support for new models"
Ensure that the kernels used in the computation implement dynamic dimension parameterization; otherwise the graph may break and kernels may need to be re-implemented.
## Related Documentation
- For more detailed Graph Mode design and implementation notes, including ACL Graph / CUDA Graph fundamentals, dynamic dimension parameterization, Piecewise Graph, and multi-shape memory reuse, see: [Graph Mode Design Document](../design/graph_mode_design.md)

40
upstream_ref/xllm/docs/en/features/groupgemm.md Normal file
View File

@@ -0,0 +1,40 @@
# GroupGEMM Operator Optimization
## Background
The Mixture of Experts (MoE) architecture has become an important paradigm for scaling large language models. Its core idea is to dynamically route input tokens to different expert sub-networks for processing. During inference, the GroupGEMM operator is a key computational unit in the MoE architecture, responsible for efficiently executing parallel computations of multiple expert matrix multiplications, which dominate the overall inference time.
## Function Introduction
Considering the current performance bottleneck of GroupGEMM being I/O constrained, an optimization scheme is proposed that replaces data copying with index reordering, eliminating multiple copies of the token vector and instead maintaining an index table for expert allocation. Through this line number index, tokens are directly mapped to the corresponding expert computation units, and the allocation scheduling of tokens is fused with matrix multiplication into a single kernel.
## User Interface
### Operator Direct Call API
```c++
aclnnStatus aclnnIndexGroupMatmulGetWorkspaceSize(
const aclTensorList *x,
const aclTensorList *weight,
const aclTensorList *scale,
const aclTensorList *perTokenScale,
const aclTensor *groupList,
const aclTensorList *out,
uint64_t *workspaceSize,
aclOpExecutor **executor);
aclnnStatus aclnnIndexGroupMatmul(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream);
```
- `x`: The input tensor list containing the data to be processed.
- `weight`: The weight tensor containing the model parameters.
- `scale`: The scaling factor used to adjust the values of the input tensors.
- `perTokenScale`: The scaling factor for each token, used for dynamic adjustment.
- `groupList`: The list of expert groups indicating which experts participate in the computation.
- `out`: The output tensor list that stores the computation results.
## Performance Effects
![groupmatmul](../../assets/groupmatmul_performance.png)
* The optimized GroupMatmul operator shows significant advantages in computation time, especially when \( k = 128 \) and \( m = 64 \). As shown in the figure, the computation delay of the optimized operator is **reduced by 50%**.

24
upstream_ref/xllm/docs/en/features/moe_params.md Normal file
View File

@@ -0,0 +1,24 @@
# EP Parallelism
## Background
When deploying the DeepSeek-R1 671B parameter-scale model, traditional distributed deployment faces core bottlenecks including low GPU memory utilization, high communication overhead, and expensive hardware costs. Therefore, Expert Parallelism (EP) is introduced.
Key advantages:
+ With the same resources, fewer Experts per GPU means more memory available for KV Cache, allowing more tokens to be cached.
+ Due to MLA characteristics, smaller TP Size with the same resources means less redundant KV Cache, enabling more tokens to be cached.
+ Large-scale EP deployment can concentrate token computations for the same expert on the same device, improving hardware utilization.
## Parameter Configuration
+ **dp_size**: Sets the data parallelism scale for the Attention part. Default: 1, can be set to powers of 2. When dp_size doesn't equal the total number of devices, TP parallelism is used within the DP group.
+ **ep_size**: Sets the expert parallelism scale for the MoE part. Default: 1, can be set to powers of 2. When ep_size doesn't equal the total number of devices, TP parallelism is used within the DP group.
+ **expert_parallel_degree**: EP-related parameter. Default is 0 when EP is disabled. When EP is enabled, default is 1 (EP Level 1). When ep_size equals the total number of devices, can be set to 2 to enable EP Level 2.
## Solution Design
+ When EP is enabled (default EP Level 1), after computing both Attention and MoE parts, data is sent to the next stage via All Gather communication across all devices. Example for 64 devices (Attention: dp32tp2, MoE: ep32tp2):
![Alt text](../../assets/moe_eplevel1.jpg)
+ When ep_size equals the total number of devices, EP Level 2 can be enabled. Communication between Attention and MoE parts changes to ALL2ALL, sending data only to required devices to reduce communication volume and overhead. Example for 64 devices:
![Alt text](../../assets/moe_eplevel2.jpg)

149
upstream_ref/xllm/docs/en/features/mtp.md Normal file
View File

@@ -0,0 +1,149 @@
# MTP Speculative Inference
## Background
MTP (Multi-Token Prediction) is an innovative inference acceleration technique that addresses efficiency bottlenecks in large language model generation. By incorporating specialized pre-training designs, MTP provides efficient draft token prediction capabilities during inference, significantly improving generation speed. Its core value lies in balancing inference efficiency with output quality, offering an optimal solution for long-sequence generation problems in LLMs, ultimately optimizing inference performance.
## Key Features
MTP offers the following core acceleration capabilities:
- **Efficient Draft Generation**: Uses a lightweight MTP architecture to rapidly generate draft tokens that serve as input for the main model's verification, dramatically reducing computation overhead compared to traditional autoregressive generation.
- **Batch Verification Mechanism**: The main model can simultaneously verify multiple MTP-generated draft tokens in batch, rather than processing them sequentially, significantly boosting inference speed.
- **High Sampling Accuracy**: MTP solves the critical pain point of low token acceptance rates in post-training draft modules (like Eagle and Medusa). By optimizing draft generation during pre-training, MTP produces tokens with higher accuracy, reducing the verification burden on the main model.
- **Reduced Inference Latency**: By pre-generating multiple potential subsequent tokens, MTP effectively decreases cumulative latency during long-text generation, creating a smoother user experience.
- **Optimized Resource Consumption**: Compared to other inference acceleration techniques, MTP maintains acceleration effects while requiring fewer additional computational resources, making it suitable for deployment in resource-constrained environments.
MTP technology provides a novel efficiency optimization solution for LLM inference, particularly well-suited for real-time applications requiring rapid responses, representing an important direction in language model inference optimization.
!!! note "Model Support"
Currently supports MTP structure export for the following models:
- DeepSeek-V3 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v3_mtp)
- DeepSeek-V3.2 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v32_mtp)
- DeepSeek-R1 (input model_type: deepseek_v3, exported MTP model_type: deepseek_v3_mtp)
- GLM4 MoE (e.g., GLM-4.5-Air, exported MTP model_type: glm4_moe_mtp)
Note:
- DeepSeek V3 and R1 both have input model_type "deepseek_v3", and the exported MTP model will have model_type "deepseek_v3_mtp"
- DeepSeek V3.2 has input model_type "deepseek_v3" (but can be auto-detected by index_head_dim fields), and the exported MTP model will have model_type "deepseek_v32_mtp"
## Usage Example
### Export Model
The script will automatically detect the model type, or you can manually specify it.
#### DeepSeek-V3
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-V3 \
--output-dir /path/to/DeepSeek-V3-mtp
```
#### DeepSeek-V3.2
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-V3.2 \
--output-dir /path/to/DeepSeek-V3.2-mtp
```
#### DeepSeek-R1
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/DeepSeek-R1 \
--output-dir /path/to/DeepSeek-R1-mtp
```
#### GLM4 MoE
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/GLM-4.5-Air \
--output-dir /path/to/GLM-4.5-Air-mtp
```
#### Manually Specify Model Type
If auto-detection fails, you can manually specify the model type:
```bash
python3 tools/export_mtp.py \
--input-dir /path/to/model \
--output-dir /path/to/model-mtp \
--model-type deepseek_v3 # Options: deepseek_v3 (for V3/R1), deepseek_v32 (for V3.2), glm4_moe
```
Input model references:
- [DeepSeek-V3](https://huggingface.co/deepseek-ai/DeepSeek-V3)
- [DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)
- [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1)
- [GLM-4.5-Air](https://huggingface.co/zai-org/GLM-4.5-Air)
### Launch Script
When using MTP for inference, you need to specify both the main model and the draft model (MTP model).
#### DeepSeek-V3/V3.2/R1 Launch Example
```bash
MODEL_PATH="/models/DeepSeek-V3"
DRAFT_MODEL_PATH="/models/DeepSeek-V3-mtp"
MASTER_NODE_ADDR="127.0.0.1:42123"
START_PORT=13222
START_DEVICE=0
LOG_DIR="log"
NNODES=16
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
nohup ./xllm \
--model $MODEL_PATH \
--devices="npu:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--draft_model $DRAFT_MODEL_PATH \
--draft_devices="npu:$DEVICE" \
--num_speculative_tokens 1 \
--max_memory_utilization=0.90 \
--max_tokens_per_batch=10000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--ep_size=1 \
--dp_size=1 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--node_rank=$i > $LOG_FILE 2>&1 &
sleep 0.5
done
```
#### GLM4 MoE Launch Example
```bash
MODEL_PATH="/models/GLM-4.5-Air"
DRAFT_MODEL_PATH="/models/GLM-4.5-Air-mtp"
# ... same other configurations
```
# Performance Data
Based on ShareGPT dataset with input length=2500, output length=1500, total requests=80.
| method | Concurrency | Mean TPOT(ms) | Mean TTFT(ms) | Output Tokens/s | Total Tokens/s |
|:---------:|:-----------:|:-------------:|:-------------:|:---------------:|:--------------:|
| baseline | 1 | 40.61 | 141.80 | 24.20 | 65.77 |
| mtp | 1 | 28.33 | 142.35 | 35.19 | 95.52 |
| baseline | 2 | 42.69 | 178.59 | 45.16 | 122.74 |
| mtp | 2 | 29.81 | 187.97 | 64.75 | 175.78 |
| baseline | 4 | 46.18 | 172.34 | 79.83 | 216.96 |
| mtp | 4 | 33.54 | 194.22 | 111.18 | 301.81 |
| baseline | 8 | 53.16 | 181.49 | 110.68 | 300.81 |
| mtp | 8 | 40.99 | 203.37 | 154.46 | 419.34 |
| baseline | 16 | 68.50 | 213.89 | 143.81 | 390.84 |
| mtp | 16 | 57.04 | 254.99 | 201.89 | 548.04 |
| baseline | 20 | 74.72 | 228.80 | 154.77 | 420.65 |
| mtp | 20 | 61.73 | 264.34 | 206.24 | 559.84 |
| baseline | 40 | 119.68 | 559.32 | 180.22 | 489.80 |
| mtp | 40 | 105.70 | 544.54 | 252.91 | 686.74 |
| baseline | 80 | 180.89 | 2996.21 | 192.09 | 522.06 |
| mtp | 80 | 152.19 | 2163.72 | 278.07 | 755.12 |

31
upstream_ref/xllm/docs/en/features/multi_streams.md Normal file
View File

@@ -0,0 +1,31 @@
# Multi-stream parallel
## Background
In distributed inference scenarios for large-scale models, additional communication operations are required to aggregate computation results from different devices. Taking large-scale MoE models like Deepseek as an example, the distributed scale is typically substantial, leading to increased communication overhead.
If both computation and communication are performed on the same stream, the device’s computing resources will remain idle while waiting for communication to complete, resulting in wasted computational capacity before subsequent calculations can begin.
## Introduction
xLLM implements multi-stream parallelism at the model layer, where the input batch is split into 2 micro-batches. One stream handles computation for the first micro-batch, another concurrently executes communication for the second micro-batch.
This overlap of computation and communication effectively hides the communication latency.![multi_streams_parallel](../../assets/multi_streams_architecture.jpg)
## Usage
xLLM provides the gflags parameter `enable_multi_stream_parallel`, which defaults to false. To enable this feature, set it to true in xLLM’s service startup script, as:
```shell
--enable_multi_stream_parallel=true
```
## Performance
With prefill dual-stream parallelism enabled, it can effectively mask over 75% of communication overhead.
On the DeepSeek-R1 model, when generating just 1 token, this achieves:
- **7%** reduction in TTFT.
- **7%** throughput improvement.
## Notice
The dual-stream parallelism currently only supports the prefill phase, with greater performance benefits observed for longer input requests.
Only Support DeepSeek, Qwen3 dense(non-MoE) models.

51
upstream_ref/xllm/docs/en/features/overview.md Normal file
View File

@@ -0,0 +1,51 @@
# Overall Architecture
## Backgroud
In recent years, with the groundbreaking progress of large language models (LLMs) ranging from tens of billions to trillions of parameters (such as GPT, Claude, DeepSeek, LLaMA, etc.) in the fields of natural language processing and multimodal interaction, the industry has an urgent need for efficient inference engines and service systems. How to reduce cluster inference costs and improve inference efficiency has become a key challenge for achieving large-scale commercial deployment.
Although a number of optimization engines for large model inference have emerged, several technical bottlenecks remain in practical deployment:
* **Hardware Adaptability Challenges:** Existing inference engines lack sufficient support for the architectural characteristics of specialized accelerators like domestic chips, making it difficult to fully exploit the performance potential of heterogeneous computing hardware, leading to low computational resource utilization.
* **MoE Architecture Optimization Difficulties:** The token distribution process in expert parallelism mechanisms generates significant All-to-All communication overhead, while dynamic routing strategies cause expert load imbalance, severely constraining system scalability.
* **Long Context Management Bottlenecks:** As model context windows continue to expand, the efficiency of optimizations in KV cache handling—such as memory fragmentation management and cross-node synchronization—directly impacts overall inference throughput performance.
* **Hybrid Deployment Efficiency Limitations:** Existing inference clusters struggle to simultaneously guarantee service quality (SLO) and optimize resource utilization when handling both online services and offline tasks.
* **Insufficient Dynamic PD Adaptation:** When input/output sequence lengths fluctuate drastically, static PD resource partitioning lacks the ability to adjust PD resource configurations in real-time. This can lead to idle GPU resources and poses a risk of SLO violations.
To address these challenges, we present xLLM—an efficient and user-friendly open-source intelligent inference framework that provides enterprise-grade service guarantees and high-performance engine computing capabilities for model inference on domestic chips.
## Feature Introduction
xLLM provides intelligent computing capabilities, implementing joint inference acceleration across multiple computational system layers and algorithm-driven layers:
### Computational System Layer
#### Multi-Level Pipeline Execution Orchestration
Asynchronizes CPU scheduling at the framework layer to pipeline it with chip inference computation, reducing computation bubbles; at the model graph layer, splits single batches to create pipelines between micro-batches, overlapping computation and communication; at the operator kernel layer, pipelines different computing units, overlapping computation and memory access.
#### Dynamic Shape Graph Execution Optimization
Addressing the static graph adaptation problem for large language models processing dynamic inputs (e.g., variable sequence lengths and batch sizes), xLLM achieves dynamic adaptation through parametric design capturing input dimensions. It combines a multi-graph caching scheme to reduce compilation overhead and uses a managed memory pool instead of absolute addresses to ensure safe reuse, ultimately achieving high execution efficiency while maintaining high flexibility.
#### Operator Optimization
xLLM implements specific optimizations for key operators in LLMs on domestic hardware chips, including GroupMatmul, Chunked Prefill, and others.
#### xTensor Memory Management
The xTensor memory management framework employs a method of *pre-allocated physical memory page pools + contiguous virtual address mapping*. It achieves efficient dynamic memory management through dynamic on-demand mapping of physical pages, reuse of reusable memory pages (Reusable), and optimized scheduling with asynchronous pre-mapping. Combined with NPU operator adaptation (such as virtual address FlashMLA), it results in improved memory utilization and reduced latency.
### Algorithm-Driven Layer
#### PD Separation
xLLM fully supports PD separation scenarios, enabling efficient management of PD instances, communication between PD instances, and KV cache transfer.
#### Global Scheduling
xLLM provides intelligent, full-lifecycle resource scheduling management for requests and instances.
##### Instance Scheduling
We have implemented various instance scheduling strategies to select how to assign instances to more suitable ones. These include a simple Round Robin strategy, a prefix cache-aware strategy based on the prefix cache hit rate of requests on each instance, and a KV cache-aware strategy based on the free memory level of instances. Furthermore, for PD separation scenarios, where static PD ratios often struggle with traffic fluctuations and sudden changes in request input/output lengths, we implemented an adaptive PD dynamic scheduler responsible for global instance allocation for online requests and runtime PD dynamic adjustment.
##### Request Scheduling
We have implemented various request scheduling strategies supporting continuous batching, including chunked prefill, prefill priority, decode priority, and other batch strategies, all while fully supporting PD separation scenarios.
#### Global KV Cache Management
Utilizes ETCD as a metadata service middleware at the global level for cluster service registration, load information synchronization, and global cache state management. Each compute instance maintains a local multi-level cache pool. Regarding scheduling strategy, the system adopts a dynamic decision-making mechanism based on KV cache: it first performs prefix matching detection, calculates the KV cache reuse rate of candidate nodes, and finally selects the node with the optimal comprehensive performance for processing, achieving dynamic offloading and migration of KV cache.
#### Speculative Inference
xLLM incorporates an optimized speculative inference algorithm that generates multiple tokens at once to boost throughput. xLLM reduces communication costs by sinking the speculative module and optimizes speculative inference computation through methods like overlapping scheduling and computation timelines and reducing operator data movement in speculative scenarios.
#### MoE Load Balancing
xLLM implements expert weight updates based on historical expert load statistics for MoE models. During inference, it achieves effective dynamic load balancing through efficient expert load statistics and double-buffered, seamless expert weight updates.
### Multimodal Support
xLLM provides comprehensive support for various multimodal models, including Qwen2-VL and MiniCPMV.

36
upstream_ref/xllm/docs/en/features/ppmatmul.md Normal file
View File

@@ -0,0 +1,36 @@
# PpMatmul Operator Optimization
## Background
In the inference of large models, matrix multiplication accounts for a high proportion and takes a long time. We have optimized the implementation of the matrix multiplication operator.
## Feature Introduction
The PpMatmul operator uses a Tiling strategy to decompose matrix multiplication into multiple smaller matrix multiplication tasks. However, when the number of tiles is small, tasks cannot be evenly distributed across all NPU cores, leading to the tail effect problem, which affects computational efficiency. We optimize the performance of the PpMatmul operator by prefetching memory or redistributing tasks.
## User Interface
### Operator Direct Call API
```cpp
aclnnStatus aclnnPpMatmulOptGetWorkspaceSize(
const aclTensor *a,
const aclTensor *b,
const aclTensor *out,
uint64_t *workspaceSize,
aclOpExecutor **executor);
aclnnStatus aclnnPpMatmulOpt(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream);
```
- `a`: Input matrix A.
- `b`: Input matrix B.
- `out`: Output matrix, storing the computation result.
## Performance Effect
For cases with a small number of tiles (e.g., when M is small, corresponding to a small batch size), there is an **18%** performance improvement of the operator compared to before optimization when (TP=4).

19
upstream_ref/xllm/docs/en/features/prefix_cache.md Normal file
View File

@@ -0,0 +1,19 @@
# Prefix Cache Optimization
## Feature Introduction
xLLM supports prefix cache matching. The prefix cache is based on `murmur_hash` and uses an LRU eviction policy, delivering superior matching efficiency and increased prefix cache hit rates.
Additionally, the prefix cache has been optimized to support the `continuous_scheduler`, `chunked_scheduler`, and `zero_evict_scheduler`. The cache is updated immediately after prefill operations, enhancing matching timeliness. For the `chunked_scheduler`, multi-stage chunked prefill matching is supported, reducing computational overhead and minimizing KV cache usage as much as possible.
## Usage
The prefix cache is implemented in xLLM and exposed through gflags parameters to control its functionality.
- Enable prefix cache with specific policy and settings:
```
--enable_prefix_cache=true
```
## Performance Impact
After enabling prefix cache, on the Qwen3-8B model with a TPOT constraint of 50ms, the E2E latency **decreased by 10%**.
!!! warning "Note"
PD separation scheduler is not currently supported.

27
upstream_ref/xllm/docs/en/features/topk_topp.md Normal file
View File

@@ -0,0 +1,27 @@
# Topk & Topp Operator Optimization
## Background
In natural language generation tasks, the topK and topP sampling strategies are widely used to control the diversity and quality of generated text. However, in small models, the computation time for these two strategies is relatively long. This is mainly due to the fewer parameters in small models, which leads to reduced efficiency in sorting and filtering when processing probability distributions, thereby affecting generation speed. Therefore, optimizing the implementation of topK and topP in small models can enhance their sampling efficiency.
## Feature Introduction
The implementation of the topKtopP operator merges multiple small operators, such as sorting, topK, softmax, and topP, into a single large operator, thereby improving computational efficiency and performance.
## User Interface
### Operator Call API
```c++
void top_k_top_p(torch::Tensor& logits,
const torch::Tensor& topK,
const torch::Tensor& topP);
```
- `logits`: The input logits tensor containing the model's output scores.
- `topK`: The threshold tensor for selecting the top K probabilities.
- `topP`: The threshold tensor for selecting the cumulative probabilities.
## Performance Effect
* After using the topKtopP fused operator, in the qwen2-0.5B model, TTOT **decreased by 37%**, and TTFT **increased by 10%**.

54
upstream_ref/xllm/docs/en/features/xllm_service_overview.md Normal file
View File

@@ -0,0 +1,54 @@
# xLLM Service
<p align="right">[:simple-github: xLLM Service](https://github.com/jd-opensource/xllm-service)</p>
## Project Overview
**xLLM-service** is a service-layer framework developed based on the **xLLM** inference engine, providing efficient, fault-tolerant, and flexible LLM inference services for clustered deployment.
xLLM-service targets to address key challenges in enterprise-level service scenarios:
- How to ensure the SLA of online services and improve resource utilization of offline tasks in a hybrid online-offline deployment environment.
- How to react to changing request loads in actual businesses, such as fluctuations in input/output lengths.
- Resolving performance bottlenecks of multimodal model requests.
- Ensuring high reliability of computing instances.
#### Background
LLM with parameter scales ranging from tens of billions to trillions are being rapidly deployed in core business scenarios such as intelligent customer service, real-time recommendation, and content generation. Efficient support for domestic computing hardware has become a core requirement for low-cost inference deployment. Existing inference engines struggle to effectively adapt to the architectural characteristics of dedicated accelerators like domestic chips. Performance issues such as low utilization of computing units, load imbalance and communication overhead bottlenecks under the MoE architecture, and difficulties in kv cache management have restricted the efficient inference of requests and the scalability of the system. The xLLM-service + xLLM inference engine improves the efficiency of the entire performance link and it provides crucial technical support for the large-scale implementation of LLM in real-world business scenarios.
---
## Overall Architecture
The overall architecture of xLLM-service is shown in the figure below:
![1](../../assets/service_arch.png)
## Core Components
### ETCD Cluster
It is used for metadata management, including the storage and management of metadata such as models, xllm instances, and requests. It also provides xllm node registration and discovery services.
### Fault Tolerance
xLLM-service provides fault tolerance management to ensure service quality and stability.
### Global Scheduler
It implements globally aware scheduling. Based on the current system status, it accurately dispatches requests to the optimal instances for execution, effectively improving the overall service response efficiency and resource utilization.
### Global KV Cache Manager
It is responsible for global KV Cache management. Its core capabilities include distributed KV cache awareness, Prefix matching, and dynamic migration of KV Cache, which optimize the efficiency of cache resource usage.
### Instance Manager
It focuses on the full-lifecycle management of instances. All xllm instances must register to service after startup. Based on preset policies, the module provides support for instances such as scheduling adaptation and fault tolerance handling.
### Event Plane
As the metrics and event hub, it receives Metrics data reported by various instances, uniformly collects and organizes statistical indicators, and provides data support for decisions such as service scheduling, fault tolerance, and scaling.
### Planner
It undertakes the functions of strategy analysis and decision-making. Based on the Metrics data reported by the Event Plane (including instance runtime indicators, machine load indicators, etc.), it analyzes the service scaling needs and the necessity of expanding hot instances, and outputs resource adjustment and instance optimization strategies.

17
upstream_ref/xllm/docs/en/features/zero_evict_scheduler.md Normal file
View File

@@ -0,0 +1,17 @@
# Zero Evict Scheduler
## Feature Introduction
xLLM supports the zero evict scheduling strategy. The zero evict scheduling strategy is an algorithm designed to minimize request eviction rates, reducing the need for prefill computation on evicted requests and consequently improving TPOT (Time Per Output Token).
This scheduling algorithm employs simulation rounds to detect whether a request can be scheduled without causing the eviction of other requests.
## Usage
The aforementioned strategy has been implemented in xLLM and is exposed through gflags parameters to control the feature's on/off state.
- Enable the zero evict strategy and set the maximum decode tokens per sequence.
```
--use_zero_evict=true
--max_decode_token_per_sequence=256
```
## Performance Impact
After enabling zero evict, on the Qwen3-8B model with an E2E latency constraint, the TPOT latency **decreased by 27%**.

99
upstream_ref/xllm/docs/en/getting_started/disagg_pd.md Normal file
View File

@@ -0,0 +1,99 @@
# PD disaggregation
`xllm` supports PD disaggregation deployment, which requires integration with our other open-source library [xllm service](https://github.com/jd-opensource/xllm-service).
## xLLM Service Dependencies
First, download and install `xllm service`, similar to installing and compiling `xllm`:
```bash
git clone https://github.com/jd-opensource/xllm-service
cd xllm_service
git submodule init
git submodule update
```
### etcd Installation
`xllm_service` compilation and operation depend on [etcd](https://github.com/etcd-io/etcd).Use the [installation script](https://github.com/etcd-io/etcd/releases) provided by etcd for installation. The default installation path provided by the script is `/tmp/etcd-download-test/etcd`. You can either manually modify the installation path in the script or manually migrate after running the script:
```bash
mv /tmp/etcd-download-test/etcd /path/to/your/etcd
```
### xLLM Service Compilation
Apply patch:
```bash
sh prepare.sh
```
Then execute the compilation:
```bash
mkdir -p build
cd build
cmake ..
make -j 8
cd ..
```
!!! warning "Potential Errors"
You may encounter installation errors related to `boost-locale` and `boost-interprocess`: `vcpkg-src/packages/boost-locale_x64-linux/include: No such file or directory`, `/vcpkg-src/packages/boost-interprocess_x64-linux/include: No such file or directory`
Reinstall these packages using `vcpkg`:
```bash
/path/to/vcpkg remove boost-locale boost-interprocess
/path/to/vcpkg install boost-locale:x64-linux
/path/to/vcpkg install boost-interprocess:x64-linux
```
## PD Disaggregation Execution
Start etcd:
```bash
./etcd-download-test/etcd --listen-peer-urls 'http://localhost:2390' --listen-client-urls 'http://localhost:2389' --advertise-client-urls 'http://localhost:2391'
```
Start xllm service:
```bash
ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
```
Taking Qwen2-7B as an example:
- Start Prefill Instance
```bash
/path/to/xllm --model=path/to/Qwen2-7B-Instruct \
--port=8010 \
--devices="npu:0" \
--master_node_addr="127.0.0.1:18888" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=PREFILL \
--etcd_addr=127.0.0.1:12389 \
--transfer_listen_port=26000 \
--disagg_pd_port=7777 \
--node_rank=0 \
--nnodes=1
```
- Start Decode Instance
```bash
/path/to/xllm --model=path/to/Qwen2-7B-Instruct \
--port=8020 \
--devices="npu:1" \
--master_node_addr="127.0.0.1:18898" \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_disagg_pd=true \
--instance_role=DECODE \
--etcd_addr=127.0.0.1:12389 \
--transfer_listen_port=26100 \
--disagg_pd_port=7787 \
--node_rank=0 \
--nnodes=1
```
Important notes:
- PD disaggregation requires reading the `/etc/hccn.conf` file. Make sure this file on the physical machine is mapped into the container.
- `etcd_addr` must match the `etcd_addr` of `xllm_service`
The test command is similar to above. Note that the `PORT` in `curl http://localhost:{PORT}/v1/chat/completions ...` should be the `port` of the `http_server_port` of xLLM service.

125
upstream_ref/xllm/docs/en/getting_started/launch_xllm.md Normal file
View File

@@ -0,0 +1,125 @@
# Launch xllm
Taking Qwen3 as an example, the script for launching xllm is as follows. The provided script is suitable for both single-node single-device and single-node multi-device scenarios. When using multiple devices on a single node, you need to modify `NNODES` (one device represents one node), as well as environment variables such as `ASCEND_RT_VISIBLE_DEVICES`, `CUDA_VISIBLE_DEVICES`, or `MLU_VISIBLE_DEVICES`.
## NPU
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_RT_VISIBLE_DEVICES=0
export HCCL_IF_BASE_PORT=43432 # HCCL communication base port
MODEL_PATH="/path/to/model/Qwen3-8B" # Model path
MASTER_NODE_ADDR="127.0.0.1:9748" # Master node address (must be globally consistent)
START_PORT=18000 # Service starting port
START_DEVICE=0 # Starting logical device number
LOG_DIR="log" # Log directory
NNODES=1 # Number of nodes (current script launches 1 process)
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="npu:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--block_size=128 \
--communication_backend="hccl" \
--enable_prefix_cache=false \
--enable_chunked_prefill=true \
--enable_schedule_overlap=true \
--enable_shm=true \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
## NVIDIA GPU
```bash
#!/bin/bash
set -e
rm -rf core.*
export CUDA_VISIBLE_DEVICES=0
# for debug
# export CUDA_LAUNCH_BLOCKING=1
MODEL_PATH="/path/to/model/Qwen3-8B"
MASTER_NODE_ADDR="127.0.0.1:9748"
START_PORT=18000
START_DEVICE=0
LOG_DIR="log"
NNODES=1
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="cuda:$DEVICE" \
--port $PORT \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--block_size=32 \
--max_memory_utilization=0.8 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--enable_schedule_overlap=true \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
## MLU
```bash
#!/bin/bash
set -e
rm -rf core.*
export MLU_VISIBLE_DEVICES=0
MODEL_PATH="/path/to/model/Qwen3-8B"
MASTER_NODE_ADDR="127.0.0.1:9748"
START_PORT=18000
START_DEVICE=0
LOG_DIR="log"
NNODES=1
mkdir -p $LOG_DIR
for (( i=0; i<$NNODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--devices="mlu:$DEVICE" \
--port $PORT \
--nnodes=$NNODES \
--master_node_addr=$MASTER_NODE_ADDR \
--block_size=16 \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```

109
upstream_ref/xllm/docs/en/getting_started/multi_machine.md Normal file
View File

@@ -0,0 +1,109 @@
# Multi-Node Deployment
This example demonstrates how to launch a 32-GPU (NPU) deployment across 2 machines.
Launching Services on the First Machine:
```shell
bash start_deepseek_machine_1.sh
```
The start_deepseek_machine_1.sh script is as follows:
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export HCCL_IF_BASE_PORT=43432 # HCCL communication base port
# 4. Start distributed service
MODEL_PATH="/path/to/your/DeepSeek-R1" # Model path
MASTER_NODE_ADDR="123.123.123.123:9748" # Master node address (must be globally consistent)
LOCAL_HOST=123.123.123.123 # Local IP for service launch
START_PORT=18000 # Service starting port
START_DEVICE=0 # Starting NPU logical device number
LOCAL_NODES=16 # Number of local processes (this script launches 16 processes)
LOG_DIR="log" # Log directory
NNODES=32 # Total number of GPUs/NPUs (32 in this 2-machine example)
mkdir -p $LOG_DIR
for (( i=0; i<$LOCAL_NODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--max_tokens_per_batch=40000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--communication_backend="hccl" \
--enable_schedule_overlap=true \
--rank_tablefile=./ranktable_2s_32p.json \
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
Launching Services on the Second Machine:
```shell
bash start_deepseek_machine_2.sh
```
The start_deepseek_machine_2.sh script is as follows:
```bash
#!/bin/bash
set -e
rm -rf core.*
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export HCCL_IF_BASE_PORT=43432 # HCCL communication base port
MODEL_PATH="/path/to/your/DeepSeek-R1" # Model path
MASTER_NODE_ADDR="123.123.123.123:9748" # Master node address (must be globally consistent)
LOCAL_HOST=456.456.456.456 # Local IP for service launch
START_PORT=18000 # Service starting port
START_DEVICE=0 # Starting NPU logical device number
LOCAL_NODES=16 # Number of local processes (this script launches 16 processes)
LOG_DIR="log" # Log directory
NNODES=32 # Total number of GPUs/NPUs (32 in this 2-machine example)
mkdir -p $LOG_DIR
for (( i=0; i<$LOCAL_NODES; i++ ))
do
PORT=$((START_PORT + i))
DEVICE=$((START_DEVICE + i))
LOG_FILE="$LOG_DIR/node_$i.log"
/path/to/xllm \
--model $MODEL_PATH \
--host $LOCAL_HOST \
--port $PORT \
--devices="npu:$DEVICE" \
--master_node_addr=$MASTER_NODE_ADDR \
--nnodes=$NNODES \
--max_memory_utilization=0.86 \
--max_tokens_per_batch=40000 \
--max_seqs_per_batch=256 \
--block_size=128 \
--enable_prefix_cache=false \
--enable_chunked_prefill=false \
--communication_backend="hccl" \
--enable_schedule_overlap=true \
--rank_tablefile=./ranktable_2s_32p.json \
--node_rank=$((i + LOCAL_NODES)) \ > $LOG_FILE 2>&1 &
done
```
This example uses 2 machines. You can set the total number of GPUs/NPUs via `--nnodes`, where `--node_rank` specifies the global rank ID for each node.
The `--rank_tablefile=./ranktable_2s_32p.json`parameter points to the configuration file required for establishing the NPU communication domain. For instructions on generating this file, refer to [Ranktable Generation](https://gitee.com/mindspore/models/blob/master/utils/hccl_tools/README.md).

15
upstream_ref/xllm/docs/en/getting_started/offline_service.md Normal file
View File

@@ -0,0 +1,15 @@
# Offline Inference
To facilitate users in quickly using xLLM for offline inference, we provide Python script examples for launching offline inference.
## LLM
LLM inference example: [:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate.py)
## Embedding
Generate embedding example: [:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate_embedding.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate_embedding.py)
## VLM
VLM inference example: [:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py)

225
upstream_ref/xllm/docs/en/getting_started/online_service.md Normal file
View File

@@ -0,0 +1,225 @@
# Online Service
First, start the xllm service according to the [xllm launch documentation](launch_xllm.md). Below are examples of client calls for LLM and VLM. Please modify the parameters according to your actual situation.
## LLM Client Calls
### HTTP Call
Chat mode:
```bash
curl http://localhost:9977/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"max_tokens": 10,
"temperature": 0,
"stream": true,
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "hello xllm"
}
]
}'
```
Completions mode:
```bash
curl http://127.0.0.1:9977/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"prompt": "hello xllm",
"max_tokens": 10,
"temperature": 0,
"stream": true
}'
```
Sample mode:
```bash
curl http://127.0.0.1:9977/v1/sample \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B-Instruct",
"prompt": "Question: <emb_0> matched or not. Conclusion: <emb_0>",
"selector": {
"type": "literal",
"value": "<emb_0>"
},
"logprobs": 5,
"request_id": "sample-demo-001"
}'
```
Typical response:
```json
{
"id": "sample-demo-001",
"object": "sample_completion",
"created": 1773369600,
"model": "Qwen2-7B-Instruct",
"choices": [
{
"index": 0,
"text": "True",
"logprobs": {
"tokens": ["True", "False"],
"token_ids": [3456, 7890],
"token_logprobs": [-0.12, -2.31]
},
"finish_reason": "selector_match"
},
{
"index": 1,
"text": "",
"logprobs": {
"tokens": [],
"token_ids": [],
"token_logprobs": []
},
"finish_reason": "empty_logprobs"
}
],
"usage": {
"prompt_tokens": 20,
"completion_tokens": 2,
"total_tokens": 22
}
}
```
`/v1/sample` notes:
- Only `--backend=llm` is supported. VLM/DiT/Rec are not supported yet.
- `selector.type` is currently fixed to `literal`. `selector.value` is matched against prompt text in full and in order.
- `logprobs` defaults to `5`, with an allowed range of `[1, 5]`.
- `choices[i].index` is the matched `sample_id`, corresponding one-to-one with the matched order in prompt.
- If no selector match is found, the service returns `200` with `choices=[]`. If a matched position has no available logprobs, it returns `finish_reason="empty_logprobs"`.
- Service logs only summary fields such as `request_id`, `sample_id`, `match_count`, and `model`, and do not log the full prompt.
`/v1/sample` common error semantics:
- Missing `model/prompt/selector/selector.value`, `selector.type != literal`, or out-of-range `logprobs` returns `INVALID_ARGUMENT`.
- If the model does not exist or the backend is not `llm`, it returns `UNKNOWN`.
- When concurrency reaches the upper limit, it returns `RESOURCE_EXHAUSTED`.
- When the model is in sleep state, it returns `UNAVAILABLE`.
### Python Call
```python
import requests
import json
url = f"http://localhost:9977/v1/chat/completions"
messages = [
{'role': 'user', 'content': "List three countries and their capitals."}
]
request_data = {
"model": "Qwen2-7B-Instruct",
"messages": messages,
"stream": False,
"temperature": 0.6,
"max_tokens": 2048,
}
response = requests.post(url, json=request_data)
if response.status_code != 200:
print(response.status_code, response.text)
else:
ans = json.loads(response.text)["choices"]
print(ans[0]['message'])
```
## VLM Client Calls
### HTTP API
```python
import base64
import requests
api_url = "http://localhost:12345/v1/chat/completions"
image_url = ""
def encode_image(url: str) -> str:
with requests.get(url) as response:
response.raise_for_status()
result = base64.b64encode(response.content).decode("utf-8")
return result
image_base64 = encode_image(image_url)
payload = {
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{image_base64}"},
},
],
}
],
"model": "Qwen2.5-VL-7B-Instruct",
"max_completion_tokens": 128,
}
response = requests.post(
api_url,
json=payload,
headers={"Content-Type": "application/json"}
)
print(response.json())
```
### OpenAI API
```python
from openai import OpenAI
import base64
import requests
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:12345/v1"
image_url = ""
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
def encode_image(url: str) -> str:
with requests.get(url) as response:
response.raise_for_status()
result = base64.b64encode(response.content).decode("utf-8")
return result
image_base64 = encode_image(image_url)
chat_completion = client.chat.completions.create(
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{image_base64}"},
},
],
}
],
model="Qwen2.5-VL-7B-Instruct",
max_completion_tokens=128,
)
result = chat_completion.choices[0].message.content
print("Chat completion output:", result)
```

107
upstream_ref/xllm/docs/en/getting_started/quick_start.md Normal file
View File

@@ -0,0 +1,107 @@
# Quick Start
## Environment Setup
All images are stored [here](https://quay.io/repository/jd_xllm/xllm-ai?tab=tags). The docker startup command below uses the dev image as an example.
### NPU
Below are our pre-built dev image.
```bash
# A2 x86
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a2-x86-20260429
# A2 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a2-arm-20260429
# A3 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-a3-arm-20260429
```
Container startup command:
```bash
docker run -it \
--ipc=host \
-u 0 \
--name xllm-npu \
--privileged \
--network=host \
--device=/dev/davinci0 \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/local/sbin/:/usr/local/sbin/ \
-v /var/log/npu/conf/slog/slog.conf:/var/log/npu/conf/slog/slog.conf \
-v /var/log/npu/slog/:/var/log/npu/slog \
-v /var/log/npu/profiling/:/var/log/npu/profiling \
-v /var/log/npu/dump/:/var/log/npu/dump \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
### NVIDIA GPU
We provide a [Dockerfile](../../../docker/Dockerfile.cuda) for NVIDIA GPU usage, which can be used to build custom image. Of course, you can also use dev image we built based on the default Dockerfile:
```bash
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-cuda-x86
```
Container startup command:
```bash
sudo docker run -it \
--privileged \
--shm-size '128gb' \
--ipc=host \
--net=host \
--pid=host \
--name=xllm-cuda \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
### MLU
We cannot provide MLU image. If you already have the dev image, you can start the container with the following command:
```bash
sudo docker run -it \
--privileged \
--shm-size '128gb' \
--ipc=host \
--net=host \
--pid=host \
--name xllm-mlu \
-v $HOME:$HOME \
-w $HOME \
<docker_image_name> \
/bin/bash
```
## Build xllm
If you download a release image, i.e., an image with a version number in the tag, you can skip this step because the release image comes with a pre-compiled xllm binary, located at `/usr/local/bin/xllm`.
Download xllm and dependencies:
```bash
git clone https://github.com/jd-opensource/xllm
cd xllm
# Install pre-commit for the first time
pip install pre-commit
pre-commit install
git submodule update --init --recursive
```
The compiled binary file is located at `/path/to/xllm/build/xllm/core/server/xllm`. In a new image, the first compilation of xllm takes a long time because all dependencies in vcpkg need to be compiled, but subsequent compilations will be much faster.
```bash
python setup.py build
```
## Launch xllm
Please refer to [How to Launch xllm](launch_xllm.md).

57
upstream_ref/xllm/docs/en/index.md Normal file
View File

@@ -0,0 +1,57 @@
---
hide:
- navigation
---
<style>
.md-content h1:first-of-type {
display: none;
}
</style>
<div style="text-align:center">
<img src="../assets/logo_with_llm.png" alt="xLLM" style="width:50%; height:auto;">
</div>
## Project Overview
**xLLM** is an efficient and user-friendly LLM intelligent inference framework that provides enterprise-level service guarantees and high-performance engine computing capabilities for model inference on domestic AI accelerators.
#### Background
LLM with parameter scales ranging from tens of billions to trillions are being rapidly deployed in core business scenarios such as intelligent customer service, real-time recommendation, and content generation. Efficient support for domestic computing hardware has become a core requirement for low-cost inference deployment. Existing inference engines struggle to effectively adapt to the architectural characteristics of dedicated accelerators like domestic chips. Performance issues such as low utilization of computing units, load imbalance and communication overhead bottlenecks under the MoE architecture, and difficulties in kv cache management have restricted the efficient inference of requests and the scalability of the system. The xLLM inference engine improves the resource efficiency of the entire "communication-computation-storage" performance link and it provides crucial technical support for the large-scale implementation of LLM in real-world business scenarios.
## Core Features
**xLLM** delivers robust intelligent computing capabilities. By leveraging hardware system optimization and algorithm-driven decision control, it jointly accelerates the inference process, enabling high-throughput, low-latency distributed inference services.
**Full Graph Pipeline Execution Orchestration**
- Asynchronous decoupled scheduling at the requests scheduling layer, to reduce computational bubbles.
- Asynchronous parallelism of computation and communication at the model graph layer, overlapping computation and communication.
- Pipelining of heterogeneous computing units at the operator kernel layer, overlapping computation and memory access.
**Graph Optimization for Dynamic Shapes**
- Dynamic shape adaptation based on parameterization and multi-graph caching methods to enhance the flexibility of static graph.
- Controlled tensor memory pool to ensure address security and reusability.
- Integration and adaptation of performance-critical custom operators (e.g., *PageAttention*, *AllReduce*).
**MoE Kernel Optimization**
- *GroupMatmul* optimization to improve computational efficiency.
- Chunked Prefill optimization to support long-sequence inputs.
**Efficient Memory Optimization**
- Mapping management between discrete physical memory and continuous virtual memory.
- On-demand memory allocation to reduce memory fragmentation.
- Intelligent scheduling of memory pages to increase memory reusability.
- Adaptation of corresponding operators for domestic accelerators.
**Global KV Cache Management**
- Intelligent offloading and prefetching of KV in hierarchical caches.
- KV cache-centric distributed storage architecture.
- Intelligent KV routing among computing nodes.
**Algorithm-driven Acceleration**
- Speculative decoding optimization to improve efficiency through multi-core parallelism.
- Dynamic load balancing of MoE experts to achieve efficient adjustment of expert distribution.

49
upstream_ref/xllm/docs/en/supported_models.md Normal file
View File

@@ -0,0 +1,49 @@
# Model Support List
## LLM
| | NPU | MLU | ILU |
| ------------------------ | :---: | :---: | :---: |
| DeepSeek-V3/R1/V3.1 | ✅ | ✅ | ❌ |
| DeepSeek-V3.2 | ✅ | ✅ | ❌ |
| DeepSeek-R1-Distill-Qwen | ✅ | ❌ | ❌ |
| Qwen2/2.5/QwQ | ✅ | ✅ | ✅ |
| Qwen3 | ✅ | ✅ | ✅ |
| Qwen3 Moe | ✅ | ✅ | ✅ |
| Kimi-k2 | ✅ | ❌ | ❌ |
| Llama2/3 | ✅ | ❌ | ✅ |
| GLM4.5 | ✅ | ❌ | ❌ |
| GLM4.6 | ✅ | ❌ | ❌ |
| GLM-4.7 | ✅ | ❌ | ❌ |
| GLM-5 | ✅ | ❌ | ❌ |
## VLM
| | NPU | MLU | ILU |
| ------------ | :---: | :---: | :---: |
| MiniCPM-V | ✅ | ❌ | ❌ |
| MiMo-VL | ✅ | ❌ | ❌ |
| Qwen2.5-VL | ✅ | ✅ | ❌ |
| Qwen3-VL | ✅ | ✅ | ❌ |
| Qwen3-VL-MoE | ✅ | ✅ | ❌ |
| GLM-4.6V | ✅ | ❌ | ❌ |
| VLM-R1 | ✅ | ❌ | ❌ |
## Rerank
| | NPU | MLU | ILU |
| -------------- | :---: | :---: | :---: |
| Qwen3-Reranker | ✅ | ❌ | ❌ |
## DiT
| | NPU | MLU | ILU |
| ---- | :---: | :---: | :---: |
| Flux | ✅ | ❌ | ❌ |
## Rec
| | NPU | MLU | ILU |
| --- | :---: | :---: | :---: |
| OneRec | ✅ | ❌ | ❌ |
| Qwen2 | ✅ | ❌ | ❌ |
| Qwen2.5 | ✅ | ❌ | ❌ |
| Qwen3 | ✅ | ❌ | ❌ |

19
upstream_ref/xllm/docs/mkdocs/javascripts/mathjax.js Normal file
View File

@@ -0,0 +1,19 @@
window.MathJax = {
tex: {
inlineMath: [["\\(", "\\)"]],
displayMath: [["\\[", "\\]"]],
processEscapes: true,
processEnvironments: true
},
options: {
ignoreHtmlClass: ".*|",
processHtmlClass: "arithmatex"
}
};
document$.subscribe(() => {
MathJax.startup.output.clearCache()
MathJax.typesetClear()
MathJax.texReset()
MathJax.typesetPromise()
})

1
upstream_ref/xllm/docs/mkdocs/overrides/.icons/email-fill.svg Normal file
View File

@@ -0,0 +1 @@
<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1755326167427" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="13611" xmlns:xlink="http://www.w3.org/1999/xlink" width="256" height="256"><path d="M926.47619 355.644952V780.190476a73.142857 73.142857 0 0 1-73.142857 73.142857H170.666667a73.142857 73.142857 0 0 1-73.142857-73.142857V355.644952l304.103619 257.828572a170.666667 170.666667 0 0 0 220.745142 0L926.47619 355.644952zM853.333333 170.666667a74.044952 74.044952 0 0 1 26.087619 4.778666 72.704 72.704 0 0 1 30.622477 22.186667 73.508571 73.508571 0 0 1 10.678857 17.67619c3.169524 7.509333 5.12 15.652571 5.607619 24.210286L926.47619 243.809524v24.380952L559.469714 581.241905a73.142857 73.142857 0 0 1-91.306666 2.901333l-3.632762-2.925714L97.52381 268.190476v-24.380952a72.899048 72.899048 0 0 1 40.155428-65.292191A72.97219 72.97219 0 0 1 170.666667 170.666667h682.666666z" p-id="13612" data-spm-anchor-id="a313x.search_index.0.i13.7cb13a81l9Qfa6" class="selected" fill="#acacac"></path></svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.1 KiB

1
upstream_ref/xllm/docs/mkdocs/overrides/.icons/gitcodeai.svg Normal file
View File

@@ -0,0 +1 @@
<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1755324951889" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="5508" width="200" height="200" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M660.48 212.352c5.952-3.392 12.352-6.912 19.2-10.88l2.368 12.032c1.28 6.272 2.368 11.456 2.88 16.512 4.032 44.288 26.304 76.928 57.856 84.224 46.144 10.752 89.664-7.04 113.088-46.08 28.16-46.848 16-104.32-32-138.88C690.56 33.216 544.128 6.72 386.304 53.056 46.144 153.344-70.272 571.264 171.2 827.52c103.296 109.632 234.432 156.928 383.36 153.344 190.72-4.48 328-99.52 415.744-264.256 62.08-116.864-5.44-244.608-134.848-271.168a837.376 837.376 0 0 0-224.768-14.4c-24.96 2.432-49.28 9.792-71.424 21.76-24.768 12.8-31.872 39.488-29.12 65.92 2.56 24.064 21.056 38.528 43.008 42.176a1837.44 1837.44 0 0 0 133.76 14.592c12.928 1.088 25.984 1.28 39.04 1.344 18.752 0.192 37.376 0.384 55.68 3.392 52.032 8.576 69.888 50.816 43.136 96-6.592 10.88-14.208 21.056-22.848 30.336a259.392 259.392 0 0 1-131.392 77.76c-92.416 22.592-184.896 23.872-276.8-5.12-104.704-33.088-167.168-109.952-169.344-213.504-0.832-63.872 15.36-126.72 46.976-182.272 14.272-25.856 22.08-52.48 19.84-81.856-0.896-12.48-1.408-24.96-1.92-38.4a2748.8 2748.8 0 0 0-1.024-22.4c10.624 2.176 21.12 4.992 31.36 8.32 40.064 16.128 79.488 23.488 122.88 11.648A222.08 222.08 0 0 1 517.76 256a188.608 188.608 0 0 0 115.648-28.288c8.64-5.056 17.344-9.92 26.944-15.36z" fill="#ACACAC" p-id="5509"></path></svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.6 KiB

Some files were not shown because too many files have changed in this diff Show More