ref(upstream): FULL TREE — Deep-Spark xllm (1470) + ds_vllm csrc/models (703)

Replaces cherry-picked upstream_ref with complete source trees. xllm/ — Iluvatar official C++ inference engine (15MB, 1470 files) Complete: kernels → layers → models → runtime → scheduler → api Excluded: .git, binary images, third_party submodule checkouts ds_vllm/ — Iluvatar official vllm fork (8MB, 703 files) Included: csrc/ (ALL CUDA kernels), fused_moe/, qwen3_5 model, _custom_ops Excluded: tests, benchmarks, docs, examples (not needed for reference) Critical call chains now fully traceable: MoE: moe_topk_softmax_kernels.cuh → ixformer.h → fused_moe.cpp → layer GDN: qwen3_gated_delta_net_base.cpp → qwen3_5_gated_delta_net.cpp Attention: ixformer.h → xllm_paged_attention → attention.cpp
2026-08-10 02:53:54 +00:00
parent 9e4fb3712f
commit 002f9879b2
2179 changed files with 494021 additions and 79 deletions
--- a/upstream_ref/xllm/docs/project/README_zh.md
+++ b/upstream_ref/xllm/docs/project/README_zh.md
@@ -0,0 +1,173 @@
+<!-- Copyright 2022 JD Co.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this project except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License. -->
+
+[English](../../README.md) | [中文](./README_zh.md)
+
+<div align="center">
+<img src="../assets/logo_with_llm.png" alt="xLLM" style="width:50%; height:auto;">
+    
+[![Document](https://img.shields.io/badge/Document-black?logo=html5&labelColor=grey&color=red)](https://xllm.readthedocs.io/zh-cn/latest/) [![Docker](https://img.shields.io/badge/Docker-black?logo=docker&labelColor=grey&color=%231E90FF)](https://hub.docker.com/r/xllm/xllm-ai) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen?labelColor=grey)](https://opensource.org/licenses/Apache-2.0) [![report](https://img.shields.io/badge/Technical%20Report-red?logo=arxiv&logoColor=%23B31B1B&labelColor=%23F0EBEB&color=%23D42626)](https://arxiv.org/abs/2510.14686) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/jd-opensource/xllm) 
+    
+</div>
+
+---------------------
+<p align="center">
+| <a href="https://xllm.readthedocs.io/zh-cn/latest/"><b>Documentation</b></a> |  <a href="https://arxiv.org/abs/2510.14686"><b>Technical Report</b></a> |
+</p>
+
+### 📢 新闻
+- 2026-04-24: 🎉 我们 day-0 支持了[DeepSeek-V4](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) 模型的推理服务，部署请参考[部署文档](https://github.com/jd-opensource/xllm/blob/preview/deepseek-v4-mlu/testspace/run_deepseek_v4.sh)。
+- 2026-02-12: 🎉 我们 day-0 支持了最新的[GLM-5](https://github.com/zai-org/GLM-5) 模型的高效推理服务，部署请参考[部署文档](https://github.com/zai-org/GLM-5/blob/main/example/ascend.md)。
+- 2025-12-21: 🎉 我们在第一时间内支持了[GLM-4.7](https://github.com/zai-org)模型的高效推理。
+- 2025-12-08: 🎉 我们在第一时间内支持了[GLM-4.6V](https://github.com/zai-org/GLM-V)模型的高效推理。
+- 2025-12-05: 🎉 我们支持了[GLM-4.5/GLM-4.6](https://github.com/zai-org/GLM-4.5/blob/main/README_zh.md)系列模型.
+- 2025-12-05: 🎉 我们支持了[VLM-R1](https://github.com/om-ai-lab/VLM-R1) 模型.
+- 2025-12-05: 🎉 我们基于[Mooncake](https://github.com/kvcache-ai/Mooncake)构建了混合 KV 缓存管理机制，支持具备智能卸载与预取能力的全局 KV 缓存管理。
+- 2025-10-16: 🎉 我们最近在 arXiv 上发布了我们的 [xLLM 技术报告](https://arxiv.org/abs/2510.14686)，提供了全面的技术蓝图和实施见解。
+
+## 简介
+
+**xLLM** 是一个高效的开源大模型推理框架，专为**国产芯片**优化设计，提供企业级的服务部署，使得性能更高、成本更低。该框架采用**服务-引擎分离的推理架构**，通过服务层的在离线请求弹性调度、动态PD分离、EPD混合机制及高可用容错设计，结合引擎层的多流并行计算、图融合优化、投机推理、动态负载均衡及全局KV缓存管理，实现推理效率突破性提升。xLLM整体架构和功能如下图所示：
+
+<div align="center">
+<img src="../assets/xllm_arch.png" alt="xllm_arch" style="width:90%; height:auto;">
+</div>
+
+**xLLM** 已支持主流大模型（如 *DeepSeek-V3.1*，*Qwen2/3*等）在国产芯片上的高效部署，助力企业实现高性能、低成本的 AI 大模型应用落地。xLLM已全面落地京东零售核心业务，涵盖智能客服、风控、供应链优化、广告推荐等多种场景。
+
+
+## 核心特性
+xLLM 提供了强大的智能计算能力，通过硬件系统的算力优化与算法驱动的决策控制，联合加速推理过程，实现高吞吐、低延迟的分布式推理服务。
+
+**全图化/多层流水线执行编排**
+- 框架调度层的异步解耦调度，减少计算空泡；
+- 模型图层的计算和通信异步并行，重叠计算与通信；
+- 算子内核层的异构计算单元深度流水，重叠计算与访存。
+
+**动态shape的图执行优化**
+- 基于参数化与多图缓存方法的动态尺寸适配，提升静态图灵活性；
+- 受管控的显存池，保证地址安全可复用；
+- 集成适配性能关键的自定义算子（如 *PageAttention*, *AllReduce*）。
+
+**高效显存优化**
+- 离散物理内存与连续虚拟内存的映射管理；
+- 按需分配内存空间，减少内存碎片与浪费；
+- 智能调度内存空间，增加内存页复用，减小分配延迟；
+- 国产芯片相应算子适配。
+
+**全局多级KV Cache管理**
+- 多级缓存的kv智能卸载与预取；
+- 以kv cache为中心的分布式存储架构；
+- 多节点间kv的智能传输路由。
+
+**算法优化**
+- 投机推理优化，多核并行提升效率；
+- MoE专家的动态负载均衡，实现专家分布的高效调整。
+
+---
+## 硬件支持
+
+| 硬件类型 | 型号   | 备注            |
+| -------- | ------ | --------------- |
+| NPU      | A2, A3 | HDK Driver 25.2.0 + |
+| MLU      |        |                 |
+| ILU      | BI150  |                 |
+| MUSA     | S5000  |                 |
+
+此外，请在[模型支持列表](../zh/supported_models.md)查看不同硬件上的模型支持情况。
+
+---
+
+## 快速开始
+
+请参考[快速开始文档](../zh/getting_started/quick_start.md)。
+
+---
+
+## 成为贡献者
+您可以通过以下方法为 xLLM 作出贡献:
+
+1. 在Issue中报告问题
+2. 提供改进建议
+3. 补充文档
+    + Fork仓库
+    + 修改文档
+    + 提出pull request
+4. 修改代码
+    + Fork仓库
+    + 创建新分支
+    + 加入您的修改
+    + 提出pull request
+
+感谢您的贡献！ 🎉🎉🎉
+如果您在开发中遇到问题，请参阅**[xLLM中文指南](https://xllm.readthedocs.io/zh-cn/latest)**
+
+---
+
+## 社区支持
+如果你在xLLM的开发或使用过程中遇到任何问题，欢迎在项目的Issue区域提交可复现的步骤或日志片段。
+如果您有企业内部Slack，请直接联系xLLM Core团队。另外，我们建立了官方微信群，可以访问以下二维码加入。欢迎沟通和联系我们:
+
+<div align="center">
+  <img src="../assets/wechat_qrcode.png" alt="qrcode3" width="50%" />
+</div>
+
+---
+
+## 致谢
+本项目的实现得益于以下开源项目: 
+
+- [ScaleLLM](https://github.com/vectorch-ai/ScaleLLM) - 采用了ScaleLLM中构图方式和借鉴Runtime执行。
+- [Mooncake](https://github.com/kvcache-ai/Mooncake) - 依赖构建了多级KV Cache管理机制。
+- [brpc](https://github.com/apache/brpc) - 依赖brpc构建了高性能http service。
+- [tokenizers-cpp](https://github.com/mlc-ai/tokenizers-cpp) - 依赖tokenizers-cpp构建了c++ tokenizer。
+- [safetensors](https://github.com/huggingface/safetensors) - 依赖其c binding safetensors能力。
+- [Partial JSON Parser](https://github.com/promplate/partial-json-parser) - xLLM的C++版本JSON解析器，参考Python与Go实现的设计思路。
+- [concurrentqueue](https://github.com/cameron314/concurrentqueue) - 高性能无锁Queue.
+
+感谢以下合作的高校实验室：
+
+- [THU-MIG](https://ise.thss.tsinghua.edu.cn/mig/projects.html)（清华大学软件学院、北京信息科学与技术国家研究中心）
+- USTC-Cloudlab（中国科学技术大学云计算实验室）
+- [Beihang-HiPO](https://github.com/buaa-hipo)（北京航空航天大学HiPO研究组）
+- PKU-DS-LAB（北京大学数据结构实验室）
+- PKU-NetSys-LAB（北京大学网络系统实验室）
+- [TJU-TANKLab](https://flashserve.org/) (天津大学TANK实验室)
+
+感谢以下为xLLM作出贡献的[开发者](https://github.com/jd-opensource/xllm/graphs/contributors)
+
+<a href="https://github.com/jd-opensource/xLLM/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=jd-opensource/xllm" />
+</a>
+
+---
+
+## 许可证
+
+[Apache License](LICENSE)
+
+#### xLLM 由 JD.com 提供 
+#### 感谢您对xLLM的关心与贡献!
+
+## 引用
+
+如果你觉得这个仓库对你有帮助，欢迎引用我们：
+```
+@article{liu2025xllm,
+  title={xLLM Technical Report},
+  author={Liu, Tongxuan and Peng, Tao and Yang, Peijun and Zhao, Xiaoyang and Lu, Xiusheng and Huang, Weizhe and Liu, Zirui and Chen, Xiaoyu and Liang, Zhiwei and Xiong, Jun and others},
+  journal={arXiv preprint arXiv:2510.14686},
+  year={2025}
+}
+```
--- a/upstream_ref/xllm/docs/project/RELEASE.md
+++ b/upstream_ref/xllm/docs/project/RELEASE.md
@@ -0,0 +1,208 @@
+# Release xllm 0.9.0
+
+## **Major Features and Improvements**
+
+### Model Support
+
+#### NPU
+- Support GLM-5 model.
+- Support GLM4.7-Flash model.
+- Support Qwen3-next model.
+- Support OneRec model.
+- Support Qwen3.5/Qwen3.5-MoE model.
+#### CUDA
+- Support LongCat-Image model.
+- Support LongCat-Image-Edit model.
+#### MLU
+- Support DeepSeek-V3.2 W4A8 MoE model.
+- Support GLM-5 W8A8 model.
+#### ILU
+- Support Qwen3-8B model.
+- Support Qwen3-30B-MoE model.
+
+### Feature
+- Adapt NPU builds to CANN 8.5 and PyTorch 2.7.1.
+- Support graph mode for the LLM part of VLM models on NPU devices.
+- Support context parallelism for NPU DeepSeek-V3.2 / GLM-5.
+- Support DeepSeek-V3.2 prefill sequence parallel on MLU devices.
+- Support rolling weight loading and loading model weights with varied prefixes.
+- Support dynamic and scalable multi-model serving.
+- Support bidirectional remote-host to local-device KV cache transfer and batch offload.
+- Support Qwen3 xattention on NPU devices.
+- Support prefix cache for DeepSeek-V3.2.
+- Support chunked prefill on CUDA devices.
+- Support embedding interface for all generate LLM models.
+- Support Anthropic Messages API.
+- Support the new `v1/sample` interface.
+- Support a single xLLM instance connecting to multiple xLLM services.
+- Support startup progress bar, worker health check, and unified request statistics logging.
+- Optimize Qwen3 MoE performance on NPU devices.
+- Add CUDA Graph Executor and piecewise prefill graph.
+- Support KV cache quantization on MLU devices.
+- Add VMM-based allocators to reuse graph buffers and physical memory.
+- Improve FP8 GEMM, fused RMSNorm, fused MoE, xattention, and activation kernel performance.
+
+### Bugfix
+- Support the new `compressed-tensors` FP8 config and fix Qwen2 prompt length.
+- Fix Qwen3 MoE VL parameter settings on MLU devices.
+- Fix Qwen VL issues on MLU devices and Qwen2.5 chunked-prefill accuracy on NPU.
+- Fix DeepSeek tool-call, prefix-cache, DP/MTP, and PD-disagg related issues.
+- Fix GLM-4.7 streaming function call issues and GLM detector stability issues.
+- Fix graph mode, schedule overlap, KV cache, and REC multi-round stability issues.
+- Fix multiple compile, link, env setup, and worker lifecycle issues.
+
+
+# Release xllm 0.8.0
+
+## **Major Features and Improvements**
+
+### Model Support
+
+#### NPU
+- Support DeepSeek-v3.2 model.
+- Support GLM4.7 model.
+- Support GLM4.6Vmodel.
+- Support GME-Qwen2-VL model.
+- Support FluxControl model.
+#### CUDA
+- Support Qwen2/3 Dense model.
+#### MLU
+- Support DeepSeek-v3.2 model.
+- Support Qwen2_5_vl/Qwen3_vl/Qwen3_vl_moe model.
+#### ILU
+- Support Qwen3-0.6B model.
+
+### Feature
+- Implement chunked prefill and prefix cache for Qwen3 MoE.
+- Support GLM-4.6V model.
+- Add wrappers for ATB and ACLNN fused operators.
+- Optimize prefetch from kv cache store.
+- Support Qwen2-VL & GME-Qwen2-VL model on npu device.
+- Fix hang issue when enable schedule overlap.
+- Add GLM-4.7 detector implementation and update tool call parser.
+- Adapt hierarchy block manager for disagg PD.
+- Support deepseek-v3.2-Exp for npu.
+- Support acl_graph for qwen3/qwen3_moe.
+- Support prefix cache for deepseek-v3/r1 models.
+- Support disagg PD for MTP.
+- Add mooncake kv cache transfer.
+- Add GLM-4.7 support to reasoning detector registry.
+- Support nd-to-nz continuous memory copy.
+- Support RPC-based link/unlink for PD disaggregation.
+- Support IntraLayerAddNorm, aclgraph, etc for DeepSeek V3.2.
+- Add activation, norm and rope ops for cuda device.
+- Support fused norm for Qwen3 and DeepSeek for cuda device.
+- Build deepseek v2 decoder layer and related model files for mlu device.
+- Support qwen2_5_vl/qwen3_vl/qwen3_vl_moe on mlu device.
+- Add moe all2all kernels and deep ep layer on mlu device.
+- Support deepseek mtp on mlu device.
+- Support graph executor on mlu device.
+- Support dp+ep moe and all2all computation on mlu device.
+- Support parallelized shared experts in fused moe on mlu device.
+- Support qwen3 0.6B model on iluvatar device.
+- Add rec proto,serivce and utils for rec framework
+- Support C api for llm inference.
+- Add constrained decoding for generative recommendation.
+- Add rec scheduler master and engine for rec framework.
+- Add rec_type and onerec batch input builder for rec framework.
+- Add onerec worker impl for rec framework.
+- Add qwen3/LlmRec support in rec framework.
+
+### Bugfix
+- Reslove core dump of stream chat completion request when backend is VLM.
+- Resolve duplicate content in multi-turn tool call conversations.
+- Fix core dump issue triggered by client disconnection.
+- Fix the memory leak issue in the completions interface.
+- Fix wrong positons of validate input when enable MTP.
+- Resolve kv_cache_num mismatch in ChunkedPrefill due to H2D block copy.
+- Fix the missing index shape in the allocate kv cache transfer.
+- Fix MiMo-VL weights loading crash on NPU device.
+- Fix inaccurate metrics issue when enabling schedule overlap.
+- Fix potential out-of-range and block leaks during deallocate in D2H copy.
+- Fix allocation failure in HierarchyBlockManagerPool::allocate.
+- Fix deepseek accuracy issues with prefix cache enabled.
+- Resolve Deepseek execution failure caused by invalid input.
+- Fix DeepSeek failing to run when enabling DP.
+- Fix the rate_limit bug for stream and non-stream request in PD disagg and refactor some callback logics.
+- Correct attn mask when prefix cache and MTP are both enabled in deepseek.
+- Correct precision loss when enabling prefixcache with disagg pd.
+- Fix incorrect async implementation in rerank interface.
+- Fix acl_graph_executor not handling q_cu_seq_lens parameter for deepseekv3.2.
+- Fix precision issue when enabling MTP in PD disaggregation mode.
+- Fix mrope calculation in the multimodal situation.
+- Fix core dump of large beam width.
+
+
+# Release xllm 0.7.0
+
+## **Major Features and Improvements**
+
+### Model Support
+
+- Support GLM-4.5.
+- Support Qwen3-Embedding.
+- Support Qwen3-VL.
+- Support FluxFill.
+
+### Feature
+- Support MLU backend, currently supports Qwen3 series models.
+- Support dynamic disaggregated PD, with dynamic switching between P and D phases based on strategy.
+- Support multi-stream parallel overlap optimization.
+- Support beam-search capability in generative models.
+- Support virtual memory continuous kv-cache capability.
+- Support ACL graph executor.
+- Support unified online-offline co-location scheduling in disaggregated PD scenarios.
+- Support PrefillOnly Scheduler.
+- Support v1/rerank model service interface.
+- Support communication between devices via shared memory instead of RPC on a single machine.
+- Support function call.
+- Support reasoning output in chat interface.
+- Support top-k+add fusion in the router component of MoE models.
+- Support offline inference for LLM, VLM, and Embedding models.
+- Optimized certain runtime performance.
+
+### Bugfix
+- Skip cancelled requests when processing stream output.
+- Resolve segmentation fault during qwen3 quantized inference.
+- Fix the alignment of monitoring metrics format for Prometheus.
+- Clear outdated tensors to save memory when loading model weights.
+- Fix attention mask to support long sequence requests.
+- Fix bugs caused by enabling scheduler overlap.
+
+# Release xllm 0.6.0
+
+## **Major Features and Improvements**
+
+### Model Support
+
+- Support DeepSeek-V3/R1.
+- Support DeepSeek-R1-Distill-Qwen.
+- Support Kimi-k2.
+- Support Llama2/3.
+- Support Qwen2/2.5/QwQ.
+- Support Qwen3/Qwen3-MoE.
+- Support MiniCPM-V.
+- Support MiMo-VL.
+- Support Qwen2.5-VL .
+
+### Feature
+
+- Support KV cache store.
+- Support Expert Parallelism Load Balance.
+- Support multi-priority on/offline scheduler.
+- Support latency-aware scheduler.
+- Support serving early stop.
+- Optimize ppmatmul kernel.
+- Support image url input for VLM.
+- Support disaggregated prefill and decoding.
+- Support large-scale EP parallelism.
+- Support Hash-based PrefixCache matching.
+- Support Multi-Token Prediction for DeepSeek.
+- Support asynchronous scheduling, allowing the scheduling and computational pipeline to execute in parallel.
+- Support EP, DP, TP model parallel.
+- Support multiple process and multiple nodes.
+
+### Docs
+
+- Add getting started docs.
+- Add features docs.