From a15754c3ba5455011e9501c59a774ceeef37cae7 Mon Sep 17 00:00:00 2001 From: starkwj Date: Thu, 12 Feb 2026 11:13:26 +0800 Subject: [PATCH] add readme --- README.md | 214 ++++---------------------------------------------- README.md.bak | 214 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 228 insertions(+), 200 deletions(-) create mode 100644 README.md.bak diff --git a/README.md b/README.md index 8e7c0df..bf0b8ef 100644 --- a/README.md +++ b/README.md @@ -1,214 +1,28 @@ -![vLLM Kunlun Logo](vllm_kunlun/patches/vLLM_Kunlun.jpg) - -

- πŸ“– Documentation | - πŸš€ Quick Start | - πŸ“¦ Installation | - πŸ’¬ Slack -

- -

- GitHub License - GitHub Stars - GitHub Forks - GitHub Issues - Python Version -

- ---- - -## Latest News πŸ”₯ - -- [2026/02] 🧠 **GLM model family support** β€” Added GLM5, GLM-4.7 MTP (Multi-Token Prediction), and GLM-47 tool parser with thinking/non-thinking mode toggle -- [2026/02] ⚑ **Performance optimizations** β€” Fused MoE with small batches, optimized attention metadata building, Multi-LoRA inference achieves 80%+ of non-LoRA performance -- [2026/02] πŸ”§ **DeepSeek-V3.2 MTP support** β€” Added MTP (Multi-Token Prediction) for DeepSeek-V3.2, with RoPE and decoding stage kernel optimizations -- [2026/01] πŸ”’ **New quantization methods** β€” Support for compressed-tensors W4A16, AWQ MoE W4A16, and DeepSeek-V3.2 W8A8 quantization -- [2026/01] πŸ› οΈ **CI/CD overhaul** β€” Added E2E tests, unit test CI, ruff format checks, and modular CI workflow refactoring -- [2025/12] πŸŽ‰ **v0.11.0rc1 released** β€” Added Qwen3-Omni, Qwen3-Next, Seed-OSS support ([Release Notes](https://github.com/baidu/vLLM-Kunlun/releases/tag/v0.11.0rc1)) -- [2025/12] πŸ“¦ **v0.10.1.1 released** β€” 5+ multimodal models, AWQ/GPTQ quantization for dense models, Piecewise CUDA Graph, vLLM V1 engine, Flash-Infer Top-K/Top-P sampling with 10-100Γ— speedup ([Release Notes](https://github.com/baidu/vLLM-Kunlun/releases/tag/v0.10.1.1)) -- [2025/12] 🌟 Initial release of vLLM Kunlun β€” Open sourced on Dec 8, 2025 - ---- +# XC-LLM: A Specially Optimized LLM Inference Engine for ModelHub XC ## Overview -**vLLM Kunlun** (`vllm-kunlun`) is a community-maintained hardware plugin designed to seamlessly run [vLLM](https://github.com/vllm-project/vllm) on the **Kunlun XPU**. It is the recommended approach for integrating the Kunlun backend within the vLLM community, adhering to the principles outlined in the [RFC Hardware Pluggable](https://github.com/vllm-project/vllm/issues/11162). +The project is optimized based on the popular LLM inference project vLLM. The repository supports Kunlun3 P800. -This plugin provides a hardware-pluggable interface that decouples the integration of the Kunlun XPU with vLLM. By utilizing vLLM Kunlun, popular open-source models β€” including Transformer-like, Mixture-of-Expert (MoE), Embedding, and Multi-modal LLMs β€” can run effortlessly on the Kunlun XPU. +One of the key features of this project is efficient memory coordination, enabling multiple vLLM instances share and dynamically hold GPU(XPU)'s physical memory. When an instance is idle, model parameters are offloaded to host memory. Upon a new inference request, the model parameters are quickly restored to the GPU’s memory (if not exist), without the need to initialize the engine and load the model from scratch. As a result, from the application’s perspective, multiple LLM inference engines can run on the GPU even when their total memory requirements exceed the physical memory limit. This technique is referred to as `InfiniVRAM`. -### ✨ Key Features -- **Seamless Plugin Integration** β€” Works as a standard vLLM platform plugin via Python entry points, no need to modify vLLM source code -- **Broad Model Support** β€” Supports 15+ mainstream LLMs including Qwen, Llama, DeepSeek, Kimi-K2, and multimodal models -- **Quantization Support** β€” INT8 and other quantization methods for MoE and dense models -- **LoRA Fine-Tuning** β€” LoRA adapter support for Qwen series models -- **Piecewise Kunlun Graph** β€” Hardware-accelerated graph optimization for high-performance inference -- **FlashMLA Attention** β€” Optimized multi-head latent attention for DeepSeek MLA architectures -- **Tensor Parallelism** β€” Multi-device parallel inference with distributed execution support -- **OpenAI-Compatible API** β€” Serve models with the standard OpenAI API interface +## Installation ---- +### Build from Dockerfile -## Prerequisites - -- **Hardware**: Kunlun3 P800 -- **OS**: Ubuntu 22.04 -- **Software**: - - Python >= 3.10 - - PyTorch >= 2.5.1 - - vLLM (same version as vllm-kunlun) - - transformers >= 4.57.0 - ---- - -## Supported Models - -### Generative Models - -| Model | Support | Quantization | LoRA | Kunlun Graph | -|:------|:-------:|:------------:|:----:|:----------------------:| -| Qwen2 | βœ… | βœ…| βœ… | βœ… | -| Qwen2.5 | βœ… |βœ… | βœ… | βœ… | -| Qwen3 | βœ… |βœ… | βœ… | βœ… | -| Qwen3-Moe | βœ… | βœ… | | βœ… | -| Qwen3-Next | βœ… | βœ… | | βœ… | -| MiMo-V2-Flash | βœ… | βœ…| | βœ… | -| Llama2 | βœ… | βœ…| βœ…| βœ… | -| Llama3 | βœ… |βœ… | βœ… | βœ… | -| Llama3.1 | βœ… |βœ… | | βœ… | -| gpt-oss | βœ… | βœ…| | | -| GLM4.5 | βœ… | βœ…| | βœ… | -| GLM4.5Air | βœ… |βœ… | | βœ… | -| GLM4.7 | βœ… | βœ…| | βœ… | -| GLM5 | βœ… | βœ…| | βœ… | -| Kimi-K2 | βœ… | βœ… | | βœ… | -| DeepSeek-R1 | βœ… | βœ… | | βœ… | -| DeepSeek-V3 | βœ… | βœ… | | βœ… | -| DeepSeek-V3.2 | βœ… | βœ… | | βœ… | - -### Multimodal Language Models - -| Model | Support | Quantization | LoRA | Kunlun Graph | -|:------|:-------:|:------------:|:----:|:----------------------:| -| Qwen2-VL | βœ… | βœ…| | βœ… | -| Qwen2.5-VL | βœ… | βœ…| | βœ… | -| Qwen3-VL | βœ… | βœ…| | βœ… | -| Qwen3-VL-MoE | βœ… | βœ… | | βœ… | -| Qwen3-Omni-MoE | βœ… | | | βœ… | -| InternVL-2.5 | βœ… | | | βœ… | -| InternVL-3.5 | βœ… | | | βœ… | -| InternS1 | βœ… | | | βœ… | - ---- - -## Performance Visualization πŸš€ - -### High-performance computing at work: How different models perform on the Kunlun3 P800. - -Current environment: 16-way concurrency, input/output size 2048. - -![Models and tgs](./vllm_kunlun/patches/performance.png) - ---- - -### Quick Start - -#### Start an OpenAI-Compatible API Server +Clone this repository: ```bash -python -m vllm.entrypoints.openai.api_server \ - --host 0.0.0.0 \ - --port 8356 \ - --model \ - --gpu-memory-utilization 0.9 \ - --trust-remote-code \ - --max-model-len 32768 \ - --tensor-parallel-size 1 \ - --dtype float16 \ - --max_num_seqs 128 \ - --max_num_batched_tokens 32768 \ - --block-size 128 \ - --distributed-executor-backend mp \ - --served-model-name +docker build -t $build_image -f ./Dockerfile . ``` -#### Send a Request +## Usage -```bash -curl http://localhost:8356/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "", - "messages": [{"role": "user", "content": "Hello!"}], - "max_tokens": 512 - }' -``` +0. To share GPU, processes coordinate via shm, so you need to set all containers with `ipc=host`. +1. Start a daemon process in a standalone container, by running `vllm_vxpu_daemon` installed inside the image. +2. Start LLM services with this image, following the official usage instructions. -### Version Matrix - -| Version | Release Type | Documentation | -|---------|:------------:|:-------------:| -| v0.11.0 | Latest stable version | [Quick Start](https://vllm-kunlun.readthedocs.io/en/latest/quick_start.html) Β· [Installation](https://vllm-kunlun.readthedocs.io/en/latest/installation.html) | - ---- - -## Architecture - -``` -vllm-kunlun/ -β”œβ”€β”€ vllm_kunlun/ # Core plugin package -β”‚ β”œβ”€β”€ platforms/ # Kunlun XPU platform implementation -β”‚ β”œβ”€β”€ models/ # Model implementations (DeepSeek, Qwen, Llama, etc.) -β”‚ β”œβ”€β”€ ops/ # Custom operators (attention, linear, sampling, etc.) -β”‚ β”‚ β”œβ”€β”€ attention/ # FlashMLA, paged attention, merge attention states -β”‚ β”‚ β”œβ”€β”€ fla/ # Flash linear attention operations -β”‚ β”‚ └── sample/ # Sampling operators -β”‚ β”œβ”€β”€ v1/ # vLLM V1 engine adaptations -β”‚ β”œβ”€β”€ compilation/ # Torch compile wrapper for Kunlun Graph -β”‚ β”œβ”€β”€ csrc/ # C++ extensions (custom CUDA-compatible kernels) -β”‚ └── config/ # Model configuration overrides -β”œβ”€β”€ tests/ # Test suite -β”œβ”€β”€ docs/ # Documentation (Sphinx-based, ReadTheDocs hosted) -β”œβ”€β”€ ci/ # CI pipeline configurations -β”œβ”€β”€ setup.py # Legacy build script (with C++ extensions) -└── pyproject.toml # Modern Python build configuration (hatchling) -``` - ---- - -## Contributing - -We welcome contributions from the community! Please read our [Contributing Guide](CONTRIBUTING.md) before submitting a PR. - -### PR Classification - -Use the following prefixes for PR titles: - -- `[Attention]` β€” Attention mechanism features/optimizations -- `[Core]` β€” Core vllm-kunlun logic (platform, attention, communicators, model runner) -- `[Kernel]` β€” Compute kernels and ops -- `[Bugfix]` β€” Bug fixes -- `[Doc]` β€” Documentation improvements -- `[Test]` β€” Tests -- `[CI]` β€” CI/CD improvements -- `[Misc]` β€” Other changes - ---- - -## Star History πŸ”₯ - -We opened the project at Dec 8, 2025. We love open source and collaboration ❀️ - -[![Star History Chart](https://api.star-history.com/svg?repos=baidu/vLLM-Kunlun&type=date&legend=bottom-right)](https://www.star-history.com/#baidu/vLLM-Kunlun&type=date&legend=bottom-right) - ---- - -## Sponsors πŸ‘‹ - -We sincerely appreciate the [**KunLunXin**](https://www.kunlunxin.com/) team for their support in providing XPU resources, which enabled efficient model adaptation debugging, comprehensive end-to-end testing, and broader model compatibility. - ---- - -## License - -Apache License 2.0, as found in the [LICENSE](./LICENSE) file. +### Environment Variables +- `VXPU_RESERVED_VRAM_SIZE_GB`: The amonut of reserved GPU memory for other miscellaneous memory. Only needs to be set for `vllm_vxpu_daemon`. Try increasing the variable if you launch multiple LLM services and encounter OOM. Default: `8`. +- `VLLM_VXPU_SHM_NAME`: The name of the shm file. Needs to be set for all containers of the shared vxpu group. Default: `/vllm_kunlun_vxpu_offload_shm`. \ No newline at end of file diff --git a/README.md.bak b/README.md.bak new file mode 100644 index 0000000..775db22 --- /dev/null +++ b/README.md.bak @@ -0,0 +1,214 @@ +![vLLM Kunlun Logo](vllm_kunlun/patches/vLLM_Kunlun.jpg) + +

+ πŸ“– Documentation | + πŸš€ Quick Start | + πŸ“¦ Installation | + πŸ’¬ Slack +

+ +

+ GitHub License + GitHub Stars + GitHub Forks + GitHub Issues + Python Version +

+ +--- + +## Latest News πŸ”₯ + +- [2026/02] 🧠 **GLM model family support** β€” Added GLM5, GLM-4.7 MTP (Multi-Token Prediction), and GLM-47 tool parser with thinking/non-thinking mode toggle +- [2026/02] ⚑ **Performance optimizations** β€” Fused MoE with small batches, optimized attention metadata building, Multi-LoRA inference achieves 80%+ of non-LoRA performance +- [2026/02] πŸ”§ **DeepSeek-V3.2 MTP support** β€” Added MTP (Multi-Token Prediction) for DeepSeek-V3.2, with RoPE and decoding stage kernel optimizations +- [2026/01] πŸ”’ **New quantization methods** β€” Support for compressed-tensors W4A16, AWQ MoE W4A16, and DeepSeek-V3.2 W8A8 quantization +- [2026/01] πŸ› οΈ **CI/CD overhaul** β€” Added E2E tests, unit test CI, ruff format checks, and modular CI workflow refactoring +- [2025/12] πŸŽ‰ **v0.11.0rc1 released** β€” Added Qwen3-Omni, Qwen3-Next, Seed-OSS support ([Release Notes](https://github.com/baidu/vLLM-Kunlun/releases/tag/v0.11.0rc1)) +- [2025/12] πŸ“¦ **v0.10.1.1 released** β€” 5+ multimodal models, AWQ/GPTQ quantization for dense models, Piecewise CUDA Graph, vLLM V1 engine, Flash-Infer Top-K/Top-P sampling with 10-100Γ— speedup ([Release Notes](https://github.com/baidu/vLLM-Kunlun/releases/tag/v0.10.1.1)) +- [2025/12] 🌟 Initial release of vLLM Kunlun β€” Open sourced on Dec 8, 2025 + +--- + +## Overview + +**vLLM Kunlun** (`vllm-kunlun`) is a community-maintained hardware plugin designed to seamlessly run [vLLM](https://github.com/vllm-project/vllm) on the **Kunlun XPU**. It is the recommended approach for integrating the Kunlun backend within the vLLM community, adhering to the principles outlined in the [RFC Hardware Pluggable](https://github.com/vllm-project/vllm/issues/11162). + +This plugin provides a hardware-pluggable interface that decouples the integration of the Kunlun XPU with vLLM. By utilizing vLLM Kunlun, popular open-source models β€” including Transformer-like, Mixture-of-Expert (MoE), Embedding, and Multi-modal LLMs β€” can run effortlessly on the Kunlun XPU. + +### ✨ Key Features + +- **Seamless Plugin Integration** β€” Works as a standard vLLM platform plugin via Python entry points, no need to modify vLLM source code +- **Broad Model Support** β€” Supports 15+ mainstream LLMs including Qwen, Llama, DeepSeek, Kimi-K2, and multimodal models +- **Quantization Support** β€” INT8 and other quantization methods for MoE and dense models +- **LoRA Fine-Tuning** β€” LoRA adapter support for Qwen series models +- **Piecewise Kunlun Graph** β€” Hardware-accelerated graph optimization for high-performance inference +- **FlashMLA Attention** β€” Optimized multi-head latent attention for DeepSeek MLA architectures +- **Tensor Parallelism** β€” Multi-device parallel inference with distributed execution support +- **OpenAI-Compatible API** β€” Serve models with the standard OpenAI API interface + +--- + +## Prerequisites + +- **Hardware**: Kunlun3 P800 +- **OS**: Ubuntu 22.04 +- **Software**: + - Python >= 3.10 + - PyTorch >= 2.5.1 + - vLLM (same version as vllm-kunlun) + - transformers >= 4.57.0 + +--- + +## Supported Models + +### Generative Models + +| Model | Support | Quantization | LoRA | Kunlun Graph | +|:------|:-------:|:------------:|:----:|:----------------------:| +| Qwen2 | βœ… | βœ…| βœ… | βœ… | +| Qwen2.5 | βœ… |βœ… | βœ… | βœ… | +| Qwen3 | βœ… |βœ… | βœ… | βœ… | +| Qwen3-Moe | βœ… | βœ… | | βœ… | +| Qwen3-Next | βœ… | βœ… | | βœ… | +| MiMo-V2-Flash | βœ… | βœ…| | βœ… | +| Llama2 | βœ… | βœ…| βœ…| βœ… | +| Llama3 | βœ… |βœ… | βœ… | βœ… | +| Llama3.1 | βœ… |βœ… | | βœ… | +| gpt-oss | βœ… | βœ…| | | +| GLM4.5 | βœ… | βœ…| | βœ… | +| GLM4.5Air | βœ… |βœ… | | βœ… | +| GLM4.7 | βœ… | βœ…| | βœ… | +| GLM5 | βœ… | βœ…| | βœ… | +| Kimi-K2 | βœ… | βœ… | | βœ… | +| DeepSeek-R1 | βœ… | βœ… | | βœ… | +| DeepSeek-V3 | βœ… | βœ… | | βœ… | +| DeepSeek-V3.2 | βœ… | βœ… | | βœ… | + +### Multimodal Language Models + +| Model | Support | Quantization | LoRA | Kunlun Graph | +|:------|:-------:|:------------:|:----:|:----------------------:| +| Qwen2-VL | βœ… | βœ…| | βœ… | +| Qwen2.5-VL | βœ… | βœ…| | βœ… | +| Qwen3-VL | βœ… | βœ…| | βœ… | +| Qwen3-VL-MoE | βœ… | βœ… | | βœ… | +| Qwen3-Omni-MoE | βœ… | | | βœ… | +| InternVL-2.5 | βœ… | | | βœ… | +| InternVL-3.5 | βœ… | | | βœ… | +| InternS1 | βœ… | | | βœ… | + +--- + +## Performance Visualization πŸš€ + +### High-performance computing at work: How different models perform on the Kunlun3 P800. + +Current environment: 16-way concurrency, input/output size 2048. + +![Models and tgs](./vllm_kunlun/patches/performance.png) + +--- + +### Quick Start + +#### Start an OpenAI-Compatible API Server + +```bash +python -m vllm.entrypoints.openai.api_server \ + --host 0.0.0.0 \ + --port 8356 \ + --model \ + --gpu-memory-utilization 0.9 \ + --trust-remote-code \ + --max-model-len 32768 \ + --tensor-parallel-size 1 \ + --dtype float16 \ + --max_num_seqs 128 \ + --max_num_batched_tokens 32768 \ + --block-size 128 \ + --distributed-executor-backend mp \ + --served-model-name +``` + +#### Send a Request + +```bash +curl http://localhost:8356/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "", + "messages": [{"role": "user", "content": "Hello!"}], + "max_tokens": 512 + }' +``` + +### Version Matrix + +| Version | Release Type | Documentation | +|---------|:------------:|:-------------:| +| v0.11.0 | Latest stable version | [Quick Start](https://vllm-kunlun.readthedocs.io/en/latest/quick_start.html) Β· [Installation](https://vllm-kunlun.readthedocs.io/en/latest/installation.html) | + +--- + +## Architecture + +``` +vllm-kunlun/ +β”œβ”€β”€ vllm_kunlun/ # Core plugin package +β”‚ β”œβ”€β”€ platforms/ # Kunlun XPU platform implementation +β”‚ β”œβ”€β”€ models/ # Model implementations (DeepSeek, Qwen, Llama, etc.) +β”‚ β”œβ”€β”€ ops/ # Custom operators (attention, linear, sampling, etc.) +β”‚ β”‚ β”œβ”€β”€ attention/ # FlashMLA, paged attention, merge attention states +β”‚ β”‚ β”œβ”€β”€ fla/ # Flash linear attention operations +β”‚ β”‚ └── sample/ # Sampling operators +β”‚ β”œβ”€β”€ v1/ # vLLM V1 engine adaptations +β”‚ β”œβ”€β”€ compilation/ # Torch compile wrapper for Kunlun Graph +β”‚ β”œβ”€β”€ csrc/ # C++ extensions (custom CUDA-compatible kernels) +β”‚ └── config/ # Model configuration overrides +β”œβ”€β”€ tests/ # Test suite +β”œβ”€β”€ docs/ # Documentation (Sphinx-based, ReadTheDocs hosted) +β”œβ”€β”€ ci/ # CI pipeline configurations +β”œβ”€β”€ setup.py # Legacy build script (with C++ extensions) +└── pyproject.toml # Modern Python build configuration (hatchling) +``` + +--- + +## Contributing + +We welcome contributions from the community! Please read our [Contributing Guide](CONTRIBUTING.md) before submitting a PR. + +### PR Classification + +Use the following prefixes for PR titles: + +- `[Attention]` β€” Attention mechanism features/optimizations +- `[Core]` β€” Core vllm-kunlun logic (platform, attention, communicators, model runner) +- `[Kernel]` β€” Compute kernels and ops +- `[Bugfix]` β€” Bug fixes +- `[Doc]` β€” Documentation improvements +- `[Test]` β€” Tests +- `[CI]` β€” CI/CD improvements +- `[Misc]` β€” Other changes + +--- + +## Star History πŸ”₯ + +We opened the project at Dec 8, 2025. We love open source and collaboration ❀️ + +[![Star History Chart](https://api.star-history.com/svg?repos=baidu/vLLM-Kunlun&type=date&legend=bottom-right)](https://www.star-history.com/#baidu/vLLM-Kunlun&type=date&legend=bottom-right) + +--- + +## Sponsors πŸ‘‹ + +We sincerely appreciate the [**KunLunXin**](https://www.kunlunxin.com/) team for their support in providing XPU resources, which enabled efficient model adaptation debugging, comprehensive end-to-end testing, and broader model compatibility. + +--- + +## License + +Apache License 2.0, as found in the [LICENSE](./LICENSE) file. \ No newline at end of file