002f9879b2
Replaces cherry-picked upstream_ref with complete source trees. xllm/ — Iluvatar official C++ inference engine (15MB, 1470 files) Complete: kernels → layers → models → runtime → scheduler → api Excluded: .git, binary images, third_party submodule checkouts ds_vllm/ — Iluvatar official vllm fork (8MB, 703 files) Included: csrc/ (ALL CUDA kernels), fused_moe/, qwen3_5 model, _custom_ops Excluded: tests, benchmarks, docs, examples (not needed for reference) Critical call chains now fully traceable: MoE: moe_topk_softmax_kernels.cuh → ixformer.h → fused_moe.cpp → layer GDN: qwen3_gated_delta_net_base.cpp → qwen3_5_gated_delta_net.cpp Attention: ixformer.h → xllm_paged_attention → attention.cpp
2.3 KiB
Global Multi-Level KV Cache
Background
In the decoding phase of large language models (LLMs), frequent access to historical KV cache due to autoregressive generation creates a bottleneck in memory bandwidth. As model sizes and context windows expand (e.g., 128K Tokens consuming over 40GB of memory), the pressure on single-device memory increases dramatically. Existing solutions (such as vLLM) exhibit significant limitations in long-context scenarios: prefill time surges, severe memory bandwidth contention during decoding, and the need for excessive resource reservation to meet SLO requirements (TTFT < 2s, TBT < 100ms). This often results in GPU utilization below 40% and difficulties in leveraging cross-server resources. To address this, we propose a distributed global multi-level KV cache management system, adopting a memory-compute integrated architecture to break through single-machine resource constraints.
Feature Introduction
The xLLM Global KV Cache feature is primarily implemented through the following three modules:
- etcd: For cluster service registration, load information synchronization, and global cache state management.
- xLLM Service: For scheduling requests and managing all compute instances.
- xLLM: The compute instances handling requests.
The overall architecture is shown in the diagram below:
Usage Example
Preparation
Install Dependencies
- xLLM: Refer to Quick Start
- xLLM Service: Refer to PD disaggregation
Usage Instructions
-
etcd Startup Configuration:
./etcd --listen-peer-urls=http://0.0.0.0:10999 --listen-client-urls=http://0.0.0.0:10998 -
xLLM Service Startup Configuration:
./xllm_master_serving --etcd_addr="127.0.0.1:10998" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/ -
xLLM Startup Configuration: Add the following gflag parameters when starting xLLM:
--enable_service_routing=true --enable_cache_upload=true # PD separation currently does not support Global KVCache Management --enable_disagg_pd=false