002f9879b2
Replaces cherry-picked upstream_ref with complete source trees. xllm/ — Iluvatar official C++ inference engine (15MB, 1470 files) Complete: kernels → layers → models → runtime → scheduler → api Excluded: .git, binary images, third_party submodule checkouts ds_vllm/ — Iluvatar official vllm fork (8MB, 703 files) Included: csrc/ (ALL CUDA kernels), fused_moe/, qwen3_5 model, _custom_ops Excluded: tests, benchmarks, docs, examples (not needed for reference) Critical call chains now fully traceable: MoE: moe_topk_softmax_kernels.cuh → ixformer.h → fused_moe.cpp → layer GDN: qwen3_gated_delta_net_base.cpp → qwen3_5_gated_delta_net.cpp Attention: ixformer.h → xllm_paged_attention → attention.cpp
1.7 KiB
Multi-stream parallel
Background
In distributed inference scenarios for large-scale models, additional communication operations are required to aggregate computation results from different devices. Taking large-scale MoE models like Deepseek as an example, the distributed scale is typically substantial, leading to increased communication overhead.
If both computation and communication are performed on the same stream, the device’s computing resources will remain idle while waiting for communication to complete, resulting in wasted computational capacity before subsequent calculations can begin.
Introduction
xLLM implements multi-stream parallelism at the model layer, where the input batch is split into 2 micro-batches. One stream handles computation for the first micro-batch, another concurrently executes communication for the second micro-batch.
This overlap of computation and communication effectively hides the communication latency.
Usage
xLLM provides the gflags parameter enable_multi_stream_parallel, which defaults to false. To enable this feature, set it to true in xLLM’s service startup script, as:
--enable_multi_stream_parallel=true
Performance
With prefill dual-stream parallelism enabled, it can effectively mask over 75% of communication overhead. On the DeepSeek-R1 model, when generating just 1 token, this achieves:
- 7% reduction in TTFT.
- 7% throughput improvement.
Notice
The dual-stream parallelism currently only supports the prefill phase, with greater performance benefits observed for longer input requests. Only Support DeepSeek, Qwen3 dense(non-MoE) models.