From 5d13bbe796d9bbc15e7db5de87f2b9da72f7b4a5 Mon Sep 17 00:00:00 2001 From: offline893 <158537145+offline893@users.noreply.github.com> Date: Thu, 25 Sep 2025 17:01:51 +0800 Subject: [PATCH] [BugFix]Modify eplb feature guide. (#3183) ### What this PR does / why we need it? Revise the EPLB feature guide content.Add eplb params to ascend config. ### Does this PR introduce any user-facing change? ### How was this patch tested? - vLLM version: v0.10.2 - vLLM main: https://github.com/vllm-project/vllm/commit/52d0cb845866869d587fc013a7c59e60a86ebcf2 Co-authored-by: offline0806 <3337230449@qq.com> --- .../configuration/additional_config.md | 6 + .../feature_guide/eplb_swift_balancer.md | 119 ++++++++++++------ 2 files changed, 90 insertions(+), 35 deletions(-) diff --git a/docs/source/user_guide/configuration/additional_config.md b/docs/source/user_guide/configuration/additional_config.md index 076f7f2..e709b3a 100644 --- a/docs/source/user_guide/configuration/additional_config.md +++ b/docs/source/user_guide/configuration/additional_config.md @@ -36,6 +36,12 @@ The following table lists the additional configuration options available in vLLM | `lmhead_tensor_parallel_size` | int | `None` | The custom tensor parallel size of lmhead. | | `oproj_tensor_parallel_size` | int | `None` | The custom tensor parallel size of oproj. | | `multistream_overlap_shared_expert`| bool | `False` | Whether to enable multistream shared expert. This option only takes effects on moe models with shared experts. | +| `dynamic_eplb` | bool | `False` | Whether to enable dynamic eplb | +|`num_iterations_eplb_update`| int | `400` | Forward iterations when eplb would begin | +|`gate_eplb`| bool | `False` | Whether to enale eplb only once. | +|`num_wait_worker_iterations`| int | `30` | The forward iterations when eplb worker will finish cpu task. In our test default value 30 would cover most cases. | +|`expert_map_record_path`| str | `None` | When dynamic eplb is completed, save the current expert load heatmap to the specified path. | +|`init_redundancy_expert`| int | `0` |Specify redundant experts during initialization.| The details of each config option are as follows: diff --git a/docs/source/user_guide/feature_guide/eplb_swift_balancer.md b/docs/source/user_guide/feature_guide/eplb_swift_balancer.md index a4d1e7a..707b263 100644 --- a/docs/source/user_guide/feature_guide/eplb_swift_balancer.md +++ b/docs/source/user_guide/feature_guide/eplb_swift_balancer.md @@ -1,45 +1,94 @@ -# Swift Balancer +# Expert Load Balance (EPLB) ## Overview -Experts rebalancing of MoE models for LLM serving is a mandatory option.Changing experts dynamically would have a negative impact on TTFT and TPOT while stop-the-world. -Asynchronously expert load balancing would be a better choice. -We have launched SwiftBalancer to support dynamic experts load balancing with Zero-overhead experts movement. -## Design +Expert balancing for MoE models in LLM serving is essential for optimal performance. Dynamically changing experts during inference can negatively impact TTFT (Time To First Token) and TPOT (Tokens Per Output Token) due to stop-the-world operations. SwiftBalancer enables asynchronous expert load balancing with zero-overhead expert movement, ensuring seamless service continuity. -![img.png](images/eplb_img.png) +## EPLB Effects -The overall workflow involves: -1. Record experts distribution during forward. We using expert_token_num after dispatch instead of topk_ids, thus we got much smaller tensor shape to reduce cost of hbm - recording and add-operator. -2. Do all-gather for experts distribution. Using all-gather instead of all-reduce as less traffic volume. -3. Wake up eplb worker process with experts distribution when num_iterations comes. Run eplb algorithm in eplb worker. -4. Generate p2p send/recv ops and other operator such as log2phy would cost long cpu time. -5. Lanch ibatch_send_recv in async_stream before forward. -6. After forward, wait for the ibatch_send_recv finish, then do uapte expert map and expert weights. +- Reduced Latency: Dynamically balances expert loads to minimize TTFT and TPOT by distributing workloads evenly across experts. +- Enhanced Throughput: Optimizes GPU utilization, increasing token generation speed under high-concurrency scenarios. +- Zero-Overhead Movement: Expert redistribution occurs asynchronously without interrupting ongoing inference requests. +- Adaptive Scaling: Automatically adjusts to workload fluctuations while maintaining stable performance. +- Fault Tolerance: Redundant expert placement ensures system resilience during hardware failures. -In our profiling shows experts transforming is hidden in the bubble between forward iterations. Cpu time cost of eplb algo. and other python operator such as log2phy -would be hidden by eplb worker process too. +## How to Use EPLB -## Config Params +### Dynamic EPLB -Currently swift balancer optimize 5ms TPOT with ep size 64 while cost less than 2ms for every layer expert movement. +Enable dynamic balancing with auto-tuned parameters. Adjust num_iterations_eplb_update and num_wait_worker_iterations based on workload patterns. -We add new parameters for eplb: -"dynamic_eplb":true --- enable dynamic eplb -"num_iterations_eplb_update": 400 -- forward iterations when eplb would begin -"gate_eplb":true -- eplb would update only once, false by default. -"num_wait_worker_iterations":30 -- forward iterations when eplb worker will finish cpu task. In our test default value 30 would cover most cases. -"expert_map_record_path" -- When dynamic eplb is completed, save the current expert load heatmap to the specified path. -"init_redundancy_expert" -- Specify redundant experts during initialization. +```shell +vllm serve Qwen/Qwen3-235B-A22 \ + --tensor-parallel-size 16 \ + --enable-expert-parallel \ + --additional-config '{ + "dynamic_eplb": true, + "num_iterations_eplb_update": 400, + "gate_eplb": true, + "num_wait_worker_iterations": 30 + }' +``` -## Examples -### Dynamic eplb -Enable dynamic eplb and specify the trigger rounds. ---additional-config '{ "dynamic_eplb":true,"num_iterations_eplb_update":400, "gate_eplb":true, "num_wait_worker_iterations":30}' -### Record expert map for static eplb -Specify the path for the static eplb initialization file. ---additional-config '{ "expert_map_record_path": "/xx/xx.json", "init_redundancy_expert": 16, dynamic_eplb":true,"num_iterations_eplb_update":400, "gate_eplb":true, "num_wait_worker_iterations":30}' -### Static eplb -If expert map has been recorded, enable static eplb with expert map path. ---additional-config '{ "expert_map_path": "/xx/xx.json"}' +### Static EPLB +#### Initial Setup (Record Expert Map) + +Generate the initial expert distribution map using expert_map_record_path. This creates a baseline configuration for future deployments. + +```shell +vllm serve Qwen/Qwen3-235B-A22 \ + --tensor-parallel-size 16 \ + --enable-expert-parallel \ + --additional-config '{ + "expert_map_record_path": "/path/to/eplb.json", + "init_redundancy_expert": 16, + "dynamic_eplb": true, + "num_iterations_eplb_update": 400, + "gate_eplb": true, + "num_wait_worker_iterations": 30 + }' +``` + +#### Subsequent Deployments (Use Recorded Map) +Load the pre-recorded expert map for consistent performance. This avoids recalculating distributions at runtime. + +```shell +vllm serve Qwen/Qwen3-235B-A22 \ + --tensor-parallel-size 16 \ + --enable-expert-parallel \ + --additional-config '{ + "expert_map_path": "/path/to/eplb.json" + }' +``` + +## Critical Considerations +1. Parameter Tuning: + - num_iterations_eplb_update: Higher values (e.g., 400+) for stable workloads; lower values (e.g., 100-200) for fluctuating traffic. + - num_wait_worker_iterations: Should be ≥30 to avoid premature balancing during startup. + - init_redundancy_expert: Must match tensor-parallel size (e.g., 16 for 16 GPUs) to ensure sufficient redundancy. + +2. Hardware Requirements: + - Ensure all GPUs have identical memory capacity and compute capabilities. + - Network bandwidth must support expert redistribution traffic (≥10Gbps recommended). + +3. Model Compatibility: + - Only MoE models with explicit expert parallelism support (e.g., Qwen3-235B-A22) are compatible. + - Verify model architecture supports dynamic expert routing via --enable-expert-parallel. + +4. Gating Configuration: + - When gate_eplb=true, validate that the gating mechanism can handle expert movement without routing errors. + - Test with synthetic workloads before production deployment. + +5. Monitoring & Validation: + - Track metrics: expert_load_balance_ratio, ttft_p99, tpot_avg, and gpu_utilization. + - Use vllm monitor to detect imbalances during runtime. + - Always verify expert map JSON structure before loading (validate with jq or similar tools). + +6. Startup Behavior: + - Initial requests may experience higher latency during the first balancing cycle (typically 1-2 minutes). + - Avoid sudden traffic spikes during warm-up phase. + +7. Common Pitfalls: + - Incorrect tensor-parallel-size vs. actual GPU count → causes resource underutilization. + - Using expert_map_path without generating the map first → runtime errors. + - Setting init_redundancy_expert > available GPUs → system failure.