EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

[NVIDIA] Change to use num_local_experts (#8453)

Browse Source
This commit is contained in:
Kaixi Hou
2025-07-28 10:38:19 -07:00
committed by GitHub
parent ccfe52a057
commit 134fa43e19
2 changed files with 3 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/backend/server_arguments.md
View File

@@ -214,7 +214,8 @@ Please consult the documentation below and [server_args.py](https://github.com/s
| `--ep-size` | The expert parallelism size. | 1 | | `--ep-size` | The expert parallelism size. | 1 |
| `--enable-ep-moe` | Enabling expert parallelism for moe. The ep size is equal to the tp size. | False | | `--enable-ep-moe` | Enabling expert parallelism for moe. The ep size is equal to the tp size. | False |
| `--enable-deepep-moe` | Enabling DeepEP MoE implementation for EP MoE. | False | | `--enable-deepep-moe` | Enabling DeepEP MoE implementation for EP MoE. | False |
| `--enable-flashinfer-moe` | Enabling Flashinfer MoE implementation. | False | | `--enable-flashinfer-cutlass-moe` | Enabling Flashinfer Cutlass MoE implementation for high throughput. | False |
| `--enable-flashinfer-trtllm-moe` | Enabling Flashinfer Trtllm MoE implementation for low latency. | False |
| `--deepep-mode` | Select the mode when enable DeepEP MoE, could be `normal`, `low_latency` or `auto`. Default is `auto`, which means `low_latency` for decode batch and `normal` for prefill batch. | auto | | `--deepep-mode` | Select the mode when enable DeepEP MoE, could be `normal`, `low_latency` or `auto`. Default is `auto`, which means `low_latency` for decode batch and `normal` for prefill batch. | auto |
| `--ep-num-redundant-experts` | Allocate this number of redundant experts in expert parallel. | 0 | | `--ep-num-redundant-experts` | Allocate this number of redundant experts in expert parallel. | 0 |
| `--ep-dispatch-algorithm` | The algorithm to choose ranks for redundant experts in expert parallel. | None | | `--ep-dispatch-algorithm` | The algorithm to choose ranks for redundant experts in expert parallel. | None |

2
python/sglang/srt/layers/moe/ep_moe/layer.py
View File

@@ -1268,7 +1268,7 @@ class FlashInferEPMoE(EPMoE):
topk_group=self.topk_group, topk_group=self.topk_group,
intermediate_size=self.w2_weight.shape[2], intermediate_size=self.w2_weight.shape[2],
local_expert_offset=self.start_expert_id, local_expert_offset=self.start_expert_id,
local_num_experts=self.num_experts_per_partition, local_num_experts=self.num_local_experts,
routed_scaling_factor=self.routed_scaling_factor, routed_scaling_factor=self.routed_scaling_factor,
tile_tokens_dim=_get_tile_tokens_dim( tile_tokens_dim=_get_tile_tokens_dim(
hidden_states.shape[0], self.top_k, self.num_experts hidden_states.shape[0], self.top_k, self.num_experts