【feat】switch for fusion ops gmmswigluquant (#5992)
### What this PR does / why we need it?
Set a additional config parameter to control whether the gmmswigluequant
fuseion operator is enabled; it is enabled by True. / When enabled with
a small number of GPUs, the gmmswigluquant fused operator can cause some
performance degradation.
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
- vLLM version: v0.13.0
- vLLM main:
2c24bc6996
#### Perf
test model: GLM 4.6(w8a8)
- single A3 node(ep16, tp16), async-scheduling, mtp, FULL_DECODE_ONLY
- bs=1, input_lens=32000, ouput_lens=1024
Without this PR: TPOT 32.22.ms
With this PR: TPOT 30.23ms
---------
Signed-off-by: zjks98 <zhangjiakang4@huawei.com>
Co-authored-by: zjks98 <zhangjiakang4@huawei.com>
This commit is contained in:
@@ -37,6 +37,9 @@ class AscendConfig:
|
||||
ascend_compilation_config = additional_config.get("ascend_compilation_config", {})
|
||||
self.ascend_compilation_config = AscendCompilationConfig(**ascend_compilation_config)
|
||||
|
||||
ascend_fusion_config = additional_config.get("ascend_fusion_config", {})
|
||||
self.ascend_fusion_config = AscendFusionConfig(**ascend_fusion_config)
|
||||
|
||||
finegrained_tp_config = additional_config.get("finegrained_tp_config", {})
|
||||
self.finegrained_tp_config = FinegrainedTPConfig(finegrained_tp_config, vllm_config)
|
||||
|
||||
@@ -190,6 +193,24 @@ class AscendCompilationConfig:
|
||||
self.fuse_allreduce_rms = fuse_allreduce_rms
|
||||
|
||||
|
||||
class AscendFusionConfig:
|
||||
"""
|
||||
Configuration for controlling whether to use a fused operator gmmswigluquant.
|
||||
"""
|
||||
|
||||
def __init__(self, fusion_ops_gmmswigluquant: bool = True, **kwargs):
|
||||
"""
|
||||
Initialize the configuration.
|
||||
|
||||
Args:
|
||||
fusion_ops_gmmswigluquant (bool): Whether to use a fused operator gmmswigluquant.
|
||||
When set to True, the system will use a fused operator gmmswigluquant.
|
||||
Default: True
|
||||
**kwargs: Additional optional parameters for forward compatibility and configuration extension.
|
||||
"""
|
||||
self.fusion_ops_gmmswigluquant = fusion_ops_gmmswigluquant
|
||||
|
||||
|
||||
class XliteGraphConfig:
|
||||
"""
|
||||
Configuration Object for xlite_graph_config from additional_config
|
||||
|
||||
Reference in New Issue
Block a user