14c71b19e1
### What this PR does / why we need it?
This PR adds comprehensive documentation for the CPU binding feature on
Ascend NPUs. It includes:
- A detailed developer guide
(`docs/source/developer_guide/feature_guide/cpu_binding.md`) covering
the design, internal logic, allocation examples, and troubleshooting for
the CPU binding mechanism.
- A concise user guide
(`docs/source/user_guide/feature_guide/cpu_binding.md`) explaining the
core concepts, usage, and common issues for end-users.
- An update to `additional_config.md` to use consistent terminology for
binding strategies (`global-slicing` and `topo-affinity`).
This documentation is needed to help both developers and users
understand, use, and debug the CPU binding feature, which is critical
for performance on ARM+Ascend platforms.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only update.
### How was this patch tested?
The documentation has been reviewed for clarity and technical accuracy.
The examples and descriptions align with the implementation in
`vllm_ascend/cpu_binding.py`.
- vLLM version: v0.16.0
- vLLM main:
4034c3d32e
---------
Signed-off-by: chenchuw886 <chenchuw@huawei.com>
Signed-off-by: c00818886 <chenchuwei@huawei.com>
Co-authored-by: chenchuw886 <chenchuw@huawei.com>
129 lines
7.9 KiB
Markdown
129 lines
7.9 KiB
Markdown
# Additional Configuration
|
|
|
|
Additional configuration is a mechanism provided by vLLM to allow plugins to control internal behavior by themselves. VLLM Ascend uses this mechanism to make the project more flexible.
|
|
|
|
## How to use
|
|
|
|
With either online mode or offline mode, users can use additional configuration. Take Qwen3 as an example:
|
|
|
|
**Online mode**:
|
|
|
|
```bash
|
|
vllm serve Qwen/Qwen3-8B --additional-config='{"config_key":"config_value"}'
|
|
```
|
|
|
|
**Offline mode**:
|
|
|
|
```python
|
|
from vllm import LLM
|
|
|
|
LLM(model="Qwen/Qwen3-8B", additional_config={"config_key":"config_value"})
|
|
```
|
|
|
|
### Configuration options
|
|
|
|
The following table lists additional configuration options available in vLLM Ascend:
|
|
|
|
| Name | Type | Default | Description |
|
|
|-------------------------------------|------|---------|-----------------------------------------------------------------------------------------------------------|
|
|
| `xlite_graph_config` | dict | `{}` | Configuration options for Xlite graph mode |
|
|
| `weight_prefetch_config` | dict | `{}` | Configuration options for weight prefetch |
|
|
| `finegrained_tp_config` | dict | `{}` | Configuration options for module tensor parallelism |
|
|
| `ascend_compilation_config` | dict | `{}` | Configuration options for ascend compilation |
|
|
| `eplb_config` | dict | `{}` | Configuration options for ascend compilation |
|
|
| `refresh` | bool | `false` | Whether to refresh global Ascend configuration content. This is usually used by rlhf or ut/e2e test case. |
|
|
| `dump_config_path` | str | `None` | Configuration file path for msprobe dump(eager mode). |
|
|
| `enable_async_exponential` | bool | `False` | Whether to enable asynchronous exponential overlap. To enable asynchronous exponential, set this config to True. |
|
|
| `enable_shared_expert_dp` | bool | `False` | When the expert is shared in DP, it delivers better performance but consumes more memory. Currently only DeepSeek series models are supported. |
|
|
| `multistream_overlap_shared_expert` | bool | `False` | Whether to enable multi-stream shared expert. This option only takes effect on MoE models with shared experts. |
|
|
| `multistream_overlap_gate` | bool | `False` | Whether to enable multi-stream overlap gate. This option only takes effect on MoE models with shared experts. |
|
|
| `recompute_scheduler_enable` | bool | `False` | Whether to enable recompute scheduler. |
|
|
| `enable_cpu_binding` | bool | `True` | Whether to enable CPU binding. Only takes effect on ARM CPUs; A3 uses the global-slicing CPU allocation strategy and other device types use the topo-affinity CPU allocation strategy. |
|
|
| `SLO_limits_for_dynamic_batch` | int | `-1` | SLO limits for dynamic batch. This is new scheduler to support dynamic batch feature |
|
|
| `enable_npugraph_ex` | bool | `False` | Whether to enable npugraph_ex graph mode. |
|
|
| `pa_shape_list` | list | `[]` | The custom shape list of page attention ops. |
|
|
| `enable_kv_nz` | bool | `False` | Whether to enable KV cache NZ layout. This option only takes effects on models using MLA (e.g., DeepSeek). |
|
|
| `layer_sharding` | dict | `{}` | Configuration options for Layer Sharding Linear |
|
|
| `sp_threshold` | int | `1000` | For dense models, only num_tokens > threshold will enable sequence parallelism. |
|
|
|
|
The details of each configuration option are as follows:
|
|
|
|
**xlite_graph_config**
|
|
|
|
| Name | Type | Default | Description |
|
|
| ---- | ---- | ------- | ----------- |
|
|
| `enabled` | bool | `False` | Whether to enable Xlite graph mode. Currently only Llama, Qwen dense series models, and Qwen3-VL are supported. |
|
|
| `full_mode` | bool | `False` | Whether to enable Xlite for both the prefill and decode stages. By default, Xlite is only enabled for the decode stage. |
|
|
|
|
**weight_prefetch_config**
|
|
|
|
| Name | Type | Default | Description |
|
|
|------------------|------|-------------------------------------------------------------|------------------------------------|
|
|
| `enabled` | bool | `False` | Whether to enable weight prefetch. |
|
|
| `prefetch_ratio` | dict | `{"attn": {"qkv": 1.0, "o": 1.0}, "moe": {"gate_up": 0.8}, "mlp": { "gate_up": 1.0, "down": 1.0}}` | Prefetch ratio of each weight. |
|
|
|
|
**finegrained_tp_config**
|
|
|
|
| Name | Type | Default | Description |
|
|
| ---- | ---- | ------- | ----------- |
|
|
| `lmhead_tensor_parallel_size` | int | `0` | The custom tensor parallel size of lm_head. |
|
|
| `oproj_tensor_parallel_size` | int | `0` | The custom tensor parallel size of o_proj. |
|
|
| `embedding_tensor_parallel_size` | int | `0` | The custom tensor parallel size of embedding. |
|
|
| `mlp_tensor_parallel_size` | int | `0` | The custom tensor parallel size of mlp. |
|
|
|
|
**ascend_compilation_config**
|
|
|
|
| Name | Type | Default | Description |
|
|
| ---- | ---- | ------- | ----------- |
|
|
| `enable_npugraph_ex` | bool | `True` | Whether to enable npugraph_ex backend. |
|
|
| `enable_static_kernel` | bool | `False` | Whether to enable static kernel. Suitable for scenarios where shape changes are minimal and some time is available for static kernel compilation. |
|
|
| `fuse_norm_quant` | bool | `True` | Whether to enable fuse_norm_quant pass. |
|
|
| `fuse_qknorm_rope` | bool | `True` | Whether to enable fuse_qknorm_rope pass. If Triton is not in the environment, set it to False. |
|
|
| `fuse_allreduce_rms` | bool | `False` | Whether to enable fuse_allreduce_rms pass. It's set to False because of conflict with SP. |
|
|
| `fuse_muls_add` | bool | `True` | Whether to enable fuse_muls_add pass.|
|
|
|
|
**eplb_config**
|
|
|
|
| Name | Type | Default | Description |
|
|
| ---- | ---- | ------- | ----------- |
|
|
| `dynamic_eplb` | bool| `False`| Whether to enable dynamic EPLB. |
|
|
| `expert_map_path` | str | `None` | When using expert load balancing for an MoE model, an expert map path needs to be passed in.|
|
|
| `expert_heat_collection_interval`| int | `400` | Forward iterations when EPLB begins. |
|
|
| `algorithm_execution_interval` | int | `30` | The forward iterations when the EPLB worker will finish CPU tasks. |
|
|
| `expert_map_record_path` | str | `None` | Save the expert load calculation results to a new expert table in the specified directory.|
|
|
| `num_redundant_experts` | int | `0` | Specify redundant experts during initialization. |
|
|
|
|
### Example
|
|
|
|
An example of additional configuration is as follows:
|
|
|
|
```python
|
|
{
|
|
"weight_prefetch_config": {
|
|
"enabled": True,
|
|
"prefetch_ratio": {
|
|
"attn": {
|
|
"qkv": 1.0,
|
|
"o": 1.0,
|
|
},
|
|
"moe": {
|
|
"gate_up": 0.8
|
|
},
|
|
"mlp": {
|
|
"gate_up": 1.0,
|
|
"down": 1.0
|
|
}
|
|
},
|
|
},
|
|
"finegrained_tp_config": {
|
|
"lmhead_tensor_parallel_size": 8,
|
|
"oproj_tensor_parallel_size": 8,
|
|
"embedding_tensor_parallel_size": 8,
|
|
"mlp_tensor_parallel_size": 8,
|
|
},
|
|
"enable_kv_nz": False,
|
|
"multistream_overlap_shared_expert": True,
|
|
"refresh": False
|
|
}
|
|
```
|