EngineX/xc-llm-ascend
3
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

[2/N][Feat] Attention and MoE weight prefetch in Qwen3MoE models (#3203)

Browse Source 
### What this PR does / why we need it?

- Refacotr and integrate a unified `WeightPrefetchMethod`
- Integrate `gate_up_proj.weight` in quantized Attention modules
- Prefetching these weights ahead of matmul-like operators imporves
performance by reducing L2 cache transfer latency

### Does this PR introduce _any_ user-facing change?

Add a new config in `--additional-config` for configuration:
```json
{
    "weight_prefetch_config": {
        "enabled": True,
        "prefetch_ratio": {
            "moe": {
                "gate_up": 0.8
            },
        },
    },
}
```
This feature is enabled by default, and can be disabled through this
configuration

### How was this patch tested?


- vLLM version: v0.11.0rc3
- vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.0

---------

Signed-off-by: yuzhup <15705211260@163.com>
This commit is contained in:
yuzhup
2025-10-14 20:16:33 +08:00
committed by GitHub
parent 07e39620ea
commit 78777237a9
9 changed files with 160 additions and 100 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
docs/source/user_guide/configuration/additional_config.md
View File

@@ -73,10 +73,10 @@ ascend_scheduler_config also support the options from [vllm scheduler config](ht
**weight_prefetch_config**
| Name | Type | Default | Description |
|------------------|------|------------------------------------|------------------------------------|
| `enabled` | bool | `False` | Whether to enable weight prefetch. |
| `prefetch_ratio` | dict | `{"attn": {"qkv": 1.0, "o": 1.0}}` | Prefetch ratio of each weights. |
| Name | Type | Default | Description |
|------------------|------|-------------------------------------------------------------|------------------------------------|
| `enabled` | bool | `False` | Whether to enable weight prefetch. |
| `prefetch_ratio` | dict | `{"attn": {"qkv": 1.0, "o": 1.0}, "moe": {"gate_up": 0.8}}` | Prefetch ratio of each weights. |
### Example
@@ -104,6 +104,9 @@ An example of additional configuration is as follows:
"qkv": 1.0,
"o": 1.0,
},
"moe": {
"gate_up": 0.8
}
},
},
"multistream_overlap_shared_expert": True,