[main][refactor] Refactoring forward_context and model_runner_v1 (#1979)
### What this PR does / why we need it?
A refactoring of forward_context and model_runner_v1, add some context
which is necessary in model inference into forward_context, and refactor
dummy_run logic, make it more reasonable.
Some details for this PR:
Add `ascend_forward_context`;
Update mc2_v2 op, and support `active_mask` param;
Update scripts in examples dir;
refactor `dummy_run` logic;
Add soc_version for A2 and A3;
### Does this PR introduce _any_ user-facing change?
No change at user-facing.
### How was this patch tested?
- vLLM version: v0.10.0
- vLLM main:
57c22e57f9
Signed-off-by: zzzzwwjj <1183291235@qq.com>
This commit is contained in:
@@ -377,20 +377,14 @@ class CustomDeepseekV2MoE(nn.Module):
|
||||
attn_metadata: Optional[AttentionMetadata] = None,
|
||||
replace_allreduce: bool = False) -> torch.Tensor:
|
||||
|
||||
if attn_metadata is None:
|
||||
attn_metadata = get_forward_context().attn_metadata
|
||||
forward_context = get_forward_context()
|
||||
# when profile runs, force experts to load balanced tokens
|
||||
# to avoid high memory consumption on a single rank.
|
||||
# TODO: need a better flag to indicate whether in profile run or not.
|
||||
if attn_metadata is None:
|
||||
# for profile run
|
||||
is_prefill = True
|
||||
enable_force_load_balance = True
|
||||
else:
|
||||
is_prefill = attn_metadata.num_prefills > 0
|
||||
enable_force_load_balance = False
|
||||
if hasattr(attn_metadata, 'with_prefill_across_dp'):
|
||||
is_prefill = is_prefill or attn_metadata.with_prefill_across_dp
|
||||
|
||||
enable_force_load_balance = forward_context.in_profile_run
|
||||
|
||||
is_prefill = forward_context.with_prefill
|
||||
|
||||
# If this node is kv_consumer, we force the moe always runs in decode path to make sure
|
||||
# the behaviour aligned between dummy_run and normal model_execute.
|
||||
if self.kv_consumer:
|
||||
@@ -572,9 +566,10 @@ class CustomDeepseekV2MLAAttention(DeepseekV2MLAAttention):
|
||||
hidden_states: torch.Tensor,
|
||||
kv_cache: Optional[torch.Tensor] = None,
|
||||
attn_metadata: Optional[AttentionMetadata] = None) -> torch.Tensor:
|
||||
forward_context = get_forward_context()
|
||||
enable_multistream_mla = (self.enable_multistream_mla
|
||||
and attn_metadata is not None
|
||||
and not attn_metadata.with_prefill_across_dp
|
||||
and not forward_context.with_prefill
|
||||
and attn_metadata.num_decodes > 0)
|
||||
forward_kwargs = {"enable_multistream_mla": enable_multistream_mla}
|
||||
if self.q_lora_rank is not None:
|
||||
|
||||
Reference in New Issue
Block a user