[Misc] Remove CP Redundant Variables after FIA operator enables for CANN 8.5 (#6013)
### What this PR does / why we need it?
PCP/DCP splits the kv-cache onto different cards. After introducing the
parameter cp-kv-cache-interleave-size, the first size tokens will be
cached at Card 0, and so on.
However, if there are too few tokens, some cards will not store the
key-value pairs, resulting in values of 0, corrupted values, and
precision issues. Currently, additional operations are introduced to
avoid this precision problem.
After we integrate FIA operator in mla_cp._forward_decode and CANN
updates to 8.5.0, we now can remove these additional operations.
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
passed all CI by CANN 8.5.0
- vLLM version: v0.13.0
- vLLM main:
2c24bc6996
Signed-off-by: dsxsteven <dsxsteven@sina.com>
Signed-off-by: dsxsteven <36877507+dsxsteven@users.noreply.github.com>
This commit is contained in:
@@ -68,10 +68,6 @@ class AscendMlaCPMetadataBuilder(AscendMLAMetadataBuilder):
|
||||
self.dcp_rank = get_decode_context_model_parallel_rank() if self.dcp_size > 1 else 0
|
||||
self.cp_local_block_size = vllm_config.parallel_config.cp_kv_cache_interleave_size
|
||||
self.cp_virtual_block_size = self.cp_local_block_size * self.dcp_size * self.pcp_size
|
||||
scheduler_config = vllm_config.scheduler_config
|
||||
decode_max_num_seqs = getattr(scheduler_config, "decode_max_num_seqs", 0)
|
||||
max_num_seqs = max(scheduler_config.max_num_seqs, decode_max_num_seqs)
|
||||
self.batch_seq_mask_buf = torch.empty(max_num_seqs * self.decode_threshold, dtype=torch.uint8, device=device)
|
||||
self.block_size = (self.block_size * self.cp_virtual_block_size) // np.gcd(
|
||||
self.block_size, self.cp_virtual_block_size
|
||||
)
|
||||
@@ -238,12 +234,7 @@ class AscendMlaCPMetadataBuilder(AscendMLAMetadataBuilder):
|
||||
|
||||
cp_seq_len = num_computed_tokens_of_cp_dcp_array[:, self.pcp_rank, self.dcp_rank]
|
||||
cp_seq_len = torch.tensor(cp_seq_len, dtype=torch.int32)
|
||||
batch_seq_mask = cp_seq_len == 0
|
||||
self.batch_seq_mask_buf[: batch_seq_mask.shape[0]].copy_(batch_seq_mask, non_blocking=True)
|
||||
batch_seq_mask = self.batch_seq_mask_buf[: batch_seq_mask.shape[0]]
|
||||
cp_seq_len = torch.where(cp_seq_len == 0, 1, cp_seq_len)
|
||||
decode_metadata.cp_seq_len = cp_seq_len.tolist()
|
||||
decode_metadata.batch_seq_mask = batch_seq_mask
|
||||
|
||||
actual_seq_lengths_q = torch.arange(self.num_decodes_flatten) + 1
|
||||
decode_metadata.actual_seq_lengths_q = actual_seq_lengths_q
|
||||
@@ -651,7 +642,7 @@ class AscendMlaCPImpl(AscendMLAImpl):
|
||||
softmax_lse = softmax_lse.permute(0, 2, 1, 3).reshape(B_lse * Q_S, N_lse, 1)
|
||||
|
||||
# Update out&lse
|
||||
attn_out_lse = _process_attn_out_lse(attn_output, softmax_lse, decode_meta.batch_seq_mask)
|
||||
attn_out_lse = _process_attn_out_lse(attn_output, softmax_lse)
|
||||
attn_output = _npu_attention_update(self.kv_lora_rank, attn_out_lse)
|
||||
return self._v_up_proj(attn_output)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user