f1f6370ed9
### What this PR does / why we need it?
This PR integrate suffix decoding (https://arxiv.org/abs/2411.04975)
from vllm (https://github.com/vllm-project/vllm/pull/25784)
#
Suffix Decoding is a dynamic n-gram matching method that:
1. Uses suffix trees to generate speculative tokens quickly using branch
frequency counts.
2. Can keep a history of prior model responses, which tends to work very
well with repetitive agentic use cases.
3. Can be dynamically updated with newly generated tokens, and FIFO
eviction of older requests.
#
### Does this PR introduce _any_ user-facing change?
This feature should be implemented as opt-in and remain seamless for
users who do not require suffix speculative decoding.
For users who wish to enable it, they must first install
arctic-inference:
`pip install arctic-inference
`
After installation, the suffix speculative decoding feature can be
enabled using the following speculative config:
`--speculative_config '{"method": "suffix", "num_speculative_tokens":
5}'
`
### How was this patch tested?
This PR is currently being tested on vLLM
main:83f478bb19
with PR https://github.com/vllm-project/vllm/pull/25784
In our previous testing, suffix decoding achieved a 13%-30% throughput
improvement over n-gram on the sonnet dataset, tested on vllm-ascend
v0.9.1 with concurrency ranging from 2 to 40.
- vLLM version: v0.11.2
---------
Signed-off-by: fluctlux <38945811+fluctlux@users.noreply.github.com>
44 lines
1.9 KiB
Python
44 lines
1.9 KiB
Python
#
|
|
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
|
|
# Copyright 2023 The vLLM team.
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
# This file is a part of the vllm-ascend project.
|
|
# Adapted from vllm-project/vllm/vllm/worker/gpu_model_runner.py
|
|
#
|
|
from vllm_ascend.spec_decode.eagle_proposer import EagleProposer
|
|
from vllm_ascend.spec_decode.mtp_proposer import MtpProposer
|
|
from vllm_ascend.spec_decode.ngram_proposer import NgramProposer
|
|
from vllm_ascend.spec_decode.suffix_proposer import SuffixDecodingProposer
|
|
from vllm_ascend.torchair.torchair_mtp_proposer import TorchairMtpProposer
|
|
|
|
|
|
def get_spec_decode_method(method,
|
|
vllm_config,
|
|
device,
|
|
runner,
|
|
is_torchair_graph=False):
|
|
if method == "ngram":
|
|
return NgramProposer(vllm_config, device, runner)
|
|
elif method in ("eagle", "eagle3"):
|
|
return EagleProposer(vllm_config, device, runner)
|
|
elif method in ('deepseek_mtp', 'qwen3_next_mtp'):
|
|
if is_torchair_graph:
|
|
return TorchairMtpProposer(vllm_config, device, runner)
|
|
return MtpProposer(vllm_config, device, runner)
|
|
elif method == 'suffix':
|
|
return SuffixDecodingProposer(vllm_config, device, runner)
|
|
else:
|
|
raise ValueError("Unknown speculative decoding method: "
|
|
f"{method}")
|