Eagle3 mm support, enablement on qwen3vl (#4848)
### What this PR does / why we need it?
follow pr
[https://github.com/vllm-project/vllm/pull/20788](https://github.com/vllm-project/vllm/pull/20788)
, Eagle3 mm support, enablement on qwen3vl
target model
[Qwen/Qwen3-VL-8B-Instruct]([https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct])
eagle3
[MNN/Qwen3-VL-8B-Instruct-Eagle3](https://www.modelscope.cn/models/MNN/Qwen3-VL-8B-Instruct-Eagle3)
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
pytest ./tests/e2e/singlecard/test_completion_with_prompt_embeds.py -vv
vLLM with eagle3 :
```bash
vllm serve /model/Qwen3-VL-8B-Instruct --enforce-eager --port 9100 --max-model-len 32768 --max-num-seqs 32 --tensor-parallel-size 2 --allowed-local-media-path /model/gx/images --speculative-config '{
"method": "eagle3",
"model": "/model/hf/Qwen3-VL-8B-Instruct-Eagle3",
"num_speculative_tokens": 3
}'
```
vLLM without eagle3 :
```bash
vllm serve /model/Qwen3-VL-8B-Instruct --enforce-eager --port 9100 --max-model-len 32768 --max-num-seqs 32 --tensor-parallel-size 2 --allowed-local-media-path /model/gx/images
```
bench:
```
vllm bench serve --backend openai-chat --base-url http://127.0.0.1:9100 --tokenizer /model/Qwen3-VL-8B-Instruct --endpoint /v1/chat/completions --model /model/Qwen3-VL-8B-Instruct --dataset-name random --num-prompts 50 --max-concurrency 5 --temperature 0 --top-p 1.0 --seed 123
```
- vLLM version: v0.12.0
- vLLM main:
ad32e3e19c
---------
Signed-off-by: jesse <szxfml@gmail.com>
This commit is contained in:
@@ -50,6 +50,7 @@ class TestEagleProposerInitialization(TestBase):
|
||||
self.vllm_config.speculative_config.draft_model_config.get_hidden_size.return_value = 4096
|
||||
self.vllm_config.compilation_config.mode = CompilationMode.VLLM_COMPILE
|
||||
self.vllm_config.model_config.enforce_eager = False
|
||||
self.vllm_config.model_config.uses_mrope = False
|
||||
self.vllm_config.speculative_config.enforce_eager = False
|
||||
self.vllm_config.scheduler_config.async_scheduling = False
|
||||
init_ascend_config(self.vllm_config)
|
||||
@@ -156,6 +157,7 @@ class TestEagleProposerLoadModel(TestBase):
|
||||
}]
|
||||
|
||||
mock_model = MagicMock()
|
||||
mock_model.supports_multimodal = False
|
||||
mock_model.model.embed_tokens = MagicMock()
|
||||
mock_model.lm_head = MagicMock()
|
||||
mock_model.multimodal_cpu_fields = None
|
||||
@@ -226,7 +228,7 @@ class TestEagleProposerLoadModel(TestBase):
|
||||
self.proposer.name = SpecDcodeType.EAGLE
|
||||
|
||||
self.proposer.load_model(mock_model)
|
||||
mock_model.get_language_model.assert_called_once()
|
||||
self.assertEqual(mock_model.get_language_model.call_count, 2)
|
||||
self.assertIs(self.proposer.model.lm_head,
|
||||
mock_model.get_language_model.return_value.lm_head)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user