Add EAGLE mtbench benchmark script (#4676)

Co-authored-by: chromecast56 <jamesll@mit.edu>
2025-03-23 04:34:01 +08:00
parent e7a8610d51
commit 8f163b1653
2 changed files with 148 additions and 0 deletions
--- a/benchmark/mtbench/README.md
+++ b/benchmark/mtbench/README.md
@@ -15,6 +15,17 @@ python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port
 python3 bench_sglang.py --num-questions 80
 ```

+### Benchmark sglang EAGLE
+```
+python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3-8B-Instruct --speculative-algo EAGLE \
+    --speculative-draft lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \
+    --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --dtype float16 --port 30000
+```
+
+```
+python3 bench_sglang_eagle.py --num-questions 80 --parallel 1
+```
+

 ### Benchmark vllm
 ```