EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

Add a new api configure_logging to allow dumping the requests (#2875)

Browse Source
This commit is contained in:
Lianmin Zheng
2025-01-13 14:24:00 -08:00
committed by GitHub
parent 923f518337
commit 46d4431889
13 changed files with 164 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
benchmark/blog_v0_2/405b_sglang.sh
View File

@@ -6,7 +6,7 @@
# wget https://huggingface.co/neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w8a8/resolve/main/tokenizer_config.json
# Launch sglang
# python -m sglang.launch_server --model-path ~/llama-3.1-405b-fp8-dummy/ --load-format dummy --tp 8 --quant fp8 --disable-radix --mem-frac 0.87
# python -m sglang.launch_server --model-path ~/llama-3.1-405b-fp8-dummy/ --load-format dummy --tp 8 --quantization fp8 --disable-radix --mem-frac 0.87
# offline
python3 -m sglang.bench_serving --backend sglang --dataset-name random --num-prompt 3000 --random-input 1024 --random-output 1024 > sglang_log11