EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

update docs for Yi-VL

Browse Source
This commit is contained in:
Lianmin Zheng
2024-02-01 22:44:05 +00:00
parent 864425300f
commit 03e04b2331
2 changed files with 6 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
README.md
View File

@@ -357,9 +357,11 @@ python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port
- Llama
- Mistral
- Mixtral
- Qwen / Qwen 2
- LLaVA
- `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.5-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --chat-template vicuna_v1.1 --port 30000`
- Qwen / Qwen 2
- Yi-VL
- see [srt_example_yi_vl.py](examples/quick_start/srt_example_yi_vl.py).
- AWQ quantization
## Benchmark And Performance

9
examples/quick_start/srt_example_yi_vl.py
View File

@@ -46,12 +46,9 @@ def batch():
if __name__ == "__main__":
runtime = sgl.Runtime(model_path="BabyChou/Yi-VL-6B",
tokenizer_path="BabyChou/Yi-VL-6B")
runtime = sgl.Runtime(model_path="BabyChou/Yi-VL-6B")
# runtime = sgl.Runtime(model_path="BabyChou/Yi-VL-34B")
sgl.set_default_backend(runtime)
# Or you can use API models
# sgl.set_default_backend(sgl.OpenAI("gpt-4-vision-preview"))
# sgl.set_default_backend(sgl.VertexAI("gemini-pro-vision"))
# Run a single request
print("\n========== single ==========\n")
@@ -65,4 +62,4 @@ if __name__ == "__main__":
print("\n========== batch ==========\n")
batch()
runtime.shutdown()
runtime.shutdown()