Improve docs & Rename Gemini -> VertexAI (#19)

This commit is contained in:
Lianmin Zheng
2024-01-17 02:54:41 -08:00
committed by GitHub
parent fd7c479239
commit bf51ddc6e5
13 changed files with 56 additions and 583 deletions

View File

@@ -1,4 +1,5 @@
# SGLang
| [**Blog**](https://lmsys.org/blog/2024-01-17-sglang/) | [**Paper**](https://arxiv.org/abs/2312.07104) |
SGLang is a structured generation language designed for large language models (LLMs).
It makes your interaction with LLMs faster and more controllable by co-designing the frontend language and the runtime system.
@@ -42,7 +43,7 @@ The example below shows how to use sglang to answer a mulit-turn question.
### Using OpenAI Models
Set the OpenAI API Key
```
export OPENAI_API_KEY=sk-xxxxxx
export OPENAI_API_KEY=sk-******
```
Then, answer a multi-turn question.
@@ -100,6 +101,7 @@ for m in state.messages():
### More Examples
Anthropic and VertexAI (Gemini) models are also supported.
You can find more examples at [examples/quick_start](examples/quick_start).
## Frontend: Structured Generation Langauge (SGLang)
@@ -251,6 +253,7 @@ python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port
- Mixtral
- LLaVA
- `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.5-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --port 30000`
- AWQ quantization
## Benchmark And Performance