Improve docs & Rename Gemini -> VertexAI (#19)
This commit is contained in:
@@ -1,4 +1,5 @@
|
||||
# SGLang
|
||||
| [**Blog**](https://lmsys.org/blog/2024-01-17-sglang/) | [**Paper**](https://arxiv.org/abs/2312.07104) |
|
||||
|
||||
SGLang is a structured generation language designed for large language models (LLMs).
|
||||
It makes your interaction with LLMs faster and more controllable by co-designing the frontend language and the runtime system.
|
||||
@@ -42,7 +43,7 @@ The example below shows how to use sglang to answer a mulit-turn question.
|
||||
### Using OpenAI Models
|
||||
Set the OpenAI API Key
|
||||
```
|
||||
export OPENAI_API_KEY=sk-xxxxxx
|
||||
export OPENAI_API_KEY=sk-******
|
||||
```
|
||||
|
||||
Then, answer a multi-turn question.
|
||||
@@ -100,6 +101,7 @@ for m in state.messages():
|
||||
|
||||
### More Examples
|
||||
|
||||
Anthropic and VertexAI (Gemini) models are also supported.
|
||||
You can find more examples at [examples/quick_start](examples/quick_start).
|
||||
|
||||
## Frontend: Structured Generation Langauge (SGLang)
|
||||
@@ -251,6 +253,7 @@ python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port
|
||||
- Mixtral
|
||||
- LLaVA
|
||||
- `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.5-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --port 30000`
|
||||
- AWQ quantization
|
||||
|
||||
## Benchmark And Performance
|
||||
|
||||
|
||||
Reference in New Issue
Block a user