[Docs] Update runtime/engine/readme.md (#5737)
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
This commit is contained in:
@@ -1,12 +1,12 @@
|
|||||||
# SGLang Engine
|
# SGLang Engine
|
||||||
|
|
||||||
## Introduction
|
SGLang provides a direct inference engine without the need for an HTTP server. There are generally these use cases:
|
||||||
SGLang provides a direct inference engine without the need for an HTTP server. There are generally two use cases:
|
|
||||||
|
|
||||||
1. **Offline Batch Inference**
|
- [Offline Batch Inference](#offline-batch-inference)
|
||||||
2. **Embedding Generation**
|
- [Embedding Generation](#embedding-generation)
|
||||||
3. **Custom Server on Top of the Engine**
|
- [Custom Server](#custom-server)
|
||||||
4. **Inference Using FastAPI**
|
- [Token-In-Token-Out for RLHF](#token-in-token-out-for-rlhf)
|
||||||
|
- [Inference Using FastAPI](#inference-using-fastapi)
|
||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|
||||||
@@ -22,28 +22,28 @@ In this example, we launch an SGLang engine and feed a batch of inputs for embed
|
|||||||
|
|
||||||
This example demonstrates how to create a custom server on top of the SGLang Engine. We use [Sanic](https://sanic.dev/en/) as an example. The server supports both non-streaming and streaming endpoints.
|
This example demonstrates how to create a custom server on top of the SGLang Engine. We use [Sanic](https://sanic.dev/en/) as an example. The server supports both non-streaming and streaming endpoints.
|
||||||
|
|
||||||
#### Steps:
|
#### Steps
|
||||||
|
|
||||||
1. Install Sanic:
|
1. Install Sanic:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install sanic
|
pip install sanic
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Run the server:
|
2. Run the server:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python custom_server
|
python custom_server
|
||||||
```
|
```
|
||||||
|
|
||||||
3. Send requests:
|
3. Send requests:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -X POST http://localhost:8000/generate -H "Content-Type: application/json" -d '{"prompt": "The Transformer architecture is..."}'
|
curl -X POST http://localhost:8000/generate -H "Content-Type: application/json" -d '{"prompt": "The Transformer architecture is..."}'
|
||||||
curl -X POST http://localhost:8000/generate_stream -H "Content-Type: application/json" -d '{"prompt": "The Transformer architecture is..."}' --no-buffer
|
curl -X POST http://localhost:8000/generate_stream -H "Content-Type: application/json" -d '{"prompt": "The Transformer architecture is..."}' --no-buffer
|
||||||
```
|
```
|
||||||
|
|
||||||
This will send both non-streaming and streaming requests to the server.
|
This will send both non-streaming and streaming requests to the server.
|
||||||
|
|
||||||
### [Token-In-Token-Out for RLHF](../token_in_token_out)
|
### [Token-In-Token-Out for RLHF](../token_in_token_out)
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user