adding the triton docker build minimal example (#242)

This commit is contained in:
Arsalan
2024-03-12 03:16:06 -04:00
committed by GitHub
parent b2eb080501
commit eb4308c4c9
4 changed files with 119 additions and 0 deletions

View File

@@ -0,0 +1,41 @@
# sglang_triton
Build the docker image:
```
docker build -t sglang-triton .
```
Then do:
```
docker run -ti --gpus=all --network=host --name sglang-triton -v ./models:/mnt/models sglang-triton
```
inside the docker container:
```
cd sglang
python3 -m sglang.launch_server --model-path mistralai/Mistral-7B-Instruct-v0.2 --port 30000 --mem-fraction-static 0.9
```
with another shell, inside the docker container:
```
docker exec -ti sglang-triton /bin/bash
cd /mnt
tritonserver --model-repository=/mnt/models
```
Send request to the server:
```
curl -X POST http://localhost:8000/v2/models/character_generation/generate \
-H "Content-Type: application/json" \
-d '{
"inputs": [
{
"name": "INPUT_TEXT",
"datatype": "STRING",
"shape": [1],
"data": ["Name1"]
}
]
}'
```