Improve docs & Add JSON decode example (#121)

This commit is contained in:
Lianmin Zheng
2024-01-30 05:45:27 -08:00
committed by GitHub
parent 0617528632
commit 97aa9b3284
19 changed files with 212 additions and 61 deletions

View File

@@ -4,13 +4,21 @@ This doc describes the sampling parameters of the SGLang Runtime.
The `/generate` endpoint accepts the following arguments in the JSON format.
```python
@dataclass
class GenerateReqInput:
# The input prompt
text: Union[List[str], str]
# The image input
image_data: Optional[Union[List[str], str]] = None
# The sampling_params
sampling_params: Union[List[Dict], Dict] = None
# The request id
rid: Optional[Union[List[str], str]] = None
# Whether return logprobs of the prompts
return_logprob: Optional[Union[List[bool], bool]] = None
# The start location of the prompt for return_logprob
logprob_start_len: Optional[Union[List[int], int]] = None
# Whether to stream output
stream: bool = False
```
@@ -84,3 +92,7 @@ for chunk in response.iter_lines(decode_unicode=False, delimiter=b"\0"):
prev = len(output)
print("")
```
### Multi modal
See [test_httpserver_llava.py](../test/srt/test_httpserver_llava.py).