EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Lianmin Zheng
2024-01-23 03:43:19 -08:00
committed by GitHub
parent 9076386d90
commit 9e037c822c
1 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
README.md
View File

@@ -218,6 +218,10 @@ for out in state.text_iter():
print(out, end="", flush=True) print(out, end="", flush=True)
``` ```
### Tips and Implementation Details
- The `choices` argument in `sgl.gen` is implemented by computing the normalized log probabilities of all choices and selecting the one with the highest probability.
- The `regex` argument in `sgl.gen` is implemented through autoregressive decoding with logit bias masking, according to the constraints set by the regex.
## Backend: SGLang Runtime (SRT) ## Backend: SGLang Runtime (SRT)
The SGLang Runtime (SRT) is designed to work best with the SGLang frontend. The SGLang Runtime (SRT) is designed to work best with the SGLang frontend.
However, it can also be used as a standalone API server. However, it can also be used as a standalone API server.