EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

[Feat]Add support for optional start len of logprobs (#1035)

Browse Source 
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
This commit is contained in:
yichuan~
2024-08-18 23:45:41 -07:00
committed by GitHub
parent d8627ed16d
commit b997a18d74
8 changed files with 113 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
python/sglang/srt/server.py
View File

@@ -559,12 +559,14 @@ class Runtime:
prompt: str,
sampling_params: Optional[Dict] = None,
return_logprob: Optional[Union[List[bool], bool]] = False,
logprob_start_len: Optional[Union[List[int], int]] = None,
top_logprobs_num: Optional[Union[List[int], int]] = None,
):
json_data = {
"text": prompt,
"sampling_params": sampling_params,
"return_logprob": return_logprob,
"logprob_start_len": logprob_start_len,
"top_logprobs_num": top_logprobs_num,
}
response = requests.post(