[Feat]Add support for optional start len of logprobs (#1035)

Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
This commit is contained in:
yichuan~
2024-08-18 23:45:41 -07:00
committed by GitHub
parent d8627ed16d
commit b997a18d74
8 changed files with 113 additions and 31 deletions

View File

@@ -209,6 +209,7 @@ class SRTRunner:
prompt,
sampling_params=sampling_params,
return_logprob=True,
logprob_start_len=0,
top_logprobs_num=NUM_TOP_LOGPROBS,
)
response = json.loads(response)