fix(serving_chat): shallow copy bug — [[]] * n and [parser] * n share references
When n>=2, all_previous_token_ids entries pointed to the SAME list, so appending tokens for choice 0 corrupted choice 1's history. Same for tool_parsers: all choices shared one stateful parser instance. Changed to list comprehensions that create independent objects. Found via CCCL result_policy.cuh read: distributed result delivery requires isolated per-rank state — same principle applies to per-choice token tracking in vLLM streaming.
This commit is contained in:
@@ -348,7 +348,7 @@ class OpenAIServingChat(OpenAIServing):
|
||||
# parsing and reasoning parsing (both require full-history context).
|
||||
if tool_choice_auto or use_reasoning:
|
||||
previous_texts = [""] * num_choices
|
||||
all_previous_token_ids = [[]] * num_choices
|
||||
all_previous_token_ids = [[] for _ in range(num_choices)]
|
||||
else:
|
||||
previous_texts, all_previous_token_ids = None, None
|
||||
|
||||
@@ -357,7 +357,8 @@ class OpenAIServingChat(OpenAIServing):
|
||||
if tool_choice_auto and self.tool_parser:
|
||||
tool_parsers: List[Optional[ToolParser]] = [
|
||||
self.tool_parser(tokenizer)
|
||||
] * num_choices
|
||||
for _ in range(num_choices)
|
||||
]
|
||||
else:
|
||||
tool_parsers = [None] * num_choices
|
||||
except RuntimeError as e:
|
||||
|
||||
Reference in New Issue
Block a user