fix(critical): wrap get_scheduler_config in try-catch — prevent n=2 engine crash cascade
Sub508 log: t2_n_2 → HTTP 500 → engine crash → ALL subsequent 24+ tests HTTP 500. The scheduler config call may throw if engine is in a bad state. Wrapping in try-catch ensures we return 400 (not 500) and the engine stays alive.
This commit is contained in:
@@ -182,8 +182,14 @@ class OpenAIServingChat(OpenAIServing):
|
||||
# n > max_num_seqs deadlock guard: scheduler uses break (not continue)
|
||||
# when can_schedule(num_new_seqs=n) fails, so an n that exceeds
|
||||
# max_num_seqs permanently blocks the entire waiting queue with no error.
|
||||
_sched_cfg = await self.engine_client.get_scheduler_config()
|
||||
_max_seqs = _sched_cfg.max_num_seqs
|
||||
# CRITICAL: Also guard against n=2+ with our competition config (max_num_seqs=1)
|
||||
# to prevent engine crash (sub508: t2_n_2 → HTTP 500 → ALL subsequent 500).
|
||||
try:
|
||||
_sched_cfg = await self.engine_client.get_scheduler_config()
|
||||
_max_seqs = _sched_cfg.max_num_seqs
|
||||
except Exception:
|
||||
# If we can't get scheduler config, use a safe default
|
||||
_max_seqs = 1
|
||||
if request.n is not None and request.n > _max_seqs:
|
||||
return self.create_error_response(
|
||||
f"n={request.n} exceeds max_num_seqs={_max_seqs}. "
|
||||
|
||||
Reference in New Issue
Block a user