[v0.11.0][BugFix][P/D] Modify the recalculation logic to prevent waiting requests from filling up the D node KVCache (#3686)
### What this PR does / why we need it? Modify the recalculation logic to prevent waiting requests from filling up the D node KVCache Signed-off-by: underfituu <hzhucong@163.com>
This commit is contained in:
@@ -361,7 +361,7 @@ class RecomputeScheduler(SchedulerInterface):
|
||||
skipped_waiting_requests = create_request_queue(self.policy)
|
||||
|
||||
# Next, schedule the WAITING requests.
|
||||
if not preempted_reqs:
|
||||
if not preempted_reqs and not recomputed_reqs:
|
||||
while self.waiting and token_budget > 0:
|
||||
if len(self.running) == self.max_num_running_reqs:
|
||||
break
|
||||
|
||||
Reference in New Issue
Block a user