Wenxuan Tan 0305c5053f Reduce memory accumulation in long-running server (#8306) ...

Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>

2025-08-03 15:03:16 +08:00

..

cache_controller.py

bugfix(hicache): Fix 'MooncakeStore' not defined error. (#8668)

2025-08-01 15:58:17 -07:00

configure_logging.py

Improve streaming, log_level, memory report, weight loading, and benchmark script (#7632)

2025-06-29 23:16:19 -07:00

data_parallel_controller.py

[Feature] Hybrid EP and TP (#8590)

2025-07-31 02:53:25 -07:00

detokenizer_manager.py

Support incremental streaming of logprob/token_ids between scheduler and detokenizer (#6225)

2025-05-12 14:33:38 -07:00

io_struct.py

Support limiting max loaded loras in CPU. (#8650)

2025-08-03 00:02:23 -07:00

mm_utils.py

fix: minor fix TransportProxyTensor under tp (#8382)

2025-07-27 00:38:49 -07:00

multimodal_processor.py

vlm: optimize tensor transport (#6003)

2025-07-26 17:41:01 +08:00

schedule_batch.py

Add support for NCCL symmetric memory for TP allreduces (#8238)

2025-08-01 23:30:55 +00:00

schedule_policy.py

Fix: resolve prefill of retracted request out-of-memory issue when ignore_eos is enabled (#7434)

2025-08-02 14:43:45 +08:00

scheduler_input_blocker.py

Support colocating requests (#7973)

2025-07-28 22:51:49 -07:00

scheduler_metrics_mixin.py

Split the scheduler into multiple mixin classes to reduce the file size (#8483)

2025-07-29 12:46:50 -07:00

scheduler_output_processor_mixin.py

feat: add tp_rank, pp_rank and dp_rank labels for scheduler metrics (#7597)

2025-07-16 17:55:59 -07:00

scheduler_profiler_mixin.py

Split the scheduler into multiple mixin classes to reduce the file size (#8483)

2025-07-29 12:46:50 -07:00

scheduler_update_weights_mixin.py

Split the scheduler into multiple mixin classes to reduce the file size (#8483)

2025-07-29 12:46:50 -07:00

scheduler.py

Reduce memory accumulation in long-running server (#8306)

2025-08-03 15:03:16 +08:00

session_controller.py

Move mem_fraction_static adjustment for multimodal models to server_args.py & Fix session control & Other cleanups (#7748)

2025-07-04 16:33:33 -07:00

template_manager.py

Fix chat template handling for OpenAI serving (#8635)

2025-07-31 21:49:45 -07:00

tokenizer_manager.py

Support limiting max loaded loras in CPU. (#8650)

2025-08-03 00:02:23 -07:00

tp_worker_overlap_thread.py

[Feature] Hybrid EP and TP (#8590)

2025-07-31 02:53:25 -07:00

tp_worker.py

[Feature] Hybrid EP and TP (#8590)

2025-07-31 02:53:25 -07:00

utils.py

Correctly abort the failed grammar requests & Improve the handling of abort (#6803)

2025-06-01 19:00:07 -07:00