Files

eigen 6ad6c8c9e6 feat: openai oss attention sink support with trtllm-gen backend #8825 (#8834 )

Co-authored-by: averyhuang <averyh@nvidia.com>

2025-08-06 19:18:27 -07:00

eval

[script] update loogle test (#7975 )

2025-07-12 00:06:17 -07:00

lang

model: support intern-s1 (#8350 )

2025-07-26 13:48:51 -07:00

srt

feat: openai oss attention sink support with trtllm-gen backend #8825 (#8834 )

2025-08-06 19:18:27 -07:00

test

Support limiting max loaded loras in CPU. (#8650 )

2025-08-03 00:02:23 -07:00

__init__.py

Frontend language separate reasoning support (#6031 )

2025-06-10 17:11:29 -07:00

api.py

Frontend language separate reasoning support (#6031 )

2025-06-10 17:11:29 -07:00

bench_offline_throughput.py

Feature/modelscope model download (#8083 )

2025-07-31 17:29:31 -07:00

bench_one_batch_server.py

[CI][Router] Fix bench_one_batch_server for pd router test (#7731 )

2025-07-02 23:18:24 -07:00

bench_one_batch.py

[Feature] Hybrid EP and TP (#8590 )

2025-07-31 02:53:25 -07:00

bench_serving.py

[Multimodal][Perf] Use pybase64 instead of base64 (#7724 )

2025-07-08 14:00:58 -07:00

check_env.py

Revert "feat: replace Decord with video_reader-rs" (#8077 )

2025-07-15 20:04:56 -07:00

compile_deep_gemm.py

[bugfix] Add 'disaggregation_mode' parameter to warmup function when compile deep_gemm manually (#8618 )

2025-08-01 16:02:53 -07:00

global_config.py

Reduce memory accumulation in long-running server (#8306 )

2025-08-03 15:03:16 +08:00

launch_server.py

Separate two entry points: Engine and HTTP server (#2996 )

2025-01-19 22:09:24 -08:00

profiler.py

Improve profiler and integrate profiler in bench_one_batch_server (#6787 )

2025-05-31 15:53:55 -07:00

README.md

feat: support loogle eval (#6190 )

2025-05-10 23:52:44 -07:00

utils.py

Split the scheduler into multiple mixin classes to reduce the file size (#8483 )

2025-07-29 12:46:50 -07:00

version.py

chore: bump v0.4.10.post2 (#8727 )

2025-08-03 03:43:29 -07:00

README.md

Code Structures

eval: The evaluation utilities.
lang: The frontend language.
srt: The backend engine for running local models. (SRT = SGLang Runtime).
test: The test utilities.
api.py: The public APIs.
bench_offline_throughput.py: Benchmark the throughput in the offline mode.
bench_one_batch.py: Benchmark the latency of running a single static batch without a server.
bench_one_batch_server.py: Benchmark the latency of running a single batch with a server.
bench_serving.py: Benchmark online serving with dynamic requests.
check_env.py: Check the environment variables and dependencies.
global_config.py: The global configs and constants.
launch_server.py: The entry point for launching the local server.
utils.py: Common utilities.
version.py: Version info.