Files

DarkSharpness 47367b768d [Refactor] Clean up radix cache related API (#7303 )

Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>

2025-06-20 00:58:48 +08:00

eval

chore: add hf_xet dep (#6243 )

2025-05-12 13:08:40 -07:00

lang

Frontend language separate reasoning support (#6031 )

2025-06-10 17:11:29 -07:00

srt

[Refactor] Clean up radix cache related API (#7303 )

2025-06-20 00:58:48 +08:00

test

Multi-Stage Awake: Support Resume and Pause KV Cache and Weights separately (#7099 )

2025-06-19 00:56:37 -07:00

__init__.py

Frontend language separate reasoning support (#6031 )

2025-06-10 17:11:29 -07:00

api.py

Frontend language separate reasoning support (#6031 )

2025-06-10 17:11:29 -07:00

bench_offline_throughput.py

Fix torch profiler bugs for bench_offline_throughput.py (#6557 )

2025-06-09 20:33:41 +08:00

bench_one_batch_server.py

Improve profiler and integrate profiler in bench_one_batch_server (#6787 )

2025-05-31 15:53:55 -07:00

bench_one_batch.py

Use monotonic clock for interval measurement (#6211 )

2025-05-17 16:49:18 -07:00

bench_serving.py

Support VILA models (#6106 )

2025-06-11 11:47:25 -07:00

check_env.py

Fix check_env script (#5901 )

2025-04-29 18:54:54 -07:00

compile_deep_gemm.py

Use monotonic clock for interval measurement (#6211 )

2025-05-17 16:49:18 -07:00

global_config.py

Misc clean up; Remove the support of jump forward (#4032 )

2025-03-03 07:02:14 -08:00

launch_server.py

Separate two entry points: Engine and HTTP server (#2996 )

2025-01-19 22:09:24 -08:00

math_utils.py

Re-quantize DeepSeek model weights to support DeepGEMM new input format (#7156 )

2025-06-13 15:57:45 -07:00

profiler.py

Improve profiler and integrate profiler in bench_one_batch_server (#6787 )

2025-05-31 15:53:55 -07:00

README.md

feat: support loogle eval (#6190 )

2025-05-10 23:52:44 -07:00

utils.py

Feat/support rerank (#6058 )

2025-06-16 10:50:01 -07:00

version.py

chore: bump v0.4.7.post1 (#7248 )

2025-06-16 15:20:29 -07:00

README.md

Code Structures

eval: The evaluation utilities.
lang: The frontend language.
srt: The backend engine for running local models. (SRT = SGLang Runtime).
test: The test utilities.
api.py: The public APIs.
bench_offline_throughput.py: Benchmark the throughput in the offline mode.
bench_one_batch.py: Benchmark the latency of running a single static batch without a server.
bench_one_batch_server.py: Benchmark the latency of running a single batch with a server.
bench_serving.py: Benchmark online serving with dynamic requests.
check_env.py: Check the environment variables and dependencies.
global_config.py: The global configs and constants.
launch_server.py: The entry point for launching the local server.
utils.py: Common utilities.
version.py: Version info.