Files

Lianmin Zheng 8e66fbecee Improve DP attention (#4390 )

Co-authored-by: dhou-xai <dhou@x.ai>
Co-authored-by: SangBin Cho <rkooo567@gmail.com>

2025-03-13 08:23:56 -07:00

lang

model: Support Janus-pro (#3203 )

2025-03-12 11:02:11 -07:00

srt

Improve DP attention (#4390 )

2025-03-13 08:23:56 -07:00

test

Revert "[XPU][CPU] Enable the native path of DeepSeek" (#4367 )

2025-03-12 23:45:52 -07:00

__init__.py

Improve metrics, logging, and importing orders (#2992 )

2025-01-19 18:36:59 -08:00

api.py

Remove outdated test utils and fix links for the doc of sampling params (#3999 )

2025-03-03 09:41:38 -08:00

bench_offline_throughput.py

Support penalty in overlap mode; return logprob with chunked prefill; improve benchmark scripts (#3988 )

2025-03-03 00:12:04 -08:00

bench_one_batch_server.py

Separate two entry points: Engine and HTTP server (#2996 )

2025-01-19 22:09:24 -08:00

bench_one_batch.py

[Eagle] Refactor eagle speculative decoding (#3986 )

2025-03-05 08:06:07 -08:00

bench_serving.py

Fix bench_serving flush cache not recognizing OPENAI_API_KEY (#4181 )

2025-03-08 01:03:38 -08:00

check_env.py

support version in sgl-kernel (#3439 )

2025-02-10 03:49:52 +08:00

global_config.py

Misc clean up; Remove the support of jump forward (#4032 )

2025-03-03 07:02:14 -08:00

launch_server.py

Separate two entry points: Engine and HTTP server (#2996 )

2025-01-19 22:09:24 -08:00

llama3_eval.py

docs: update README (#2516 )

2024-12-19 13:44:02 +08:00

README.md

Misc clean up; Remove the support of jump forward (#4032 )

2025-03-03 07:02:14 -08:00

utils.py

Memory pool fix for upstream change about eagle (#4170 )

2025-03-07 00:58:20 -08:00

version.py

chore: bump v0.4.4 (#4041 )

2025-03-13 02:49:58 -07:00

README.md

Code Structures

lang: The frontend language.
srt: The backend engine for running local models. (SRT = SGLang Runtime).
test: The test utilities.
api.py: The public APIs.
bench_offline_throughput.py: Benchmark the throughput in the offline mode.
bench_one_batch.py: Benchmark the latency of running a single static batch without a server.
bench_one_batch_server.py: Benchmark the latency of running a single batch with a server.
bench_serving.py: Benchmark online serving with dynamic requests.
check_env.py: Check the environment variables and dependencies.
global_config.py: The global configs and constants.
launch_server.py: The entry point for launching the local server.
llama3_eval.py: Evaluation of Llama 3 using the Meta Llama dataset.
utils.py: Common utilities.
version.py: Version info.