EngineX-Hygon/sglang

Files

History

Pratyush Patel 8f15789314 Add more metrics to serving benchmark. (#2819 )

2025-01-10 23:30:44 +08:00

..

Add generator-style run_batch function (#2513 )

2025-01-06 15:04:55 -08:00

Fix typo in cuda_graph_bs (#2813 )

2025-01-09 03:03:24 -08:00

Fix nightly accuracy tests (#2780 )

2025-01-07 21:02:35 -08:00

__init__.py

minor: limit the range of vllm versions (#2350 )

2024-12-05 02:00:34 +08:00

api.py

Merged three native APIs into one: get_server_info (#2152 )

2024-11-24 01:37:58 -08:00

bench_latency.py

Rename sglang.bench_latency to sglang.bench_one_batch (#2118 )

2024-11-21 20:07:48 -08:00

bench_offline_throughput.py

Clean up wrapper in flashinfer backend (#2638 )

2024-12-29 00:45:57 -08:00

bench_one_batch_server.py

Crash the server correctly during error (#2231 )

2024-11-28 00:22:39 -08:00

bench_one_batch.py

Eagle speculative decoding part 3: small modifications to the general scheduler (#2709 )

2025-01-02 02:09:08 -08:00

bench_serving.py

Add more metrics to serving benchmark. (#2819 )

2025-01-10 23:30:44 +08:00

check_env.py

ROCm support for sglang.check_env (#2426 )

2024-12-17 03:45:14 -08:00

global_config.py

Improve the user control of new_token_ratio (#1811 )

2024-10-26 16:39:41 -07:00

launch_server_llavavid.py

Update vllm to 0.6.3 (#1711 ) (#1720 )

2024-10-19 20:45:41 -07:00

launch_server.py

fix missing launch server import (#2242 )

2024-11-28 21:24:47 +08:00

llama3_eval.py

docs: update README (#2516 )

2024-12-19 13:44:02 +08:00

README.md

feat: use CUDA 12.4 by default (for FA3) (#2682 )

2024-12-31 15:52:09 +08:00

utils.py

[Core] in batch prefix caching by delay scheduling (#2442 )

2024-12-11 12:51:50 -08:00

version.py

chore: bump v0.4.1.post4 (#2713 )

2025-01-06 01:29:54 +08:00

README.md

Code Structures

lang: The frontend language.
srt: The backend engine for running local models. (SRT = SGLang Runtime).
test: The test utilities.
api.py: The public APIs.
bench_offline_throughput.py: Benchmark the throughput in the offline mode.
bench_one_batch.py: Benchmark the latency of running a single static batch without a server.
bench_one_batch_server.py: Benchmark the latency of running a single batch with a server.
bench_serving.py: Benchmark online serving with dynamic requests.
check_env.py: Check the environment variables.
global_config.py: The global configs and constants.
launch_server.py: The entry point for launching the local server.
llama3_eval.py: Evaluation of Llama 3.1 using the Meta Llama dataset.
utils.py: Common utilities.