Files

yizhang2077 d04899d7ca stop_str of qwen2-vl template should be a tuple not a str (#1834 )

Co-authored-by: Byron Hsu <byronhsu1230@gmail.com>

2024-10-29 20:30:41 +00:00

lang

stop_str of qwen2-vl template should be a tuple not a str (#1834 )

2024-10-29 20:30:41 +00:00

srt

fix get_memory_pool_size deadlock for DP (#1830 )

2024-10-28 23:07:14 -07:00

test

Add a watch dog thread (#1816 )

2024-10-27 02:00:50 -07:00

__init__.py

Provide an offline engine API (#1567 )

2024-10-06 20:27:03 -07:00

api.py

Update links (#1805 )

2024-10-26 04:46:01 -07:00

bench_latency.py

Add a watch dog thread (#1816 )

2024-10-27 02:00:50 -07:00

bench_server_latency.py

Add a watch dog thread (#1816 )

2024-10-27 02:00:50 -07:00

bench_serving.py

Improve benchmark scripts (#1672 )

2024-10-14 21:53:01 -07:00

check_env.py

misc: add hypervisor vendor (#1165 )

2024-08-21 02:14:51 +10:00

global_config.py

Improve the user control of new_token_ratio (#1811 )

2024-10-26 16:39:41 -07:00

launch_server_llavavid.py

Update vllm to 0.6.3 (#1711 ) (#1720 )

2024-10-19 20:45:41 -07:00

launch_server.py

Add a watch dog thread (#1816 )

2024-10-27 02:00:50 -07:00

README.md

Update readme and workflow (#1716 )

2024-10-19 13:01:44 -07:00

utils.py

Simplify our docs with complicated functions into utils (#1807 )

2024-10-26 17:44:11 +00:00

version.py

Release v0.3.4.post2 (#1796 )

2024-10-25 11:07:19 -07:00

README.md

Code Structures

lang: The frontend language.
srt: The backend engine for running local models. (SRT = SGLang Runtime).
test: The test utilities.
api.py: The public APIs.
bench_latency.py: Benchmark the latency of running a single static batch.
bench_server_latency.py: Benchmark the latency of serving a single batch with a real server.
bench_serving.py: Benchmark online serving with dynamic requests.
global_config.py: The global configs and constants.
launch_server.py: The entry point for launching the local server.
utils.py: Common utilities.