Files

Lzhang-hub a146d9990e support prometheus metrics (#1853 )

Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Byron Hsu <byronhsu1230@gmail.com>

2024-11-05 20:42:53 -08:00

lang

Unify the model type checking (#1905 )

2024-11-03 12:25:39 -08:00

srt

support prometheus metrics (#1853 )

2024-11-05 20:42:53 -08:00

test

Let reward model take text inputs instead of message lists (#1907 )

2024-11-03 13:27:12 -08:00

__init__.py

Provide an offline engine API (#1567 )

2024-10-06 20:27:03 -07:00

api.py

Update links (#1805 )

2024-10-26 04:46:01 -07:00

bench_latency.py

Unify the model type checking (#1905 )

2024-11-03 12:25:39 -08:00

bench_server_latency.py

Add a watch dog thread (#1816 )

2024-10-27 02:00:50 -07:00

bench_serving.py

feat: support truss endpoint for benchmark serving (#1906 )

2024-11-03 12:56:10 -08:00

check_env.py

misc: add hypervisor vendor (#1165 )

2024-08-21 02:14:51 +10:00

global_config.py

Improve the user control of new_token_ratio (#1811 )

2024-10-26 16:39:41 -07:00

launch_server_llavavid.py

Update vllm to 0.6.3 (#1711 ) (#1720 )

2024-10-19 20:45:41 -07:00

launch_server.py

Add a watch dog thread (#1816 )

2024-10-27 02:00:50 -07:00

README.md

Update readme and workflow (#1716 )

2024-10-19 13:01:44 -07:00

utils.py

Escape backwards slash (#1902 )

2024-11-03 12:27:11 -08:00

version.py

Release v0.3.5 (#1908 )

2024-11-03 13:48:11 -08:00

README.md

Code Structures

lang: The frontend language.
srt: The backend engine for running local models. (SRT = SGLang Runtime).
test: The test utilities.
api.py: The public APIs.
bench_latency.py: Benchmark the latency of running a single static batch.
bench_server_latency.py: Benchmark the latency of serving a single batch with a real server.
bench_serving.py: Benchmark online serving with dynamic requests.
global_config.py: The global configs and constants.
launch_server.py: The entry point for launching the local server.
utils.py: Common utilities.