EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

[Submodule] Change FlashInfer to import (#156)

Browse Source
This commit is contained in:
Cody Yu
2024-02-06 19:28:29 -08:00
committed by GitHub
parent cb8e1982f8
commit 26c3494152
5 changed files with 17 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/flashinfer.md
View File

@@ -5,13 +5,15 @@ It can be used in SGLang runtime to accelerate attention computation.
### Install flashinfer
Note: The compilation can take a very long time.
You can install flashinfer via pip as follows for CUDA 12.1.
```bash
git submodule update --init --recursive
pip install 3rdparty/flashinfer/python
pip install flashinfer -i https://flashinfer.ai/whl/cu121/
```
You can look for other CUDA versions in https://github.com/flashinfer-ai/flashinfer?tab=readme-ov-file#installation. If there is no desire version for your environment,
please build it from source (the compilation takes a long time).
### Run a Server With Flashinfer Mode
Add `--model-mode flashinfer` argument to enable flashinfer when launching a server.