Misc clean up; Remove the support of jump forward (#4032)
This commit is contained in:
Lianmin Zheng
@@ -17,3 +17,59 @@ For CUDA 12.1 or CUDA 12.4:
|
||||
```bash
|
||||
pip3 install sgl-kernel
|
||||
```
|
||||
|
||||
# Developer Guide
|
||||
|
||||
## Development Environment Setup
|
||||
|
||||
Use Docker to set up the development environment. See [Docker setup guide](https://github.com/sgl-project/sglang/blob/main/docs/developer/development_guide_using_docker.md#setup-docker-container).
|
||||
|
||||
Create and enter development container:
|
||||
```bash
|
||||
docker run -itd --shm-size 32g --gpus all -v $HOME/.cache:/root/.cache --ipc=host --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
|
||||
docker exec -it sglang_zhyncs /bin/zsh
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
### Dependencies
|
||||
|
||||
Third-party libraries:
|
||||
|
||||
- [CCCL](https://github.com/NVIDIA/cccl)
|
||||
- [CUTLASS](https://github.com/NVIDIA/cutlass)
|
||||
- [FlashInfer](https://github.com/flashinfer-ai/flashinfer)
|
||||
- [TurboMind](https://github.com/InternLM/turbomind)
|
||||
|
||||
### Kernel Development
|
||||
|
||||
Steps to add a new kernel:
|
||||
|
||||
1. Implement in [src/sgl-kernel/csrc/](https://github.com/sgl-project/sglang/tree/main/sgl-kernel/src/sgl-kernel/csrc)
|
||||
2. Expose interface in [src/sgl-kernel/include/sgl_kernels_ops.h](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/src/sgl-kernel/include/sgl_kernels_ops.h)
|
||||
3. Create torch extension in [src/sgl-kernel/torch_extension.cc](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/src/sgl-kernel/torch_extension.cc)
|
||||
4. Create Python wrapper in [src/sgl-kernel/ops/\_\_init\_\_.py](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/src/sgl-kernel/ops/__init__.py)
|
||||
5. Expose Python interface in [src/sgl-kernel/\_\_init\_\_.py](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/src/sgl-kernel/__init__.py)
|
||||
6. Update [setup.py](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/setup.py) to include new CUDA source
|
||||
|
||||
### Build & Install
|
||||
|
||||
Development build:
|
||||
|
||||
```bash
|
||||
make build
|
||||
```
|
||||
|
||||
Note:
|
||||
|
||||
The `sgl-kernel` is rapidly evolving. If you experience a compilation failure, try using `make rebuild`.
|
||||
|
||||
### Testing & Benchmarking
|
||||
|
||||
1. Add pytest tests in [tests/](https://github.com/sgl-project/sglang/tree/main/sgl-kernel/tests)
|
||||
2. Add benchmarks using [triton benchmark](https://triton-lang.org/main/python-api/generated/triton.testing.Benchmark.html) in [benchmark/](https://github.com/sgl-project/sglang/tree/main/sgl-kernel/benchmark)
|
||||
3. Run test suite
|
||||
|
||||
### Release new version
|
||||
|
||||
Update version in [pyproject.toml](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/pyproject.toml) and [version.py](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/src/sgl-kernel/version.py)
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
# Developer Guide for sgl-kernel
|
||||
|
||||
## Development Environment Setup
|
||||
|
||||
Use Docker to set up the development environment. See [Docker setup guide](https://github.com/sgl-project/sglang/blob/main/docs/developer/development_guide_using_docker.md#setup-docker-container).
|
||||
|
||||
Create and enter development container:
|
||||
```bash
|
||||
docker run -itd --shm-size 32g --gpus all -v $HOME/.cache:/root/.cache --ipc=host --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
|
||||
docker exec -it sglang_zhyncs /bin/zsh
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
### Dependencies
|
||||
|
||||
Third-party libraries:
|
||||
|
||||
- [CCCL](https://github.com/NVIDIA/cccl)
|
||||
- [CUTLASS](https://github.com/NVIDIA/cutlass)
|
||||
- [FlashInfer](https://github.com/flashinfer-ai/flashinfer)
|
||||
- [TurboMind](https://github.com/InternLM/turbomind)
|
||||
|
||||
### Kernel Development
|
||||
|
||||
Steps to add a new kernel:
|
||||
|
||||
1. Implement in [src/sgl-kernel/csrc/](https://github.com/sgl-project/sglang/tree/main/sgl-kernel/src/sgl-kernel/csrc)
|
||||
2. Expose interface in [src/sgl-kernel/include/sgl_kernels_ops.h](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/src/sgl-kernel/include/sgl_kernels_ops.h)
|
||||
3. Create torch extension in [src/sgl-kernel/torch_extension.cc](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/src/sgl-kernel/torch_extension.cc)
|
||||
4. Create Python wrapper in [src/sgl-kernel/ops/\_\_init\_\_.py](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/src/sgl-kernel/ops/__init__.py)
|
||||
5. Expose Python interface in [src/sgl-kernel/\_\_init\_\_.py](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/src/sgl-kernel/__init__.py)
|
||||
6. Update [setup.py](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/setup.py) to include new CUDA source
|
||||
|
||||
### Build & Install
|
||||
|
||||
Development build:
|
||||
|
||||
```bash
|
||||
make build
|
||||
```
|
||||
|
||||
Note:
|
||||
|
||||
The `sgl-kernel` is rapidly evolving. If you experience a compilation failure, try using `make rebuild`.
|
||||
|
||||
### Testing & Benchmarking
|
||||
|
||||
1. Add pytest tests in [tests/](https://github.com/sgl-project/sglang/tree/main/sgl-kernel/tests)
|
||||
2. Add benchmarks using [triton benchmark](https://triton-lang.org/main/python-api/generated/triton.testing.Benchmark.html) in [benchmark/](https://github.com/sgl-project/sglang/tree/main/sgl-kernel/benchmark)
|
||||
3. Run test suite
|
||||
|
||||
### Release new version
|
||||
|
||||
Update version in [pyproject.toml](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/pyproject.toml) and [version.py](https://github.com/sgl-project/sglang/blob/main/sgl-kernel/src/sgl-kernel/version.py)
|
||||
@@ -100,6 +100,7 @@ sources = [
|
||||
"src/sgl-kernel/csrc/activation/fused_add_rms_norm_kernel.cu",
|
||||
"src/sgl-kernel/csrc/allreduce/trt_reduce_internal.cu",
|
||||
"src/sgl-kernel/csrc/allreduce/trt_reduce_kernel.cu",
|
||||
"src/sgl-kernel/csrc/attention/lightning_attention_decode_kernel.cu",
|
||||
"src/sgl-kernel/csrc/gemm/cublas_grouped_gemm.cu",
|
||||
"src/sgl-kernel/csrc/gemm/fp8_gemm_kernel.cu",
|
||||
"src/sgl-kernel/csrc/gemm/fp8_blockwise_gemm_kernel.cu",
|
||||
@@ -108,7 +109,6 @@ sources = [
|
||||
"src/sgl-kernel/csrc/moe/moe_align_kernel.cu",
|
||||
"src/sgl-kernel/csrc/speculative/eagle_utils.cu",
|
||||
"src/sgl-kernel/csrc/speculative/speculative_sampling.cu",
|
||||
"src/sgl-kernel/csrc/lightning_attention_decode_kernel.cu",
|
||||
"3rdparty/flashinfer/csrc/activation.cu",
|
||||
"3rdparty/flashinfer/csrc/bmm_fp8.cu",
|
||||
"3rdparty/flashinfer/csrc/norm.cu",
|
||||
|
||||
@@ -62,6 +62,11 @@ TORCH_LIBRARY_EXPAND(sgl_kernels, m) {
|
||||
m.def("register_graph_buffers(int fa, int[][] handles, int[][] offsets) -> ()");
|
||||
m.impl("register_graph_buffers", torch::kCUDA, ®ister_graph_buffers);
|
||||
|
||||
/*
|
||||
* From csrc/attention
|
||||
*/
|
||||
m.impl("lightning_attention_decode", torch::kCUDA, &lightning_attention_decode);
|
||||
|
||||
/*
|
||||
* From csrc/gemm
|
||||
*/
|
||||
@@ -163,11 +168,6 @@ TORCH_LIBRARY_EXPAND(sgl_kernels, m) {
|
||||
"apply_rope_pos_ids_cos_sin_cache(Tensor q, Tensor k, Tensor! q_rope, Tensor! k_rope, Tensor cos_sin_cache, "
|
||||
"Tensor pos_ids, bool interleave, int cuda_stream) -> ()");
|
||||
m.impl("apply_rope_pos_ids_cos_sin_cache", torch::kCUDA, &apply_rope_pos_ids_cos_sin_cache);
|
||||
|
||||
/*
|
||||
* Other
|
||||
*/
|
||||
m.impl("lightning_attention_decode", torch::kCUDA, &lightning_attention_decode);
|
||||
}
|
||||
|
||||
REGISTER_EXTENSION(_kernels)
|
||||
|
||||
Reference in New Issue
Block a user