4065248214
Co-authored-by: laixinn <xielx@shanghaitech.edu.cn> Co-authored-by: sleepcoo <sleepcoo@gmail.com> Co-authored-by: zhyncs <me@zhyncs.com>
quantization compressed_tensors module
To support compressed_tensors format quantization models, we adapted https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization/compressed_tensors into SGLang.
For practical purposes, we have only applied the compressed_tensors format of w8a8_fp8. If you have requirements for other formats, you can submit an issue through this link.