EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

Commit Graph

  • 8796cebb2c fix typo SGLang supports three grammar backends (#4679) BroadbentJim 2025-03-22 21:33:48 +00:00
  • c2bd094d6e Optimize Permute Kernel in DeepEP (#4643) xutizhou 2025-03-23 05:30:34 +08:00
  • f8f9244a61 [Bug Fix] Add partial rotary factor support for Phi-4 and upgrade to transformers v4.50.0 (#3984) Adarsh Shirawalmath 2025-03-23 02:57:39 +05:30
  • ecbfe58bb0 Bug fix for metrics counter (#4660) Zhiqiang Xie 2025-03-22 13:39:21 -07:00
  • 8f163b1653 Add EAGLE mtbench benchmark script (#4676) Ke Bao 2025-03-23 04:34:01 +08:00
  • e7a8610d51 fix flaky ut (#4670) Yineng Zhang 2025-03-22 12:36:50 -07:00
  • a2cc62a6db [CI fix] test skipping modelopt on AMD (#4677) Adarsh Shirawalmath 2025-03-23 01:06:02 +05:30
  • fb8886037c [Docs] Update docs for gemma3 and VLM chat templates (#4674) Adarsh Shirawalmath 2025-03-22 20:32:19 +05:30
  • 321ab756bc [1/3] fix dsv3 awq issue (#4556) AniZpZ 2025-03-22 16:07:17 +08:00
  • 38f25e87fc Correcting default configuration when benchmarking fused_moe (#4665) penguin_wwy 2025-03-22 15:52:34 +08:00
  • 8cd4250401 [quantization] fix channelwise conversion with scalar weight scale (#4596) Yun Dai 2025-03-22 00:47:52 -07:00
  • 6a384d5c01 Speed up per token and per tensor quant by 15% (#4639) Chunan Zeng 2025-03-22 00:37:57 -07:00
  • f69e0696bc update news for README (#4664) Yineng Zhang 2025-03-21 23:25:59 -07:00
  • f6ab4ca6bc fix: fix ipython running error for Engine due to outlines nest_asyncio (#4582) mlmz 2025-03-22 10:11:15 +08:00
  • c7c7dbebbe [PD] Release initial code (#4654) Byron Hsu 2025-03-21 14:47:47 -07:00
  • 417fc72f6f Align completion and chat_completion response to OpenAI API (#4637) Yuhong Guo 2025-03-21 13:59:04 +08:00
  • c6ec70290f [docs] Add links and fix grammars in deploy_on_k8s.md (#4641) Michael Yao 2025-03-21 13:55:23 +08:00
  • 4c56e5dbee Set deepgemm to the default value in the hopper architecture. (#4613) lukec 2025-03-21 13:03:00 +08:00
  • 7b5fc71972 fix SUPPORT_CUTLASS_BLOCK_FP8 flag (#4640) Cheng Wan 2025-03-21 00:45:07 -04:00
  • ad4e58bf67 Support fp8 gemm for blackwell (#4558) Shu Wang 2025-03-20 14:40:28 -05:00
  • bfb03c6182 Update doc for MTP and DP attention (#4622) Ke Bao 2025-03-21 02:31:48 +08:00
  • b36ab493b3 Enable setting sglang logger from Env Variable SGLANG_LOGGING_CONFIG_PATH (#4592) Yuhong Guo 2025-03-20 17:10:32 +08:00
  • 9e93ef3f8e [fix] fix illegal mem access and clean up triton attention backend (#4571) JieXin Liang 2025-03-20 17:01:52 +08:00
  • fad86a6863 Support n in OpenAI API completions (#3446) Chuyue Sun 2025-03-19 22:46:46 -07:00
  • df7014a8d2 avoid cudaStreamSynchronize in DeepSeekV2AttentionMLA (#4577) strgrb 2025-03-20 01:02:26 +08:00
  • 4942074174 [fix] fix initialization of _ENABLE_TORCH_INFERENCE_MODE (#4549) JieXin Liang 2025-03-20 00:57:59 +08:00
  • ba52fd1868 Add clang-format to pre-commit config (#4583) Hongbo Xu 2025-03-20 00:50:19 +08:00
  • b6944f97a6 Support FlashMLA backend cuda graph (#4514) lukec 2025-03-19 23:25:34 +08:00
  • f44db16c8e [Feature] Integrate DeepEP into SGLang (#4232) Jinyan Chen 2025-03-19 23:16:31 +08:00
  • f9c53cbb42 Create col-major and tma-aligned x_scale for deep_gemm.gemm_fp8_fp8_bf16_nt (#4515) strgrb 2025-03-19 15:02:43 +08:00
  • 90532b7627 [Fix] Fix raw_bs bug when using flashinfer mla and eagle (#4557) Baizhou Zhang 2025-03-18 21:26:53 -07:00
  • c0e9a36c5f Optimize Triton decoding kernel for dynamic workload (#4553) JieXin Liang 2025-03-19 12:25:38 +08:00
  • 588865f0e0 [Feature] Support Tensor Parallelism and Weight Slicing for Lora (#4274) aoshen524 2025-03-18 23:33:07 -04:00
  • 3196999f63 Reduce computation and communication in DP attention (#4521) Cheng Wan 2025-03-18 16:41:36 -04:00
  • 9e0186f352 [Feature] Support EAGLE 3 (#4247) James Liu 2025-03-18 10:35:23 -04:00
  • 8baf9a0c18 [Fix] Type annotation correction for UpdateWeightsFromTensorReqInput (#4532) Wei Wu 2025-03-18 15:52:47 +08:00
  • c787298547 use sgl custom all reduce (#4441) Yineng Zhang 2025-03-18 00:46:41 -07:00
  • 45212ce18b Add deepseek v2 torch compile pr test (#4538) Ke Bao 2025-03-18 15:29:24 +08:00
  • c16b33ccac cleanup deps 3/n (#4541) Yineng Zhang 2025-03-18 00:11:36 -07:00
  • 2d0045125f Fix the incorrect args in benchmark_and_profiling.md (#4542) Albert 2025-03-18 15:07:06 +08:00
  • 804d250a0d remove useless backend forward in rotary_embedding (#4500) Xiaoyu Zhang 2025-03-18 14:54:00 +08:00
  • dd865befde [Hotfix] solve fp8 w8a8 ci test fail (#4531) Xiaoyu Zhang 2025-03-18 14:17:04 +08:00
  • d373a48c98 fix: second_per_grid_ts should be used to get mrope position (#3682) Mick 2025-03-18 09:12:38 +08:00
  • 98be3bd306 refactor: rewrite bench-mmmu-sglang (#4458) Mick 2025-03-18 09:11:47 +08:00
  • a98290aea3 Unit test for Hierarchical Caching (#4486) Zhiqiang Xie 2025-03-17 17:45:00 -07:00
  • 9b81f9bd34 sglang quant module remove vllm dependency (#4507) Xiaoyu Zhang 2025-03-18 06:51:59 +08:00
  • f81a27f65e upgrade sgl-kernel 0.0.5.post3 (#4522) Yineng Zhang 2025-03-17 14:49:56 -07:00
  • 988ab646ec bump v0.0.5.post3 (#4520) Yineng Zhang 2025-03-17 13:05:59 -07:00
  • 3ded4b215d Revert "feat: update grouped_topk to support softmax and sigmoid" (#4505) Ke Bao 2025-03-18 02:30:26 +08:00
  • f4d7ab7a63 [sgl-router] improvement to avoid hang (#4482) Yinghai Lu 2025-03-17 10:37:50 -07:00
  • c38ca4fc8e Update readme (#4517) Lianmin Zheng 2025-03-17 08:22:42 -07:00
  • 82dec1f70b Remove redundant type conversion (#4513) Lianmin Zheng 2025-03-17 05:57:35 -07:00
  • 5f9b2c62ff [ROCm] fix dtype (#4510) yiakwy-xpu-ml-framework-team 2025-03-17 20:20:50 +08:00
  • 5493c3343e Fix data parallel + tensor parallel (#4499) Lianmin Zheng 2025-03-17 05:13:16 -07:00
  • f2ab37e500 [Doc] add doc for quantization w8a8_fp8 or w8a8_int8 (#4495) HandH1998 2025-03-17 17:25:00 +08:00
  • 91ba98fe50 [Fix] Resolve GPU Memory Leak in update_weights_from_tensor (#4446) Wei Wu 2025-03-17 16:54:30 +08:00
  • c614dbdf95 Nicer standalone engine inferface (#4480) Yinghai Lu 2025-03-17 01:42:04 -07:00
  • 927ca935a7 Constraint Decoding: Tool call with text (#4067) Xihuai Wang 2025-03-17 16:06:46 +08:00
  • ef3c2dd08e Support Online Quantization for W8A8 (#4485) Stefan He 2025-03-17 00:28:56 -07:00
  • 75b656488a Support serving DeepSeek-R1-Channel-INT8 with 32 L40S. (#4418) Wenbo Yang 2025-03-17 15:03:43 +08:00
  • 0f52fb55ec config: Update fused moe config (#4493) Mick 2025-03-17 14:51:58 +08:00
  • d6d21640d3 [Feature] Support Deepseek-VL2 (#2798) 萝卜菜 2025-03-17 14:07:59 +08:00
  • 0212d2e288 [Fix] use torch.inference_mode() instead of torch.no_grad() (#4372) JieXin Liang 2025-03-17 13:54:16 +08:00
  • 8cc300f536 Fix router test (#4483) Byron Hsu 2025-03-16 22:49:47 -07:00
  • 452db50808 Constraint Decoding: Set xgrammar as the default grammar backend (#4386) mlmz 2025-03-17 09:53:43 +08:00
  • d1112d8548 Add endpoint for file support, purely to speed up processing of input_embeds. (#2797) Rin Intachuen 2025-03-17 08:30:37 +07:00
  • 48efec7b05 Feature: support code completion (#3612) woodx 2025-03-17 09:26:19 +08:00
  • 9b8333d992 [ROCm] enable moe topk softmax in amd (#4448) yiakwy-xpu-ml-framework-team 2025-03-17 09:16:55 +08:00
  • f5bbf6037d Fix: Complete int32 to int64 conversion (#4465) Zhiqiang Xie 2025-03-16 18:14:27 -07:00
  • 5cbd709ea1 Fix: modelscope env comment (#4474) huiwq1990 2025-03-17 09:11:33 +08:00
  • 2e4a1e2d05 Initialize image processor for skip-tokenizer-init codepath (#4479) Yinghai Lu 2025-03-16 18:10:09 -07:00
  • 9d02bb3e2a Urgent model support: support gemma-3-it (#4424) Mick 2025-03-17 08:37:32 +08:00
  • 402db5c58c Benchmark: Statistical Analysis of the Output Stability of the Deepseek Model (#4202) ZelinTan 2025-03-17 08:32:57 +08:00
  • 754a0e8278 Update CODEOWNERS (#4484) Lianmin Zheng 2025-03-16 17:10:15 -07:00
  • 799fb5f455 400 on empty input_ids (#4481) Yinghai Lu 2025-03-16 14:01:23 -07:00
  • 25e1816eff fix custom allreduce performance/accuracy problem (#4477) Yi Zhang 2025-03-17 03:16:30 +08:00
  • a53fe428f9 Support FlashMLA backend (#4472) lukec 2025-03-17 00:07:06 +08:00
  • 1b859295f4 [Eagle] Remove the greedy branch and some redundant code (#4363) Ying Sheng 2025-03-16 02:48:55 -07:00
  • 9971dc2283 Revert "feat: Add FlashMLA submodule (#4449)" (#4470) Yineng Zhang 2025-03-16 01:30:05 -07:00
  • 3db35c1af4 Release sgl-kernel v0.0.5.post2 (#4469) Lianmin Zheng 2025-03-16 01:01:53 -07:00
  • 52a34d7448 Add greedy verification kernel (#4383) Ying Sheng 2025-03-16 00:58:26 -07:00
  • 06d12b39d3 Remove filter for pr-tests (#4468) Lianmin Zheng 2025-03-16 00:57:26 -07:00
  • c30976fb41 Fix finish step for pr tests and notebook tests (#4467) Lianmin Zheng 2025-03-16 00:52:06 -07:00
  • 1a3fa75f2f [Fix] use torch.cat instead of torch.concat to prevent entering the Autograd backends. (#4466) JieXin Liang 2025-03-16 15:02:47 +08:00
  • 81f431eded feat: Add FlashMLA submodule (#4449) Shi Shuai 2025-03-16 14:30:25 +08:00
  • 65b7c9b78f cleanup deps 2/n (#4464) Yineng Zhang 2025-03-15 23:06:17 -07:00
  • 2c4f5ccac1 Fix minor style (#4460) Lianmin Zheng 2025-03-15 21:51:12 -07:00
  • 158430473e Fix typos (#4368) Wang Ran (汪然) 2025-03-16 12:27:58 +08:00
  • 8ec2ce0726 perf: update fused moe config (#4459) Mick 2025-03-16 12:23:57 +08:00
  • 1fd0cf8a7b Update comment in qwen2.py (#4447) Michael Feil 2025-03-15 21:14:29 -07:00
  • bf63ee54ed Auto-detect device if not specified in server arguments. (#4423) vikram singh shekhawat 2025-03-16 09:43:51 +05:30
  • 22c96f78a6 typos: Update sampling_params.md (#4391) Wang Ran (汪然) 2025-03-16 07:40:18 +08:00
  • 2892b9bb97 bugfix: Update sampling_params.py (#4413) Wang Ran (汪然) 2025-03-16 07:39:19 +08:00
  • 470b474075 Update bench_serving.py (#4454) Xu Song 2025-03-16 07:33:58 +08:00
  • 26c372c13c docs: Add Llama 3.3 to supported models (#4453) 江家瑋 2025-03-16 07:33:43 +08:00
  • 86d9baedc2 [Fix] Fix errors when using the device except cuda. (#4455) Chen Shengzhi 2025-03-16 07:33:00 +08:00
  • 21d485f835 Fix test_create_kvindices unit test (#4452) lukec 2025-03-16 07:01:04 +08:00
  • 035ac2ab74 ci: update transformers==4.48.3 (#4451) Mick 2025-03-16 04:27:26 +08:00
  • e1a5e7e47d docs: hot fix torch compile cache (#4442) Chayenne 2025-03-14 19:05:59 -07:00
  • ad1ae7f7cd use topk_softmax with sgl-kernel (#4439) Yineng Zhang 2025-03-14 15:59:06 -07:00