EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity
2,298 Commits 8 Branches 0 Tags
0fe7c13be18d1edd8682747ce558b430a1aa1c9e
Commit Graph

1539 Commits

Author SHA1 Message Date
Mingshan
0fe7c13be1 Fix bench_serving flush cache not recognizing OPENAI_API_KEY (#4181) 
Signed-off-by: Mingshan <git@brighill.com>
 2025-03-08 01:03:38 -08:00
Lianmin Zheng
08c4d764a5 lazy import attn backends (#4200) 2025-03-08 00:41:35 -08:00
Lianmin Zheng
d4017a6b63 [EAGLE] many fixes for eagle (#4195) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Sehoon Kim <sehoon@x.ai>
 2025-03-07 22:12:13 -08:00
Lianmin Zheng
d052f4c8a9 New clang format for sgl kernel (#4194) 2025-03-07 20:21:08 -08:00
Ke Bao
20c8119915 Fix eagle hang issue for max_new_tokens=1 (#4185) 2025-03-07 12:11:18 -08:00
Yineng Zhang
eb61f5c9af Revert "ROCm: Flex Attention Enablement with custom backends (#4178)" (#4186) 2025-03-07 10:27:52 -08:00
HAI
0beea4503f ROCm: Flex Attention Enablement with custom backends (#4178) 
Co-authored-by: linsun12 <linsun12@amd.com>
 2025-03-07 04:38:53 -08:00
Zhiqiang Xie
9376ac361d Memory pool fix for upstream change about eagle (#4170) 2025-03-07 00:58:20 -08:00
HandH1998
c7f254468f [Feature] DeepSeek V3/R1 INT8 Quantization (channel-wise) (#3888) 
Co-authored-by: yych0745 <1398089567@qq.com>
Co-authored-by: sleepcoo <sleepcoo@gmail.com>
Co-authored-by: b0urnee <2769086541@qq.com>
 2025-03-06 20:54:52 -08:00
Pan Lyu
361971b859 Add Support for Qwen2-VL Multi-modal Embedding Models (#3694) 2025-03-06 16:46:20 -08:00
HAI
13bc39c5d6 ROCm: enable trillion-parameter MoE models with INT4-FP8 single node (#4152) 2025-03-06 15:33:02 -08:00
Lianmin Zheng
9c58e68b4c Release v0.4.3.post4 (#4140) 2025-03-06 12:50:28 -08:00
Oliver Stanley
d03b3467b8 Fix constrained generation errors by adding datasets dependency (#4142) 2025-03-06 12:07:51 -08:00
yinfan98
ab7fba0ece Fix nightly ci Gsm8k & Fix flashinfer backend kvcache quant (#4147) 2025-03-06 11:50:07 -08:00
Lianmin Zheng
bc1534ff32 Fix a draft model accuracy bug in eagle; support step=1; return logprob in eagle (#4134) 
Co-authored-by: Sehoon Kim <kssteven418@gmail.com>
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Sehoon Kim <sehoon@x.ai>
 2025-03-06 06:13:59 -08:00
Lzhang-hub
3a3918121f fix bench serving bug (#4135) 2025-03-06 05:34:02 -08:00
Lianmin Zheng
98c73d71cb [Minor] make the __init__ function of model_runner.py shorter (#4132) 2025-03-06 01:51:12 -08:00
Lianmin Zheng
fcc2e37f69 Split the __init__ of scheduler as smaller functions. Improve the eagle tests (#4128) 2025-03-06 00:13:20 -08:00
Zhiqiang Xie
aee30630d8 Add a pointer to the real KV cache pool (#4113) 2025-03-05 21:39:07 -08:00
Lianmin Zheng
286e6540a6 Remove prefill-only-one-req (#4117) 2025-03-05 20:58:48 -08:00
Wenxuan Tan
718c391fd7 [Hoxfix] Fix incomplete token_to_kv_pool refactor (#4121) 2025-03-05 19:32:42 -08:00
Yineng Zhang
fc671f66c1 chore: bump v0.4.3.post3 (#4114) 2025-03-05 17:26:10 -08:00
Yueyang Pan
25482edb5c Online serving benchmarks of real datasets for hierarchical KV caching (#3211) 
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-03-05 16:16:43 -08:00
luzengxiangcn
62b362b1f1 Debug radixcache: refactor recursive helper methods (#3029) 
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-03-05 16:11:42 -08:00
Jhin
70b3c6eeb1 Add update_weights_from_disk endpoint to Engine (#4102) 
Co-authored-by: zhaochenyang20 <zhaochen20@outlook.com>
 2025-03-05 12:25:18 -08:00
Ke Bao
ef9d3b3c2c Fix triton kernel illegal memory issue for eagle (#4100) 2025-03-05 11:23:53 -08:00
Baizhou Zhang
fc91d08a8f [Revision] Add fast decode plan for flashinfer mla (#4012) 2025-03-05 11:20:41 -08:00
HAI
71ab0dabe0 Fix the moe padding conditional logic (#4081) 2025-03-05 10:56:51 -08:00
Ying Sheng
d3d4d76758 [Eagle] Refactor eagle speculative decoding (#3986) 
Co-authored-by: Ke Bao <ISPObaoke@163.com>
 2025-03-05 08:06:07 -08:00
yigex
5be8f1ed98 ROCM: AITER BLOCK GEMM (#4075) 2025-03-05 03:10:49 -08:00
Qubitium-ModelCloud
56a724eba3 [QUANT] Add GPTQModel Dynamic Quantization + lm_head Quantization (#3790) 
Signed-off-by: ZX-ModelCloud <zx@modelcloud.ai>
Co-authored-by: ZX-ModelCloud <zx@modelcloud.ai>
 2025-03-05 01:11:00 -08:00
Mick
583d6af71b example: add vlm to token in & out example (#3941) 
Co-authored-by: zhaochenyang20 <zhaochen20@outlook.com>
 2025-03-04 22:18:26 -08:00
Lianmin Zheng
e074d84e5b [Minor] more code cleanup (#4077) 2025-03-04 21:23:47 -08:00
Lianmin Zheng
77a3954bf7 Simplify eagle tests and TP sync in grammar backend (#4066) 2025-03-04 13:40:40 -08:00
Lianmin Zheng
2dd7d0c533 Revert "Fix nightly-test CI" (#4065) 2025-03-04 05:38:24 -08:00
William
0d4e3228cf [Feature] Add test for speculative_token_map (#4016) 2025-03-04 04:26:24 -08:00
Xiuyu Li
9545bfb28a fix: support gelu_new activation function in gpt2 (#3712) 2025-03-04 04:09:52 -08:00
Chen Shengzhi
61261b3996 [XCCL] Use xccl for xpu backend since xccl is ready in latest PyTorch. (#3954) 2025-03-04 04:05:56 -08:00
DarkSharpness
19120f71f3 [Fix & Style] Refactor the grammar backend to reduce human errors and improve readability (#4030) 2025-03-04 03:56:45 -08:00
Qubitium-ModelCloud
87f671ab58 Fix debug_tensor_dump_output_folder optional key missing (#4046) 2025-03-04 03:42:48 -08:00
HAI
51d25405a7 ROCm: update aiter and its usage to fused moe (bloat16, fp8, fp8 block-quant) (#4053) 2025-03-04 03:00:46 -08:00
kk
e0a2c96308 Fix breakage problem when using custom_ar (#4052) 2025-03-04 02:59:03 -08:00
Xihuai Wang
12f2e6c3f1 Fix: #3988 using blockwise_int8 (#4023) 2025-03-03 23:49:58 -08:00
Xihuai Wang
95575aa76a Reasoning parser (#4000) 
Co-authored-by: Lucas Pickup <lupickup@microsoft.com>
 2025-03-03 21:16:36 -08:00
kk
11eea69e70 Fix assert options.num_stages != 0 error in the latest ROCm build image (#4049) 
Co-authored-by: wunhuang <wunhuang@amd.com>
 2025-03-03 20:37:03 -08:00
Ke Bao
9fafa62db7 Share target model embed and head weights for nextn (#4033) 2025-03-03 13:30:04 -08:00
Qiaolin Yu
57a404fd55 Remove outdated test utils and fix links for the doc of sampling params (#3999) 2025-03-03 09:41:38 -08:00
Lianmin Zheng
935cda944b Misc clean up; Remove the support of jump forward (#4032) 2025-03-03 07:02:14 -08:00
Lianmin Zheng
1a8f995c46 remove cache configs in model definitions (#4031) 2025-03-03 05:00:50 -08:00
Lianmin Zheng
a3ab768a2b Clean up custom allreduce (#4029) 2025-03-03 04:59:53 -08:00