Lianmin Zheng
|
bea2bb9eea
|
Improve multi-node stability (#1171)
|
2024-08-20 22:35:05 -07:00 |
|
Shan Yu
|
cd10654e7e
|
[Feat] Support update weights without restart server (#1157)
|
2024-08-20 13:48:24 -07:00 |
|
Yineng Zhang
|
350a81609b
|
fix: resolve README render (#1166)
|
2024-08-21 03:23:52 +10:00 |
|
Lucien
|
6242c399ab
|
Generate 1 token to verify the health of the inference service in /health (#1154)
Co-authored-by: Yineng Zhang <me@zhyncs.com>
|
2024-08-21 03:14:34 +10:00 |
|
Yineng Zhang
|
04707b09b7
|
misc: add hypervisor vendor (#1165)
|
2024-08-21 02:14:51 +10:00 |
|
Xu-Chen
|
ff2cfdb1a2
|
[Feature] add disable-custom-all-reduce (#1148)
Co-authored-by: chenxu02 <chenxu02@zhihu.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
|
2024-08-20 08:44:12 -07:00 |
|
Lianmin Zheng
|
a8ae640328
|
Improve docs and warnings (#1164)
|
2024-08-20 08:31:29 -07:00 |
|
Juwan Yoo
|
d8476818ef
|
feat: allow streaming for multi-prompt and/or parallel sampling (#1134)
|
2024-08-20 08:06:55 -07:00 |
|
Ke Bao
|
df191254ab
|
Optimize MLA/GQA/MQA Triton decoding (#1138)
Co-authored-by: Yineng Zhang <me@zhyncs.com>
|
2024-08-19 20:23:07 +10:00 |
|
yichuan~
|
b997a18d74
|
[Feat]Add support for optional start len of logprobs (#1035)
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
|
2024-08-18 23:45:41 -07:00 |
|
Zhanghao Wu
|
d8627ed16d
|
[Docs] Add instruction for running on clouds and kubernetes with SkyPilot (#1144)
Co-authored-by: Zongheng Yang <zongheng.y@gmail.com>
|
2024-08-19 14:01:55 +08:00 |
|
min-xu-et
|
fa13b95d6b
|
fixed a typo (#1143)
|
2024-08-18 14:29:09 -07:00 |
|
Lianmin Zheng
|
3c1f5a9220
|
Fix duplicated imports in hf_transformers_utils.py (#1141)
|
2024-08-17 18:03:00 -07:00 |
|
Lianmin Zheng
|
57d0bd91ec
|
Improve benchmark (#1140)
|
2024-08-17 17:43:23 -07:00 |
|
Lianmin Zheng
|
cdc8d60752
|
Improve the code style: more comments and remove useless packages (#1139)
|
2024-08-17 14:37:52 -07:00 |
|
Yineng Zhang
|
9208591f05
|
fix: use fp16 dtype for sm75 (#1136)
|
2024-08-18 00:45:42 +10:00 |
|
Liangsheng Yin
|
5d0d40d0eb
|
Fix CI accuracy && time out limit (#1133)
|
2024-08-16 21:41:11 -07:00 |
|
Liangsheng Yin
|
f624f6a6cc
|
Fix port conflicts between local CI and runner CI. (#1131)
|
2024-08-16 15:12:38 -07:00 |
|
Liangsheng Yin
|
3694f8f996
|
Mixed style of chunked prefill (#1013)
|
2024-08-16 09:13:00 +00:00 |
|
Lianmin Zheng
|
5a261bd055
|
Fix the deadlock in multi-node tp (#1122)
|
2024-08-16 01:39:24 -07:00 |
|
Yineng Zhang
|
6aa8ad14f8
|
fix: resolve Python.h header missing (#1119)
|
2024-08-16 15:46:43 +10:00 |
|
Yineng Zhang
|
26e9c12c15
|
ci: compatible with fork repo (#1115)
|
2024-08-16 04:26:44 +10:00 |
|
Lianmin Zheng
|
87a0db82b8
|
update hyperparameter guide (#1114)
|
2024-08-15 10:54:24 -07:00 |
|
Yineng Zhang
|
5bd953749b
|
chore: bump v0.2.13 (#1111)
|
2024-08-16 03:50:43 +10:00 |
|
Lianmin Zheng
|
0cb099e20a
|
set CUDA_DEVICE_MAX_CONNECTIONS=1 (#1113)
|
2024-08-16 03:47:39 +10:00 |
|
Ying Sheng
|
93d4e354d8
|
[Fix] Window attention compatible with RadixAttention and chunked prefill (#1112)
|
2024-08-15 10:33:20 -07:00 |
|
Yineng Zhang
|
9195d1362a
|
misc: rm unused model_loader (#1110)
|
2024-08-15 08:29:35 -07:00 |
|
Ying Sheng
|
14cb544d56
|
[Fix] fix flashinfer usage for window attention (#1107)
|
2024-08-15 00:53:24 -07:00 |
|
Lianmin Zheng
|
e86b1ccbf0
|
Enable chunked prefill by default (#1040)
|
2024-08-14 21:56:20 -07:00 |
|
Ying Sheng
|
8d2d876fc8
|
[Fix] fix the typo bug for window attention (#1106)
|
2024-08-14 21:56:01 -07:00 |
|
Lianmin Zheng
|
326df4bab2
|
Use a single workspace for flashinfer (#1077)
|
2024-08-14 19:25:37 -07:00 |
|
Ying Sheng
|
6767e2229f
|
Support jinja as chat template file (#1104)
|
2024-08-14 17:43:14 -07:00 |
|
Liangsheng Yin
|
73cf6834f2
|
Support stop_token_ids in sglang API (#1092)
|
2024-08-15 00:31:39 +00:00 |
|
Yineng Zhang
|
1c2b5f5240
|
docs: update nsys usage (#1103)
|
2024-08-15 01:39:15 +08:00 |
|
Ying Sheng
|
96a2093ef0
|
[Fix] Compatibility of window attention and cuda graph (#1090)
|
2024-08-14 10:37:01 -07:00 |
|
Liangsheng Yin
|
a34dd86a7d
|
Use dtype to control generate (#1082)
Co-authored-by: zhyncs <me@zhyncs.com>
|
2024-08-14 15:58:07 +00:00 |
|
Yineng Zhang
|
67c0d832a6
|
docs: update pr template (#1099)
|
2024-08-14 22:25:39 +10:00 |
|
Lianmin Zheng
|
a59636bb5e
|
Update grok 1 model (#1095)
|
2024-08-14 04:40:44 -07:00 |
|
Yineng Zhang
|
fe5024325b
|
docs: update README (#1098)
|
2024-08-14 04:40:05 -07:00 |
|
Yineng Zhang
|
f14569f64a
|
ci: remove workflow path trigger (#1096)
|
2024-08-14 20:36:24 +10:00 |
|
Lianmin Zheng
|
8f790ac100
|
Fix a bug in cuda graph runner (#1094)
|
2024-08-14 03:25:38 -07:00 |
|
rainred
|
616b59f384
|
[Feature] modify Runtime to support skip_tokenizer_init (#1088)
Co-authored-by: lzhang <zhanglei@modelbest.cn>
|
2024-08-14 00:28:04 -07:00 |
|
Yineng Zhang
|
c8423ca311
|
ci: update timeout and retry (#1086)
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
|
2024-08-14 00:27:35 -07:00 |
|
Liangsheng Yin
|
e205527cb1
|
Fix jump forward final state circular path bug. (#1084)
|
2024-08-13 21:14:05 -07:00 |
|
Ying Sheng
|
0909bb0d2f
|
[Feat] Add window attention for gemma-2 (#1056)
|
2024-08-13 17:01:26 -07:00 |
|
Lianmin Zheng
|
ad3e4f1619
|
Update the mixtral to use the better FusedMoE layer (#1081)
|
2024-08-13 15:44:25 -07:00 |
|
Lucien
|
312e849255
|
Example file for docker compose and k8s (#1006)
|
2024-08-13 15:07:57 -07:00 |
|
rainred
|
95f5fbf1a7
|
Fix create_abort_task, GenerateReqInput does not have rids. (#1079)
Co-authored-by: lzhang <zhanglei@modelbest.cn>
|
2024-08-13 12:47:22 +00:00 |
|
Yineng Zhang
|
cebd78d83e
|
ci: add accuracy timeout (#1078)
|
2024-08-13 22:12:58 +10:00 |
|
Yineng Zhang
|
0076f11541
|
fix: use devel for Triton's compiler requirements (#1074)
|
2024-08-13 04:08:43 -07:00 |
|