Ying Sheng 04b262cd91 [Fix] Fix major performance bug in certain cases (#1563) ...

Co-authored-by: hnyls2002 <hnyls2002@gmail.com>

2024-10-04 08:51:11 +00:00

..

configs

minor: fix config (#1524)

2024-09-27 01:49:16 -07:00

constrained

[FIX] Catch syntax error of Regex Guide to avoid crash (#1521)

2024-09-28 12:42:06 -07:00

layers

[Performance, Hardware] MoE tuning on AMD MI300x GPUs (#1554)

2024-10-02 09:52:46 -07:00

lora

[Fix] Fix AttributeError in Qwen2.5 LoRA: 'Qwen2ForCausalLM' object has no attribute 'get_hidden_dim' (#1536)

2024-10-02 20:41:15 -07:00

managers

[Fix] Fix major performance bug in certain cases (#1563)

2024-10-04 08:51:11 +00:00

mem_cache

Make input_ids a torch.Tensor (#1568)

2024-10-04 01:09:59 -07:00

model_executor

Make input_ids a torch.Tensor (#1568)

2024-10-04 01:09:59 -07:00

models

[FP8 KV Cache] Avoid KeyError at loading pre-quantized FP8 model with kv_scale (#1559)

2024-10-03 01:52:26 -07:00

openai_api

[API, Feature] Support response prefill for openai API (#1490)

2024-09-22 06:46:17 -07:00

sampling

Organize sampling batch info better (#1562)

2024-10-03 18:29:49 -07:00

conversation.py

[Minor] move triton attention kernels into a separate folder (#1379)

2024-09-10 15:15:08 -07:00

hf_transformers_utils.py

[Fix] Fix clean_up_tokenization_spaces in tokenizer (#1510)

2024-09-24 21:37:33 -07:00

mm_utils.py

[Feat/WIP] add llava-onevision, with support for (1) siglip encoder, (2) qwen2 decoder (3) openai api compatible server. (#1123)

2024-08-23 14:11:16 -07:00

server_args.py

Use ipc instead of tcp in zmq (#1566)

2024-10-04 00:45:52 -07:00

server.py

Use ipc instead of tcp in zmq (#1566)

2024-10-04 00:45:52 -07:00

utils.py

Use ipc instead of tcp in zmq (#1566)

2024-10-04 00:45:52 -07:00