EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

improve docs

Browse Source
This commit is contained in:
Lianmin Zheng
2024-02-05 11:22:06 +00:00
parent 26f0bedc8f
commit 8ff870bf3e
2 changed files with 5 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
README.md
View File

@@ -9,8 +9,9 @@ The core features of SGLang include:
- **A High-Performance Runtime with RadixAttention**: This feature significantly accelerates the execution of complex LLM programs by automatic KV cache reuse across multiple calls. It also supports other common techniques like continuous batching and tensor parallelism. - **A High-Performance Runtime with RadixAttention**: This feature significantly accelerates the execution of complex LLM programs by automatic KV cache reuse across multiple calls. It also supports other common techniques like continuous batching and tensor parallelism.
## News ## News
- [2024/01] 🔥 SGLang powers the serving of the offical LLaVA v1.6 release demo ([blog](https://llava-vl.github.io/blog/2024-01-30-llava-1-6/)). - [2024/02] 🔥 SGLang enables **3x faster JSON decoding** with compressed finite state machine ([blog](https://lmsys.org/blog/2024-02-05-compressed-fsm/)).
- [2024/01] SGLang provides up to 5x faster inference with RadixAttention ([blog](https://lmsys.org/blog/2024-01-17-sglang/)). - [2024/01] 🔥 SGLang powers the serving of the offical **LLaVA v1.6** release demo ([usage](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#demo)).
- [2024/01] SGLang provides up to **5x faster inference** with RadixAttention ([blog](https://lmsys.org/blog/2024-01-17-sglang/)).
## Contents ## Contents
- [Install](#install) - [Install](#install)
@@ -216,7 +217,7 @@ character_regex = (
@sgl.function @sgl.function
def character_gen(s, name): def character_gen(s, name):
s += name + " is a character in Harry Potter. Please fill in the following information about him/her.\n" s += name + " is a character in Harry Potter. Please fill in the following information about this character.\n"
s += sgl.gen("json_output", max_tokens=256, regex=character_regex) s += sgl.gen("json_output", max_tokens=256, regex=character_regex)
``` ```

2
benchmark/json_jump_forward/bench_other.py
View File

@@ -64,7 +64,7 @@ def character_maker(lm, name):
regex_str_no_quote = r"[\w\d\s]+" regex_str_no_quote = r"[\w\d\s]+"
regex_float = r"[0-9]+\.[0-9]+" regex_float = r"[0-9]+\.[0-9]+"
lm += f"""\ lm += f"""\
{name} is a character in Harry Potter. Please fill in the following information about him/her. {name} is a character in Harry Potter. Please fill in the following information about this character.
{{ {{
"name": "{guidance.gen("name", max_tokens=16, regex=regex_str_no_quote)}", "name": "{guidance.gen("name", max_tokens=16, regex=regex_str_no_quote)}",
"house": "{guidance.select(options=['Gryffindor', 'Slytherin', 'Ravenclaw', 'Hufflepuff'], name='house')}", "house": "{guidance.select(options=['Gryffindor', 'Slytherin', 'Ravenclaw', 'Hufflepuff'], name='house')}",