Revert "fix some typos" (#6244)

benchmark/benchmark_vllm_060/README.md

@@ -1,6 +1,6 @@
 ## How to reproduce the benchmark results for SGLang v0.3.0 compared to vLLM v0.6.0
 
-In short, with multi step enabled, in online scenarios that we benchmarked, the Median TTFT of vLLM is **3 times** that of SGLang, and the Median ITL is **10 times** that of SGLang. Lower Median TTFT and ITL are better. vLLM's multi-step optimization did not improve throughput while ensuring lower Median TTFT and ITL. Also, under maximum throughput benchmark, if vLLM does not set GPU utilization to 0.95 separately and uses the default configuration instead, its maximum throughput is **lower** than that of SGLang.
+In short, with multi step enabled, in online scenarios that we benchmarked, the Median TTFT of vLLM is **3 times** that of SGLang, and the Median ITL is **10 times** that of SGLang. Lower Median TTFT and ITL are better. vLLM's multi-step optimization did not improve throughput while ensuring lower Median TTFT and ITL. Also, under maximum throughput benchmark, if vLLM does not set gpu util to 0.95 separately and uses the default configuration instead, its maximum throughput is **lower** than that of SGLang.
 
 ## Online benchmark results
 
@@ -41,12 +41,12 @@ In short, with multi step enabled, in online scenarios that we benchmarked, the
 ## Installation
 
 ```bash
-# install SGLang v0.3.0
+# install sglang v0.3.0
 pip install --upgrade pip
 pip install "sglang[all]"==0.3.0
 pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/
 
-# install vLLM v0.6.0
+# install vllm v0.6.0
 pip install vllm==0.6.0
 ```
 

benchmark/deepseek_v3/README.md

@@ -45,10 +45,10 @@ Add [performance optimization options](#performance-optimization-options) as nee
 
 ### Performance Optimization Options
 
-[MLA optimizations](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) are enabled by default. Here are some optional optimizations that can be enabled as needed.
+[MLA optimizations](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) are enabled by default. Here are some optional optimizations can be enabled as needed.
 
 - [Data Parallelism Attention](https://lmsys.org/blog/2024-12-04-sglang-v0-4/#data-parallelism-attention-for-deepseek-models): For high QPS scenarios, add the `--enable-dp-attention` argument to boost throughput.
-- [Torch.compile Optimization](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#torchcompile-latency-optimizations): Add `--enable-torch-compile` argument to enable it. This will take some time while the server starts. The maximum batch size for torch.compile optimization can be controlled with `--torch-compile-max-bs`. It's recommended to set it between `1` and `8`. (e.g., `--torch-compile-max-bs 8`)
+- [Torch.compile Optimization](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#torchcompile-latency-optimizations): Add `--enable-torch-compile` argument to enable it. This will take some time while server starts. The maximum batch size for torch.compile optimization can be controlled with `--torch-compile-max-bs`. It's recommended to set it between `1` and `8`. (e.g., `--torch-compile-max-bs 8`)
 
 ### Example: Sending requests with OpenAI API
 
@@ -90,7 +90,7 @@ If you have two H100 nodes, the usage is similar to the aforementioned H20.
 
 > **Note that the launch command here does not enable Data Parallelism Attention or `torch.compile` Optimization**. For optimal performance, please refer to the command options in [Performance Optimization Options](#option_args).
 
-### Example: Serving with two H200\*8 nodes and Docker
+### Example: Serving with two H200\*8 nodes and docker
 
 There are two H200 nodes, each with 8 GPUs. The first node's IP is `192.168.114.10`, and the second node's IP is `192.168.114.11`. Configure the endpoint to expose it to another Docker container using `--host 0.0.0.0` and `--port 40000`, and set up communications with `--dist-init-addr 192.168.114.10:20000`.
 A single H200 with 8 devices can run DeepSeek V3, the dual H200 setup is just to demonstrate multi-node usage.
@@ -147,7 +147,7 @@ docker run --gpus all \
 
 To serve DeepSeek-V3 with A100 GPUs, we need to convert the [FP8 model checkpoints](https://huggingface.co/deepseek-ai/DeepSeek-V3) to BF16 with [script](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py) mentioned [here](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py) first.
 
-Since the BF16 model is over 1.3 TB, we need to prepare four A100 nodes, each with 8 80GB GPUs. Assuming the first node's IP is `10.0.0.1`, and the converted model path is `/path/to/DeepSeek-V3-BF16`, we can run the following commands to launch the server.
+Since the BF16 model is over 1.3 TB, we need to prepare four A100 nodes, each with 8 80GB GPUs. Assume the first node's IP is `10.0.0.1`, and the converted model path is `/path/to/DeepSeek-V3-BF16`, we can have following commands to launch the server.
 
 ```bash
 # node 1
@@ -178,7 +178,7 @@ python3 -m sglang.bench_one_batch_server --model None --base-url http://10.0.0.1
 
 ### Example: Serving with 8 A100/A800 with AWQ Quantization
 
-Add the `--quantization moe_wna16` flag to enable the MoE wna16 kernel for better performance.
+Add `--quantization moe_wna16` flag to enable moe wna16 kernel for better performance.
 One example is as follows:
 
 ```bash
@@ -188,12 +188,12 @@ python3 -m sglang.launch_server --model cognitivecomputations/DeepSeek-R1-AWQ --
 
 ### Example: Serving with 16 A100/A800 with int8 Quantization
 
-There are block-wise and per-channel quantization methods, and the quantization parameters have already been uploaded to HuggingFace. One example is as follows:
+There are block-wise and per-channel quantization methods, and the quantization parameters have already been uploaded to Huggingface. One example is as follows:
 
 - [meituan/DeepSeek-R1-Block-INT8](https://huggingface.co/meituan/DeepSeek-R1-Block-INT8)
 - [meituan/DeepSeek-R1-Channel-INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8)
 
-Assuming that the master node IP is `MASTER_IP`, checkpoint path is `/path/to/DeepSeek-R1-INT8` and port=5000, we can run the following commands to launch the server:
+Assuming that master node IP is `MASTER_IP`, checkpoint path is `/path/to/DeepSeek-R1-INT8` and port=5000, we can have following commands to launch the server:
 ```bash
 #master
 python3 -m sglang.launch_server \
@@ -225,7 +225,7 @@ Running with per-channel quantization model:
 
 - [meituan/DeepSeek-R1-Channel-INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8)
 
-Assuming that the master node IP is `MASTER_IP`, checkpoint path is `/path/to/DeepSeek-R1-Channel-INT8` and port=5000, we can run the following commands to launch the server:
+Assuming that master node IP is `MASTER_IP`, checkpoint path is `/path/to/DeepSeek-R1-Channel-INT8` and port=5000, we can have following commands to launch the server:
 
 ```bash
 #master

benchmark/gsm8k/README.md

@@ -1,6 +1,6 @@
-## Run Benchmark
+## Run benchmark
 
-### Benchmark SGLang
+### Benchmark sglang
 ```
 python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
 ```
@@ -10,7 +10,7 @@ python3 bench_sglang.py --num-questions 200
 ```
 
 
-### Benchmark vLLM
+### Benchmark vllm
 ```
 python3 -m vllm.entrypoints.api_server --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000
 ```
@@ -20,7 +20,7 @@ python3 bench_other.py --num-questions 200 --backend vllm
 ```
 
 
-### Benchmark LightLLM
+### Benchmark lightllm
 ```
 # A10G
 python -m lightllm.server.api_server --tokenizer_mode auto --model_dir ~/model_weights/llama-2-7b-chat-hf --max_total_token_num 16000 --port 22000
@@ -31,13 +31,13 @@ python3 bench_other.py --num-questions 200 --backend lightllm
 ```
 
 
-### Benchmark Guidance
+### Benchmark guidance
 ```
 python3 bench_other.py --num-questions 200 --backend guidance --parallel 1 --n-ctx 4096 --model-path path/to/gguf
 ```
 
 
-### Benchmark LMQL
+### Benchmark lmql
 ```
 CUDA_VISIBLE_DEVICES=0,1 lmql serve-model meta-llama/Llama-2-7b-chat-hf --cuda --port 23000
 ```

benchmark/hellaswag/README.md

@@ -1,6 +1,6 @@
 ## Run benchmark
 
-### Benchmark SGLang
+### Benchmark sglang
 ```
 python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
 ```
@@ -10,7 +10,7 @@ python3 bench_sglang.py --num-questions 200
 ```
 
 
-### Benchmark vLLM
+### Benchmark vllm
 ```
 python3 -m vllm.entrypoints.api_server --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000
 ```
@@ -20,7 +20,7 @@ python3 bench_other.py --num-questions 200 --backend vllm
 ```
 
 
-### Benchmark LightLLM
+### Benchmark lightllm
 ```
 # A10G
 python -m lightllm.server.api_server --tokenizer_mode auto --model_dir ~/model_weights/llama-2-7b-chat-hf --max_total_token_num 16000 --port 22000
@@ -31,13 +31,13 @@ python3 bench_other.py --num-questions 200 --backend lightllm
 ```
 
 
-### Benchmark Guidance
+### Benchmark guidance
 ```
 CUDA_VISIBLE_DEVICES=0,1 python3 bench_other.py --num-questions 200 --backend guidance --parallel 1 --n-ctx 4096 --model-path path/to/gguf
 ```
 
 
-### Benchmark LMQL
+### Benchmark lmql
 ```
 lmql serve-model meta-llama/Llama-2-7b-chat-hf --cuda --port 23000
 ```

benchmark/kernels/fused_moe_triton/README.md

@@ -4,7 +4,7 @@ This directory contains benchmarking tools for MoE (Mixture of Experts) kernels.
 
 ### Tuning Tool
 
-- `tuning_fused_moe_triton.py`: A tool for tuning the `fused_moe_triton` kernel. Adapted from [vLLM's benchmark_moe.py](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_moe.py), with added support for various model architectures.
+- `tuning_fused_moe_triton.py`: A tool for tuning the `fused_moe_triton` kernel. Adapted from [vllm's benchmark_moe.py](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_moe.py), with added support for various model architectures.
 
 Example usage:
 ```bash
@@ -48,7 +48,7 @@ After tuning, a configuration file (e.g., `E=64,N=640,device_name=NVIDIA_GeForce
 
 ### Performance Comparison Tool
 
-- `benchmark_vllm_vs_sglang_fused_moe_triton.py`: A tool for comparing the performance of fused MoE kernels between vLLM and SGLang implementations. Supports various model architectures and data types.
+- `benchmark_vllm_vs_sglang_fused_moe_triton.py`: A tool for comparing the performance of fused MoE kernels between vllm and sglang implementations. Supports various model architectures and data types.
 
 Example usage:
 ```bash

benchmark/mmlu/README.md

@@ -1,11 +1,11 @@
-## Download Data
+## Download data
 ```
 bash download_data.sh
 ```
 
-## Run Benchmark
+## Run benchmark
 
-### Benchmark SGLang
+### Benchmark sglang
 ```
 python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
 ```
@@ -19,7 +19,7 @@ python3 bench_sglang.py --nsub 10
 python3 bench_sglang.py --backend gpt-3.5-turbo --parallel 8
 ```
 
-### Benchmark vLLM
+### Benchmark vllm
 ```
 python3 -m vllm.entrypoints.api_server --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000
 ```
@@ -29,7 +29,7 @@ python3 bench_other.py --nsub 10 --backend vllm
 ```
 
 
-### Benchmark LightLLM
+### Benchmark lightllm
 ```
 # A10G
 python -m lightllm.server.api_server --tokenizer_mode auto --model_dir ~/model_weights/llama-2-7b-chat-hf --max_total_token_num 16000 --port 22000
@@ -43,13 +43,13 @@ python3 bench_other.py --nsub 10 --backend lightllm
 ```
 
 
-### Benchmark Guidance
+### Benchmark guidance
 ```
 python3 bench_other.py --nsub 10 --backend guidance --parallel 1 --n-ctx 4096 --model-path path/to/gguf
 ```
 
 
-### Benchmark LMQL
+### Benchmark lmql
 ```
 CUDA_VISIBLE_DEVICES=0,1 lmql serve-model meta-llama/Llama-2-7b-chat-hf --cuda --port 23000
 ```

benchmark/mtbench/README.md

@@ -4,9 +4,9 @@
 wget -O question.jsonl https://raw.githubusercontent.com/lm-sys/FastChat/main/fastchat/llm_judge/data/mt_bench/question.jsonl
 ```
 
-## Run Benchmark
+## Run benchmark
 
-### Benchmark SGLang
+### Benchmark sglang
 ```
 python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
 ```
@@ -15,7 +15,7 @@ python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port
 python3 bench_sglang.py --num-questions 80
 ```
 
-### Benchmark SGLang EAGLE
+### Benchmark sglang EAGLE
 ```
 python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3-8B-Instruct --speculative-algo EAGLE \
     --speculative-draft lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \
@@ -27,7 +27,7 @@ python3 bench_sglang_eagle.py --num-questions 80 --parallel 1
 ```
 
 
-### Benchmark vLLM
+### Benchmark vllm
 ```
 python3 -m vllm.entrypoints.api_server --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000
 ```
@@ -37,7 +37,7 @@ python3 bench_other.py --num-questions 80 --backend vllm
 ```
 
 
-### Benchmark LightLLM
+### Benchmark lightllm
 ```
 # A10G
 python -m lightllm.server.api_server --tokenizer_mode auto --model_dir ~/model_weights/llama-2-7b-chat-hf --max_total_token_num 16000 --port 22000

benchmark/multi_chain_reasoning/README.md

@@ -1,11 +1,11 @@
-## Download Data
+## Download data
 ```
 wget https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/test.jsonl
 ```
 
-## Run Benchmark
+## Run benchmark
 
-### Benchmark SGLang
+### Benchmark sglang
 ```
 python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000  --schedule-conservativeness 1.3
 ```
@@ -16,7 +16,7 @@ python3 bench_sglang.py --num-questions 32 --parallel 1
 ```
 
 
-### Benchmark vLLM
+### Benchmark vllm
 ```
 python3 -m vllm.entrypoints.api_server --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000
 ```
@@ -26,7 +26,7 @@ python3 bench_other.py --num-questions 64 --backend vllm
 ```
 
 
-### Benchmark LightLLM
+### Benchmark lightllm
 ```
 # A10G
 python -m lightllm.server.api_server --tokenizer_mode auto --model_dir ~/model_weights/llama-2-7b-chat-hf --max_total_token_num 16000 --port 22000
@@ -37,12 +37,12 @@ python3 bench_other.py --num-questions 64 --backend lightllm
 ```
 
 
-### Benchmark Guidance
+### Benchmark guidance
 ```
 python3 bench_other.py --num-questions 8 --backend guidance --parallel 1 --n-ctx 4096 --model-path path/to/gguf
 ```
 
-### Benchmark LMQL
+### Benchmark lmql
 
 ```
 python3 bench_other.py --num-questions 64 --backend lmql --parallel 1

benchmark/tree_of_thought_deep/README.md

@@ -1,13 +1,13 @@
-## Download Data
+## Download data
 ```
 wget https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/test.jsonl
 ```
 
-## Run Benchmark
+## Run benchmark
 
 NOTE: This is an implementation for throughput/latency benchmark purposes. The prompts are not tuned to achieve good accuracy on the GSM-8K tasks.
 
-### Benchmark SGLang
+### Benchmark sglang
 ```
 python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
 ```
@@ -18,7 +18,7 @@ python3 bench_sglang.py --num-questions 16 --parallel 1
 ```
 
 
-### Benchmark vLLM
+### Benchmark vllm
 ```
 python3 -m vllm.entrypoints.api_server --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000
 ```
@@ -28,7 +28,7 @@ python3 bench_other.py --num-questions 32 --backend vllm
 ```
 
 
-### Benchmark LightLLM
+### Benchmark lightllm
 ```
 # A10G
 python -m lightllm.server.api_server --tokenizer_mode auto --model_dir ~/model_weights/llama-2-7b-chat-hf --max_total_token_num 16000 --port 22000
@@ -39,12 +39,12 @@ python3 bench_other.py --num-questions 32 --backend lightllm
 ```
 
 
-### Benchmark Guidance
+### Benchmark guidance
 ```
 python3 bench_other.py --num-questions 8 --backend guidance --parallel 1 --n-ctx 4096 --model-path path/to/gguf
 ```
 
-### Benchmark LMQL
+### Benchmark lmql
 
 ```
 python3 bench_other.py --num-questions 8 --backend lmql --parallel 1

benchmark/tree_of_thought_v0/README.md

@@ -1,11 +1,11 @@
-## Download Data
+## Download data
 ```
 wget https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/test.jsonl
 ```
 
 ## Run benchmark
 
-### Benchmark SGLang
+### Benchmark sglang
 ```
 python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
 ```
@@ -16,7 +16,7 @@ python3 bench_sglang.py --num-questions 10 --parallel 1
 ```
 
 
-### Benchmark vLLM
+### Benchmark vllm
 ```
 python3 -m vllm.entrypoints.api_server --tokenizer-mode auto --model meta-llama/Llama-2-7b-chat-hf --disable-log-requests --port 21000
 ```
@@ -26,7 +26,7 @@ python3 bench_other.py --num-questions 32 --backend vllm
 ```
 
 
-### Benchmark LightLLM
+### Benchmark lightllm
 ```
 # A10G
 python -m lightllm.server.api_server --tokenizer_mode auto --model_dir ~/model_weights/llama-2-7b-chat-hf --max_total_token_num 16000 --port 22000
@@ -37,7 +37,7 @@ python3 bench_other.py --num-questions 32 --backend lightllm
 ```
 
 
-### Benchmark Guidance
+### Benchmark guidance
 ```
 python3 bench_other.py --num-questions 32 --backend guidance --parallel 1 --n-ctx 4096 --model-path path/to/gguf
 ```