60e2be1b36
[RFC](https://github.com/vllm-project/vllm-ascend/issues/3328) for more details. Add dynamic batch feature in chunked prefilling strategy, the token budget can be refined to achieve better effective throughput and TPOT. !!! NOTE: only 910B3 is supported till now, we are working on further improvements. Additional file for lookup table is required. - vLLM version: v0.11.0rc3 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.0 --------- Signed-off-by: Cheng Wang <wangchengkyrie@outlook.com>
2.7 KiB
Dynamic Batch
Dynamic batch is a technique that dynamically adjusts the chunksize during each inference iteration within the chunked prefilling strategy according to the resources and SLO targets, thereby improving the effective throughput and decreasing the TBT.
Dynamic batch is controlled by the value of the --SLO_limits_for_dynamic_batch.
Notably, only 910 B3 is supported with decode token numbers scales below 2048 so far.
Especially, the improvements are quite obvious on Qwen, Llama models.
We are working on further improvements and this feature will support more XPUs in the future.
Getting started
Prerequisites
-
Dynamic batch now depends on a offline cost model saved in a look-up table to refine the token budget. The lookup-table is saved in '.csv' file, which should be first downloaded from here, renamed, and saved to the path
vllm_ascend/core/profile_table.csv -
Pandasis needed to load the look-up table, in casepandasis not installed.
pip install pandas
```
### Tuning Parameter
`--SLO_limits_for_dynamic_batch` is the tuning parameters (integer type) for the dynamic batch feature, greater values impose more constraints on the latency limitation, leading to higher effective throughput. The parameter can be selected according to the specific models or service requirements.
```python
--SLO_limits_for_dynamic_batch =-1 # default value, dynamic batch disabled.
--SLO_limits_for_dynamic_batch = 0 # baseline value for dynamic batch, dynamic batch disabled, FCFS and decode-first chunked prefilling strategy is used.
--SLO_limits_for_dynamic_batch > 0 # user-defined value for dynamic batch, dynamic batch enabled with FCFS and decode-first chunked prefilling strategy.
Supported Models
So far, dynamic batch performs better on several dense models including Qwen and Llama (from 8B to 32B) with tensor_parallel_size=8. For different models, a proper SLO_limits_for_dynamic_batch parameter is needed. The empirical value of this parameter is generally 35, 50, or 75. Therefore, some additional tests are needed to select the best parameter.
Usage
Dynamic batch is used in the online inference. A fully executable example is as follows:
SLO_LITMIT=50
vllm serve Qwen/Qwen2.5-14B-Instruct\
--additional_config '{"SLO_limits_for_dynamic_batch":'${SLO_LITMIT}'}' \
--max-num-seqs 256 \
--block-size 128 \
--tensor_parallel_size 8 \
--load_format dummy \
--max_num_batched_tokens 1024 \
--max_model_len 9000 \
--host localhost \
--port 12091 \
--gpu-memory-utilization 0.9 \
--trust-remote-code