diff --git a/docs/source/assets/multi_node_dp.png b/docs/source/assets/multi_node_dp.png deleted file mode 100644 index 598270f..0000000 Binary files a/docs/source/assets/multi_node_dp.png and /dev/null differ diff --git a/docs/source/assets/multi_node_dp_deepseek.png b/docs/source/assets/multi_node_dp_deepseek.png new file mode 100644 index 0000000..1809dc3 Binary files /dev/null and b/docs/source/assets/multi_node_dp_deepseek.png differ diff --git a/docs/source/assets/multi_node_dp_kimi.png b/docs/source/assets/multi_node_dp_kimi.png new file mode 100644 index 0000000..a4ffc08 Binary files /dev/null and b/docs/source/assets/multi_node_dp_kimi.png differ diff --git a/docs/source/tutorials/index.md b/docs/source/tutorials/index.md index c73d5eb..39c7a83 100644 --- a/docs/source/tutorials/index.md +++ b/docs/source/tutorials/index.md @@ -13,4 +13,5 @@ multi_npu_qwen3_moe multi_npu_quantization single_node_300i multi_node +multi_node_kimi ::: diff --git a/docs/source/tutorials/multi_node.md b/docs/source/tutorials/multi_node.md index 4d425d7..cccde2d 100644 --- a/docs/source/tutorials/multi_node.md +++ b/docs/source/tutorials/multi_node.md @@ -90,12 +90,12 @@ docker run --rm \ -it $IMAGE bash ``` -:::{note} -Before launch the inference server, ensure some environment variables are set for multi node communication -::: - Run the following scripts on two nodes respectively +:::{note} +Before launch the inference server, ensure the following environment variables are set for multi node communication +::: + **node0** ```shell @@ -178,7 +178,7 @@ vllm serve /root/.cache/ds_v3 \ ``` The Deployment view looks like: -![alt text](../assets/multi_node_dp.png) +![alt text](../assets/multi_node_dp_deepseek.png) Once your server is started, you can query the model with input prompts: diff --git a/docs/source/tutorials/multi_node_kimi.md b/docs/source/tutorials/multi_node_kimi.md new file mode 100644 index 0000000..4848eaa --- /dev/null +++ b/docs/source/tutorials/multi_node_kimi.md @@ -0,0 +1,153 @@ +# Multi-Node-DP (Kimi-K2) + +## Verify Multi-Node Communication Environment + +referring to [multi_node.md](https://vllm-ascend.readthedocs.io/en/latest/tutorials/multi_node.html#verification-process) + +## Run with docker +Assume you have two Atlas 800 A3(64G*16) nodes(or 4 *A2* 8), and want to deploy the `Kimi-K2-Instruct-W8A8` quantitative model across multi-node. + +```{code-block} bash + :substitutions: +# Update the vllm-ascend image +export IMAGE=m.daocloud.io/quay.io/ascend/vllm-ascend:|vllm_ascend_version| +export NAME=vllm-ascend + +# Run the container using the defined variables +# Note if you are running bridge network with docker, Please expose available ports for multiple nodes communication in advance +docker run --rm \ +--name $NAME \ +--net=host \ +--device /dev/davinci0 \ +--device /dev/davinci1 \ +--device /dev/davinci2 \ +--device /dev/davinci3 \ +--device /dev/davinci4 \ +--device /dev/davinci5 \ +--device /dev/davinci6 \ +--device /dev/davinci7 \ +--device /dev/davinci8 \ +--device /dev/davinci9 \ +--device /dev/davinci10 \ +--device /dev/davinci11 \ +--device /dev/davinci12 \ +--device /dev/davinci13 \ +--device /dev/davinci14 \ +--device /dev/davinci15 \ +--device /dev/davinci_manager \ +--device /dev/devmm_svm \ +--device /dev/hisi_hdc \ +-v /usr/local/dcmi:/usr/local/dcmi \ +-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \ +-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ +-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ +-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ +-v /etc/ascend_install.info:/etc/ascend_install.info \ +-v /mnt/sfs_turbo/.cache:/home/cache \ +-it $IMAGE bash +``` + +Run the following scripts on two nodes respectively + +:::{note} +Before launch the inference server, ensure the following environment variables are set for multi node communication +::: + +**node0** + +```shell +#!/bin/sh + +# this obtained through ifconfig +# nic_name is the network interface name corresponding to local_ip +nic_name="xxxx" +local_ip="xxxx" + +export HCCL_IF_IP=$local_ip +export GLOO_SOCKET_IFNAME=$nic_name +export TP_SOCKET_IFNAME=$nic_name +export HCCL_SOCKET_IFNAME=$nic_name +export OMP_PROC_BIND=false +export OMP_NUM_THREADS=100 +export VLLM_USE_V1=1 +export HCCL_BUFFSIZE=1024 + +# The w8a8 weight can obtained from https://www.modelscope.cn/models/vllm-ascend/Kimi-K2-Instruct-W8A8 +# If you want to the quantization manually, please refer to https://vllm-ascend.readthedocs.io/en/latest/user_guide/feature_guide/quantization.html +vllm serve /home/cache/weights/Kimi-K2-Instruct-W8A8 \ +--host 0.0.0.0 \ +--port 8004 \ +--data-parallel-size 4 \ +--api-server-count 2 \ +--data-parallel-size-local 2 \ +--data-parallel-address $local_ip \ +--data-parallel-rpc-port 13389 \ +--seed 1024 \ +--served-model-name kimi \ +--quantization ascend \ +--tensor-parallel-size 8 \ +--enable-expert-parallel \ +--max-num-seqs 16 \ +--max-model-len 32768 \ +--max-num-batched-tokens 4096 \ +--trust-remote-code \ +--no-enable-prefix-caching \ +--gpu-memory-utilization 0.9 \ +--additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true}}' +``` + +**node1** + +```shell +#!/bin/sh + +nic_name="xxxx" +local_ip="xxxx" + +export HCCL_IF_IP=$local_ip +export GLOO_SOCKET_IFNAME=$nic_name +export TP_SOCKET_IFNAME=$nic_name +export HCCL_SOCKET_IFNAME=$nic_name +export OMP_PROC_BIND=false +export OMP_NUM_THREADS=100 +export VLLM_USE_V1=1 +export HCCL_BUFFSIZE=1024 + +vllm serve /home/cache/weights/Kimi-K2-Instruct-W8A8 \ +--host 0.0.0.0 \ +--port 8004 \ +--headless \ +--data-parallel-size 4 \ +--data-parallel-size-local 2 \ +--data-parallel-start-rank 2 \ +--data-parallel-address $node0_ip \ +--data-parallel-rpc-port 13389 \ +--seed 1024 \ +--tensor-parallel-size 8 \ +--served-model-name kimi \ +--max-num-seqs 16 \ +--max-model-len 32768 \ +--quantization ascend \ +--max-num-batched-tokens 4096 \ +--enable-expert-parallel \ +--trust-remote-code \ +--no-enable-prefix-caching \ +--gpu-memory-utilization 0.92 \ +--additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true}}' +``` + +The Deployment view looks like: +![alt text](../assets/multi_node_dp_kimi.png) + +Once your server is started, you can query the model with input prompts: + +```shell +curl http://{ node0 ip:8004 }/v1/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "kimi", + "prompt": "The future of AI is", + "max_tokens": 50, + "temperature": 0 + }' +``` diff --git a/docs/source/user_guide/feature_guide/quantization.md b/docs/source/user_guide/feature_guide/quantization.md index abdf344..bde91ea 100644 --- a/docs/source/user_guide/feature_guide/quantization.md +++ b/docs/source/user_guide/feature_guide/quantization.md @@ -8,12 +8,13 @@ Since 0.9.0rc2 version, quantization feature is experimentally supported in vLLM To quantize a model, users should install [ModelSlim](https://gitee.com/ascend/msit/blob/master/msmodelslim/README.md) which is the Ascend compression and acceleration tool. It is an affinity-based compression tool designed for acceleration, using compression as its core technology and built upon the Ascend platform. -Currently, only the specific tag [modelslim-VLLM-8.1.RC1.b020_001](https://gitee.com/ascend/msit/blob/modelslim-VLLM-8.1.RC1.b020_001/msmodelslim/README.md) of modelslim works with vLLM Ascend. Please do not install other version until modelslim master version is available for vLLM Ascend in the future. - Install modelslim: ```bash -git clone https://gitee.com/ascend/msit -b modelslim-VLLM-8.1.RC1.b020_001 +git clone https://gitee.com/ascend/msit +# Optional, this commit has been verified +git checkout f8ab35a772a6c1ee7675368a2aa4bafba3bedd1a + cd msit/msmodelslim bash install.sh pip install accelerate @@ -21,41 +22,43 @@ pip install accelerate ## Quantize model -Take [DeepSeek-V2-Lite](https://modelscope.cn/models/deepseek-ai/DeepSeek-V2-Lite) as an example, you just need to download the model, and then execute the convert command. The command is shown below. More info can be found in modelslim doc [deepseek w8a8 dynamic quantization docs](https://gitee.com/ascend/msit/blob/modelslim-VLLM-8.1.RC1.b020_001/msmodelslim/example/DeepSeek/README.md#deepseek-v2-w8a8-dynamic%E9%87%8F%E5%8C%96). +:::{note} +You can choose to convert the model yourself or use the quantized model we uploaded, +see https://www.modelscope.cn/models/vllm-ascend/Kimi-K2-Instruct-W8A8 +This conversion process will require a larger CPU memory, please ensure that the RAM size is greater than 2TB +::: + +### Adapts and change +1. Ascend does not support the `flash_attn` library. To run the model, you need to follow the [guide](https://gitee.com/ascend/msit/blob/master/msmodelslim/example/DeepSeek/README.md#deepseek-v3r1) and comment out certain parts of the code in `modeling_deepseek.py` located in the weights folder. +2. The current version of transformers does not support loading weights in FP8 quantization format. you need to follow the [guide](https://gitee.com/ascend/msit/blob/master/msmodelslim/example/DeepSeek/README.md#deepseek-v3r1) and delete the quantization related fields from `config.json` in the weights folder + +### Generate the w8a8 weights ```bash cd example/DeepSeek -python3 quant_deepseek.py --model_path {original_model_path} --save_directory {quantized_model_save_path} --device_type cpu --act_method 2 --w_bit 8 --a_bit 8 --is_dynamic True + +export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:False +export MODEL_PATH="/root/.cache/Kimi-K2-Instruct" +export SAVE_PATH="/root/.cache/Kimi-K2-Instruct-W8A8" + +python3 quant_deepseek_w8a8.py --model_path $MODEL_PATH --save_path $SAVE_PATH --batch_size 4 ``` -:::{note} -You can also download the quantized model that we uploaded. Please note that these weights should be used for test only. For example, https://www.modelscope.cn/models/vllm-ascend/DeepSeek-V2-Lite-W8A8 -::: - -Once convert action is done, there are two important files generated. - -1. [config.json](https://www.modelscope.cn/models/vllm-ascend/DeepSeek-V2-Lite-W8A8/file/view/master/config.json?status=1). Please make sure that there is no `quantization_config` field in it. - -2. [quant_model_description.json](https://www.modelscope.cn/models/vllm-ascend/DeepSeek-V2-Lite-W8A8/file/view/master/quant_model_description.json?status=1). All the converted weights info are recorded in this file. - -Here is the full converted model files: +Here is the full converted model files except safetensors: ```bash . -├── config.json -├── configuration_deepseek.py -├── configuration.json -├── generation_config.json -├── quant_model_description.json -├── quant_model_weight_w8a8_dynamic-00001-of-00004.safetensors -├── quant_model_weight_w8a8_dynamic-00002-of-00004.safetensors -├── quant_model_weight_w8a8_dynamic-00003-of-00004.safetensors -├── quant_model_weight_w8a8_dynamic-00004-of-00004.safetensors -├── quant_model_weight_w8a8_dynamic.safetensors.index.json -├── README.md -├── tokenization_deepseek_fast.py -├── tokenizer_config.json -└── tokenizer.json +|-- config.json +|-- configuration.json +|-- configuration_deepseek.py +|-- generation_config.json +|-- modeling_deepseek.py +|-- quant_model_description.json +|-- quant_model_weight_w8a8_dynamic.safetensors.index.json +|-- tiktoken.model +|-- tokenization_kimi.py +`-- tokenizer_config.json ``` ## Run the model @@ -90,10 +93,7 @@ for output in outputs: ### Online inference -```bash -# Enable quantization by specifying `--quantization ascend` -vllm serve {quantized_model_save_path} --served-model-name "deepseek-v2-lite-w8a8" --max-model-len 2048 --quantization ascend --trust-remote-code -``` +Enable quantization by specifying `--quantization ascend`, for more details, see DeepSeek-V3-W8A8 [tutorial](https://vllm-ascend.readthedocs.io/en/latest/tutorials/multi_node.html) ## FAQs diff --git a/vllm_ascend/ascend_config.py b/vllm_ascend/ascend_config.py index b8fd24e..659f441 100644 --- a/vllm_ascend/ascend_config.py +++ b/vllm_ascend/ascend_config.py @@ -17,7 +17,7 @@ from typing import Optional from vllm.logger import logger -TORCHAIR_MODEL_LIST = ["deepseek", "pangu"] +TORCHAIR_MODEL_LIST = ["deepseek", "pangu", "kimi_k2"] def _check_torchair_supported(model_type: str):