EngineX-Ascend/enginex-ascend-910-vllm
11
0
Fork 1
Code Issues Pull Requests Actions Projects Releases Wiki Activity

init v0.23.0

Browse Source 
Signed-off-by: Sun Ruoxi <sunruoxi@4paradigm.com>
This commit is contained in:
Sun Ruoxi
2026-08-27 15:11:51 +08:00
parent b582a8e7d1
commit 7f8a1b1f7a
2849 changed files with 712887 additions and 22001 deletions
Download Patch File Download Diff File Expand all files Collapse all files

129
docs/source/developer_guide/evaluation/using_lm_eval.md
View File

@@ -1,9 +1,12 @@
# Using lm-eval
This document will guide you have a accuracy testing using [lm-eval][1].
This document guides you to conduct accuracy testing using [lm-eval][1].
## Online Server
### 1. start the vLLM server
You can run docker container to start the vLLM server on a single NPU:
### 1. Start the vLLM server
You can run a docker container to start the vLLM server on a single NPU:
```{code-block} bash
:substitutions:
@@ -13,6 +16,7 @@ export DEVICE=/dev/davinci7
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
@@ -31,46 +35,54 @@ docker run --rm \
vllm serve Qwen/Qwen2.5-0.5B-Instruct --max_model_len 4096 &
```
Started the vLLM server successfully,if you see log as below:
The vLLM server is started successfully, if you see logs as below:
```
```shell
INFO: Started server process [9446]
INFO: Waiting for application startup.
INFO: Application startup complete.
```
### 2. Run gsm8k accuracy test using lm-eval
### 2. Run GSM8K using the vLLM server (curl) and then run lm-eval for accuracy testing
You can query result with input prompts:
You can query the result with input prompts:
```shell
PROMPT='<|im_start|>system
You are a professional accountant. Answer questions using accounting knowledge, output only the option letter (A/B/C/D).<|im_end|>
<|im_start|>user
Question: A company'"'"'s balance sheet as of December 31, 2023 shows:
Current assets: Cash and equivalents 5 million yuan, Accounts receivable 8 million yuan, Inventory 6 million yuan
Non-current assets: Net fixed assets 12 million yuan
Current liabilities: Short-term loans 4 million yuan, Accounts payable 3 million yuan
Non-current liabilities: Long-term loans 9 million yuan
Owner'"'"'s equity: Paid-in capital 10 million yuan, Retained earnings ?
Requirement: Calculate the company'"'"'s Asset-Liability Ratio and Current Ratio (round to two decimal places).
Options:
A. Asset-Liability Ratio=58.33%, Current Ratio=1.90
B. Asset-Liability Ratio=62.50%, Current Ratio=2.17
C. Asset-Liability Ratio=65.22%, Current Ratio=1.75
D. Asset-Liability Ratio=68.00%, Current Ratio=2.50<|im_end|>
<|im_start|>assistant
'
```
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-0.5B-Instruct",
"prompt": "'"<|im_start|>system\nYou are a professional accountant. Answer questions using accounting knowledge, output only the option letter (A/B/C/D).<|im_end|>\n"\
"<|im_start|>user\nQuestion: A company's balance sheet as of December 31, 2023 shows:\n"\
" Current assets: Cash and equivalents 5 million yuan, Accounts receivable 8 million yuan, Inventory 6 million yuan\n"\
" Non-current assets: Net fixed assets 12 million yuan\n"\
" Current liabilities: Short-term loans 4 million yuan, Accounts payable 3 million yuan\n"\
" Non-current liabilities: Long-term loans 9 million yuan\n"\
" Owner's equity: Paid-in capital 10 million yuan, Retained earnings ?\n"\
"Requirement: Calculate the company's Asset-Liability Ratio and Current Ratio (round to two decimal places).\n"\
"Options:\n"\
"A. Asset-Liability Ratio=58.33%, Current Ratio=1.90\n"\
"B. Asset-Liability Ratio=62.50%, Current Ratio=2.17\n"\
"C. Asset-Liability Ratio=65.22%, Current Ratio=1.75\n"\
"D. Asset-Liability Ratio=68.00%, Current Ratio=2.50<|im_end|>\n"\
"<|im_start|>assistant\n"'",
"max_tokens": 1,
"temperature": 0,
"stop": ["<|im_end|>"]
}' | python3 -m json.tool
-d "$(jq -n \
--arg model "Qwen/Qwen2.5-0.5B-Instruct" \
--arg prompt "$PROMPT" \
'{
model: $model,
prompt: $prompt,
max_completion_tokens: 1,
temperature: 0,
stop: ["<|im_end|>"]
}')" | python3 -m json.tool
```
The output format matches the following:
```
```json
{
"id": "cmpl-2f678e8bdf5a4b209a3f2c1fa5832e25",
"object": "text_completion",
@@ -98,16 +110,24 @@ The output format matches the following:
}
```
Install lm-eval in the container.
Install lm-eval in the container:
```bash
export HF_ENDPOINT="https://hf-mirror.com"
export USE_MODELSCOPE_HUB=0
pip install lm-eval[api]
```
:::{note}
The Docker container is launched with `VLLM_USE_MODELSCOPE=True`, which may
cause lm-eval to download datasets from ModelScope instead of HuggingFace.
Setting `USE_MODELSCOPE_HUB=0` disables this behavior so that lm-eval can
fetch datasets from HuggingFace correctly.
:::
Run the following command:
```
```shell
# Only test gsm8k dataset in this demo
lm_eval \
--model local-completions \
@@ -116,19 +136,20 @@ lm_eval \
--output_path ./
```
After 30 mins, the output is as shown below:
After 30 minutes, the output is as shown below:
```
The markdown format results is as below:
```shell
The results in Markdown format are as follows:
Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3215|± |0.0129|
| | |strict-match | 5|exact_match|↑ |0.2077|± |0.0112|
|gsm8k| 3|strict-match | 5|exact_match|↑ |0.2077|± |0.0112|
```
## Offline Server
### 1. Run docker container
You can run docker container on a single NPU:
@@ -141,6 +162,7 @@ export DEVICE=/dev/davinci7
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
@@ -158,17 +180,26 @@ docker run --rm \
/bin/bash
```
### 2. Run gsm8k accuracy test using lm-eval
Install lm-eval in the container.
### 2. Run GSM8K using lm-eval for accuracy testing
Install lm-eval in the container:
```bash
export HF_ENDPOINT="https://hf-mirror.com"
export USE_MODELSCOPE_HUB=0
pip install lm-eval
```
:::{note}
The Docker container is launched with `VLLM_USE_MODELSCOPE=True`, which may
cause lm-eval to download datasets from ModelScope instead of HuggingFace.
Setting `USE_MODELSCOPE_HUB=0` disables this behavior so that lm-eval can
fetch datasets from HuggingFace correctly.
:::
Run the following command:
```
```shell
# Only test gsm8k dataset in this demo
lm_eval \
--model vllm \
@@ -177,21 +208,21 @@ lm_eval \
--batch_size auto
```
After 1-2 mins, the output is as shown below:
After 1 to 2 minutes, the output is shown below:
```
The markdown format results is as below:
```shell
The markdown format results are as below:
Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3412|± |0.0131|
| | |strict-match | 5|exact_match|↑ |0.3139|± |0.0128|
|gsm8k| 3|strict-match | 5|exact_match|↑ |0.3139|± |0.0128|
```
## Use offline Datasets
## Use Offline Datasets
Take gsm8k(single dataset) and mmlu(multi-subject dataset) as examples, and you can see more from [here][2].
Take GSM8K (single dataset) and MMLU (multi-subject dataset) as examples, and you can see more from [using-local-datasets][2].
```bash
# set HF_DATASETS_OFFLINE when using offline datasets
@@ -205,7 +236,7 @@ cd lm_eval/tasks/gsm8k
cd lm_eval/tasks/mmlu/default
```
set [gsm8k.yaml][3] as follows:
Set [gsm8k.yaml][3] as follows:
```yaml
tag:
@@ -230,7 +261,7 @@ training_split: train
fewshot_split: train
test_split: test
doc_to_text: 'Q: {{question}}
A(Please follow the summarize the result at the end with the format of "The answer is xxx", where xx is the result.):'
A(Please follow the summarized result at the end with the format of "The answer is xxx", where xx is the result.):'
doc_to_target: "{{answer}}" #" {{answer.split('### ')[-1].rstrip()}}"
metric_list:
- metric: exact_match
@@ -268,7 +299,7 @@ metadata:
version: 3.0
```
set [_default_template_yaml][4] as follows:
Set [_default_template_yaml][4] as follows:
```yaml
# set dataset_path according to the downloaded dataset