xc-llm-ascend/tests/e2e/models/configs/Qwen3-8B-Base.yaml

model_name: "Qwen/Qwen3-8B-Base"
runner: "linux-aarch64-a2-1"
hardware: "Atlas A2 Series"
tasks:
- name: "gsm8k"
  metrics:
  - name: "exact_match,strict-match"
    value: 0.82
  - name: "exact_match,flexible-extract"
    value: 0.83
- name: "ceval-valid"
  metrics:
  - name: "acc,none"
    value: 0.82
num_fewshot: 5
Enable pytest and yaml style accuracy test (#2073) ### What this PR does / why we need it? This PR enabled pytest and yaml style accuracy test, users now can enable accuracy test by running: ```bash cd ~/vllm-ascend pytest -sv ./tests/e2e/singlecard/models/test_lm_eval_correctness.py \ --config ./tests/e2e/singlecard/models/configs/Qwen3-8B-Base.yaml \ --report_output ./benchmarks/accuracy/Qwen3-8B-Base.md pytest -sv ./tests/e2e/singlecard/models/test_lm_eval_correctness.py \ --config-list-file ./tests/e2e/singlecard/models/configs/accuracy.txt ``` Closes: https://github.com/vllm-project/vllm-ascend/issues/1970 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? - vLLM version: v0.10.0 - vLLM main: https://github.com/vllm-project/vllm/commit/2836dd73f13015ee386c544760ca0d16888203f3 --------- Signed-off-by: Icey <1790571317@qq.com> 2025-07-31 21:39:13 +08:00			`model_name: "Qwen/Qwen3-8B-Base"`
[Test] Update the format of the accuracy report (#3081) ### What this PR does / why we need it? Update the format of the accuracy report ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.10.2 - vLLM main: https://github.com/vllm-project/vllm/commit/c60e6137f0bf2034853919b3a9d705d7e06b93cf Signed-off-by: hfadzxy <starmoon_zhang@163.com> 2025-09-22 14:10:03 +08:00			`runner: "linux-aarch64-a2-1"`
			`hardware: "Atlas A2 Series"`
Enable pytest and yaml style accuracy test (#2073) ### What this PR does / why we need it? This PR enabled pytest and yaml style accuracy test, users now can enable accuracy test by running: ```bash cd ~/vllm-ascend pytest -sv ./tests/e2e/singlecard/models/test_lm_eval_correctness.py \ --config ./tests/e2e/singlecard/models/configs/Qwen3-8B-Base.yaml \ --report_output ./benchmarks/accuracy/Qwen3-8B-Base.md pytest -sv ./tests/e2e/singlecard/models/test_lm_eval_correctness.py \ --config-list-file ./tests/e2e/singlecard/models/configs/accuracy.txt ``` Closes: https://github.com/vllm-project/vllm-ascend/issues/1970 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? - vLLM version: v0.10.0 - vLLM main: https://github.com/vllm-project/vllm/commit/2836dd73f13015ee386c544760ca0d16888203f3 --------- Signed-off-by: Icey <1790571317@qq.com> 2025-07-31 21:39:13 +08:00			`tasks:`
			`- name: "gsm8k"`
			`metrics:`
			`- name: "exact_match,strict-match"`
			`value: 0.82`
			`- name: "exact_match,flexible-extract"`
			`value: 0.83`
			`- name: "ceval-valid"`
			`metrics:`
			`- name: "acc,none"`
			`value: 0.82`
			`num_fewshot: 5`