EngineX-Ascend/enginex-ascend-910-vllm
11
0
Fork 1
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
71b752346bed5784caa89e2fd69193d78a56f76d
enginex-ascend-910-vllm/docs/source/developer_guide/contribution/nightly_ci_test.md
Sun Ruoxi 7f8a1b1f7a init v0.23.0 
Signed-off-by: Sun Ruoxi <sunruoxi@4paradigm.com>
2026-08-27 15:11:51 +08:00

12 KiB
Raw Blame History

Nightly CI Test

This document explains how to trigger nightly hardware CI tests against your own PR code on Ascend NPU hardware (A2/A3), without waiting for the scheduled nightly run.

Background

By default, nightly CI tests run on a fixed schedule using pre-built nightly images. Contributors can self-service trigger these tests directly against their PR changes by combining a GitHub label with a comment command.

How to Trigger

1. Post a comment

Post one of the following comments in the PR to specify which tests to run. The comment itself triggers the workflow — no label is required.

Comment Effect
/nightly Run all nightly tests
/nightly all Run all nightly tests (same as above)
/nightly test1 test2 ... Run only the named tests

:::{note} Only repository Contributors (Triage role) and Maintainers (Write role) can trigger the /nightly command. If you do not have this permission, ask a maintainer to post the comment for you. You can find the list of maintainers and contributors in the project's Governance page or by checking the CODEOWNERS file. :::

2. Wait for results

GitHub Actions will trigger the Nightly-A2 or Nightly-A3 workflow. Only tests matching the filter will be dispatched, which saves hardware resources.

Differences Between PR and Scheduled Runs

Scheduled / Manual Dispatch PR-triggered
Trigger Cron (daily) or workflow_dispatch /nightly comment
Code tested Pre-built nightly image Your PR's HEAD commit (source installed fresh)
Test scope All tests Configurable via /nightly <names>
vLLM + vllm-ascend From image Checked out and installed from source
Test matrix From main branch's matrix YAML From PR branch's matrix YAML

When a PR run is detected (is_pr_test: true), the workflow additionally:

  1. Uninstalls any existing vllm packages in the container.
  2. Checks out the specific vllm version and your PR's vllm-ascend commit from source.
  3. Installs all dependencies from source.
  4. Installs the aisbench benchmark suite.

Test Matrix Data Source

The set of nightly test cases (their names, runners, test paths, model configs) is declared in a single data file:

.github/workflows/configs/nightly_config.yaml

The file is organized as a2: and a3: top-level keys (one per SoC). Under each SoC, tests are grouped by execution shape (single-node, multi-node, double-node, multi-card, accuracy) and each group holds a test_config (or nightly / pr_only for accuracy) list whose entries carry a name plus the fields consumed by the downstream reusable workflows (os, tests, config_file_path, size, etc.).

Both the Nightly-A2 and Nightly-A3 workflows dynamically read this file at run time — there is no hardcoded test matrix in the workflow YAMLs. The /nightly <name> slash command resolves names by walking the same file from the PR branch, so newly added entries can be exercised on a PR before they land on main.

Adding a New Nightly Test Case

To add a new test case (no need to touch the workflow YAMLs):

  1. Append an entry under the appropriate section in .github/workflows/configs/nightly_config.yaml. Each entry needs at least:
    • name: unique identifier used in /nightly <name> filters
    • os (for single-node / multi-card pytest+yaml tests) or runner is inferred
    • one of tests: (pytest directory) or config_file_path: (YAML-driven model config)
    • size (multi-node / double-node only)
  2. Add the actual test files (pytest modules under tests/e2e/nightly/... or YAML model configs in tests/e2e/nightly/.../configs/).
  3. Open a PR. Once CI is green, you can validate the new entry against real NPU hardware without merging the PR — see Examples below.

Available Test Names

The test names you can pass to /nightly correspond to the name fields under the matching section in .github/workflows/configs/nightly_config.yaml. The tables below mirror the current contents of that file.

A2 workflow (.github/workflows/schedule_nightly_test_a2.yaml)

Single-node tests (a2.single_node.test_config):

Test name Description
test_custom_op_multi_card Custom operator tests (multi card)
qwen3-vl-32b-instruct-w8a8 Qwen3-VL-32B-Instruct W8A8
qwen3-32b-int8 Qwen3-32B INT8 quantization
Qwen3.5-27B-w8a8-A2 Qwen3.5-27B W8A8
Qwen3.5-397B-A17B-w4a8-mtp Qwen3.5-397B-A17B W4A8 + MTP

Multi-node tests (a2.multi_node.test_config):

Test name Description
multi-node-qwen3-235b-dp Qwen3-235B-A22B, 2-node DP
multi-node-GLM-5.1-w8a8-A2 GLM-5.1 W8A8, 2 nodes
multi-node-Kimi-K2.5-W4A8-A2 Kimi-K2.5 W4A8, 2 nodes

Accuracy tests (a2.accuracy.nightly and a2.accuracy.pr_only):

Test name Description Scope
accuracy-group-1 Qwen3-VL-8B, Qwen3-8B, Qwen2-Audio-7B, etc. nightly
accuracy-group-2 ERNIE-4.5, Molmo-7B, Llama-3.2-3B, etc. nightly
accuracy-group-3 Qwen3-30B-A3B, Qwen3-VL-30B-A3B, etc. nightly
accuracy-group-4 Qwen3-Next-80B-A3B, Qwen3-Omni-30B-A3B, etc. nightly
pr-accuracy-group-1 gemma-3-4b-it, internlm3-8b-instruct, etc. pr_only
pr-accuracy-group-2 Qwen2.5-Math-RM-72B, Hunyuan-A13B-Instruct pr_only

The pr-accuracy-group-* entries only run on /nightly (PR-triggered) runs; /nightly all on the schedule skips them.

A3 workflow (.github/workflows/schedule_nightly_test_a3.yaml)

Multi-node tests (a3.multi_node.test_config, 4-node):

Test name Description
multi-node-deepseek-v3.2-W8A8-EP DeepSeek-V3.2-W8A8 with EP, 4-node

Double-node tests (a3.double_node.test_config, 2-node, run after multi-node):

Test name Description
multi-node-deepseek-r1-w8a8-longseq DeepSeek-R1-W8A8 long sequence, 2-node
multi-node-qwen3-dp Qwen3-235B-A22B, 2-node DP
multi-node-qwenw8a8-2node-eplb Qwen3-235B-W8A8 with EPLB, 2-node
multi-node-dpsk3.2-2node DeepSeek-V3.2-W8A8, 2-node
multi-node-qwenw8a8-2node-longseq Qwen3-235B-W8A8 long sequence, 2-node
multi-node-qwen-disagg-pd Qwen3-235B disaggregated PD, 2-node
multi-node-qwen-vl-disagg-pd Qwen3-VL-235B disaggregated PD, 2-node
multi-node-deepseek-v3.1 DeepSeek-V3.1-BF16, 2-node
multi-node-deepseek-v3.2-W8A8-EP DeepSeek-V3.2-W8A8 with EP, 4-node
multi-node-glm-5.2 GLM-5.1-W8A8, 2-node

Single-node tests (a3.single_node.test_config):

Test name Description
mtpx-deepseek-r1-0528-w8a8 MTP-X + DeepSeek-R1-0528-W8A8
deepseek-r1-0528-w8a8 DeepSeek-R1-0528-W8A8
kimi-k2-thinking Kimi-K2-Thinking
qwen3-vl-235b-a22b-instruct-w8a8 Qwen3-VL-235B-A22B-Instruct-W8A8
deepseek-r1-0528-w8a8-prefix-cache DeepSeek-R1-0528-W8A8 prefix cache
deepseek-v3-2-w8a8 DeepSeek-V3.2-W8A8
glm-4.7-w8a8 GLM-4.7 W8A8
kimi-k2.5 Kimi-K2.5
qwen3-235b-a22b-w8a8 Qwen3-235B-A22B-W8A8
Qwen3.5-397B-A17B-w8a8-mtp Qwen3.5-397B-A17B W8A8 + MTP
MiniMax-M2.5-w8a8-QuaRot-A3 MiniMax-M2.5 W8A8 + QuaRot
Qwen3.5-27B-w8a8-A3 Qwen3.5-27B W8A8
Qwen3.5-122B-A10B-W8A8-A3 Qwen3.5-122B-A10B W8A8
DeepSeek-V4-Flash-W8A8-A3 DeepSeek-V4-Flash W8A8

Multi-card tests (a3.multi_card.test_config):

Test name Description
qwen3-30b-acc Qwen3-30B accuracy test
qwen3-30b-a3b-w8a8 Qwen3-30B-A3B-W8A8
qwen3-32b-int8 Qwen3-32B-Int8
qwen3-32b-int8-prefix-cache Qwen3-32B-Int8 prefix cache
Qwen3-30B-A3B-W4A8-llm-compressor Qwen3-30B-A3B W4A8 via llm-compressor
Qwen3-30B-QuaRot Qwen3-30B QuaRot + eagle3
Qwen3-32B-QuaRot Qwen3-32B QuaRot + eagle3

:::{warning} The A3 resource pool has a maximum concurrency of 5×16 NPUs. Multi-node tests run with max-parallel: 2 to avoid resource exhaustion. Running /nightly all on A3 will queue a large number of jobs — prefer targeting specific test names when possible. :::

Examples

Run all available nightly tests against your PR:

/nightly

Run only the custom operator multi-card test:

/nightly test_custom_op_multi_card

Run two specific tests at once (one per SoC):

/nightly test_custom_op_multi_card mtpx-deepseek-r1-0528-w8a8

Run a single accuracy group (with all of its models):

/nightly accuracy-group-1

Run a single accuracy model (only that model from a group):

/nightly accuracy-group-1/Qwen3-8B

Re-trigger after fixing an issue: just push a new commit. The synchronize event re-runs the workflow and picks up the existing /nightly comment automatically — no need to post a new comment.

Adding a New Test Case — Worked Example

To add my-new-test to the A2 single-node section:

  1. Edit .github/workflows/configs/nightly_config.yaml, append under a2.single_node.test_config:

      - name: my-new-test
    os: linux-aarch64-a2b3-4
    tests: tests/e2e/nightly/single_node/ops/multicard_ops_a2/test_my_new.py

  2. Commit the new pytest file (test_my_new.py) in the same PR.

  3. Trigger from the PR:

    /nightly my-new-test

The workflow will:

  • pr_nightly_command.yml reads your PR's nightly_config.yaml and resolves my-new-test → dispatch A2 only.
  • Nightly-A2 is dispatched at main, but generate-a2-matrix checks out your PR commit and reads the new entry from the matrix.
  • single-node-tests runs one matrix job for my-new-test, with should_run=true. The reusable workflow checks out your PR code (via vllm_ascend_ref) and runs your pytest.

Troubleshooting

The workflow didn't start after I posted the comment.

  • Check that the comment starts exactly with /nightly with no leading spaces or extra characters before the slash.
  • Confirm you have at least Triage permission on the repository; unauthorized users' comments are ignored.
  • To re-trigger after fixing an issue, simply push a new commit — the workflow will reuse the existing /nightly comment automatically.

Only some tests ran, not the ones I expected.

  • Test names are case-sensitive and must match the name field in .github/workflows/configs/nightly_config.yaml exactly (see the tables above).
  • For a PR-triggered run, the matrix is loaded from your PR's nightly_config.yaml, not main. If a name isn't in your PR's file, it won't be recognized and the dispatch will be skipped.
  • Check the parse-trigger job output in GitHub Actions for the resolved test_filter value.

The workflow ran with the scheduled image, not my PR code.

  • Confirm the workflow was triggered by repository_dispatch (slash command), not bare workflow_dispatch. The pr_nightly_command.yml workflow is what actually dispatches schedule_nightly_test_a2.yaml / _a3.yaml with vllm_ascend_ref pointing at your PR SHA.

A new test I added isn't being recognized.

  • Confirm the entry is well-formed YAML under .github/workflows/configs/nightly_config.yaml. The name field is required and must be unique within the SoC's section.
  • The matrix is loaded from your PR branch, so make sure the file is committed to the same branch the /nightly comment was posted on.

How to obtain more detailed logs to pinpoint problems for multi-node tests

  • For most issues, the stdout pop-up logs from GitHub actions are sufficient (this log always represents the logs from the first node).

  • If the logs from a first node are no longer sufficient to provide effective logging information, see the summary of your jobs to download log archive for the corresponding test, which includes the framework-side logs and plog information for each node, structured as follows:

    .
├── node0
│   ├── root
│   │   └── ascend
│   │       └── log
│   └── var
│       └── log
│           └── vllm-deepseek-v3-0f233d-0_logs.txt
└── node1
    ├── root
    │   └── ascend
    │       └── log
    └── var
        └── log
            └── vllm-deepseek-v3-0f233d-0-1_logs.txt