[CI] Enable auto upgrade e2e estimated time for auto-partition suites (#6840)
### What this PR does / why we need it? This patch add a schedule triggered workflow for auto upgrade e2e estimated-time for batter load balance 1. The workflow will run the full e2e test to get the duration of each test. 2. The script `update_estimated_time.py` will upgrade the [config.json](https://github.com/vllm-project/vllm-ascend/blob/main/.github/workflows/scripts/config.yaml) according to the latest time 3. The workflow will submit a pull request that includes changes to `config.json` automatically <img width="2484" height="764" alt="image" src="https://github.com/user-attachments/assets/02f3459c-bb3b-4f8e-9966-8bb2e5c1bbea" /> ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.15.0 - vLLM main:83b47f67b1- ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.15.0 - vLLM main:83b47f67b1--------- Signed-off-by: wangli <wangli858794774@gmail.com>
This commit is contained in:
111
.github/workflows/schedule_update_estimated_time.yaml
vendored
Normal file
111
.github/workflows/schedule_update_estimated_time.yaml
vendored
Normal file
@@ -0,0 +1,111 @@
|
||||
name: Update estimated test times
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 2 * * 1' # Every Monday at 02:00 UTC
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
branches:
|
||||
- 'main'
|
||||
paths:
|
||||
- '.github/workflows/schedule_update_estimated_time.yaml'
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
|
||||
concurrency:
|
||||
group: update-estimated-times-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
e2e-test:
|
||||
name: e2e-test
|
||||
strategy:
|
||||
matrix:
|
||||
vllm_version: [15d76f74e2fdb12a95ea00f0ca283acf6219a2b7]
|
||||
type: [full, light]
|
||||
uses: ./.github/workflows/_e2e_test.yaml
|
||||
with:
|
||||
vllm: ${{ matrix.vllm_version }}
|
||||
image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/vllm-ascend:main
|
||||
contains_310: false
|
||||
type: ${{ matrix.type }}
|
||||
continue_on_error: true # Continue even if some tests fail, we want to collect as much timing data as possible
|
||||
|
||||
update-estimated-times:
|
||||
name: Update estimated_time in config.yaml
|
||||
needs: [e2e-test]
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout repo
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Download all timing artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: timing-data-*
|
||||
path: timing-artifacts/
|
||||
merge-multiple: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install pyyaml
|
||||
|
||||
- name: Update config.yaml from timing data
|
||||
run: |
|
||||
python3 .github/workflows/scripts/update_estimated_time.py \
|
||||
--timing-dir timing-artifacts/ \
|
||||
--config .github/workflows/scripts/config.yaml
|
||||
|
||||
- name: Check for changes
|
||||
id: check_changes
|
||||
run: |
|
||||
if git diff --quiet .github/workflows/scripts/config.yaml; then
|
||||
echo "changed=false" >> "$GITHUB_OUTPUT"
|
||||
echo "No changes to config.yaml."
|
||||
else
|
||||
echo "changed=true" >> "$GITHUB_OUTPUT"
|
||||
echo "config.yaml has been updated:"
|
||||
git diff .github/workflows/scripts/config.yaml
|
||||
fi
|
||||
|
||||
- name: Create pull request
|
||||
if: steps.check_changes.outputs.changed == 'true' && github.event_name != 'pull_request'
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
BRANCH="auto/update-estimated-times-${{ github.run_id }}"
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "github-actions[bot]@users.noreply.github.com"
|
||||
git checkout -b "$BRANCH"
|
||||
git add .github/workflows/scripts/config.yaml
|
||||
git commit -m "[CI] Auto-update estimated test times in config.yaml
|
||||
|
||||
Computed from timing-data artifacts of workflow run ${{ github.run_id }}.
|
||||
Buffer ratio: 1.1x median, rounded to the nearest 10 s."
|
||||
git push origin "$BRANCH"
|
||||
gh pr create \
|
||||
--repo "${{ github.repository }}" \
|
||||
--base main \
|
||||
--head "$BRANCH" \
|
||||
--title "chore: Auto-update estimated test times in config.yaml" \
|
||||
--body "## Summary
|
||||
|
||||
This PR was auto-generated by the **Update estimated test times** workflow.
|
||||
|
||||
It updates the \`estimated_time\` values in \`.github/workflows/scripts/config.yaml\`
|
||||
based on actual elapsed times collected from workflow run \`${{ github.run_id }}\`.
|
||||
|
||||
### Methodology
|
||||
- Timing data is uploaded as \`timing-data-*\` artifacts by each e2e test job.
|
||||
- For each test file, the **median** of all collected elapsed times is taken.
|
||||
- A **10 % safety buffer** is applied and the result is rounded to the nearest 10 s.
|
||||
|
||||
Please review the diff and merge if the new values look reasonable."
|
||||
Reference in New Issue
Block a user