feat: add adaptive GPU scheduling

This commit is contained in:
CoolBoy
2026-08-02 16:59:44 +08:00
parent 80a1b8518d
commit eab5ab6dce
14 changed files with 1216 additions and 52 deletions

View File

@@ -8,6 +8,7 @@ It currently supports:
- continuous queue refill via `run_poll.sh`
- multiple ModelHub tokens read from `KEY.md` and `KEYS.md`
- automatic task/framework/template selection across the supported GPU catalog
- adaptive long-term/exploration/recent GPU scheduling with a persistent local snapshot
## Layout
@@ -72,13 +73,21 @@ bash run_poll.sh --dry-run
## Behavior
- The runner auto-discovers all safe GPU/template combinations from the public submit catalog.
- Automatic GPU selection uses smooth 50/30/20 scheduling: long-term Wilson-ranked top 3 GPUs,
all supported GPUs, and the best GPU from the latest 1,000 terminal tasks.
- A strategy generation lasts exactly 200 platform-accepted submissions. Rejected API calls and
duplicates do not advance it. The next cycle refreshes platform history before submitting again.
- Strategy state is stored in `.modelhub_state/gpu_strategy.json`; a generation never recalculates
during candidate submission.
- Each model can be submitted at most once per GPU.
- Multiple ModelHub tokens are pooled and used to route submissions to the account with available async capacity.
- Concurrent submissions reserve account slots locally, and an account-capacity race automatically falls through to another account.
- Concurrent local processes claim model/GPU pairs in `.modelhub_state/submission_claims.jsonl`; shared ledger, history, and outcome files use process locks and atomic replacement.
- Isolated agent containers diversify candidate order by instance identity to reduce cross-container duplicate submissions.
- History stats are online-only. The local ledger is used for local accounting, but platform history is only used after the local ledger reaches the configured threshold.
- The default history threshold is `500` records.
- ModelScope list pages are paced and cached for 15 minutes. HTTP 429 responses use exponential
backoff and `Retry-After`; pages already downloaded remain usable and the failed page is retried
on the next cycle.
- Every third poll cycle, a full account gets one controlled capacity probe. A successful probe
raises that account's persisted known limit; a capacity rejection enters cooldown.
## Important Flags
@@ -95,6 +104,8 @@ Common flags:
- `--skip-outcome-sync`: skip outcome sync before scanning
- `--skip-history-archive`: skip history archive download for this run
- `--dry-run`: plan only, do not submit
- `--gpu-strategy-refresh-submissions`: accepted tasks per strategy generation (default `200`)
- `--disable-gpu-strategy`: restore legacy ordering; explicit `--gpu/--gpus` also bypasses adaptive selection
- `run_daily.sh` injects `--daily-target 3` when no daily-target flag is provided. Set `SUBMIT_DAILY_TARGET` or pass `--daily-target` explicitly for a different target.
`run_poll.sh` adds:
@@ -127,6 +138,11 @@ Each run typically includes:
- `skipped.jsonl`
- `failed.jsonl`
Persistent local scheduler state is written under `.modelhub_state/`:
- `gpu_strategy.json`: GPU ranks, generation progress, and 50/30/20 accepted counters
- `account_capacity.json`: learned per-account active-task limits
## Verification
Run the full test suite: