feat: add adaptive GPU scheduling
This commit is contained in:
@@ -8,6 +8,7 @@ It currently supports:
|
||||
- continuous queue refill via `run_poll.sh`
|
||||
- multiple ModelHub tokens read from `KEY.md` and `KEYS.md`
|
||||
- automatic task/framework/template selection across the supported GPU catalog
|
||||
- adaptive long-term/exploration/recent GPU scheduling with a persistent local snapshot
|
||||
|
||||
## Layout
|
||||
|
||||
@@ -72,13 +73,21 @@ bash run_poll.sh --dry-run
|
||||
## Behavior
|
||||
|
||||
- The runner auto-discovers all safe GPU/template combinations from the public submit catalog.
|
||||
- Automatic GPU selection uses smooth 50/30/20 scheduling: long-term Wilson-ranked top 3 GPUs,
|
||||
all supported GPUs, and the best GPU from the latest 1,000 terminal tasks.
|
||||
- A strategy generation lasts exactly 200 platform-accepted submissions. Rejected API calls and
|
||||
duplicates do not advance it. The next cycle refreshes platform history before submitting again.
|
||||
- Strategy state is stored in `.modelhub_state/gpu_strategy.json`; a generation never recalculates
|
||||
during candidate submission.
|
||||
- Each model can be submitted at most once per GPU.
|
||||
- Multiple ModelHub tokens are pooled and used to route submissions to the account with available async capacity.
|
||||
- Concurrent submissions reserve account slots locally, and an account-capacity race automatically falls through to another account.
|
||||
- Concurrent local processes claim model/GPU pairs in `.modelhub_state/submission_claims.jsonl`; shared ledger, history, and outcome files use process locks and atomic replacement.
|
||||
- Isolated agent containers diversify candidate order by instance identity to reduce cross-container duplicate submissions.
|
||||
- History stats are online-only. The local ledger is used for local accounting, but platform history is only used after the local ledger reaches the configured threshold.
|
||||
- The default history threshold is `500` records.
|
||||
- ModelScope list pages are paced and cached for 15 minutes. HTTP 429 responses use exponential
|
||||
backoff and `Retry-After`; pages already downloaded remain usable and the failed page is retried
|
||||
on the next cycle.
|
||||
- Every third poll cycle, a full account gets one controlled capacity probe. A successful probe
|
||||
raises that account's persisted known limit; a capacity rejection enters cooldown.
|
||||
|
||||
## Important Flags
|
||||
|
||||
@@ -95,6 +104,8 @@ Common flags:
|
||||
- `--skip-outcome-sync`: skip outcome sync before scanning
|
||||
- `--skip-history-archive`: skip history archive download for this run
|
||||
- `--dry-run`: plan only, do not submit
|
||||
- `--gpu-strategy-refresh-submissions`: accepted tasks per strategy generation (default `200`)
|
||||
- `--disable-gpu-strategy`: restore legacy ordering; explicit `--gpu/--gpus` also bypasses adaptive selection
|
||||
- `run_daily.sh` injects `--daily-target 3` when no daily-target flag is provided. Set `SUBMIT_DAILY_TARGET` or pass `--daily-target` explicitly for a different target.
|
||||
|
||||
`run_poll.sh` adds:
|
||||
@@ -127,6 +138,11 @@ Each run typically includes:
|
||||
- `skipped.jsonl`
|
||||
- `failed.jsonl`
|
||||
|
||||
Persistent local scheduler state is written under `.modelhub_state/`:
|
||||
|
||||
- `gpu_strategy.json`: GPU ranks, generation progress, and 50/30/20 accepted counters
|
||||
- `account_capacity.json`: learned per-account active-task limits
|
||||
|
||||
## Verification
|
||||
|
||||
Run the full test suite:
|
||||
|
||||
Reference in New Issue
Block a user