feat: add queue-aware adaptive scheduling
This commit is contained in:
@@ -9,6 +9,7 @@ It currently supports:
|
||||
- multiple ModelHub tokens read from `KEY.md` and `KEYS.md`
|
||||
- automatic task/framework/template selection across the supported GPU catalog
|
||||
- adaptive long-term/exploration/recent GPU scheduling with a persistent local snapshot
|
||||
- live queue/throughput-aware GPU weighting and confidence-ranked framework selection
|
||||
|
||||
## Layout
|
||||
|
||||
@@ -73,8 +74,19 @@ bash run_poll.sh --dry-run
|
||||
## Behavior
|
||||
|
||||
- The runner auto-discovers all safe GPU/template combinations from the public submit catalog.
|
||||
- Automatic GPU selection uses smooth 50/30/20 scheduling: long-term Wilson-ranked top 3 GPUs,
|
||||
all supported GPUs, and the best GPU from the latest 1,000 terminal tasks.
|
||||
- Automatic GPU selection uses exact 50/30/20 accepted-task scheduling: long-term
|
||||
Wilson-ranked top 3 GPUs, all supported GPUs, and the top GPUs from the latest
|
||||
1,000 terminal tasks.
|
||||
- Within each category, weighted-fair scheduling uses estimated queue backlog hours,
|
||||
recent public throughput/success, machine availability, and worker concurrency.
|
||||
Unavailable or stalled GPU pools are circuit-broken instead of continuing to absorb work.
|
||||
- Compatible frameworks are ranked by ModelHub public aggregate success statistics
|
||||
plus capped local GPU+framework evidence, with a 100-sample public minimum and
|
||||
Wilson confidence bounds. The legacy safe order is retained whenever evidence
|
||||
is missing or too small.
|
||||
- New frameworks are discovered from the live catalog but get no novelty bonus.
|
||||
They are eligible only with a complete official build config that passes local
|
||||
validation; cached local templates remain the fallback if live config sync fails.
|
||||
- A strategy generation lasts exactly 200 platform-accepted submissions. Rejected API calls and
|
||||
duplicates do not advance it. The next cycle refreshes platform history before submitting again.
|
||||
- Strategy state is stored in `.modelhub_state/gpu_strategy.json`; a generation never recalculates
|
||||
@@ -116,6 +128,7 @@ Common flags:
|
||||
- `--dry-run`: plan only, do not submit
|
||||
- `--gpu-strategy-refresh-submissions`: accepted tasks per strategy generation (default `200`)
|
||||
- `--disable-gpu-strategy`: restore legacy ordering; explicit `--gpu/--gpus` also bypasses adaptive selection
|
||||
- `--disable-market-intelligence`: disable live queue/throughput and framework-stat weighting
|
||||
- `run_daily.sh` injects `--daily-target 3` when no daily-target flag is provided. Set `SUBMIT_DAILY_TARGET` or pass `--daily-target` explicitly for a different target.
|
||||
|
||||
`run_poll.sh` adds:
|
||||
@@ -151,6 +164,7 @@ Each run typically includes:
|
||||
Persistent local scheduler state is written under `.modelhub_state/`:
|
||||
|
||||
- `gpu_strategy.json`: GPU ranks, generation progress, and 50/30/20 accepted counters
|
||||
- `market_intelligence.json`: cached public queue, throughput, health, and framework statistics
|
||||
- `account_capacity.json`: learned per-account active-task limits
|
||||
- `submission_exclusions.jsonl`: non-retryable model/GPU uniqueness rejections
|
||||
|
||||
|
||||
Reference in New Issue
Block a user