feat: add queue-aware adaptive scheduling

This commit is contained in:
CoolBoy
2026-08-04 20:22:08 +08:00
parent ccae7ff8f3
commit 882479e43e
12 changed files with 1586 additions and 95 deletions

View File

@@ -9,6 +9,7 @@ It currently supports:
- multiple ModelHub tokens read from `KEY.md` and `KEYS.md`
- automatic task/framework/template selection across the supported GPU catalog
- adaptive long-term/exploration/recent GPU scheduling with a persistent local snapshot
- live queue/throughput-aware GPU weighting and confidence-ranked framework selection
## Layout
@@ -73,8 +74,19 @@ bash run_poll.sh --dry-run
## Behavior
- The runner auto-discovers all safe GPU/template combinations from the public submit catalog.
- Automatic GPU selection uses smooth 50/30/20 scheduling: long-term Wilson-ranked top 3 GPUs,
all supported GPUs, and the best GPU from the latest 1,000 terminal tasks.
- Automatic GPU selection uses exact 50/30/20 accepted-task scheduling: long-term
Wilson-ranked top 3 GPUs, all supported GPUs, and the top GPUs from the latest
1,000 terminal tasks.
- Within each category, weighted-fair scheduling uses estimated queue backlog hours,
recent public throughput/success, machine availability, and worker concurrency.
Unavailable or stalled GPU pools are circuit-broken instead of continuing to absorb work.
- Compatible frameworks are ranked by ModelHub public aggregate success statistics
plus capped local GPU+framework evidence, with a 100-sample public minimum and
Wilson confidence bounds. The legacy safe order is retained whenever evidence
is missing or too small.
- New frameworks are discovered from the live catalog but get no novelty bonus.
They are eligible only with a complete official build config that passes local
validation; cached local templates remain the fallback if live config sync fails.
- A strategy generation lasts exactly 200 platform-accepted submissions. Rejected API calls and
duplicates do not advance it. The next cycle refreshes platform history before submitting again.
- Strategy state is stored in `.modelhub_state/gpu_strategy.json`; a generation never recalculates
@@ -116,6 +128,7 @@ Common flags:
- `--dry-run`: plan only, do not submit
- `--gpu-strategy-refresh-submissions`: accepted tasks per strategy generation (default `200`)
- `--disable-gpu-strategy`: restore legacy ordering; explicit `--gpu/--gpus` also bypasses adaptive selection
- `--disable-market-intelligence`: disable live queue/throughput and framework-stat weighting
- `run_daily.sh` injects `--daily-target 3` when no daily-target flag is provided. Set `SUBMIT_DAILY_TARGET` or pass `--daily-target` explicitly for a different target.
`run_poll.sh` adds:
@@ -151,6 +164,7 @@ Each run typically includes:
Persistent local scheduler state is written under `.modelhub_state/`:
- `gpu_strategy.json`: GPU ranks, generation progress, and 50/30/20 accepted counters
- `market_intelligence.json`: cached public queue, throughput, health, and framework statistics
- `account_capacity.json`: learned per-account active-task limits
- `submission_exclusions.jsonl`: non-retryable model/GPU uniqueness rejections