feat: add adaptive GPU scheduling

This commit is contained in:
CoolBoy
2026-08-02 16:59:44 +08:00
parent 80a1b8518d
commit eab5ab6dce
14 changed files with 1216 additions and 52 deletions

View File

@@ -31,6 +31,8 @@ Optional tuning:
- `MODELHUB_AGENT_POST_CYCLE_COOLDOWN_SECONDS` default `2`
- `MODELHUB_AGENT_MAX_SUBMITS_PER_RUN` default `0` (fill all currently available slots)
- `MODELHUB_AGENT_ACTIVE_TASK_CAP` default `100` per account
- `MODELHUB_CAPACITY_PROBE_INTERVAL_CYCLES` default `3`
- `MODELHUB_CAPACITY_STATE_PATH` default `.modelhub_state/account_capacity.json`
- `MODELHUB_AGENT_ACTIVE_COUNTS_TTL_SECONDS` default `15`
- `MODELHUB_AGENT_RESERVATION_TTL_SECONDS` default `120`
- `MODELHUB_AGENT_INSTANCE_ID` optional stable worker identity used to spread concurrent agents across accounts and candidates
@@ -39,6 +41,27 @@ Optional tuning:
- `MODELHUB_AGENT_MIN_DOWNLOADS`
- `MODELHUB_AGENT_GPUS`
- `MODELHUB_AGENT_EXTRA_ARGS`
- `MODELHUB_GPU_STRATEGY_STATE_PATH` default `.modelhub_state/gpu_strategy.json`
- `MODELSCOPE_PAGE_INTERVAL_SECONDS` default `0.25`
- `MODELSCOPE_PAGE_CACHE_TTL_SECONDS` default `900`
## Adaptive GPU Strategy
When no explicit GPU override is supplied, the worker uses a local 50/30/20
strategy generation:
- 50%: the three long-term GPUs with the best Wilson lower confidence score and at least 100 terminal samples
- 30%: round-robin exploration across every currently supported GPU
- 20%: the best GPU among the latest 1,000 terminal tasks
Only platform-accepted tasks count. After exactly 200 accepted tasks, the next
poll cycle reloads all account history, generates a new immutable strategy snapshot,
and resets the generation counters to 100/60/40 targets. The active snapshot and
progress are stored in `.modelhub_state/gpu_strategy.json`.
ModelScope HTTP 429 responses use exponential backoff and `Retry-After`. Successful
pages remain cached, so a later cycle retries the failed page instead of restarting
the whole pagination scan.
## Concurrent Agents
@@ -56,7 +79,7 @@ authority for account capacity and model/GPU uniqueness.
If the platform reports that a model/GPU is already being validated, the claim
is retained and the runner immediately draws replacement candidates from the
same scan instead of retrying the duplicate every cycle. Startup logs and the
health response expose `agent_version`; version `2026.08.02.2` or newer includes
health response expose `agent_version`; version `2026.08.02.3` or newer includes
this behavior.
## Deploy