fix: refill from expanded candidate history

This commit is contained in:
CoolBoy
2026-08-02 17:45:21 +08:00
parent eab5ab6dce
commit 8e68c6f611
10 changed files with 418 additions and 64 deletions

View File

@@ -29,7 +29,8 @@ Optional tuning:
- `MODELHUB_AGENT_POLL_INTERVAL_SECONDS` default `15`
- `MODELHUB_AGENT_IDLE_INTERVAL_SECONDS` default `60`
- `MODELHUB_AGENT_POST_CYCLE_COOLDOWN_SECONDS` default `2`
- `MODELHUB_AGENT_MAX_SUBMITS_PER_RUN` default `0` (fill all currently available slots)
- The hosted entrypoint always uses `--max-submits-per-run 0` so stale deployment
settings cannot restrict a refill cycle to five submissions.
- `MODELHUB_AGENT_ACTIVE_TASK_CAP` default `100` per account
- `MODELHUB_CAPACITY_PROBE_INTERVAL_CYCLES` default `3`
- `MODELHUB_CAPACITY_STATE_PATH` default `.modelhub_state/account_capacity.json`
@@ -44,6 +45,7 @@ Optional tuning:
- `MODELHUB_GPU_STRATEGY_STATE_PATH` default `.modelhub_state/gpu_strategy.json`
- `MODELSCOPE_PAGE_INTERVAL_SECONDS` default `0.25`
- `MODELSCOPE_PAGE_CACHE_TTL_SECONDS` default `900`
- `MODELHUB_AGENT_VERIFY_CACHE_TTL_SECONDS` default `900`
## Adaptive GPU Strategy
@@ -63,6 +65,18 @@ ModelScope HTTP 429 responses use exponential backoff and `Retry-After`. Success
pages remain cached, so a later cycle retries the failed page instead of restarting
the whole pagination scan.
## Adaptive Candidate Discovery
The configured recent window remains the fast path. If it contains no usable
model/GPU combinations, the same run progressively expands discovery to the last
7 days, the last 30 days, and finally older history (up to 3,000 models). Scanning
stops as soon as enough replacement candidates have been found.
Model verification results are reused for 15 minutes across poll cycles, and a
locally failed model/GPU pair cools down for 24 hours instead of being excluded
forever. The `[scan]` lines show every expansion stage, while `[daily] wave_done`
includes `skip_reasons` so an empty candidate pool is directly diagnosable.
## Concurrent Agents
The token pool keeps a local reservation for every in-flight submission, so a
@@ -80,7 +94,8 @@ If the platform reports that a model/GPU is already being validated, the claim
is retained and the runner immediately draws replacement candidates from the
same scan instead of retrying the duplicate every cycle. Startup logs and the
health response expose `agent_version`; version `2026.08.02.3` or newer includes
this behavior.
duplicate replacement behavior, while version `2026.08.02.4` adds adaptive
candidate-window expansion and skip-reason reporting.
## Deploy