feat: make model age admission-only

This commit is contained in:
CoolBoy
2026-08-12 10:18:09 +08:00
parent 86088ce577
commit 91d1d3d87d
15 changed files with 438 additions and 653 deletions

View File

@@ -16,7 +16,7 @@ It currently supports:
- `main.py`: core discovery, scoring, dedup, and submission
- `daily_runner.py`: daily wave orchestration
- `poll_runner.py`: long-running queue refiller
- `queue_cleanup.py`: fail-closed cleanup for certain OOM, architecture, and age policies
- `queue_cleanup.py`: fail-closed cleanup for deterministic OOM and architecture policies
- `runner_common.py`: shared token / key file loading
- `hf_discovery.py`: ModelScope model discovery and inspection (keeps the legacy module name)
- `modelhub_client.py`: ModelHub API client and token-pool routing
@@ -137,14 +137,13 @@ bash run_poll.sh --dry-run
memory rule as new submissions. Only tasks whose own repository size times `1.20` exceeds
their selected GPU capacity are stopped, after a fresh account-scoped active-state check.
Incomplete size/capacity evidence is never used for cancellation.
- Models older than seven days may occupy only the current account capacity minus its final 10
positions. The account pool enforces this per-account boundary atomically and updates it when
capacity probing discovers a higher limit. Cleanup stops OOM tasks first, recalculates the
surviving queue order, and applies the same limit-minus-10 boundary on startup. Scheduled
cleanup relaxes to limit minus 5 to avoid excessive pruning. Age cleanup stops waiting tasks
only; running tasks are protected and rechecked after OOM cleanup, immediately before the
age-only stop batch. Recent
overflow tasks stay, and unknown ModelScope timestamps never authorize a cancellation.
- Models older than seven days may occupy only the current account capacity minus its final 5
positions. The account pool enforces this per-account admission boundary atomically and updates
it when capacity probing discovers a higher limit. A failed active-count read makes that account
ineligible for older models while other readable accounts are still tried. Recent models may use
all available positions. Age policy never cancels an existing waiting or running task; startup
and scheduled cleanup remain limited to deterministic OOM and learned architecture evidence.
Date-deferred candidates are skipped, not failed or permanently excluded.
## Important Flags
@@ -178,19 +177,25 @@ Common flags:
- `--submit-concurrency`: concurrent task submission calls used by each cycle (0 = auto)
- `--post-cycle-cooldown-seconds`: pause after a successful cycle before next cycle (default 2)
- `--max-cycles`: optional hard stop for testing or batch windows
- `--disable-queue-cleanup`: disable automatic OOM and old-overflow queue cleanup
- `--disable-queue-cleanup`: disable automatic deterministic OOM/architecture queue cleanup
The first automatic cleanup removes models older than seven days beyond each
account's discovered capacity minus 10. Later scheduled cleanups use capacity
minus 5, while admission continues to reserve the final 10 slots for recent
models. Override these suffix sizes with `MODELHUB_RECENT_MODEL_RESERVE_SLOTS`
and `MODELHUB_DYNAMIC_OLD_MODEL_CLEANUP_RESERVE_SLOTS`.
Model age is admission-only. The default `MODELHUB_RECENT_MODEL_RESERVE_SLOTS=5`
reserves each account's final five known-capacity positions for models updated
within seven days. Queue cleanup reports `ageCleanupMode=admission_only` and the
compatibility field `oldOverflowCount=0`; no restart migration or periodic
date-based stop is performed.
Every successful worker-initiated stop is persisted as `policy_cancelled` in the
outcome store. It is excluded from GPU/framework success rates, local failure
cooldowns, and circuit breakers. If a task races to a real success before the
stop takes effect, that success remains authoritative.
Unclassified failures and failures with `failureScope=unknown` are reported as
`unresolvedFailureCount`. They do not penalize GPU/framework decision success,
open attributable-failure circuits, or create model/GPU cooldowns. Explicitly
classified non-platform failures remain strategy evidence, while platform
failures retain their separate short-circuit handling.
Failure-informed preflight is enabled by default. It rejects deterministic
missing-file and predicted-OOM cases, clamps unsafe context-length arguments,
and records its decisions in `candidatePreflight` and each candidate's