feat: make model age admission-only
This commit is contained in:
@@ -16,7 +16,7 @@ It currently supports:
|
||||
- `main.py`: core discovery, scoring, dedup, and submission
|
||||
- `daily_runner.py`: daily wave orchestration
|
||||
- `poll_runner.py`: long-running queue refiller
|
||||
- `queue_cleanup.py`: fail-closed cleanup for certain OOM, architecture, and age policies
|
||||
- `queue_cleanup.py`: fail-closed cleanup for deterministic OOM and architecture policies
|
||||
- `runner_common.py`: shared token / key file loading
|
||||
- `hf_discovery.py`: ModelScope model discovery and inspection (keeps the legacy module name)
|
||||
- `modelhub_client.py`: ModelHub API client and token-pool routing
|
||||
@@ -137,14 +137,13 @@ bash run_poll.sh --dry-run
|
||||
memory rule as new submissions. Only tasks whose own repository size times `1.20` exceeds
|
||||
their selected GPU capacity are stopped, after a fresh account-scoped active-state check.
|
||||
Incomplete size/capacity evidence is never used for cancellation.
|
||||
- Models older than seven days may occupy only the current account capacity minus its final 10
|
||||
positions. The account pool enforces this per-account boundary atomically and updates it when
|
||||
capacity probing discovers a higher limit. Cleanup stops OOM tasks first, recalculates the
|
||||
surviving queue order, and applies the same limit-minus-10 boundary on startup. Scheduled
|
||||
cleanup relaxes to limit minus 5 to avoid excessive pruning. Age cleanup stops waiting tasks
|
||||
only; running tasks are protected and rechecked after OOM cleanup, immediately before the
|
||||
age-only stop batch. Recent
|
||||
overflow tasks stay, and unknown ModelScope timestamps never authorize a cancellation.
|
||||
- Models older than seven days may occupy only the current account capacity minus its final 5
|
||||
positions. The account pool enforces this per-account admission boundary atomically and updates
|
||||
it when capacity probing discovers a higher limit. A failed active-count read makes that account
|
||||
ineligible for older models while other readable accounts are still tried. Recent models may use
|
||||
all available positions. Age policy never cancels an existing waiting or running task; startup
|
||||
and scheduled cleanup remain limited to deterministic OOM and learned architecture evidence.
|
||||
Date-deferred candidates are skipped, not failed or permanently excluded.
|
||||
|
||||
## Important Flags
|
||||
|
||||
@@ -178,19 +177,25 @@ Common flags:
|
||||
- `--submit-concurrency`: concurrent task submission calls used by each cycle (0 = auto)
|
||||
- `--post-cycle-cooldown-seconds`: pause after a successful cycle before next cycle (default 2)
|
||||
- `--max-cycles`: optional hard stop for testing or batch windows
|
||||
- `--disable-queue-cleanup`: disable automatic OOM and old-overflow queue cleanup
|
||||
- `--disable-queue-cleanup`: disable automatic deterministic OOM/architecture queue cleanup
|
||||
|
||||
The first automatic cleanup removes models older than seven days beyond each
|
||||
account's discovered capacity minus 10. Later scheduled cleanups use capacity
|
||||
minus 5, while admission continues to reserve the final 10 slots for recent
|
||||
models. Override these suffix sizes with `MODELHUB_RECENT_MODEL_RESERVE_SLOTS`
|
||||
and `MODELHUB_DYNAMIC_OLD_MODEL_CLEANUP_RESERVE_SLOTS`.
|
||||
Model age is admission-only. The default `MODELHUB_RECENT_MODEL_RESERVE_SLOTS=5`
|
||||
reserves each account's final five known-capacity positions for models updated
|
||||
within seven days. Queue cleanup reports `ageCleanupMode=admission_only` and the
|
||||
compatibility field `oldOverflowCount=0`; no restart migration or periodic
|
||||
date-based stop is performed.
|
||||
|
||||
Every successful worker-initiated stop is persisted as `policy_cancelled` in the
|
||||
outcome store. It is excluded from GPU/framework success rates, local failure
|
||||
cooldowns, and circuit breakers. If a task races to a real success before the
|
||||
stop takes effect, that success remains authoritative.
|
||||
|
||||
Unclassified failures and failures with `failureScope=unknown` are reported as
|
||||
`unresolvedFailureCount`. They do not penalize GPU/framework decision success,
|
||||
open attributable-failure circuits, or create model/GPU cooldowns. Explicitly
|
||||
classified non-platform failures remain strategy evidence, while platform
|
||||
failures retain their separate short-circuit handling.
|
||||
|
||||
Failure-informed preflight is enabled by default. It rejects deterministic
|
||||
missing-file and predicted-OOM cases, clamps unsafe context-length arguments,
|
||||
and records its decisions in `candidatePreflight` and each candidate's
|
||||
|
||||
Reference in New Issue
Block a user