CoolBoy/submmit
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

feat: make model age admission-only

Browse Source
This commit is contained in:
CoolBoy
2026-08-12 10:18:09 +08:00
parent 86088ce577
commit 91d1d3d87d
15 changed files with 438 additions and 653 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
modelhub_submmit_api/README.md
View File

@@ -16,7 +16,7 @@ It currently supports:
- `main.py`: core discovery, scoring, dedup, and submission
- `daily_runner.py`: daily wave orchestration
- `poll_runner.py`: long-running queue refiller
- `queue_cleanup.py`: fail-closed cleanup for certain OOM, architecture, and age policies
- `queue_cleanup.py`: fail-closed cleanup for deterministic OOM and architecture policies
- `runner_common.py`: shared token / key file loading
- `hf_discovery.py`: ModelScope model discovery and inspection (keeps the legacy module name)
- `modelhub_client.py`: ModelHub API client and token-pool routing
@@ -137,14 +137,13 @@ bash run_poll.sh --dry-run
memory rule as new submissions. Only tasks whose own repository size times `1.20` exceeds
their selected GPU capacity are stopped, after a fresh account-scoped active-state check.
Incomplete size/capacity evidence is never used for cancellation.
- Models older than seven days may occupy only the current account capacity minus its final 10
positions. The account pool enforces this per-account boundary atomically and updates it when
capacity probing discovers a higher limit. Cleanup stops OOM tasks first, recalculates the
surviving queue order, and applies the same limit-minus-10 boundary on startup. Scheduled
cleanup relaxes to limit minus 5 to avoid excessive pruning. Age cleanup stops waiting tasks
only; running tasks are protected and rechecked after OOM cleanup, immediately before the
age-only stop batch. Recent
overflow tasks stay, and unknown ModelScope timestamps never authorize a cancellation.
- Models older than seven days may occupy only the current account capacity minus its final 5
positions. The account pool enforces this per-account admission boundary atomically and updates
it when capacity probing discovers a higher limit. A failed active-count read makes that account
ineligible for older models while other readable accounts are still tried. Recent models may use
all available positions. Age policy never cancels an existing waiting or running task; startup
and scheduled cleanup remain limited to deterministic OOM and learned architecture evidence.
Date-deferred candidates are skipped, not failed or permanently excluded.
## Important Flags
@@ -178,19 +177,25 @@ Common flags:
- `--submit-concurrency`: concurrent task submission calls used by each cycle (0 = auto)
- `--post-cycle-cooldown-seconds`: pause after a successful cycle before next cycle (default 2)
- `--max-cycles`: optional hard stop for testing or batch windows
- `--disable-queue-cleanup`: disable automatic OOM and old-overflow queue cleanup
- `--disable-queue-cleanup`: disable automatic deterministic OOM/architecture queue cleanup
The first automatic cleanup removes models older than seven days beyond each
account's discovered capacity minus 10. Later scheduled cleanups use capacity
minus 5, while admission continues to reserve the final 10 slots for recent
models. Override these suffix sizes with `MODELHUB_RECENT_MODEL_RESERVE_SLOTS`
and `MODELHUB_DYNAMIC_OLD_MODEL_CLEANUP_RESERVE_SLOTS`.
Model age is admission-only. The default `MODELHUB_RECENT_MODEL_RESERVE_SLOTS=5`
reserves each account's final five known-capacity positions for models updated
within seven days. Queue cleanup reports `ageCleanupMode=admission_only` and the
compatibility field `oldOverflowCount=0`; no restart migration or periodic
date-based stop is performed.
Every successful worker-initiated stop is persisted as `policy_cancelled` in the
outcome store. It is excluded from GPU/framework success rates, local failure
cooldowns, and circuit breakers. If a task races to a real success before the
stop takes effect, that success remains authoritative.
Unclassified failures and failures with `failureScope=unknown` are reported as
`unresolvedFailureCount`. They do not penalize GPU/framework decision success,
open attributable-failure circuits, or create model/GPU cooldowns. Explicitly
classified non-platform failures remain strategy evidence, while platform
failures retain their separate short-circuit handling.
Failure-informed preflight is enabled by default. It rejects deterministic
missing-file and predicted-OOM cases, clamps unsafe context-length arguments,
and records its decisions in `candidatePreflight` and each candidate's

4
modelhub_submmit_api/daily_runner.py
View File

@@ -88,7 +88,7 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--recent-model-reserve-slots",
type=int,
default=int(os.getenv("MODELHUB_RECENT_MODEL_RESERVE_SLOTS", "10")),
default=int(os.getenv("MODELHUB_RECENT_MODEL_RESERVE_SLOTS", "5")),
help=argparse.SUPPRESS,
)
parser.add_argument(
@@ -252,7 +252,7 @@ def make_wave_namespace(base_args: argparse.Namespace, wave: WaveSpec) -> argpar
capacity_probe_interval_cycles=getattr(base_args, "capacity_probe_interval_cycles", 3),
submit_concurrency=getattr(base_args, "submit_concurrency", 1),
max_submits_per_run=getattr(base_args, "max_submits_per_run", 0),
recent_model_reserve_slots=getattr(base_args, "recent_model_reserve_slots", 10),
recent_model_reserve_slots=getattr(base_args, "recent_model_reserve_slots", 5),
recent_model_days=getattr(base_args, "recent_model_days", 7),
disable_candidate_preflight=getattr(base_args, "disable_candidate_preflight", False),
llm_classifier_endpoint=getattr(base_args, "llm_classifier_endpoint", None),

14
modelhub_submmit_api/gpu_strategy.py
View File

@@ -17,7 +17,7 @@ DEFAULT_GPU_STRATEGY_PATH = Path(".modelhub_state/gpu_strategy.json")
DEFAULT_REFRESH_SUBMISSIONS = 200
DEFAULT_RECENT_TERMINAL_WINDOW = 1000
DEFAULT_LONG_TERM_MIN_SAMPLES = 100
STRATEGY_STATE_VERSION = 3
STRATEGY_STATE_VERSION = 4
LONG_TERM = "long_term"
RECENT = "recent"
@@ -291,6 +291,7 @@ class GPUStrategyManager:
*,
supported_gpus: list[str],
now: datetime | None = None,
history_records: list[dict[str, Any]] | None = None,
) -> dict[str, Any]:
now = now or utc_now()
supported_gpus = list(dict.fromkeys(gpu for gpu in supported_gpus if gpu))
@@ -304,7 +305,11 @@ class GPUStrategyManager:
previous_generation = int(state.get("generation", -1)) if state is not None else -1
self.log(f"[strategy] refresh_start reason={reason} supported_gpus={len(supported_gpus)}")
try:
tasks = self._load_platform_history(client)
tasks = (
list(history_records)
if history_records is not None
else self._load_platform_history(client)
)
refreshed = build_strategy_snapshot(
tasks,
supported_gpus=supported_gpus,
@@ -315,6 +320,11 @@ class GPUStrategyManager:
refresh_submissions=self.refresh_submissions,
)
refreshed["acceptedTotal"] = int((state or {}).get("acceptedTotal") or 0)
refreshed["historySource"] = (
"classified_attributable_outcomes"
if history_records is not None
else "raw_platform_history"
)
self.state = refreshed
write_json(self.path, refreshed)
self._log_state("refreshed")

73
modelhub_submmit_api/main.py
View File

@@ -111,7 +111,7 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--recent-model-reserve-slots",
type=int,
default=int(os.getenv("MODELHUB_RECENT_MODEL_RESERVE_SLOTS", "10")),
default=int(os.getenv("MODELHUB_RECENT_MODEL_RESERVE_SLOTS", "5")),
help=argparse.SUPPRESS,
)
parser.add_argument(
@@ -749,10 +749,15 @@ def submit_candidate(
}
except ModelHubAPIError as exc:
if isinstance(exc, OldModelQueuePolicyError):
print(
f"[submit] deferred repo={candidate['repoId']} gpu={candidate['targetGpu']} "
f"framework={candidate['framework']} reason=age_policy_skipped",
flush=True,
)
return {
"outcome": "old_model_policy_skipped",
"outcome": "age_policy_deferred",
"candidate": candidate,
"reason": "all_accounts_at_old_model_queue_threshold",
"reason": "age_policy_skipped",
}
if is_model_uniqueness_error(exc):
print(
@@ -839,7 +844,7 @@ def run_submission(
clients,
capacity_probe_interval_cycles=max(0, int(getattr(args, "capacity_probe_interval_cycles", 3) or 0)),
capacity_state_path=Path(getattr(args, "capacity_state_path", DEFAULT_CAPACITY_STATE_PATH)),
recent_model_reserve_slots=max(0, int(getattr(args, "recent_model_reserve_slots", 10) or 0)),
recent_model_reserve_slots=max(0, int(getattr(args, "recent_model_reserve_slots", 5) or 0)),
recent_model_days=max(1, int(getattr(args, "recent_model_days", 7) or 7)),
)
@@ -1054,7 +1059,13 @@ def run_submission(
long_term_min_samples=max(1, int(getattr(args, "gpu_strategy_min_long_samples", 100) or 100)),
market_intelligence=market_intelligence,
)
strategy_manager.prepare(modelhub_client, supported_gpus=target_gpus, now=now)
strategy_history_records = outcome_tracker.get_strategy_history_records()
strategy_manager.prepare(
modelhub_client,
supported_gpus=target_gpus,
now=now,
history_records=strategy_history_records,
)
strategy_summary = strategy_manager.summary()
if getattr(args, "skip_history_archive", False):
@@ -1108,17 +1119,18 @@ def run_submission(
pipeline_tags = pipeline_tags_for_task_types(selected_task_types)
explicit_scan_cap = max(0, int(getattr(args, "max_scan_models", 0) or 0))
recent_model_days = max(1, int(getattr(args, "recent_model_days", 7) or 7))
recent_model_reserve_slots = max(0, int(getattr(args, "recent_model_reserve_slots", 10) or 0))
recent_model_reserve_slots = max(0, int(getattr(args, "recent_model_reserve_slots", 5) or 0))
old_model_slots_before_scan: int | None = None
if hasattr(modelhub_client, "old_model_submit_slots"):
old_model_slots_before_scan = int(modelhub_client.old_model_submit_slots())
old_model_queue_thresholds: list[int] | None = None
old_model_queue_thresholds: list[int | None] | None = None
if hasattr(modelhub_client, "old_model_queue_thresholds"):
old_model_queue_thresholds = list(modelhub_client.old_model_queue_thresholds())
allow_older_models_for_scan = old_model_slots_before_scan is None or old_model_slots_before_scan > 0
print(
f"[age-policy] recent_days={recent_model_days} reserve_recent_slots={recent_model_reserve_slots} "
f"account_thresholds={','.join(str(value) for value in old_model_queue_thresholds) if old_model_queue_thresholds is not None else 'n/a'} "
f"[age-policy] mode=admission_only recent_days={recent_model_days} "
f"reserve_recent_slots={recent_model_reserve_slots} "
f"account_thresholds={','.join('n/a' if value is None else str(value) for value in old_model_queue_thresholds) if old_model_queue_thresholds is not None else 'n/a'} "
f"old_model_slots={old_model_slots_before_scan if old_model_slots_before_scan is not None else 'n/a'} "
f"scan_older={'on' if allow_older_models_for_scan else 'off'}",
flush=True,
@@ -1133,6 +1145,7 @@ def run_submission(
):
if candidate_goal <= 0 or len(candidates) >= candidate_goal:
break
skipped_before_stage = len(skipped)
stage_updated_after = stage["updatedAfter"]
stage_limit = int(stage["limit"])
query_kwargs = {
@@ -1149,15 +1162,26 @@ def run_submission(
except TypeError:
models = hf_discovery.list_recent_models(**query_kwargs)
if not allow_older_models_for_scan:
models = [
model
for model in models
recent_models: list[HFModelSummary] = []
for model in models:
if is_recent_model(
model.last_modified,
reference_time=now,
recent_model_days=recent_model_days,
):
recent_models.append(model)
continue
if model.repo_id in seen_model_ids:
continue
seen_model_ids.add(model.repo_id)
skipped.append(
{
"repoId": model.repo_id,
"reason": "age_policy_skipped",
"deferred": True,
}
)
]
models = recent_models
stage_candidates, stage_skipped, stage_failed, processed_count = collect_candidates_from_models(
models=models,
@@ -1186,14 +1210,14 @@ def run_submission(
"newModelsProcessed": processed_count,
"candidatesAdded": len(stage_candidates),
"candidateCountAfterStage": len(candidates),
"skippedAdded": len(stage_skipped),
"skippedAdded": len(skipped) - skipped_before_stage,
"failedAdded": len(stage_failed),
}
scan_stages.append(stage_summary)
print(
f"[scan] stage={stage['name']} discovered={len(models)} new_processed={processed_count} "
f"candidates_added={len(stage_candidates)} candidates_total={len(candidates)}/{candidate_goal} "
f"skipped_added={len(stage_skipped)}",
f"skipped_added={len(skipped) - skipped_before_stage}",
flush=True,
)
@@ -1235,25 +1259,11 @@ def run_submission(
# already-scanned pool until the desired number of real submissions is
# reached or account capacity is genuinely exhausted.
while len(submitted) < target_submit_count and len(attempted_candidates) < max_submit_attempts:
allow_older_models_now = (
bool(modelhub_client.can_submit_old_models())
if hasattr(modelhub_client, "can_submit_old_models")
else True
)
submission_reference_time = utc_now()
remaining_candidates = [
candidate
for candidate in diversified_candidates
if candidate_key(candidate) not in attempted_keys
and not submission_exclusion_store.is_blocked(candidate["repoId"], candidate["targetGpu"])
and (
allow_older_models_now
or is_recent_model(
candidate.get("lastModified"),
reference_time=submission_reference_time,
recent_model_days=recent_model_days,
)
)
]
remaining_candidates = one_candidate_per_model(remaining_candidates)
desired_count = min(
@@ -1328,13 +1338,13 @@ def run_submission(
}
)
continue
if result["outcome"] == "old_model_policy_skipped":
if result["outcome"] == "age_policy_deferred":
batch_policy_skipped_candidates.append(candidate)
skipped.append(
{
"repoId": candidate["repoId"],
"targetGpu": candidate["targetGpu"],
"reason": "all_accounts_at_old_model_queue_threshold",
"reason": "age_policy_skipped",
}
)
continue
@@ -1428,6 +1438,7 @@ def run_submission(
"marketIntelligence": market_summary,
"candidatePreflight": preflight_summary,
"agePolicy": {
"mode": "admission_only",
"recentModelDays": recent_model_days,
"recentModelReserveSlots": recent_model_reserve_slots,
"oldModelSubmitThresholds": old_model_queue_thresholds,

45
modelhub_submmit_api/modelhub_client.py
View File

@@ -407,7 +407,7 @@ class ModelHubClientPool:
configured_recent_reserve = (
recent_model_reserve_slots
if recent_model_reserve_slots is not None
else os.getenv("MODELHUB_RECENT_MODEL_RESERVE_SLOTS", "10")
else os.getenv("MODELHUB_RECENT_MODEL_RESERVE_SLOTS", "5")
)
configured_recent_days = (
recent_model_days
@@ -434,6 +434,10 @@ class ModelHubClientPool:
self._active_counts_ttl = max(1.0, float(configured_ttl))
self._reservation_ttl = max(self._active_counts_ttl * 2, float(configured_reservation_ttl))
self._remote_counts: list[int] = [0 for _ in clients]
# Old-model admission is fail-closed per account. A configured or
# persisted capacity is not enough without a successful active-count
# read for the current scheduling window.
self._count_known: list[bool] = [False for _ in clients]
self._active_refresh_at: float = 0.0
self._counts_initialized = False
self._state_lock = threading.Lock()
@@ -518,11 +522,11 @@ class ModelHubClientPool:
with self._state_lock:
return self._capacity_probe_enabled
def _safe_count_active_tasks(self, client: ModelHubClient, max_count: int) -> int:
def _safe_count_active_tasks(self, client: ModelHubClient, max_count: int) -> int | None:
try:
return client.count_active_tasks(max_count=max_count, page_size=200)
except Exception:
return max(0, max_count - 1)
return None
def _safe_list_tasks(self, client: ModelHubClient, kwargs: dict[str, Any]) -> list[dict[str, Any]]:
try:
@@ -550,10 +554,10 @@ class ModelHubClientPool:
with self._state_lock:
count_limits = [cap + 1 for cap in self._account_caps]
def _to_indexed_result(index: int, client: ModelHubClient) -> tuple[int, int]:
def _to_indexed_result(index: int, client: ModelHubClient) -> tuple[int, int | None]:
return index, self._safe_count_active_tasks(client, count_limits[index])
results: list[tuple[int, int]] = []
results: list[tuple[int, int | None]] = []
with ThreadPoolExecutor(max_workers=min(len(self.clients), 12)) as executor:
futures = {
executor.submit(_to_indexed_result, index, client): index
@@ -564,14 +568,18 @@ class ModelHubClientPool:
try:
results.append(future.result())
except Exception:
results.append((index, self.active_task_cap))
results.append((index, None))
refreshed_at = time.monotonic()
caps_changed = False
with self._state_lock:
for index, count in results:
if count is None:
self._count_known[index] = False
continue
old_remote_count = self._remote_counts[index]
new_remote_count = max(0, int(count))
self._count_known[index] = True
if new_remote_count > self._account_caps[index]:
self._account_caps[index] = new_remote_count
caps_changed = True
@@ -725,8 +733,11 @@ class ModelHubClientPool:
if index not in excluded
and (
reserve_capacity_slots is None
or self._effective_count_locked(index)
< max(0, self._account_caps[index] - reserve_capacity_slots)
or (
self._count_known[index]
and self._effective_count_locked(index)
< max(0, self._account_caps[index] - reserve_capacity_slots)
)
)
}
usable = {index: remaining for index, remaining in remaining_by_index.items() if remaining > 0}
@@ -765,8 +776,11 @@ class ModelHubClientPool:
and self._effective_count_locked(index) >= self._account_caps[index]
and (
reserve_capacity_slots is None
or self._effective_count_locked(index)
< max(0, self._account_caps[index] - reserve_capacity_slots)
or (
self._count_known[index]
and self._effective_count_locked(index)
< max(0, self._account_caps[index] - reserve_capacity_slots)
)
)
]
if not eligible:
@@ -810,6 +824,7 @@ class ModelHubClientPool:
self._account_caps[index] = observed_capacity
cap_changed = True
self._remote_counts[index] = self._account_caps[index]
self._count_known[index] = True
self._counts_initialized = True
self._active_refresh_at = time.monotonic()
if cap_changed:
@@ -834,15 +849,19 @@ class ModelHubClientPool:
self._old_model_count_limit_locked(index) - self._effective_count_locked(index),
)
for index in range(len(self.clients))
if self._count_known[index]
)
def _old_model_count_limit_locked(self, index: int) -> int:
return max(0, self._account_caps[index] - self.recent_model_reserve_slots)
def old_model_queue_thresholds(self) -> list[int]:
"""Return each account's current cap minus its recent-model reserve."""
def old_model_queue_thresholds(self) -> list[int | None]:
"""Return thresholds, or None when old-model admission must fail closed."""
with self._state_lock:
return [self._old_model_count_limit_locked(index) for index in range(len(self.clients))]
return [
self._old_model_count_limit_locked(index) if self._count_known[index] else None
for index in range(len(self.clients))
]
def can_submit_old_models(self) -> bool:
return self.old_model_submit_slots() > 0

54
modelhub_submmit_api/outcome_tracker.py
View File

@@ -76,6 +76,28 @@ class OutcomeTracker:
}
return contexts
def get_strategy_history_records(self) -> list[dict[str, Any]]:
"""Expose only successes and evidence-attributable failures for GPU ranking."""
records: list[dict[str, Any]] = []
for record in self._records:
outcome = record.get("outcome")
if outcome == "success":
verify_result = 1
elif _is_attributable_failure(record):
verify_result = -1
else:
continue
records.append(
{
**record,
"gpuType": record.get("targetGpu"),
"status": "success",
"verifyResult": verify_result,
"updateTime": record.get("lastSyncTime") or record.get("submitTime"),
}
)
return records
def _rebuild_indexes(self) -> None:
self._by_task_id.clear()
self._by_model_gpu.clear()
@@ -619,10 +641,13 @@ class OutcomeTracker:
self._failed_model_gpus.clear()
latest_by_combo: dict[tuple[str, str], tuple[datetime, dict[str, Any]]] = {}
for record in self._records:
# Infrastructure failures and our own policy cancellations neither
# clear nor create a model/GPU cooldown. Look through them to the
# latest attributable outcome.
if _is_platform_failure(record) or _is_policy_cancelled(record):
# Platform, unresolved, and policy outcomes neither clear nor
# create a model/GPU cooldown. The full-history audit showed that
# more than half of failures lack enough evidence for attribution.
if _is_policy_cancelled(record) or (
record.get("outcome") == "failed"
and not _is_attributable_failure(record)
):
continue
model_id = record.get("modelId") or ""
target_gpu = record.get("targetGpu") or ""
@@ -701,10 +726,13 @@ def _summarize(records: list[dict[str, Any]]) -> dict[str, Any]:
failure_count = sum(1 for r in records if r.get("outcome") == "failed")
pending_count = sum(1 for r in records if r.get("outcome") == "pending")
attributable_failure_count = sum(
1 for record in records
if record.get("outcome") == "failed" and not _is_platform_failure(record)
1 for record in records if _is_attributable_failure(record)
)
platform_failure_count = sum(1 for record in records if _is_platform_failure(record))
unresolved_failure_count = max(
0,
failure_count - attributable_failure_count - platform_failure_count,
)
platform_failure_count = failure_count - attributable_failure_count
decision_total = success_count + attributable_failure_count
failure_breakdown: dict[str, int] = defaultdict(int)
@@ -719,6 +747,7 @@ def _summarize(records: list[dict[str, Any]]) -> dict[str, Any]:
"failureCount": failure_count,
"attributableFailureCount": attributable_failure_count,
"platformFailureCount": platform_failure_count,
"unresolvedFailureCount": unresolved_failure_count,
"decisionTotal": decision_total,
"pendingCount": pending_count,
"successRate": round(success_count / total, 4) if total > 0 else 0.0,
@@ -892,6 +921,15 @@ def _is_platform_failure(record: dict[str, Any]) -> bool:
return scope == "platform" or category.startswith("platform_")
def _is_attributable_failure(record: dict[str, Any]) -> bool:
"""Require classified, non-platform evidence before penalizing a strategy."""
if record.get("outcome") != "failed" or _is_platform_failure(record):
return False
category = str(record.get("failureCategory") or "").strip().lower()
scope = str(record.get("failureScope") or "").strip().lower()
return bool(category and scope not in {"", "unknown", "platform"})
def _is_policy_cancelled(record: dict[str, Any]) -> bool:
return bool(record.get("policyCancelled")) or record.get("outcome") == "policy_cancelled"
@@ -901,7 +939,7 @@ def _consecutive_attributable_failures(records: list[dict[str, Any]]) -> int:
for record in records:
if record.get("outcome") == "success":
break
if record.get("outcome") == "failed" and not _is_platform_failure(record):
if _is_attributable_failure(record):
count += 1
return count

53
modelhub_submmit_api/poll_runner.py
View File

@@ -74,7 +74,7 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--recent-model-reserve-slots",
type=int,
default=int(os.getenv("MODELHUB_RECENT_MODEL_RESERVE_SLOTS", "10")),
default=int(os.getenv("MODELHUB_RECENT_MODEL_RESERVE_SLOTS", "5")),
help=argparse.SUPPRESS,
)
parser.add_argument(
@@ -83,12 +83,6 @@ def build_parser() -> argparse.ArgumentParser:
default=int(os.getenv("MODELHUB_RECENT_MODEL_DAYS", "7")),
help=argparse.SUPPRESS,
)
parser.add_argument(
"--dynamic-old-model-cleanup-reserve-slots",
type=int,
default=int(os.getenv("MODELHUB_DYNAMIC_OLD_MODEL_CLEANUP_RESERVE_SLOTS", "5")),
help=argparse.SUPPRESS,
)
parser.add_argument("--disable-candidate-preflight", action="store_true", help=argparse.SUPPRESS)
parser.add_argument("--llm-classifier-endpoint", default=os.getenv("MODELHUB_LLM_CLASSIFIER_ENDPOINT"), help=argparse.SUPPRESS)
parser.add_argument("--llm-classifier-model", default=os.getenv("MODELHUB_LLM_CLASSIFIER_MODEL"), help=argparse.SUPPRESS)
@@ -245,33 +239,11 @@ def _build_modelhub_client(base_args: argparse.Namespace) -> ModelHubClientPool:
clients,
capacity_probe_interval_cycles=max(0, int(getattr(base_args, "capacity_probe_interval_cycles", 3) or 0)),
capacity_state_path=Path(getattr(base_args, "capacity_state_path", DEFAULT_CAPACITY_STATE_PATH)),
recent_model_reserve_slots=max(0, int(getattr(base_args, "recent_model_reserve_slots", 10) or 0)),
recent_model_reserve_slots=max(0, int(getattr(base_args, "recent_model_reserve_slots", 5) or 0)),
recent_model_days=max(1, int(getattr(base_args, "recent_model_days", 7) or 7)),
)
def resolve_age_cleanup_policy(
base_args: argparse.Namespace,
*,
initial_cleanup_pending: bool,
) -> tuple[str, int]:
"""Reserve ten slots initially, then use a five-slot cleanup hysteresis."""
initial_reserve_slots = max(
0,
int(getattr(base_args, "recent_model_reserve_slots", 10) or 0),
)
dynamic_reserve_slots = min(
initial_reserve_slots,
max(
0,
int(getattr(base_args, "dynamic_old_model_cleanup_reserve_slots", 5) or 0),
),
)
if initial_cleanup_pending:
return "initial", initial_reserve_slots
return "dynamic", dynamic_reserve_slots
def _persist_architecture_blacklist(
report: dict[str, Any],
*,
@@ -554,7 +526,6 @@ def run_poll_loop(
submitted_total = 0
cycles = 0
stopped_reason = "max_cycles_reached"
initial_age_cleanup_pending = True
pending_architecture_cleanup = False
last_cleaned_architecture_blocks: set[str] = set()
@@ -645,14 +616,9 @@ def run_poll_loop(
)
if active_architecture_block_keys - last_cleaned_architecture_blocks:
pending_architecture_cleanup = True
age_cleanup_mode, age_cleanup_reserve_slots = resolve_age_cleanup_policy(
base_args,
initial_cleanup_pending=initial_age_cleanup_pending,
)
log(
f"[queue-cleanup] mode={'architecture_dynamic' if architecture_only_cleanup else age_cleanup_mode} "
f"reserve_recent_slots={age_cleanup_reserve_slots} "
f"recent_days={max(1, int(getattr(base_args, 'recent_model_days', 7) or 7))}"
"[queue-cleanup] age_cleanup=disabled_admission_only "
f"deterministic_cleanup={'architecture_only' if architecture_only_cleanup else 'full'}"
)
cleanup_summary = cleanup_certain_oom_tasks(
modelhub_client,
@@ -672,7 +638,6 @@ def run_poll_loop(
)
),
architecture_only=architecture_only_cleanup,
age_reserved_slots=age_cleanup_reserve_slots,
log=log,
)
policy_cancelled_recorded = outcome_tracker.mark_policy_cancellations(
@@ -697,9 +662,13 @@ def run_poll_loop(
"mode": (
"architecture_dynamic"
if architecture_only_cleanup
else age_cleanup_mode
else "deterministic_full"
),
"ageCleanupMode": "admission_only",
"ageReservedSlots": max(
0,
int(getattr(base_args, "recent_model_reserve_slots", 5) or 0),
),
"ageReservedSlots": age_cleanup_reserve_slots,
"ageQueueThresholds": cleanup_summary["oldModelQueueThresholds"],
"activeScanned": cleanup_summary["activeScanned"],
"certainOomCount": cleanup_summary["certainOomCount"],
@@ -715,8 +684,6 @@ def run_poll_loop(
"stopErrorCount": len(cleanup_summary["stopErrors"]),
}
)
if not architecture_only_cleanup:
initial_age_cleanup_pending = False
pending_architecture_cleanup = False
last_cleaned_architecture_blocks = active_architecture_block_keys
except Exception as exc:

284
modelhub_submmit_api/queue_cleanup.py
View File

@@ -3,13 +3,12 @@ from __future__ import annotations
import argparse
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any, Callable, Iterable
from architecture_compatibility import architecture_compatibility_key, architecture_profiles
from candidate_preflight import CandidatePreflightAdvisor, MODEL_LOAD_OVERHEAD
from common import utc_now, write_json
from common import write_json
from hf_discovery import HuggingFaceDiscovery, inspect_repo_tree
from modelhub_client import ModelHubClient, ModelHubClientPool
from runner_common import DEFAULT_KEY_PATH, ensure_tokens
@@ -168,41 +167,6 @@ def _load_repository_sizes(
return sizes, errors
def _load_model_last_modified(
model_ids: set[str],
*,
discovery: HuggingFaceDiscovery,
read_concurrency: int,
log: Callable[[str], None],
) -> tuple[dict[str, datetime], dict[str, str]]:
values: dict[str, datetime] = {}
errors: dict[str, str] = {}
completed = 0
workers = min(max(1, int(read_concurrency)), max(1, len(model_ids)))
with ThreadPoolExecutor(max_workers=workers) as executor:
futures = {
executor.submit(discovery.get_model_last_modified, model_id): model_id
for model_id in sorted(model_ids)
}
for future in as_completed(futures):
model_id = futures[future]
try:
value = future.result()
if value is None:
errors[model_id] = "model_last_modified_unknown"
else:
values[model_id] = value
except Exception as exc:
errors[model_id] = f"{type(exc).__name__}: {exc}"
completed += 1
if completed == len(model_ids) or completed % 50 == 0:
log(
f"[queue-cleanup] age_scan={completed}/{len(model_ids)} "
f"complete={len(values)} unknown={len(errors)}"
)
return values, errors
def _load_model_configs(
model_ids: set[str],
*,
@@ -480,74 +444,6 @@ def find_architecture_incompatible_tasks(
return decisions, skipped
def find_old_overflow_tasks(
tasks: list[OwnedTask],
*,
model_last_modified: dict[str, datetime],
queue_threshold: int | dict[int, int] = 80,
recent_model_days: int = 7,
reference_time: datetime | None = None,
incomplete_accounts: set[int] | None = None,
) -> tuple[list[dict[str, Any]], dict[str, int]]:
recent_days = max(1, int(recent_model_days))
cutoff = (reference_time or utc_now()) - timedelta(days=recent_days)
incomplete_accounts = incomplete_accounts or set()
grouped: dict[int, list[OwnedTask]] = {}
for task in tasks:
grouped.setdefault(task.account_index, []).append(task)
decisions: list[dict[str, Any]] = []
skipped = {
"accountsWithIncompleteListing": len(incomplete_accounts),
"withinFirstQueuePositions": 0,
"recentOverflowTasks": 0,
"modelAgeUnknown": 0,
"accountThresholdUnknown": 0,
"runningOverflowProtected": 0,
}
for account_index, account_tasks in sorted(grouped.items()):
if account_index in incomplete_accounts:
continue
if isinstance(queue_threshold, dict):
configured_threshold = queue_threshold.get(account_index)
if configured_threshold is None:
skipped["accountThresholdUnknown"] += 1
continue
else:
configured_threshold = queue_threshold
threshold = max(0, int(configured_threshold))
ordered = sorted(account_tasks, key=lambda item: item.task_id)
skipped["withinFirstQueuePositions"] += min(threshold, len(ordered))
for position, task in enumerate(ordered, start=1):
if position <= threshold:
continue
if task.status != "waiting":
skipped["runningOverflowProtected"] += 1
continue
last_modified = model_last_modified.get(task.model_id)
if last_modified is None:
skipped["modelAgeUnknown"] += 1
continue
if last_modified >= cutoff:
skipped["recentOverflowTasks"] += 1
continue
decisions.append(
{
"accountIndex": account_index + 1,
"taskId": task.task_id,
"modelId": task.model_id,
"gpuType": task.gpu_type,
"status": task.status,
"queuePosition": position,
"queueThreshold": threshold,
"modelLastModified": last_modified.isoformat(),
"recentCutoff": cutoff.isoformat(),
"reason": "old_model_beyond_account_queue_threshold",
}
)
return decisions, skipped
def _chunks(values: list[int], size: int) -> Iterable[list[int]]:
for offset in range(0, len(values), size):
yield values[offset : offset + size]
@@ -564,18 +460,11 @@ def cleanup_certain_oom_tasks(
architecture_compatibility_blocks: dict[str, dict[str, Any]] | None = None,
task_compatibility_contexts: dict[str, dict[str, Any]] | None = None,
architecture_only: bool = False,
age_reserved_slots: int | None = None,
reference_time: datetime | None = None,
log: Callable[[str], None] = print,
) -> dict[str, Any]:
"""Stop deterministic OOM/architecture tasks and over-threshold old tasks."""
"""Stop only deterministically impossible OOM/architecture tasks."""
clients = list(modelhub.clients)
reference_time = reference_time or utc_now()
configured_reserved_slots = (
age_reserved_slots
if age_reserved_slots is not None
else getattr(modelhub, "recent_model_reserve_slots", 10)
)
configured_reserved_slots = getattr(modelhub, "recent_model_reserve_slots", 5)
reserved_slots = max(0, int(configured_reserved_slots or 0))
recent_model_days = max(1, int(getattr(modelhub, "recent_model_days", 7) or 7))
tasks, listing_errors = collect_active_tasks(clients, read_concurrency=read_concurrency)
@@ -604,6 +493,11 @@ def cleanup_certain_oom_tasks(
index: max(0, int(account_caps[index]) - reserved_slots)
for index in range(min(len(clients), len(account_caps)))
}
log(
f"[queue-cleanup] age_cleanup=disabled deterministic_cleanup=enabled "
f"age_policy=admission_only reserve_recent_slots={reserved_slots} "
f"recent_days={recent_model_days}"
)
model_ids = {task.model_id for task in tasks}
repository_sizes: dict[str, int] = {}
@@ -717,62 +611,6 @@ def cleanup_certain_oom_tasks(
f"running_protected={architecture_skipped['runningMatchedProtected']}"
)
deterministic_task_keys = {
(int(decision["accountIndex"]) - 1, int(decision["taskId"]))
for decision in [*oom_decisions, *architecture_decisions]
}
# Deterministically impossible tasks are stopped first. Rank the age-policy
# queue as it will look afterwards to avoid over-cancelling old models.
age_rank_tasks = [
task
for task in tasks
if (task.account_index, task.task_id) not in deterministic_task_keys
]
overflow_model_ids = {
task.model_id
for account_index in range(len(clients))
if account_index not in listing_errors
for task in sorted(
(item for item in age_rank_tasks if item.account_index == account_index),
key=lambda item: item.task_id,
)[queue_thresholds.get(account_index, len(age_rank_tasks)):]
if task.status == "waiting"
}
model_last_modified: dict[str, datetime] = {}
age_errors: dict[str, str] = {}
old_overflow_decisions: list[dict[str, Any]] = []
age_skipped = {
"accountsWithIncompleteListing": len(listing_errors),
"withinFirstQueuePositions": 0,
"recentOverflowTasks": 0,
"modelAgeUnknown": 0,
"accountThresholdUnknown": 0,
"runningOverflowProtected": 0,
}
if not architecture_only:
model_last_modified, age_errors = _load_model_last_modified(
overflow_model_ids,
discovery=discovery,
read_concurrency=read_concurrency,
log=log,
)
old_overflow_decisions, age_skipped = find_old_overflow_tasks(
age_rank_tasks,
model_last_modified=model_last_modified,
queue_threshold=queue_thresholds,
recent_model_days=recent_model_days,
reference_time=reference_time,
incomplete_accounts=set(listing_errors),
)
log(
f"[queue-cleanup] old_overflow={len(old_overflow_decisions)} "
f"thresholds={','.join(str(queue_thresholds[index]) for index in sorted(queue_thresholds))} "
f"reserve_recent_slots={reserved_slots} recent_days={recent_model_days} "
f"recent_overflow={age_skipped['recentOverflowTasks']} "
f"age_unknown={age_skipped['modelAgeUnknown']} "
f"running_protected={age_skipped['runningOverflowProtected']}"
)
decisions_by_key: dict[tuple[int, int], dict[str, Any]] = {}
for decision in [*oom_decisions, *architecture_decisions]:
enriched = dict(decision)
@@ -790,17 +628,6 @@ def cleanup_certain_oom_tasks(
if field not in {"reason", "cleanupReasons"} and value is not None
}
)
for decision in old_overflow_decisions:
key = (int(decision["accountIndex"]), int(decision["taskId"]))
existing = decisions_by_key.get(key)
if existing is None:
enriched = dict(decision)
enriched["cleanupReasons"] = [decision["reason"]]
decisions_by_key[key] = enriched
continue
existing["cleanupReasons"].append(decision["reason"])
for field in ("queuePosition", "queueThreshold", "modelLastModified", "recentCutoff"):
existing[field] = decision[field]
decisions = sorted(
decisions_by_key.values(),
key=lambda item: (int(item["accountIndex"]), int(item["taskId"])),
@@ -815,26 +642,10 @@ def cleanup_certain_oom_tasks(
# query fails for that account, fail closed and do not terminate its tasks.
refreshed_tasks, refresh_errors = collect_active_tasks(clients, read_concurrency=read_concurrency)
active_ids_by_account: dict[int, set[int]] = {}
active_positions_by_account: dict[int, dict[int, int]] = {}
active_status_by_account: dict[int, dict[int, str]] = {}
for task in refreshed_tasks:
active_ids_by_account.setdefault(task.account_index, set()).add(task.task_id)
active_status_by_account.setdefault(task.account_index, {})[task.task_id] = task.status
for account_index in range(len(clients)):
planned_deterministic_ids = {
int(decision["taskId"])
for decision in [*oom_decisions, *architecture_decisions]
if int(decision["accountIndex"]) - 1 == account_index
}
ordered_ids = sorted(
task.task_id
for task in refreshed_tasks
if task.account_index == account_index
and task.task_id not in planned_deterministic_ids
)
active_positions_by_account[account_index] = {
task_id: position for position, task_id in enumerate(ordered_ids, start=1)
}
for account_index, details in sorted(refresh_errors.items()):
stop_errors.append(
@@ -854,45 +665,20 @@ def cleanup_certain_oom_tasks(
disappeared.append(decision)
continue
cleanup_reasons = set(decision.get("cleanupReasons") or [decision.get("reason")])
age_only = cleanup_reasons == {"old_model_beyond_account_queue_threshold"}
architecture_without_oom = bool(
"known_framework_architecture_incompatible" in cleanup_reasons
and "certain_oom_repository_size_exceeds_gpu_capacity" not in cleanup_reasons
)
current_position = active_positions_by_account.get(account_index, {}).get(int(decision["taskId"]))
current_status = active_status_by_account.get(account_index, {}).get(int(decision["taskId"]))
account_queue_threshold = queue_thresholds.get(account_index)
if age_only and (
current_position is None
or account_queue_threshold is None
or current_position <= account_queue_threshold
or current_status != "waiting"
):
policy_no_longer_applies.append(
{
**decision,
"recheckedQueuePosition": current_position,
"recheckedStatus": current_status,
"policyChangeReason": (
"task_started_running"
if current_status == "running"
else "queue_position_or_status_changed"
),
}
)
continue
if architecture_without_oom and current_status != "waiting":
policy_no_longer_applies.append(
{
**decision,
"recheckedQueuePosition": current_position,
"recheckedStatus": current_status,
"policyChangeReason": "task_started_running",
}
)
continue
if current_position is not None:
decision["recheckedQueuePosition"] = current_position
by_account.setdefault(account_index, []).append(decision)
batch_size = max(1, min(100, int(stop_batch_size)))
@@ -901,7 +687,7 @@ def cleanup_certain_oom_tasks(
if stop_failed:
break
decisions_by_id = {int(item["taskId"]): item for item in by_account[account_index]}
for cleanup_phase in ("oom", "architecture", "age"):
for cleanup_phase in ("oom", "architecture"):
task_ids = sorted(
task_id
for task_id, decision in decisions_by_id.items()
@@ -917,20 +703,10 @@ def cleanup_certain_oom_tasks(
and "known_framework_architecture_incompatible"
in decision["cleanupReasons"]
)
or (
cleanup_phase == "age"
and "certain_oom_repository_size_exceeds_gpu_capacity"
not in decision["cleanupReasons"]
and "known_framework_architecture_incompatible"
not in decision["cleanupReasons"]
and "old_model_beyond_account_queue_threshold"
in decision["cleanupReasons"]
)
)
if cleanup_phase != "oom" and task_ids:
# Earlier deterministic stops can change positions, and a
# waiting architecture/age task can start running after the
# account-wide recheck. Re-read before each later phase.
# A waiting architecture task can start running after the
# account-wide recheck. Re-read before this protected phase.
try:
phase_tasks: dict[int, OwnedTask] = {}
for status in ACTIVE_FILTER_STATUSES:
@@ -951,18 +727,12 @@ def cleanup_certain_oom_tasks(
stop_failed = True
break
phase_positions = {
task_id: position
for position, task_id in enumerate(sorted(phase_tasks), start=1)
}
eligible_task_ids: list[int] = []
account_queue_threshold = queue_thresholds.get(account_index)
for task_id in task_ids:
current_task = phase_tasks.get(task_id)
if current_task is None:
disappeared.append(decisions_by_id[task_id])
continue
current_position = phase_positions.get(task_id)
reasons = set(
decisions_by_id[task_id].get("cleanupReasons")
or [decisions_by_id[task_id].get("reason")]
@@ -972,29 +742,19 @@ def cleanup_certain_oom_tasks(
and "known_framework_architecture_incompatible" in reasons
and current_task.status == "waiting"
)
age_applies = bool(
cleanup_phase == "age"
and "old_model_beyond_account_queue_threshold" in reasons
and account_queue_threshold is not None
and current_position is not None
and current_position > account_queue_threshold
and current_task.status == "waiting"
)
if not architecture_applies and not age_applies:
if not architecture_applies:
policy_no_longer_applies.append(
{
**decisions_by_id[task_id],
"recheckedQueuePosition": current_position,
"recheckedStatus": current_task.status,
"policyChangeReason": (
"task_started_running"
if current_task.status == "running"
else "queue_position_status_or_policy_changed"
else "task_no_longer_architecture_cleanup_eligible"
),
}
)
continue
decisions_by_id[task_id]["recheckedQueuePosition"] = current_position
eligible_task_ids.append(task_id)
task_ids = eligible_task_ids
@@ -1030,8 +790,9 @@ def cleanup_certain_oom_tasks(
"uniqueModels": len(model_ids),
"repositorySizesComplete": len(repository_sizes),
"repositorySizeErrors": size_errors,
"modelAgeMetadataComplete": len(model_last_modified),
"modelAgeErrors": age_errors,
"ageCleanupMode": "admission_only",
"modelAgeMetadataComplete": 0,
"modelAgeErrors": {},
"listingErrors": {str(index + 1): values for index, values in listing_errors.items()},
"certainOomCount": len(oom_decisions),
"certainOomTasks": oom_decisions,
@@ -1046,8 +807,8 @@ def cleanup_certain_oom_tasks(
},
"architectureFrameworkCatalogErrors": framework_catalog_errors,
"architecturePolicySkipped": architecture_skipped,
"oldOverflowCount": len(old_overflow_decisions),
"oldOverflowTasks": old_overflow_decisions,
"oldOverflowCount": 0,
"oldOverflowTasks": [],
"oldModelQueueThresholds": [
queue_thresholds.get(index) for index in range(len(clients))
],
@@ -1057,7 +818,10 @@ def cleanup_certain_oom_tasks(
],
"recentModelReserveSlots": reserved_slots,
"recentModelDays": recent_model_days,
"agePolicySkipped": age_skipped,
"agePolicySkipped": {
"cleanupDisabled": True,
"reason": "admission_only",
},
"cleanupCandidateCount": len(decisions),
"skipped": skipped,
"cancelledCount": len(cancelled),
@@ -1071,7 +835,7 @@ def cleanup_certain_oom_tasks(
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
description="Safely stop deterministic OOM/architecture and over-threshold old ModelHub tasks."
description="Safely stop only deterministic OOM/architecture ModelHub tasks."
)
parser.add_argument("--execute", action="store_true", help="Actually terminate selected tasks; otherwise only print a preview")
parser.add_argument("--read-concurrency", type=int, default=6)
@@ -1105,7 +869,7 @@ def main(argv: list[str] | None = None) -> int:
print(
f"[queue-cleanup] finished certain_oom={summary['certainOomCount']} "
f"architecture_incompatible={summary['architectureIncompatibleCount']} "
f"old_overflow={summary['oldOverflowCount']} "
f"age_cleanup={summary['ageCleanupMode']} "
f"cancelled={summary['cancelledCount']} stop_errors={len(summary['stopErrors'])} "
f"report={report_path}",
flush=True,

2
modelhub_submmit_api/version.py
View File

@@ -1 +1 @@
AGENT_VERSION = "2026.08.12.5"
AGENT_VERSION = "2026.08.12.6"