diff --git a/README.md b/README.md index 1d05e72b01..43a497fb99 100644 --- a/README.md +++ b/README.md @@ -107,8 +107,10 @@ model/GPU uniqueness response, the entire submission cycle pauses fail-closed. For each compatible model/GPU pair, the worker also ranks the GPU's supported frameworks using ModelHub's public aggregate `modelCount` and `successCount` data, then blends in the worker's own GPU+framework outcomes with a capped weight. -Only frameworks with at least 300 public samples and a safe Wilson lower bound -are eligible. Framework statistics refresh +Only frameworks with at least 300 public samples and a Wilson lower bound of at +least 20% are eligible. GPU pools normally require a 15% lower bound; a +low-sample pool can remain available only when it has a recent success and a +framework that passes the stricter framework gate. Framework statistics refresh every 6 hours, so they do not add per-model API traffic. Newly published frameworks are discovered automatically, but receive no novelty bonus: they can win only when their confidence-adjusted success score beats the best incumbent by @@ -124,7 +126,10 @@ evidence on every scan; accepted route counters and lifetime sufficient statisti are stored in `.modelhub_state/routing_intelligence.json`. Five consecutive local failures open a 12-hour GPU/framework circuit breaker. -A sub-20% success rate over the latest 20 terminal tasks opens a 6-hour breaker. +Five or more recent terminal results below 10% success open the same breaker, +including ambiguous runtime failures that are unsafe to keep feeding. A sub-20% +attributable success rate over the latest 20 terminal tasks opens a 6-hour +breaker. Platform/infrastructure failures are excluded from long-term compatibility rates and model/profile breakers. Three consecutive platform failures on a GPU/framework instead open a short 30-minute breaker, so a temporary broken runner or lack of an @@ -481,6 +486,18 @@ calibration each Monday 10:00-10:59 Asia/Shanghai. During that window only, the worker may borrow the five recent-model positions to measure a new limit; ordinary capacity-minus-five admission resumes afterward. +Version `2026.09.22.1` restores accuracy-first admission while retaining bounded +capacity recovery. Any existing result for an exact model/GPU pair, including a +failed result, closes that pair; a different framework no longer reopens it. +Each model is limited to six GPU routes across poll cycles and at most two local +routes may await feedback simultaneously. Lower-ranked GPUs are still checked, +in score order, only when the preferred six cannot produce a safe candidate. +Download-ranked popularity fallback is removed: discovery expands by update +window instead. Public framework/GPU confidence floors and an all-terminal +failure circuit prevent high-volume low-quality routes from winning on relative +rank alone. Community-confirmed model successes are used only as a tie-breaker +inside the same confidence band. + ## Deploy Create a tag and submit the repository URL plus tag in "我的适配智能体". diff --git a/modelhub_submmit_api/README.md b/modelhub_submmit_api/README.md index d3d3050883..cb418f81bc 100644 --- a/modelhub_submmit_api/README.md +++ b/modelhub_submmit_api/README.md @@ -79,21 +79,25 @@ bash run_poll.sh --dry-run ## Behavior - The runner auto-discovers all safe GPU/template combinations from the public submit catalog. -- Automatic GPU selection uses exact 70/30 accepted-task scheduling: long-term - Wilson-ranked top 3 GPUs and the top GPUs from the latest 1,000 terminal tasks. - There is no all-GPU exploration category. -- Within each category, weighted-fair scheduling uses estimated queue backlog hours, +- Automatic GPU selection uses one success-first score built from qualified + local cohorts and ModelHub public evidence. There is no random or fixed-quota + exploration category. +- Within the qualified success band, scheduling uses estimated queue backlog hours, recent public throughput/success, machine availability, and worker concurrency. Unavailable or stalled GPU pools are circuit-broken instead of continuing to absorb work. - Compatible frameworks are ranked by ModelHub public aggregate success statistics plus capped local GPU+framework evidence, with a 300-sample public minimum and - Wilson confidence bounds. Missing or undersized public evidence receives zero - traffic rather than falling back to exploration. + a 20% Wilson lower-bound floor. GPU pools normally require a 15% floor. + Missing or undersized public evidence receives zero traffic rather than + falling back to exploration; a low-sample GPU requires a recent success and a + framework that already passes the stricter framework gate. - New frameworks are discovered from the live catalog but get no novelty bonus. They are eligible only with a complete official build config that passes local validation and a confidence score at least 10% above the best incumbent. -- Five consecutive local failures pause a GPU/framework pair for 12 hours; a - sub-20% rate over the latest 20 terminal tasks pauses it for 6 hours. +- Five consecutive local failures pause a GPU/framework pair for 12 hours. Five + or more recent terminal outcomes below 10% success also pause it for 12 hours, + including ambiguous runtime failures. A sub-20% attributable rate over the + latest 20 terminal tasks pauses it for 6 hours. - An explicit "framework does not support this model/architecture" failure learns a 30-day GPU + framework + task + architecture block. Architecture identity comes from candidate `config.json` plus exact unsupported `model_type` or @@ -114,20 +118,22 @@ bash run_poll.sh --dry-run duplicates do not advance it. The next cycle refreshes platform history before submitting again. - Strategy state is stored in `.modelhub_state/gpu_strategy.json`; a generation never recalculates during candidate submission. -- Candidate discovery starts with the configured recent window, then automatically expands to - 7 days, 30 days, and the first 3,000 raw ModelScope rows ordered by update time. If the - candidate pool is still short, it rotates through up to 200 download-ranked rows per cycle - as an older-model fallback. +- Candidate discovery starts with the configured recent window, then + automatically expands to 7 days, 30 days, and the first 3,000 raw ModelScope + rows ordered by update time. Download popularity is not treated as ModelHub + compatibility evidence. - Model verification responses are cached across poll cycles for 15 minutes. Local model/GPU failures cool down after 24 hours instead of remaining permanently blocked. -- Community deduplication is model/GPU-specific: another GPU's adaptation does not block the - current GPU. `未验证` with no records is eligible. `验证失败` is eligible only with a - different publicly vetted framework after all recorded attempts explicitly failed; - `已验证`, `验证中`, and ambiguous records remain blocked. Every actual submission - performs a fresh uncached check for its exact GPU and framework. +- Community deduplication is model/GPU-specific: another GPU's adaptation does + not block the current GPU. Only `未验证` with no records is eligible; + `验证失败`, `已验证`, `验证中`, and ambiguous records remain blocked. Every actual + submission performs a fresh uncached check for its exact GPU. - If the community lookup is unavailable, submission is deferred. A platform model-uniqueness rejection permanently excludes only that model/GPU combination from future local retries. -- The same framework is not retried for a model/GPU after a recorded failure. +- A model may use at most six GPU routes across poll cycles, and at most two + locally submitted routes for that model may await feedback at once. The first + six ranked GPUs remain preferred; lower-ranked eligible GPUs are inspected in + order only when that tranche yields no safe candidate. - Multiple ModelHub tokens are pooled and used to route submissions to the account with available async capacity. - Concurrent submissions reserve account slots locally, and an account-capacity race automatically falls through to another account. - Concurrent local processes claim model/GPU pairs in `.modelhub_state/submission_claims.jsonl`; shared ledger, history, and outcome files use process locks and atomic replacement. diff --git a/modelhub_submmit_api/main.py b/modelhub_submmit_api/main.py index 120ac2ac86..ecda225d09 100644 --- a/modelhub_submmit_api/main.py +++ b/modelhub_submmit_api/main.py @@ -66,6 +66,7 @@ DEFAULT_LEDGER_PATH = Path("ledger/submissions.jsonl") ADAPTIVE_SCAN_MAX_MODELS = 3000 ADAPTIVE_SCAN_MIN_FALLBACK_MODELS = 500 MAX_ROUTES_PER_MODEL = 6 +MAX_PENDING_ROUTES_PER_MODEL = 2 def build_parser() -> argparse.ArgumentParser: @@ -572,9 +573,11 @@ def process_model_for_candidates( if hasattr(modelhub_client, "model_submission_precheck"): precheck = modelhub_client.model_submission_precheck(model.repo_id) processed_gpus = set(precheck.get("processedGpus") or set()) + successful_gpus = set(precheck.get("successfulGpus") or set()) failed_frameworks_by_gpu = precheck.get("failedFrameworksByGpu") or {} else: processed_gpus = modelhub_client.processed_gpus_for_model(model.repo_id) + successful_gpus = set() failed_frameworks_by_gpu = {} except Exception as exc: return ( @@ -582,6 +585,31 @@ def process_model_for_candidates( [{"repoId": model.repo_id, "reason": "community_precheck_unavailable_fail_closed"}], [{"repoId": model.repo_id, "reason": f"community_precheck_error:{exc}"}], ) + if len(processed_gpus) >= MAX_ROUTES_PER_MODEL: + return ( + [], + [{ + "repoId": model.repo_id, + "reason": "model_route_limit_reached", + "processedGpuCount": len(processed_gpus), + }], + [], + ) + pending_route_count = ( + outcome_tracker.pending_route_count_for_model(model.repo_id) + if outcome_tracker is not None + else 0 + ) + if pending_route_count >= MAX_PENDING_ROUTES_PER_MODEL: + return ( + [], + [{ + "repoId": model.repo_id, + "reason": "model_pending_feedback_limit", + "pendingRouteCount": pending_route_count, + }], + [], + ) candidates: list[dict[str, Any]] = [] skipped: list[dict[str, Any]] = [] failed: list[dict[str, Any]] = [] @@ -605,33 +633,14 @@ def process_model_for_candidates( if target_gpu in processed_gpus: skipped.append({"repoId": model.repo_id, "targetGpu": target_gpu, "reason": "already_processed_for_gpu"}) continue - if target_gpu in failed_frameworks_by_gpu and market_intelligence is None: + if target_gpu in failed_frameworks_by_gpu: skipped.append({ "repoId": model.repo_id, "targetGpu": target_gpu, - "reason": "prior_failed_gpu_requires_public_framework_evidence", + "reason": "prior_failed_gpu_closed_accuracy_first", }) continue failed_frameworks = set(failed_frameworks_by_gpu.get(target_gpu) or set()) - if ( - failed_frameworks - and market_intelligence is not None - and hasattr(market_intelligence, "has_vetted_alternative_framework") - and allowed_task_types_set - and all(market_intelligence.has_framework_route(task_type, target_gpu) for task_type in allowed_task_types_set) - and not any( - market_intelligence.has_vetted_alternative_framework( - task_type, target_gpu, failed_frameworks - ) - for task_type in allowed_task_types_set - ) - ): - skipped.append({ - "repoId": model.repo_id, - "targetGpu": target_gpu, - "reason": "prior_failed_framework_no_alternative", - }) - continue if outcome_tracker and outcome_tracker.is_model_gpu_failed(model.repo_id, target_gpu): skipped.append({"repoId": model.repo_id, "targetGpu": target_gpu, "reason": "local_failure_cooldown_24h"}) continue @@ -734,6 +743,9 @@ def process_model_for_candidates( record = candidate_to_record(best) if excluded_frameworks: record["previousFailedFrameworks"] = sorted(excluded_frameworks) + record["communitySuccessfulGpuCount"] = len(successful_gpus) + if successful_gpus: + record["communitySuccessfulGpus"] = sorted(successful_gpus) if outcome_tracker is not None: record["transformersPrerequisiteSatisfied"] = ( outcome_tracker.has_non_transformers_failure(model.repo_id) @@ -810,20 +822,6 @@ def build_adaptive_scan_stages( "limit": max_models, } ) - if allow_older_than_recent_window: - # ModelScope only exposes the first 3,000 raw rows for a sort order. - # Downloads ordering reaches a different older, established slice - # after the recent-update slice has produced too few candidates. - stages.append( - { - "name": "popular_history", - "updatedAfter": None, - "limit": min(max_models, 200), - "sort": "downloads", - "rotatePages": True, - } - ) - deduped: list[dict[str, Any]] = [] seen: set[tuple[str | None, int, str]] = set() for stage in stages: @@ -951,20 +949,20 @@ def submit_candidate( "candidate": candidate, "reason": "already_processed_for_gpu", } - failed_frameworks = set( - (precheck.get("failedFrameworksByGpu") or {}).get(candidate["targetGpu"]) or set() - ) - if candidate["framework"] in failed_frameworks: + if len(set(precheck.get("processedGpus") or set())) >= MAX_ROUTES_PER_MODEL: return { "outcome": "duplicate", "candidate": candidate, - "reason": "prior_failed_framework_for_gpu", + "reason": "model_route_limit_reached", } - if failed_frameworks and not candidate.get("frameworkEvidenceQualified"): + failed_frameworks = set( + (precheck.get("failedFrameworksByGpu") or {}).get(candidate["targetGpu"]) or set() + ) + if failed_frameworks: return { - "outcome": "framework_evidence_deferred", + "outcome": "duplicate", "candidate": candidate, - "reason": "prior_failed_gpu_requires_public_framework_evidence", + "reason": "prior_failed_gpu_closed_accuracy_first", } payload = { diff --git a/modelhub_submmit_api/market_intelligence.py b/modelhub_submmit_api/market_intelligence.py index d6f6bfb953..b9a86dde7f 100644 --- a/modelhub_submmit_api/market_intelligence.py +++ b/modelhub_submmit_api/market_intelligence.py @@ -20,7 +20,8 @@ DEFAULT_THROUGHPUT_WINDOW_HOURS = 6 DEFAULT_FETCH_WORKERS = 4 DEFAULT_FRAMEWORK_MIN_SAMPLES = 300 DEFAULT_GPU_MIN_RECENT_TERMINALS = 20 -DEFAULT_FRAMEWORK_MIN_WILSON = 0.05 +DEFAULT_FRAMEWORK_MIN_WILSON = 0.20 +DEFAULT_GPU_MIN_WILSON = 0.15 DEFAULT_COMMUNITY_WINDOW_DAYS = 30 DEFAULT_COMMUNITY_MAX_MODELS = 200 DEFAULT_COMMUNITY_HYDRATE_PER_REFRESH = 50 @@ -201,7 +202,7 @@ class MarketIntelligenceManager: and health_factor >= 0.5 and int(item.get("recentTerminal") or 0) >= DEFAULT_GPU_MIN_RECENT_TERMINALS and int(item.get("recentSuccess") or 0) > 0 - and wilson >= DEFAULT_FRAMEWORK_MIN_WILSON + and wilson >= DEFAULT_GPU_MIN_WILSON ) def _base_state(self, gpus: list[str], task_types: list[str], now: datetime) -> dict[str, Any]: @@ -554,7 +555,7 @@ class MarketIntelligenceManager: and health_factor >= 0.5 and int(stats.get("recentTerminal") or 0) >= DEFAULT_GPU_MIN_RECENT_TERMINALS and int(stats.get("recentSuccess") or 0) > 0 - and wilson >= DEFAULT_FRAMEWORK_MIN_WILSON + and wilson >= DEFAULT_GPU_MIN_WILSON ) result[gpu] = stats @@ -661,7 +662,26 @@ class MarketIntelligenceManager: # Missing live data falls back to the scheduler's already-proven # long/recent pools. Explicitly failed public gates never do. if eligibility is False: - continue + low_sample = int(item.get("recentTerminal") or 0) < DEFAULT_GPU_MIN_RECENT_TERMINALS + has_recent_success = int(item.get("recentSuccess") or 0) > 0 + recent_wilson = float(item.get("recentWilsonLowerBound") or 0.0) + healthy = float(item.get("healthFactor") or 1.0) >= 0.5 + has_proven_framework = any( + bool(framework.get("officialConfigValid", False)) + and int(framework.get("modelCount") or 0) >= self.framework_min_samples + and float(framework.get("wilsonLowerBound") or 0.0) >= DEFAULT_FRAMEWORK_MIN_WILSON + for by_gpu in ((self.state or {}).get("frameworkStats") or {}).values() + for framework in ((by_gpu or {}).get(gpu) or {}).values() + ) + if ( + item.get("canVerify") is False + or not low_sample + or not has_recent_success + or recent_wilson < DEFAULT_GPU_MIN_WILSON + or not healthy + or not has_proven_framework + ): + continue ranked.append((-float(item.get("selectionWeight") or 1.0), index, gpu)) return [gpu for _weight, _index, gpu in sorted(ranked)] @@ -723,6 +743,7 @@ class MarketIntelligenceManager: recent_item = (self.local_outcome_stats.get("recentCombinationStats") or {}).get(local_key) or {} recent_success = max(0, int(recent_item.get("successCount") or 0)) + recent_all_failures = max(0, int(recent_item.get("failureCount") or 0)) recent_failure = max( 0, int(recent_item.get("attributableFailureCount", recent_item.get("failureCount") or 0)), @@ -743,6 +764,16 @@ class MarketIntelligenceManager: elif last_terminal_at is not None and consecutive_failures >= 5: circuit_reason = "five_consecutive_local_failures" circuit_until = last_terminal_at + timedelta(hours=12) + elif ( + last_terminal_at is not None + and recent_success + recent_all_failures >= 5 + and recent_success / (recent_success + recent_all_failures) < 0.10 + ): + # Runtime-empty and other unresolved failures must not create a + # permanent compatibility blacklist. Repetition is still strong + # operational evidence, so pause the route while feedback settles. + circuit_reason = "recent_terminal_success_below_10_percent" + circuit_until = last_terminal_at + timedelta(hours=12) elif last_terminal_at is not None and recent_samples >= 20 and recent_rate is not None and recent_rate < 0.20: circuit_reason = "recent_local_success_below_20_percent" circuit_until = last_terminal_at + timedelta(hours=6) diff --git a/modelhub_submmit_api/modelhub_client.py b/modelhub_submmit_api/modelhub_client.py index 4ff3273db5..707fe0e850 100644 --- a/modelhub_submmit_api/modelhub_client.py +++ b/modelhub_submmit_api/modelhub_client.py @@ -38,6 +38,7 @@ def parse_model_submission_precheck(payload: Any) -> dict[str, Any]: raise ModelHubAPIError("Community model precheck verifyResult is invalid", payload=payload) blocked_gpus: set[str] = set() + successful_gpus: set[str] = set() failed_frameworks_by_gpu: dict[str, set[str]] = {} for gpu, gpu_result in verify_result.items(): name = str(gpu) @@ -48,6 +49,12 @@ def parse_model_submission_precheck(payload: Any) -> dict[str, Any]: records = gpu_result.get("records") if status == "未验证" and isinstance(records, list) and not records: continue + blocked_gpus.add(name) + if status == "已验证" or any( + isinstance(record, dict) and _positive_verify_result(record.get("verifyResult")) + for record in (records or []) + ): + successful_gpus.add(name) if status == "验证失败" and isinstance(records, list) and records: frameworks: set[str] = set() for record in records: @@ -63,20 +70,26 @@ def parse_model_submission_precheck(payload: Any) -> dict[str, Any]: frameworks.add(framework) else: failed_frameworks_by_gpu[name] = frameworks - continue - # Verified, validating and unknown states remain closed. A failed GPU - # opens only for a different framework with explicit failed records. - blocked_gpus.add(name) + # Accuracy-first admission: any existing model/GPU state remains + # closed. Failed-framework details are diagnostic evidence only. return { "isInDB": data.get("isInDB") is True, "processedGpus": blocked_gpus, + "successfulGpus": successful_gpus, "failedFrameworksByGpu": failed_frameworks_by_gpu, "verifyResult": verify_result, "modelInfo": data.get("modelInfo") if isinstance(data.get("modelInfo"), dict) else {}, } +def _positive_verify_result(value: Any) -> bool: + try: + return int(value) > 0 + except (TypeError, ValueError): + return False + + class ModelHubClient: def __init__( self, diff --git a/modelhub_submmit_api/outcome_tracker.py b/modelhub_submmit_api/outcome_tracker.py index 8699b1e122..9023dbe5f0 100644 --- a/modelhub_submmit_api/outcome_tracker.py +++ b/modelhub_submmit_api/outcome_tracker.py @@ -344,6 +344,17 @@ class OutcomeTracker: self._by_model_gpu[(model_id, target_gpu)].append(record) append_jsonl(self.path, record) + def pending_route_count_for_model(self, model_id: str) -> int: + """Return distinct locally submitted GPU routes still awaiting an outcome.""" + pending_gpus: set[str] = set() + for record in self._records: + if record.get("modelId") != model_id or record.get("outcome") != "pending": + continue + target_gpu = str(record.get("targetGpu") or "").strip() + if target_gpu: + pending_gpus.add(target_gpu) + return len(pending_gpus) + def mark_policy_cancellations(self, decisions: list[dict[str, Any]]) -> int: """Persist intentional task stops so they never become failure evidence.""" marked = 0 diff --git a/modelhub_submmit_api/routing_engine.py b/modelhub_submmit_api/routing_engine.py index 7e1014562d..ddc5c6c685 100644 --- a/modelhub_submmit_api/routing_engine.py +++ b/modelhub_submmit_api/routing_engine.py @@ -123,6 +123,9 @@ class SuccessFirstRoutingEngine: "routingEtaHours": eta, "routingEtaUnknown": eta_unknown, "routingExpectedSuccessPerHour": expected, + "routingCommunityModelSuccesses": max( + 0, int(candidate.get("communitySuccessfulGpuCount") or 0) + ), "strategyCategory": "unified_success_first", } ) @@ -168,6 +171,7 @@ class SuccessFirstRoutingEngine: selected.sort( key=lambda item: ( -int(item["routingSuccessBand"]), + -int(item.get("routingCommunityModelSuccesses") or 0), -float(item["routingExpectedSuccessPerHour"]), -int(item["routingSamples"]), -int(item.get("downloads") or 0), diff --git a/modelhub_submmit_api/version.py b/modelhub_submmit_api/version.py index c2eb688126..669bf78e6e 100644 --- a/modelhub_submmit_api/version.py +++ b/modelhub_submmit_api/version.py @@ -1 +1 @@ -AGENT_VERSION = "2026.09.20.2" +AGENT_VERSION = "2026.09.22.1" diff --git a/tests/test_concurrency.py b/tests/test_concurrency.py index ef42107b8b..3f3945ab58 100644 --- a/tests/test_concurrency.py +++ b/tests/test_concurrency.py @@ -733,9 +733,9 @@ class ClientPoolConcurrencyTests(unittest.TestCase): ) self.assertEqual(2, summary["submittedCount"]) - self.assertEqual(len(supported_gpus), summary["skipReasonCounts"]["already_processed_for_gpu"]) - self.assertEqual("popular_history", summary["scanStages"][-1]["name"]) - self.assertEqual("downloads", summary["scanStages"][-1]["sort"]) + self.assertEqual(1, summary["skipReasonCounts"]["model_route_limit_reached"]) + self.assertEqual("all_history", summary["scanStages"][-1]["name"]) + self.assertEqual("last_modified", summary["scanStages"][-1]["sort"]) self.assertIsNone(discovery.calls[-1]) def test_adaptive_strategy_counts_only_platform_accepted_tasks(self) -> None: diff --git a/tests/test_market_intelligence.py b/tests/test_market_intelligence.py index 9199ef48cd..c4ab5e9f20 100644 --- a/tests/test_market_intelligence.py +++ b/tests/test_market_intelligence.py @@ -219,7 +219,7 @@ class MarketIntelligenceTests(unittest.TestCase): "frameworkStats": { "text-generation": { "gpu": { - "vllm": {"modelCount": 1000, "wilsonLowerBound": 0.10}, + "vllm": {"modelCount": 1000, "wilsonLowerBound": 0.30}, "vllm-mlu": {"modelCount": 800, "wilsonLowerBound": 0.35}, "vllm-customized": {"modelCount": 2, "wilsonLowerBound": 0.90}, } @@ -357,7 +357,7 @@ class MarketIntelligenceTests(unittest.TestCase): "frameworkStats": { "text-generation": { "Cambricon_mlu-370-x4": { - "vllm": {"modelCount": 1000, "wilsonLowerBound": 0.10}, + "vllm": {"modelCount": 1000, "wilsonLowerBound": 0.30}, "future-engine": { "modelCount": 1000, "successCount": 500, @@ -396,7 +396,7 @@ class MarketIntelligenceTests(unittest.TestCase): "frameworkStats": { "text-generation": { "Cambricon_mlu-370-x4": { - "vllm": {"modelCount": 1000, "wilsonLowerBound": 0.10}, + "vllm": {"modelCount": 1000, "wilsonLowerBound": 0.30}, "future-engine": { "modelCount": 2, "wilsonLowerBound": 0.90, @@ -454,6 +454,124 @@ class MarketIntelligenceTests(unittest.TestCase): ) self.assertIsNone(candidate) + def test_low_confidence_public_framework_is_rejected_even_with_many_samples(self) -> None: + manager = MarketIntelligenceManager("unused.json", framework_min_samples=100) + manager.state = { + "frameworkStats": { + "text-generation": { + "Cambricon_mlu-370-x4": { + "vllm": {"modelCount": 10_000, "wilsonLowerBound": 0.12}, + } + } + } + } + candidate = choose_candidate_for_gpu( + model=HFModelSummary( + repo_id="owner/model", + downloads=100, + last_modified=None, + pipeline_tag="text-generation", + ), + inspection=ModelInspection(repo_id="owner/model", weight_files=["model.safetensors"]), + template_selector=TemplateSelector(), + task_types=["text-generation"], + target_gpu="Cambricon_mlu-370-x4", + market_intelligence=manager, + ) + self.assertIsNone(candidate) + + def test_repeated_ambiguous_failures_open_operational_circuit(self) -> None: + manager = MarketIntelligenceManager("unused.json", framework_min_samples=100) + manager.state = { + "frameworkStats": { + "text-generation": { + "Biren_166m": { + "vllm_fix_tokenizer": {"modelCount": 10_000, "wilsonLowerBound": 0.35}, + } + } + } + } + manager.set_local_outcome_stats( + { + "recentCombinationStats": { + "Biren_166m|vllm_fix_tokenizer|text-generation": { + "successCount": 0, + "failureCount": 5, + "attributableFailureCount": 0, + "consecutiveFailures": 0, + "lastTerminalAt": datetime.now(timezone.utc).isoformat(), + } + } + } + ) + metadata = manager.framework_metadata( + "text-generation", "Biren_166m", "vllm_fix_tokenizer" + ) + self.assertTrue(metadata["frameworkCircuitOpen"]) + self.assertEqual( + "recent_terminal_success_below_10_percent", + metadata["frameworkCircuitReason"], + ) + + def test_low_sample_gpu_with_recent_success_and_proven_framework_remains_eligible(self) -> None: + manager = MarketIntelligenceManager("unused.json", framework_min_samples=100) + manager.state = { + "gpuStats": { + "Sunrise_pt-200-x1": { + "canVerify": True, + "submissionEligible": False, + "recentTerminal": 10, + "recentSuccess": 5, + "recentWilsonLowerBound": 0.24, + "healthFactor": 1.0, + "selectionWeight": 0.5, + } + }, + "frameworkStats": { + "text-generation": { + "Sunrise_pt-200-x1": { + "vllm_fix_tokenizer": { + "modelCount": 6_000, + "wilsonLowerBound": 0.41, + "officialConfigValid": True, + } + } + } + }, + } + self.assertEqual( + ["Sunrise_pt-200-x1"], + manager.eligible_gpus(["Sunrise_pt-200-x1"]), + ) + + def test_low_sample_gpu_does_not_bypass_absolute_quality_floor(self) -> None: + manager = MarketIntelligenceManager("unused.json", framework_min_samples=100) + manager.state = { + "gpuStats": { + "gpu-low-quality": { + "canVerify": True, + "submissionEligible": False, + "recentTerminal": 10, + "recentSuccess": 1, + "recentWilsonLowerBound": 0.02, + "healthFactor": 1.0, + "selectionWeight": 0.5, + } + }, + "frameworkStats": { + "text-generation": { + "gpu-low-quality": { + "vllm": { + "modelCount": 6_000, + "wilsonLowerBound": 0.41, + "officialConfigValid": True, + } + } + } + }, + } + self.assertEqual([], manager.eligible_gpus(["gpu-low-quality"])) + def test_five_consecutive_local_failures_circuit_break_framework(self) -> None: manager = MarketIntelligenceManager("unused.json", framework_min_samples=100) manager.state = { diff --git a/tests/test_submission_safety.py b/tests/test_submission_safety.py index 3feb372997..a456b9e202 100644 --- a/tests/test_submission_safety.py +++ b/tests/test_submission_safety.py @@ -21,6 +21,7 @@ from modelhub_client import ( # noqa: E402 parse_model_submission_precheck, ) from models import HFModelSummary, ModelInspection # noqa: E402 +from outcome_tracker import OutcomeTracker # noqa: E402 from submission_exclusions import SubmissionExclusionStore # noqa: E402 from template_selector import TemplateSelector # noqa: E402 @@ -286,9 +287,10 @@ class SubmissionSafetyTests(unittest.TestCase): } precheck = parse_model_submission_precheck(payload) self.assertEqual( - {"running", "success", "unknown", "failed_without_framework"}, + {"failed", "running", "success", "unknown", "failed_without_framework"}, precheck["processedGpus"], ) + self.assertEqual({"success"}, precheck["successfulGpus"]) self.assertEqual({"failed": {"vllm"}}, precheck["failedFrameworksByGpu"]) def test_unverified_gpu_becomes_candidate_but_validating_gpu_does_not(self) -> None: @@ -321,7 +323,7 @@ class SubmissionSafetyTests(unittest.TestCase): self.assertEqual([], active_candidates) self.assertEqual("already_processed_for_gpu", active_skips[0]["reason"]) - def test_failed_gpu_requires_an_alternative_framework(self) -> None: + def test_failed_gpu_remains_closed_even_when_an_alternative_framework_exists(self) -> None: model = SafeDiscovery().model client = FailedFrameworkClient() candidates, skipped, failed = process_model_for_candidates( @@ -334,23 +336,14 @@ class SubmissionSafetyTests(unittest.TestCase): market_intelligence=TwoFrameworkMarket(), # type: ignore[arg-type] official_registry=AllGpuOfficialRegistry(), # type: ignore[arg-type] ) - self.assertEqual([], skipped) + self.assertEqual("prior_failed_gpu_closed_accuracy_first", skipped[-1]["reason"]) self.assertEqual([], failed) - self.assertEqual(["vllm-mlu"], [item["framework"] for item in candidates]) - self.assertEqual(["vllm"], candidates[0]["previousFailedFrameworks"]) + self.assertEqual([], candidates) - blocked = submit_candidate({**candidates[0], "framework": "vllm"}, client) # type: ignore[arg-type] - accepted = submit_candidate(candidates[0], client) # type: ignore[arg-type] + blocked = submit_candidate(candidate("Cambricon_mlu-370-x4"), client) # type: ignore[arg-type] self.assertEqual("duplicate", blocked["outcome"]) - self.assertEqual("prior_failed_framework_for_gpu", blocked["reason"]) - self.assertEqual("submitted", accepted["outcome"]) - self.assertEqual(1, client.add_calls) - - unvetted = submit_candidate( - {**candidates[0], "frameworkEvidenceQualified": False}, client # type: ignore[arg-type] - ) - self.assertEqual("framework_evidence_deferred", unvetted["outcome"]) - self.assertEqual(1, client.add_calls) + self.assertEqual("prior_failed_gpu_closed_accuracy_first", blocked["reason"]) + self.assertEqual(0, client.add_calls) no_routes, no_route_skips, _ = process_model_for_candidates( model=model, @@ -363,7 +356,7 @@ class SubmissionSafetyTests(unittest.TestCase): official_registry=AllGpuOfficialRegistry(), # type: ignore[arg-type] ) self.assertEqual([], no_routes) - self.assertEqual("prior_failed_framework_no_alternative", no_route_skips[-1]["reason"]) + self.assertEqual("prior_failed_gpu_closed_accuracy_first", no_route_skips[-1]["reason"]) no_market_routes, no_market_skips, _ = process_model_for_candidates( model=model, @@ -375,10 +368,65 @@ class SubmissionSafetyTests(unittest.TestCase): ) self.assertEqual([], no_market_routes) self.assertEqual( - "prior_failed_gpu_requires_public_framework_evidence", + "prior_failed_gpu_closed_accuracy_first", no_market_skips[-1]["reason"], ) + def test_model_route_limit_is_global_across_poll_cycles(self) -> None: + class SixRouteClient: + @staticmethod + def model_submission_precheck(_model_id: str) -> dict: + return { + "processedGpus": {f"gpu-{index}" for index in range(1, 7)}, + "successfulGpus": {"gpu-1"}, + "failedFrameworksByGpu": {}, + } + + candidates, skipped, failed = process_model_for_candidates( + model=SafeDiscovery().model, + hf_discovery=SafeDiscovery(), # type: ignore[arg-type] + modelhub_client=SixRouteClient(), # type: ignore[arg-type] + template_selector=TemplateSelector(), + target_gpus=["gpu-7"], + allowed_task_types=["text-generation"], + ) + self.assertEqual([], candidates) + self.assertEqual([], failed) + self.assertEqual("model_route_limit_reached", skipped[0]["reason"]) + self.assertEqual(6, skipped[0]["processedGpuCount"]) + + def test_model_waits_for_feedback_after_two_pending_routes(self) -> None: + with tempfile.TemporaryDirectory() as temporary_dir: + root = Path(temporary_dir) + tracker = OutcomeTracker( + root / "outcomes.jsonl", + checkpoint_path=root / "checkpoint.json", + recent_path=root / "recent.jsonl", + ) + tracker.record_submission( + "owner/model", "gpu-1", "vllm", "text-generation", "task-1", + "2026-09-22T00:00:00+00:00", + ) + tracker.record_submission( + "owner/model", "gpu-2", "vllm", "text-generation", "task-2", + "2026-09-22T00:01:00+00:00", + ) + + candidates, skipped, failed = process_model_for_candidates( + model=SafeDiscovery().model, + hf_discovery=SafeDiscovery(), # type: ignore[arg-type] + modelhub_client=OtherGpuProcessedClient(), # type: ignore[arg-type] + template_selector=TemplateSelector(), + target_gpus=["gpu-3"], + allowed_task_types=["text-generation"], + outcome_tracker=tracker, + ) + + self.assertEqual([], candidates) + self.assertEqual([], failed) + self.assertEqual("model_pending_feedback_limit", skipped[0]["reason"]) + self.assertEqual(2, skipped[0]["pendingRouteCount"]) + def test_published_size_avoids_official_calls_for_certain_oom(self) -> None: class CountingRegistry(AllGpuOfficialRegistry): calls = 0