CoolBoy/submmit
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

fix: restore accuracy-first route admission

Browse Source
This commit is contained in:
CoolBoy
2026-09-22 15:26:42 +08:00
parent 4afce879dc
commit 6154dc738f
11 changed files with 343 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
README.md
View File

@@ -107,8 +107,10 @@ model/GPU uniqueness response, the entire submission cycle pauses fail-closed.
For each compatible model/GPU pair, the worker also ranks the GPU's supported
frameworks using ModelHub's public aggregate `modelCount` and `successCount` data,
then blends in the worker's own GPU+framework outcomes with a capped weight.
Only frameworks with at least 300 public samples and a safe Wilson lower bound
are eligible. Framework statistics refresh
Only frameworks with at least 300 public samples and a Wilson lower bound of at
least 20% are eligible. GPU pools normally require a 15% lower bound; a
low-sample pool can remain available only when it has a recent success and a
framework that passes the stricter framework gate. Framework statistics refresh
every 6 hours, so they do not add per-model API traffic. Newly published
frameworks are discovered automatically, but receive no novelty bonus: they can
win only when their confidence-adjusted success score beats the best incumbent by
@@ -124,7 +126,10 @@ evidence on every scan; accepted route counters and lifetime sufficient statisti
are stored in `.modelhub_state/routing_intelligence.json`.
Five consecutive local failures open a 12-hour GPU/framework circuit breaker.
A sub-20% success rate over the latest 20 terminal tasks opens a 6-hour breaker.
Five or more recent terminal results below 10% success open the same breaker,
including ambiguous runtime failures that are unsafe to keep feeding. A sub-20%
attributable success rate over the latest 20 terminal tasks opens a 6-hour
breaker.
Platform/infrastructure failures are excluded from long-term compatibility rates
and model/profile breakers. Three consecutive platform failures on a GPU/framework
instead open a short 30-minute breaker, so a temporary broken runner or lack of an
@@ -481,6 +486,18 @@ calibration each Monday 10:00-10:59 Asia/Shanghai. During that window only,
the worker may borrow the five recent-model positions to measure a new limit;
ordinary capacity-minus-five admission resumes afterward.
Version `2026.09.22.1` restores accuracy-first admission while retaining bounded
capacity recovery. Any existing result for an exact model/GPU pair, including a
failed result, closes that pair; a different framework no longer reopens it.
Each model is limited to six GPU routes across poll cycles and at most two local
routes may await feedback simultaneously. Lower-ranked GPUs are still checked,
in score order, only when the preferred six cannot produce a safe candidate.
Download-ranked popularity fallback is removed: discovery expands by update
window instead. Public framework/GPU confidence floors and an all-terminal
failure circuit prevent high-volume low-quality routes from winning on relative
rank alone. Community-confirmed model successes are used only as a tie-breaker
inside the same confidence band.
## Deploy
Create a tag and submit the repository URL plus tag in "我的适配智能体".

42
modelhub_submmit_api/README.md
View File

@@ -79,21 +79,25 @@ bash run_poll.sh --dry-run
## Behavior
- The runner auto-discovers all safe GPU/template combinations from the public submit catalog.
- Automatic GPU selection uses exact 70/30 accepted-task scheduling: long-term
Wilson-ranked top 3 GPUs and the top GPUs from the latest 1,000 terminal tasks.
There is no all-GPU exploration category.
- Within each category, weighted-fair scheduling uses estimated queue backlog hours,
- Automatic GPU selection uses one success-first score built from qualified
local cohorts and ModelHub public evidence. There is no random or fixed-quota
exploration category.
- Within the qualified success band, scheduling uses estimated queue backlog hours,
recent public throughput/success, machine availability, and worker concurrency.
Unavailable or stalled GPU pools are circuit-broken instead of continuing to absorb work.
- Compatible frameworks are ranked by ModelHub public aggregate success statistics
plus capped local GPU+framework evidence, with a 300-sample public minimum and
Wilson confidence bounds. Missing or undersized public evidence receives zero
traffic rather than falling back to exploration.
a 20% Wilson lower-bound floor. GPU pools normally require a 15% floor.
Missing or undersized public evidence receives zero traffic rather than
falling back to exploration; a low-sample GPU requires a recent success and a
framework that already passes the stricter framework gate.
- New frameworks are discovered from the live catalog but get no novelty bonus.
They are eligible only with a complete official build config that passes local
validation and a confidence score at least 10% above the best incumbent.
- Five consecutive local failures pause a GPU/framework pair for 12 hours; a
sub-20% rate over the latest 20 terminal tasks pauses it for 6 hours.
- Five consecutive local failures pause a GPU/framework pair for 12 hours. Five
or more recent terminal outcomes below 10% success also pause it for 12 hours,
including ambiguous runtime failures. A sub-20% attributable rate over the
latest 20 terminal tasks pauses it for 6 hours.
- An explicit "framework does not support this model/architecture" failure learns
a 30-day GPU + framework + task + architecture block. Architecture identity
comes from candidate `config.json` plus exact unsupported `model_type` or
@@ -114,20 +118,22 @@ bash run_poll.sh --dry-run
duplicates do not advance it. The next cycle refreshes platform history before submitting again.
- Strategy state is stored in `.modelhub_state/gpu_strategy.json`; a generation never recalculates
during candidate submission.
- Candidate discovery starts with the configured recent window, then automatically expands to
7 days, 30 days, and the first 3,000 raw ModelScope rows ordered by update time. If the
candidate pool is still short, it rotates through up to 200 download-ranked rows per cycle
as an older-model fallback.
- Candidate discovery starts with the configured recent window, then
automatically expands to 7 days, 30 days, and the first 3,000 raw ModelScope
rows ordered by update time. Download popularity is not treated as ModelHub
compatibility evidence.
- Model verification responses are cached across poll cycles for 15 minutes. Local model/GPU
failures cool down after 24 hours instead of remaining permanently blocked.
- Community deduplication is model/GPU-specific: another GPU's adaptation does not block the
current GPU. `未验证` with no records is eligible. `验证失败` is eligible only with a
different publicly vetted framework after all recorded attempts explicitly failed;
`已验证`, `验证中`, and ambiguous records remain blocked. Every actual submission
performs a fresh uncached check for its exact GPU and framework.
- Community deduplication is model/GPU-specific: another GPU's adaptation does
not block the current GPU. Only `未验证` with no records is eligible;
`验证失败`, `已验证`, `验证中`, and ambiguous records remain blocked. Every actual
submission performs a fresh uncached check for its exact GPU.
- If the community lookup is unavailable, submission is deferred. A platform model-uniqueness
rejection permanently excludes only that model/GPU combination from future local retries.
- The same framework is not retried for a model/GPU after a recorded failure.
- A model may use at most six GPU routes across poll cycles, and at most two
locally submitted routes for that model may await feedback at once. The first
six ranked GPUs remain preferred; lower-ranked eligible GPUs are inspected in
order only when that tranche yields no safe candidate.
- Multiple ModelHub tokens are pooled and used to route submissions to the account with available async capacity.
- Concurrent submissions reserve account slots locally, and an account-capacity race automatically falls through to another account.
- Concurrent local processes claim model/GPU pairs in `.modelhub_state/submission_claims.jsonl`; shared ledger, history, and outcome files use process locks and atomic replacement.

84
modelhub_submmit_api/main.py
View File

@@ -66,6 +66,7 @@ DEFAULT_LEDGER_PATH = Path("ledger/submissions.jsonl")
ADAPTIVE_SCAN_MAX_MODELS = 3000
ADAPTIVE_SCAN_MIN_FALLBACK_MODELS = 500
MAX_ROUTES_PER_MODEL = 6
MAX_PENDING_ROUTES_PER_MODEL = 2
def build_parser() -> argparse.ArgumentParser:
@@ -572,9 +573,11 @@ def process_model_for_candidates(
if hasattr(modelhub_client, "model_submission_precheck"):
precheck = modelhub_client.model_submission_precheck(model.repo_id)
processed_gpus = set(precheck.get("processedGpus") or set())
successful_gpus = set(precheck.get("successfulGpus") or set())
failed_frameworks_by_gpu = precheck.get("failedFrameworksByGpu") or {}
else:
processed_gpus = modelhub_client.processed_gpus_for_model(model.repo_id)
successful_gpus = set()
failed_frameworks_by_gpu = {}
except Exception as exc:
return (
@@ -582,6 +585,31 @@ def process_model_for_candidates(
[{"repoId": model.repo_id, "reason": "community_precheck_unavailable_fail_closed"}],
[{"repoId": model.repo_id, "reason": f"community_precheck_error:{exc}"}],
)
if len(processed_gpus) >= MAX_ROUTES_PER_MODEL:
return (
[],
[{
"repoId": model.repo_id,
"reason": "model_route_limit_reached",
"processedGpuCount": len(processed_gpus),
}],
[],
)
pending_route_count = (
outcome_tracker.pending_route_count_for_model(model.repo_id)
if outcome_tracker is not None
else 0
)
if pending_route_count >= MAX_PENDING_ROUTES_PER_MODEL:
return (
[],
[{
"repoId": model.repo_id,
"reason": "model_pending_feedback_limit",
"pendingRouteCount": pending_route_count,
}],
[],
)
candidates: list[dict[str, Any]] = []
skipped: list[dict[str, Any]] = []
failed: list[dict[str, Any]] = []
@@ -605,33 +633,14 @@ def process_model_for_candidates(
if target_gpu in processed_gpus:
skipped.append({"repoId": model.repo_id, "targetGpu": target_gpu, "reason": "already_processed_for_gpu"})
continue
if target_gpu in failed_frameworks_by_gpu and market_intelligence is None:
if target_gpu in failed_frameworks_by_gpu:
skipped.append({
"repoId": model.repo_id,
"targetGpu": target_gpu,
"reason": "prior_failed_gpu_requires_public_framework_evidence",
"reason": "prior_failed_gpu_closed_accuracy_first",
})
continue
failed_frameworks = set(failed_frameworks_by_gpu.get(target_gpu) or set())
if (
failed_frameworks
and market_intelligence is not None
and hasattr(market_intelligence, "has_vetted_alternative_framework")
and allowed_task_types_set
and all(market_intelligence.has_framework_route(task_type, target_gpu) for task_type in allowed_task_types_set)
and not any(
market_intelligence.has_vetted_alternative_framework(
task_type, target_gpu, failed_frameworks
)
for task_type in allowed_task_types_set
)
):
skipped.append({
"repoId": model.repo_id,
"targetGpu": target_gpu,
"reason": "prior_failed_framework_no_alternative",
})
continue
if outcome_tracker and outcome_tracker.is_model_gpu_failed(model.repo_id, target_gpu):
skipped.append({"repoId": model.repo_id, "targetGpu": target_gpu, "reason": "local_failure_cooldown_24h"})
continue
@@ -734,6 +743,9 @@ def process_model_for_candidates(
record = candidate_to_record(best)
if excluded_frameworks:
record["previousFailedFrameworks"] = sorted(excluded_frameworks)
record["communitySuccessfulGpuCount"] = len(successful_gpus)
if successful_gpus:
record["communitySuccessfulGpus"] = sorted(successful_gpus)
if outcome_tracker is not None:
record["transformersPrerequisiteSatisfied"] = (
outcome_tracker.has_non_transformers_failure(model.repo_id)
@@ -810,20 +822,6 @@ def build_adaptive_scan_stages(
"limit": max_models,
}
)
if allow_older_than_recent_window:
# ModelScope only exposes the first 3,000 raw rows for a sort order.
# Downloads ordering reaches a different older, established slice
# after the recent-update slice has produced too few candidates.
stages.append(
{
"name": "popular_history",
"updatedAfter": None,
"limit": min(max_models, 200),
"sort": "downloads",
"rotatePages": True,
}
)
deduped: list[dict[str, Any]] = []
seen: set[tuple[str | None, int, str]] = set()
for stage in stages:
@@ -951,20 +949,20 @@ def submit_candidate(
"candidate": candidate,
"reason": "already_processed_for_gpu",
}
failed_frameworks = set(
(precheck.get("failedFrameworksByGpu") or {}).get(candidate["targetGpu"]) or set()
)
if candidate["framework"] in failed_frameworks:
if len(set(precheck.get("processedGpus") or set())) >= MAX_ROUTES_PER_MODEL:
return {
"outcome": "duplicate",
"candidate": candidate,
"reason": "prior_failed_framework_for_gpu",
"reason": "model_route_limit_reached",
}
if failed_frameworks and not candidate.get("frameworkEvidenceQualified"):
failed_frameworks = set(
(precheck.get("failedFrameworksByGpu") or {}).get(candidate["targetGpu"]) or set()
)
if failed_frameworks:
return {
"outcome": "framework_evidence_deferred",
"outcome": "duplicate",
"candidate": candidate,
"reason": "prior_failed_gpu_requires_public_framework_evidence",
"reason": "prior_failed_gpu_closed_accuracy_first",
}
payload = {

37
modelhub_submmit_api/market_intelligence.py
View File

@@ -20,7 +20,8 @@ DEFAULT_THROUGHPUT_WINDOW_HOURS = 6
DEFAULT_FETCH_WORKERS = 4
DEFAULT_FRAMEWORK_MIN_SAMPLES = 300
DEFAULT_GPU_MIN_RECENT_TERMINALS = 20
DEFAULT_FRAMEWORK_MIN_WILSON = 0.05
DEFAULT_FRAMEWORK_MIN_WILSON = 0.20
DEFAULT_GPU_MIN_WILSON = 0.15
DEFAULT_COMMUNITY_WINDOW_DAYS = 30
DEFAULT_COMMUNITY_MAX_MODELS = 200
DEFAULT_COMMUNITY_HYDRATE_PER_REFRESH = 50
@@ -201,7 +202,7 @@ class MarketIntelligenceManager:
and health_factor >= 0.5
and int(item.get("recentTerminal") or 0) >= DEFAULT_GPU_MIN_RECENT_TERMINALS
and int(item.get("recentSuccess") or 0) > 0
and wilson >= DEFAULT_FRAMEWORK_MIN_WILSON
and wilson >= DEFAULT_GPU_MIN_WILSON
)
def _base_state(self, gpus: list[str], task_types: list[str], now: datetime) -> dict[str, Any]:
@@ -554,7 +555,7 @@ class MarketIntelligenceManager:
and health_factor >= 0.5
and int(stats.get("recentTerminal") or 0) >= DEFAULT_GPU_MIN_RECENT_TERMINALS
and int(stats.get("recentSuccess") or 0) > 0
and wilson >= DEFAULT_FRAMEWORK_MIN_WILSON
and wilson >= DEFAULT_GPU_MIN_WILSON
)
result[gpu] = stats
@@ -661,6 +662,25 @@ class MarketIntelligenceManager:
# Missing live data falls back to the scheduler's already-proven
# long/recent pools. Explicitly failed public gates never do.
if eligibility is False:
low_sample = int(item.get("recentTerminal") or 0) < DEFAULT_GPU_MIN_RECENT_TERMINALS
has_recent_success = int(item.get("recentSuccess") or 0) > 0
recent_wilson = float(item.get("recentWilsonLowerBound") or 0.0)
healthy = float(item.get("healthFactor") or 1.0) >= 0.5
has_proven_framework = any(
bool(framework.get("officialConfigValid", False))
and int(framework.get("modelCount") or 0) >= self.framework_min_samples
and float(framework.get("wilsonLowerBound") or 0.0) >= DEFAULT_FRAMEWORK_MIN_WILSON
for by_gpu in ((self.state or {}).get("frameworkStats") or {}).values()
for framework in ((by_gpu or {}).get(gpu) or {}).values()
)
if (
item.get("canVerify") is False
or not low_sample
or not has_recent_success
or recent_wilson < DEFAULT_GPU_MIN_WILSON
or not healthy
or not has_proven_framework
):
continue
ranked.append((-float(item.get("selectionWeight") or 1.0), index, gpu))
return [gpu for _weight, _index, gpu in sorted(ranked)]
@@ -723,6 +743,7 @@ class MarketIntelligenceManager:
recent_item = (self.local_outcome_stats.get("recentCombinationStats") or {}).get(local_key) or {}
recent_success = max(0, int(recent_item.get("successCount") or 0))
recent_all_failures = max(0, int(recent_item.get("failureCount") or 0))
recent_failure = max(
0,
int(recent_item.get("attributableFailureCount", recent_item.get("failureCount") or 0)),
@@ -743,6 +764,16 @@ class MarketIntelligenceManager:
elif last_terminal_at is not None and consecutive_failures >= 5:
circuit_reason = "five_consecutive_local_failures"
circuit_until = last_terminal_at + timedelta(hours=12)
elif (
last_terminal_at is not None
and recent_success + recent_all_failures >= 5
and recent_success / (recent_success + recent_all_failures) < 0.10
):
# Runtime-empty and other unresolved failures must not create a
# permanent compatibility blacklist. Repetition is still strong
# operational evidence, so pause the route while feedback settles.
circuit_reason = "recent_terminal_success_below_10_percent"
circuit_until = last_terminal_at + timedelta(hours=12)
elif last_terminal_at is not None and recent_samples >= 20 and recent_rate is not None and recent_rate < 0.20:
circuit_reason = "recent_local_success_below_20_percent"
circuit_until = last_terminal_at + timedelta(hours=6)

21
modelhub_submmit_api/modelhub_client.py
View File

@@ -38,6 +38,7 @@ def parse_model_submission_precheck(payload: Any) -> dict[str, Any]:
raise ModelHubAPIError("Community model precheck verifyResult is invalid", payload=payload)
blocked_gpus: set[str] = set()
successful_gpus: set[str] = set()
failed_frameworks_by_gpu: dict[str, set[str]] = {}
for gpu, gpu_result in verify_result.items():
name = str(gpu)
@@ -48,6 +49,12 @@ def parse_model_submission_precheck(payload: Any) -> dict[str, Any]:
records = gpu_result.get("records")
if status == "未验证" and isinstance(records, list) and not records:
continue
blocked_gpus.add(name)
if status == "已验证" or any(
isinstance(record, dict) and _positive_verify_result(record.get("verifyResult"))
for record in (records or [])
):
successful_gpus.add(name)
if status == "验证失败" and isinstance(records, list) and records:
frameworks: set[str] = set()
for record in records:
@@ -63,20 +70,26 @@ def parse_model_submission_precheck(payload: Any) -> dict[str, Any]:
frameworks.add(framework)
else:
failed_frameworks_by_gpu[name] = frameworks
continue
# Verified, validating and unknown states remain closed. A failed GPU
# opens only for a different framework with explicit failed records.
blocked_gpus.add(name)
# Accuracy-first admission: any existing model/GPU state remains
# closed. Failed-framework details are diagnostic evidence only.
return {
"isInDB": data.get("isInDB") is True,
"processedGpus": blocked_gpus,
"successfulGpus": successful_gpus,
"failedFrameworksByGpu": failed_frameworks_by_gpu,
"verifyResult": verify_result,
"modelInfo": data.get("modelInfo") if isinstance(data.get("modelInfo"), dict) else {},
}
def _positive_verify_result(value: Any) -> bool:
try:
return int(value) > 0
except (TypeError, ValueError):
return False
class ModelHubClient:
def __init__(
self,

11
modelhub_submmit_api/outcome_tracker.py
View File

@@ -344,6 +344,17 @@ class OutcomeTracker:
self._by_model_gpu[(model_id, target_gpu)].append(record)
append_jsonl(self.path, record)
def pending_route_count_for_model(self, model_id: str) -> int:
"""Return distinct locally submitted GPU routes still awaiting an outcome."""
pending_gpus: set[str] = set()
for record in self._records:
if record.get("modelId") != model_id or record.get("outcome") != "pending":
continue
target_gpu = str(record.get("targetGpu") or "").strip()
if target_gpu:
pending_gpus.add(target_gpu)
return len(pending_gpus)
def mark_policy_cancellations(self, decisions: list[dict[str, Any]]) -> int:
"""Persist intentional task stops so they never become failure evidence."""
marked = 0

4
modelhub_submmit_api/routing_engine.py
View File

@@ -123,6 +123,9 @@ class SuccessFirstRoutingEngine:
"routingEtaHours": eta,
"routingEtaUnknown": eta_unknown,
"routingExpectedSuccessPerHour": expected,
"routingCommunityModelSuccesses": max(
0, int(candidate.get("communitySuccessfulGpuCount") or 0)
),
"strategyCategory": "unified_success_first",
}
)
@@ -168,6 +171,7 @@ class SuccessFirstRoutingEngine:
selected.sort(
key=lambda item: (
-int(item["routingSuccessBand"]),
-int(item.get("routingCommunityModelSuccesses") or 0),
-float(item["routingExpectedSuccessPerHour"]),
-int(item["routingSamples"]),
-int(item.get("downloads") or 0),

2
modelhub_submmit_api/version.py
View File

@@ -1 +1 @@
AGENT_VERSION = "2026.09.20.2"
AGENT_VERSION = "2026.09.22.1"

6
tests/test_concurrency.py
View File

@@ -733,9 +733,9 @@ class ClientPoolConcurrencyTests(unittest.TestCase):
)
self.assertEqual(2, summary["submittedCount"])
self.assertEqual(len(supported_gpus), summary["skipReasonCounts"]["already_processed_for_gpu"])
self.assertEqual("popular_history", summary["scanStages"][-1]["name"])
self.assertEqual("downloads", summary["scanStages"][-1]["sort"])
self.assertEqual(1, summary["skipReasonCounts"]["model_route_limit_reached"])
self.assertEqual("all_history", summary["scanStages"][-1]["name"])
self.assertEqual("last_modified", summary["scanStages"][-1]["sort"])
self.assertIsNone(discovery.calls[-1])
def test_adaptive_strategy_counts_only_platform_accepted_tasks(self) -> None:

124
tests/test_market_intelligence.py
View File

@@ -219,7 +219,7 @@ class MarketIntelligenceTests(unittest.TestCase):
"frameworkStats": {
"text-generation": {
"gpu": {
"vllm": {"modelCount": 1000, "wilsonLowerBound": 0.10},
"vllm": {"modelCount": 1000, "wilsonLowerBound": 0.30},
"vllm-mlu": {"modelCount": 800, "wilsonLowerBound": 0.35},
"vllm-customized": {"modelCount": 2, "wilsonLowerBound": 0.90},
}
@@ -357,7 +357,7 @@ class MarketIntelligenceTests(unittest.TestCase):
"frameworkStats": {
"text-generation": {
"Cambricon_mlu-370-x4": {
"vllm": {"modelCount": 1000, "wilsonLowerBound": 0.10},
"vllm": {"modelCount": 1000, "wilsonLowerBound": 0.30},
"future-engine": {
"modelCount": 1000,
"successCount": 500,
@@ -396,7 +396,7 @@ class MarketIntelligenceTests(unittest.TestCase):
"frameworkStats": {
"text-generation": {
"Cambricon_mlu-370-x4": {
"vllm": {"modelCount": 1000, "wilsonLowerBound": 0.10},
"vllm": {"modelCount": 1000, "wilsonLowerBound": 0.30},
"future-engine": {
"modelCount": 2,
"wilsonLowerBound": 0.90,
@@ -454,6 +454,124 @@ class MarketIntelligenceTests(unittest.TestCase):
)
self.assertIsNone(candidate)
def test_low_confidence_public_framework_is_rejected_even_with_many_samples(self) -> None:
manager = MarketIntelligenceManager("unused.json", framework_min_samples=100)
manager.state = {
"frameworkStats": {
"text-generation": {
"Cambricon_mlu-370-x4": {
"vllm": {"modelCount": 10_000, "wilsonLowerBound": 0.12},
}
}
}
}
candidate = choose_candidate_for_gpu(
model=HFModelSummary(
repo_id="owner/model",
downloads=100,
last_modified=None,
pipeline_tag="text-generation",
),
inspection=ModelInspection(repo_id="owner/model", weight_files=["model.safetensors"]),
template_selector=TemplateSelector(),
task_types=["text-generation"],
target_gpu="Cambricon_mlu-370-x4",
market_intelligence=manager,
)
self.assertIsNone(candidate)
def test_repeated_ambiguous_failures_open_operational_circuit(self) -> None:
manager = MarketIntelligenceManager("unused.json", framework_min_samples=100)
manager.state = {
"frameworkStats": {
"text-generation": {
"Biren_166m": {
"vllm_fix_tokenizer": {"modelCount": 10_000, "wilsonLowerBound": 0.35},
}
}
}
}
manager.set_local_outcome_stats(
{
"recentCombinationStats": {
"Biren_166m|vllm_fix_tokenizer|text-generation": {
"successCount": 0,
"failureCount": 5,
"attributableFailureCount": 0,
"consecutiveFailures": 0,
"lastTerminalAt": datetime.now(timezone.utc).isoformat(),
}
}
}
)
metadata = manager.framework_metadata(
"text-generation", "Biren_166m", "vllm_fix_tokenizer"
)
self.assertTrue(metadata["frameworkCircuitOpen"])
self.assertEqual(
"recent_terminal_success_below_10_percent",
metadata["frameworkCircuitReason"],
)
def test_low_sample_gpu_with_recent_success_and_proven_framework_remains_eligible(self) -> None:
manager = MarketIntelligenceManager("unused.json", framework_min_samples=100)
manager.state = {
"gpuStats": {
"Sunrise_pt-200-x1": {
"canVerify": True,
"submissionEligible": False,
"recentTerminal": 10,
"recentSuccess": 5,
"recentWilsonLowerBound": 0.24,
"healthFactor": 1.0,
"selectionWeight": 0.5,
}
},
"frameworkStats": {
"text-generation": {
"Sunrise_pt-200-x1": {
"vllm_fix_tokenizer": {
"modelCount": 6_000,
"wilsonLowerBound": 0.41,
"officialConfigValid": True,
}
}
}
},
}
self.assertEqual(
["Sunrise_pt-200-x1"],
manager.eligible_gpus(["Sunrise_pt-200-x1"]),
)
def test_low_sample_gpu_does_not_bypass_absolute_quality_floor(self) -> None:
manager = MarketIntelligenceManager("unused.json", framework_min_samples=100)
manager.state = {
"gpuStats": {
"gpu-low-quality": {
"canVerify": True,
"submissionEligible": False,
"recentTerminal": 10,
"recentSuccess": 1,
"recentWilsonLowerBound": 0.02,
"healthFactor": 1.0,
"selectionWeight": 0.5,
}
},
"frameworkStats": {
"text-generation": {
"gpu-low-quality": {
"vllm": {
"modelCount": 6_000,
"wilsonLowerBound": 0.41,
"officialConfigValid": True,
}
}
}
},
}
self.assertEqual([], manager.eligible_gpus(["gpu-low-quality"]))
def test_five_consecutive_local_failures_circuit_break_framework(self) -> None:
manager = MarketIntelligenceManager("unused.json", framework_min_samples=100)
manager.state = {

84
tests/test_submission_safety.py
View File

@@ -21,6 +21,7 @@ from modelhub_client import ( # noqa: E402
parse_model_submission_precheck,
)
from models import HFModelSummary, ModelInspection # noqa: E402
from outcome_tracker import OutcomeTracker # noqa: E402
from submission_exclusions import SubmissionExclusionStore # noqa: E402
from template_selector import TemplateSelector # noqa: E402
@@ -286,9 +287,10 @@ class SubmissionSafetyTests(unittest.TestCase):
}
precheck = parse_model_submission_precheck(payload)
self.assertEqual(
{"running", "success", "unknown", "failed_without_framework"},
{"failed", "running", "success", "unknown", "failed_without_framework"},
precheck["processedGpus"],
)
self.assertEqual({"success"}, precheck["successfulGpus"])
self.assertEqual({"failed": {"vllm"}}, precheck["failedFrameworksByGpu"])
def test_unverified_gpu_becomes_candidate_but_validating_gpu_does_not(self) -> None:
@@ -321,7 +323,7 @@ class SubmissionSafetyTests(unittest.TestCase):
self.assertEqual([], active_candidates)
self.assertEqual("already_processed_for_gpu", active_skips[0]["reason"])
def test_failed_gpu_requires_an_alternative_framework(self) -> None:
def test_failed_gpu_remains_closed_even_when_an_alternative_framework_exists(self) -> None:
model = SafeDiscovery().model
client = FailedFrameworkClient()
candidates, skipped, failed = process_model_for_candidates(
@@ -334,23 +336,14 @@ class SubmissionSafetyTests(unittest.TestCase):
market_intelligence=TwoFrameworkMarket(), # type: ignore[arg-type]
official_registry=AllGpuOfficialRegistry(), # type: ignore[arg-type]
)
self.assertEqual([], skipped)
self.assertEqual("prior_failed_gpu_closed_accuracy_first", skipped[-1]["reason"])
self.assertEqual([], failed)
self.assertEqual(["vllm-mlu"], [item["framework"] for item in candidates])
self.assertEqual(["vllm"], candidates[0]["previousFailedFrameworks"])
self.assertEqual([], candidates)
blocked = submit_candidate({**candidates[0], "framework": "vllm"}, client) # type: ignore[arg-type]
accepted = submit_candidate(candidates[0], client) # type: ignore[arg-type]
blocked = submit_candidate(candidate("Cambricon_mlu-370-x4"), client) # type: ignore[arg-type]
self.assertEqual("duplicate", blocked["outcome"])
self.assertEqual("prior_failed_framework_for_gpu", blocked["reason"])
self.assertEqual("submitted", accepted["outcome"])
self.assertEqual(1, client.add_calls)
unvetted = submit_candidate(
{**candidates[0], "frameworkEvidenceQualified": False}, client # type: ignore[arg-type]
)
self.assertEqual("framework_evidence_deferred", unvetted["outcome"])
self.assertEqual(1, client.add_calls)
self.assertEqual("prior_failed_gpu_closed_accuracy_first", blocked["reason"])
self.assertEqual(0, client.add_calls)
no_routes, no_route_skips, _ = process_model_for_candidates(
model=model,
@@ -363,7 +356,7 @@ class SubmissionSafetyTests(unittest.TestCase):
official_registry=AllGpuOfficialRegistry(), # type: ignore[arg-type]
)
self.assertEqual([], no_routes)
self.assertEqual("prior_failed_framework_no_alternative", no_route_skips[-1]["reason"])
self.assertEqual("prior_failed_gpu_closed_accuracy_first", no_route_skips[-1]["reason"])
no_market_routes, no_market_skips, _ = process_model_for_candidates(
model=model,
@@ -375,10 +368,65 @@ class SubmissionSafetyTests(unittest.TestCase):
)
self.assertEqual([], no_market_routes)
self.assertEqual(
"prior_failed_gpu_requires_public_framework_evidence",
"prior_failed_gpu_closed_accuracy_first",
no_market_skips[-1]["reason"],
)
def test_model_route_limit_is_global_across_poll_cycles(self) -> None:
class SixRouteClient:
@staticmethod
def model_submission_precheck(_model_id: str) -> dict:
return {
"processedGpus": {f"gpu-{index}" for index in range(1, 7)},
"successfulGpus": {"gpu-1"},
"failedFrameworksByGpu": {},
}
candidates, skipped, failed = process_model_for_candidates(
model=SafeDiscovery().model,
hf_discovery=SafeDiscovery(), # type: ignore[arg-type]
modelhub_client=SixRouteClient(), # type: ignore[arg-type]
template_selector=TemplateSelector(),
target_gpus=["gpu-7"],
allowed_task_types=["text-generation"],
)
self.assertEqual([], candidates)
self.assertEqual([], failed)
self.assertEqual("model_route_limit_reached", skipped[0]["reason"])
self.assertEqual(6, skipped[0]["processedGpuCount"])
def test_model_waits_for_feedback_after_two_pending_routes(self) -> None:
with tempfile.TemporaryDirectory() as temporary_dir:
root = Path(temporary_dir)
tracker = OutcomeTracker(
root / "outcomes.jsonl",
checkpoint_path=root / "checkpoint.json",
recent_path=root / "recent.jsonl",
)
tracker.record_submission(
"owner/model", "gpu-1", "vllm", "text-generation", "task-1",
"2026-09-22T00:00:00+00:00",
)
tracker.record_submission(
"owner/model", "gpu-2", "vllm", "text-generation", "task-2",
"2026-09-22T00:01:00+00:00",
)
candidates, skipped, failed = process_model_for_candidates(
model=SafeDiscovery().model,
hf_discovery=SafeDiscovery(), # type: ignore[arg-type]
modelhub_client=OtherGpuProcessedClient(), # type: ignore[arg-type]
template_selector=TemplateSelector(),
target_gpus=["gpu-3"],
allowed_task_types=["text-generation"],
outcome_tracker=tracker,
)
self.assertEqual([], candidates)
self.assertEqual([], failed)
self.assertEqual("model_pending_feedback_limit", skipped[0]["reason"])
self.assertEqual(2, skipped[0]["pendingRouteCount"])
def test_published_size_avoids_official_calls_for_certain_oom(self) -> None:
class CountingRegistry(AllGpuOfficialRegistry):
calls = 0