CoolBoy/submmit
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

fix: restore accuracy-first route admission

Browse Source
This commit is contained in:
CoolBoy
2026-09-22 15:26:42 +08:00
parent 4afce879dc
commit 6154dc738f
11 changed files with 343 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
modelhub_submmit_api/README.md
View File

@@ -79,21 +79,25 @@ bash run_poll.sh --dry-run
## Behavior
- The runner auto-discovers all safe GPU/template combinations from the public submit catalog.
- Automatic GPU selection uses exact 70/30 accepted-task scheduling: long-term
Wilson-ranked top 3 GPUs and the top GPUs from the latest 1,000 terminal tasks.
There is no all-GPU exploration category.
- Within each category, weighted-fair scheduling uses estimated queue backlog hours,
- Automatic GPU selection uses one success-first score built from qualified
local cohorts and ModelHub public evidence. There is no random or fixed-quota
exploration category.
- Within the qualified success band, scheduling uses estimated queue backlog hours,
recent public throughput/success, machine availability, and worker concurrency.
Unavailable or stalled GPU pools are circuit-broken instead of continuing to absorb work.
- Compatible frameworks are ranked by ModelHub public aggregate success statistics
plus capped local GPU+framework evidence, with a 300-sample public minimum and
Wilson confidence bounds. Missing or undersized public evidence receives zero
traffic rather than falling back to exploration.
a 20% Wilson lower-bound floor. GPU pools normally require a 15% floor.
Missing or undersized public evidence receives zero traffic rather than
falling back to exploration; a low-sample GPU requires a recent success and a
framework that already passes the stricter framework gate.
- New frameworks are discovered from the live catalog but get no novelty bonus.
They are eligible only with a complete official build config that passes local
validation and a confidence score at least 10% above the best incumbent.
- Five consecutive local failures pause a GPU/framework pair for 12 hours; a
sub-20% rate over the latest 20 terminal tasks pauses it for 6 hours.
- Five consecutive local failures pause a GPU/framework pair for 12 hours. Five
or more recent terminal outcomes below 10% success also pause it for 12 hours,
including ambiguous runtime failures. A sub-20% attributable rate over the
latest 20 terminal tasks pauses it for 6 hours.
- An explicit "framework does not support this model/architecture" failure learns
a 30-day GPU + framework + task + architecture block. Architecture identity
comes from candidate `config.json` plus exact unsupported `model_type` or
@@ -114,20 +118,22 @@ bash run_poll.sh --dry-run
duplicates do not advance it. The next cycle refreshes platform history before submitting again.
- Strategy state is stored in `.modelhub_state/gpu_strategy.json`; a generation never recalculates
during candidate submission.
- Candidate discovery starts with the configured recent window, then automatically expands to
7 days, 30 days, and the first 3,000 raw ModelScope rows ordered by update time. If the
candidate pool is still short, it rotates through up to 200 download-ranked rows per cycle
as an older-model fallback.
- Candidate discovery starts with the configured recent window, then
automatically expands to 7 days, 30 days, and the first 3,000 raw ModelScope
rows ordered by update time. Download popularity is not treated as ModelHub
compatibility evidence.
- Model verification responses are cached across poll cycles for 15 minutes. Local model/GPU
failures cool down after 24 hours instead of remaining permanently blocked.
- Community deduplication is model/GPU-specific: another GPU's adaptation does not block the
current GPU. `未验证` with no records is eligible. `验证失败` is eligible only with a
different publicly vetted framework after all recorded attempts explicitly failed;
`已验证`, `验证中`, and ambiguous records remain blocked. Every actual submission
performs a fresh uncached check for its exact GPU and framework.
- Community deduplication is model/GPU-specific: another GPU's adaptation does
not block the current GPU. Only `未验证` with no records is eligible;
`验证失败`, `已验证`, `验证中`, and ambiguous records remain blocked. Every actual
submission performs a fresh uncached check for its exact GPU.
- If the community lookup is unavailable, submission is deferred. A platform model-uniqueness
rejection permanently excludes only that model/GPU combination from future local retries.
- The same framework is not retried for a model/GPU after a recorded failure.
- A model may use at most six GPU routes across poll cycles, and at most two
locally submitted routes for that model may await feedback at once. The first
six ranked GPUs remain preferred; lower-ranked eligible GPUs are inspected in
order only when that tranche yields no safe candidate.
- Multiple ModelHub tokens are pooled and used to route submissions to the account with available async capacity.
- Concurrent submissions reserve account slots locally, and an account-capacity race automatically falls through to another account.
- Concurrent local processes claim model/GPU pairs in `.modelhub_state/submission_claims.jsonl`; shared ledger, history, and outcome files use process locks and atomic replacement.

84
modelhub_submmit_api/main.py
View File

@@ -66,6 +66,7 @@ DEFAULT_LEDGER_PATH = Path("ledger/submissions.jsonl")
ADAPTIVE_SCAN_MAX_MODELS = 3000
ADAPTIVE_SCAN_MIN_FALLBACK_MODELS = 500
MAX_ROUTES_PER_MODEL = 6
MAX_PENDING_ROUTES_PER_MODEL = 2
def build_parser() -> argparse.ArgumentParser:
@@ -572,9 +573,11 @@ def process_model_for_candidates(
if hasattr(modelhub_client, "model_submission_precheck"):
precheck = modelhub_client.model_submission_precheck(model.repo_id)
processed_gpus = set(precheck.get("processedGpus") or set())
successful_gpus = set(precheck.get("successfulGpus") or set())
failed_frameworks_by_gpu = precheck.get("failedFrameworksByGpu") or {}
else:
processed_gpus = modelhub_client.processed_gpus_for_model(model.repo_id)
successful_gpus = set()
failed_frameworks_by_gpu = {}
except Exception as exc:
return (
@@ -582,6 +585,31 @@ def process_model_for_candidates(
[{"repoId": model.repo_id, "reason": "community_precheck_unavailable_fail_closed"}],
[{"repoId": model.repo_id, "reason": f"community_precheck_error:{exc}"}],
)
if len(processed_gpus) >= MAX_ROUTES_PER_MODEL:
return (
[],
[{
"repoId": model.repo_id,
"reason": "model_route_limit_reached",
"processedGpuCount": len(processed_gpus),
}],
[],
)
pending_route_count = (
outcome_tracker.pending_route_count_for_model(model.repo_id)
if outcome_tracker is not None
else 0
)
if pending_route_count >= MAX_PENDING_ROUTES_PER_MODEL:
return (
[],
[{
"repoId": model.repo_id,
"reason": "model_pending_feedback_limit",
"pendingRouteCount": pending_route_count,
}],
[],
)
candidates: list[dict[str, Any]] = []
skipped: list[dict[str, Any]] = []
failed: list[dict[str, Any]] = []
@@ -605,33 +633,14 @@ def process_model_for_candidates(
if target_gpu in processed_gpus:
skipped.append({"repoId": model.repo_id, "targetGpu": target_gpu, "reason": "already_processed_for_gpu"})
continue
if target_gpu in failed_frameworks_by_gpu and market_intelligence is None:
if target_gpu in failed_frameworks_by_gpu:
skipped.append({
"repoId": model.repo_id,
"targetGpu": target_gpu,
"reason": "prior_failed_gpu_requires_public_framework_evidence",
"reason": "prior_failed_gpu_closed_accuracy_first",
})
continue
failed_frameworks = set(failed_frameworks_by_gpu.get(target_gpu) or set())
if (
failed_frameworks
and market_intelligence is not None
and hasattr(market_intelligence, "has_vetted_alternative_framework")
and allowed_task_types_set
and all(market_intelligence.has_framework_route(task_type, target_gpu) for task_type in allowed_task_types_set)
and not any(
market_intelligence.has_vetted_alternative_framework(
task_type, target_gpu, failed_frameworks
)
for task_type in allowed_task_types_set
)
):
skipped.append({
"repoId": model.repo_id,
"targetGpu": target_gpu,
"reason": "prior_failed_framework_no_alternative",
})
continue
if outcome_tracker and outcome_tracker.is_model_gpu_failed(model.repo_id, target_gpu):
skipped.append({"repoId": model.repo_id, "targetGpu": target_gpu, "reason": "local_failure_cooldown_24h"})
continue
@@ -734,6 +743,9 @@ def process_model_for_candidates(
record = candidate_to_record(best)
if excluded_frameworks:
record["previousFailedFrameworks"] = sorted(excluded_frameworks)
record["communitySuccessfulGpuCount"] = len(successful_gpus)
if successful_gpus:
record["communitySuccessfulGpus"] = sorted(successful_gpus)
if outcome_tracker is not None:
record["transformersPrerequisiteSatisfied"] = (
outcome_tracker.has_non_transformers_failure(model.repo_id)
@@ -810,20 +822,6 @@ def build_adaptive_scan_stages(
"limit": max_models,
}
)
if allow_older_than_recent_window:
# ModelScope only exposes the first 3,000 raw rows for a sort order.
# Downloads ordering reaches a different older, established slice
# after the recent-update slice has produced too few candidates.
stages.append(
{
"name": "popular_history",
"updatedAfter": None,
"limit": min(max_models, 200),
"sort": "downloads",
"rotatePages": True,
}
)
deduped: list[dict[str, Any]] = []
seen: set[tuple[str | None, int, str]] = set()
for stage in stages:
@@ -951,20 +949,20 @@ def submit_candidate(
"candidate": candidate,
"reason": "already_processed_for_gpu",
}
failed_frameworks = set(
(precheck.get("failedFrameworksByGpu") or {}).get(candidate["targetGpu"]) or set()
)
if candidate["framework"] in failed_frameworks:
if len(set(precheck.get("processedGpus") or set())) >= MAX_ROUTES_PER_MODEL:
return {
"outcome": "duplicate",
"candidate": candidate,
"reason": "prior_failed_framework_for_gpu",
"reason": "model_route_limit_reached",
}
if failed_frameworks and not candidate.get("frameworkEvidenceQualified"):
failed_frameworks = set(
(precheck.get("failedFrameworksByGpu") or {}).get(candidate["targetGpu"]) or set()
)
if failed_frameworks:
return {
"outcome": "framework_evidence_deferred",
"outcome": "duplicate",
"candidate": candidate,
"reason": "prior_failed_gpu_requires_public_framework_evidence",
"reason": "prior_failed_gpu_closed_accuracy_first",
}
payload = {

39
modelhub_submmit_api/market_intelligence.py
View File

@@ -20,7 +20,8 @@ DEFAULT_THROUGHPUT_WINDOW_HOURS = 6
DEFAULT_FETCH_WORKERS = 4
DEFAULT_FRAMEWORK_MIN_SAMPLES = 300
DEFAULT_GPU_MIN_RECENT_TERMINALS = 20
DEFAULT_FRAMEWORK_MIN_WILSON = 0.05
DEFAULT_FRAMEWORK_MIN_WILSON = 0.20
DEFAULT_GPU_MIN_WILSON = 0.15
DEFAULT_COMMUNITY_WINDOW_DAYS = 30
DEFAULT_COMMUNITY_MAX_MODELS = 200
DEFAULT_COMMUNITY_HYDRATE_PER_REFRESH = 50
@@ -201,7 +202,7 @@ class MarketIntelligenceManager:
and health_factor >= 0.5
and int(item.get("recentTerminal") or 0) >= DEFAULT_GPU_MIN_RECENT_TERMINALS
and int(item.get("recentSuccess") or 0) > 0
and wilson >= DEFAULT_FRAMEWORK_MIN_WILSON
and wilson >= DEFAULT_GPU_MIN_WILSON
)
def _base_state(self, gpus: list[str], task_types: list[str], now: datetime) -> dict[str, Any]:
@@ -554,7 +555,7 @@ class MarketIntelligenceManager:
and health_factor >= 0.5
and int(stats.get("recentTerminal") or 0) >= DEFAULT_GPU_MIN_RECENT_TERMINALS
and int(stats.get("recentSuccess") or 0) > 0
and wilson >= DEFAULT_FRAMEWORK_MIN_WILSON
and wilson >= DEFAULT_GPU_MIN_WILSON
)
result[gpu] = stats
@@ -661,7 +662,26 @@ class MarketIntelligenceManager:
# Missing live data falls back to the scheduler's already-proven
# long/recent pools. Explicitly failed public gates never do.
if eligibility is False:
continue
low_sample = int(item.get("recentTerminal") or 0) < DEFAULT_GPU_MIN_RECENT_TERMINALS
has_recent_success = int(item.get("recentSuccess") or 0) > 0
recent_wilson = float(item.get("recentWilsonLowerBound") or 0.0)
healthy = float(item.get("healthFactor") or 1.0) >= 0.5
has_proven_framework = any(
bool(framework.get("officialConfigValid", False))
and int(framework.get("modelCount") or 0) >= self.framework_min_samples
and float(framework.get("wilsonLowerBound") or 0.0) >= DEFAULT_FRAMEWORK_MIN_WILSON
for by_gpu in ((self.state or {}).get("frameworkStats") or {}).values()
for framework in ((by_gpu or {}).get(gpu) or {}).values()
)
if (
item.get("canVerify") is False
or not low_sample
or not has_recent_success
or recent_wilson < DEFAULT_GPU_MIN_WILSON
or not healthy
or not has_proven_framework
):
continue
ranked.append((-float(item.get("selectionWeight") or 1.0), index, gpu))
return [gpu for _weight, _index, gpu in sorted(ranked)]
@@ -723,6 +743,7 @@ class MarketIntelligenceManager:
recent_item = (self.local_outcome_stats.get("recentCombinationStats") or {}).get(local_key) or {}
recent_success = max(0, int(recent_item.get("successCount") or 0))
recent_all_failures = max(0, int(recent_item.get("failureCount") or 0))
recent_failure = max(
0,
int(recent_item.get("attributableFailureCount", recent_item.get("failureCount") or 0)),
@@ -743,6 +764,16 @@ class MarketIntelligenceManager:
elif last_terminal_at is not None and consecutive_failures >= 5:
circuit_reason = "five_consecutive_local_failures"
circuit_until = last_terminal_at + timedelta(hours=12)
elif (
last_terminal_at is not None
and recent_success + recent_all_failures >= 5
and recent_success / (recent_success + recent_all_failures) < 0.10
):
# Runtime-empty and other unresolved failures must not create a
# permanent compatibility blacklist. Repetition is still strong
# operational evidence, so pause the route while feedback settles.
circuit_reason = "recent_terminal_success_below_10_percent"
circuit_until = last_terminal_at + timedelta(hours=12)
elif last_terminal_at is not None and recent_samples >= 20 and recent_rate is not None and recent_rate < 0.20:
circuit_reason = "recent_local_success_below_20_percent"
circuit_until = last_terminal_at + timedelta(hours=6)

21
modelhub_submmit_api/modelhub_client.py
View File

@@ -38,6 +38,7 @@ def parse_model_submission_precheck(payload: Any) -> dict[str, Any]:
raise ModelHubAPIError("Community model precheck verifyResult is invalid", payload=payload)
blocked_gpus: set[str] = set()
successful_gpus: set[str] = set()
failed_frameworks_by_gpu: dict[str, set[str]] = {}
for gpu, gpu_result in verify_result.items():
name = str(gpu)
@@ -48,6 +49,12 @@ def parse_model_submission_precheck(payload: Any) -> dict[str, Any]:
records = gpu_result.get("records")
if status == "未验证" and isinstance(records, list) and not records:
continue
blocked_gpus.add(name)
if status == "已验证" or any(
isinstance(record, dict) and _positive_verify_result(record.get("verifyResult"))
for record in (records or [])
):
successful_gpus.add(name)
if status == "验证失败" and isinstance(records, list) and records:
frameworks: set[str] = set()
for record in records:
@@ -63,20 +70,26 @@ def parse_model_submission_precheck(payload: Any) -> dict[str, Any]:
frameworks.add(framework)
else:
failed_frameworks_by_gpu[name] = frameworks
continue
# Verified, validating and unknown states remain closed. A failed GPU
# opens only for a different framework with explicit failed records.
blocked_gpus.add(name)
# Accuracy-first admission: any existing model/GPU state remains
# closed. Failed-framework details are diagnostic evidence only.
return {
"isInDB": data.get("isInDB") is True,
"processedGpus": blocked_gpus,
"successfulGpus": successful_gpus,
"failedFrameworksByGpu": failed_frameworks_by_gpu,
"verifyResult": verify_result,
"modelInfo": data.get("modelInfo") if isinstance(data.get("modelInfo"), dict) else {},
}
def _positive_verify_result(value: Any) -> bool:
try:
return int(value) > 0
except (TypeError, ValueError):
return False
class ModelHubClient:
def __init__(
self,

11
modelhub_submmit_api/outcome_tracker.py
View File

@@ -344,6 +344,17 @@ class OutcomeTracker:
self._by_model_gpu[(model_id, target_gpu)].append(record)
append_jsonl(self.path, record)
def pending_route_count_for_model(self, model_id: str) -> int:
"""Return distinct locally submitted GPU routes still awaiting an outcome."""
pending_gpus: set[str] = set()
for record in self._records:
if record.get("modelId") != model_id or record.get("outcome") != "pending":
continue
target_gpu = str(record.get("targetGpu") or "").strip()
if target_gpu:
pending_gpus.add(target_gpu)
return len(pending_gpus)
def mark_policy_cancellations(self, decisions: list[dict[str, Any]]) -> int:
"""Persist intentional task stops so they never become failure evidence."""
marked = 0

4
modelhub_submmit_api/routing_engine.py
View File

@@ -123,6 +123,9 @@ class SuccessFirstRoutingEngine:
"routingEtaHours": eta,
"routingEtaUnknown": eta_unknown,
"routingExpectedSuccessPerHour": expected,
"routingCommunityModelSuccesses": max(
0, int(candidate.get("communitySuccessfulGpuCount") or 0)
),
"strategyCategory": "unified_success_first",
}
)
@@ -168,6 +171,7 @@ class SuccessFirstRoutingEngine:
selected.sort(
key=lambda item: (
-int(item["routingSuccessBand"]),
-int(item.get("routingCommunityModelSuccesses") or 0),
-float(item["routingExpectedSuccessPerHour"]),
-int(item["routingSamples"]),
-int(item.get("downloads") or 0),

2
modelhub_submmit_api/version.py
View File

@@ -1 +1 @@
AGENT_VERSION = "2026.09.20.2"
AGENT_VERSION = "2026.09.22.1"