diff --git a/README.md b/README.md index 31d297a8..ccce6607 100644 --- a/README.md +++ b/README.md @@ -139,32 +139,21 @@ The verified capacities, safe repository-size boundaries, evidence hierarchy, and source links are recorded in `docs/gpu-memory-capacity-2026-08-10.md`. -Ambiguous custom architectures can optionally be reviewed by a small -OpenAI-compatible Qwen model. Qwen is lazy: deterministic rules handle repository -layout, model size, context length, known errors, and ordinary quantization cases -without an LLM call. Set -`MODELHUB_LLM_CLASSIFIER_ENDPOINT` to the full chat-completions URL and -`MODELHUB_LLM_CLASSIFIER_MODEL`; set `MODELHUB_LLM_CLASSIFIER_API_KEY` only when -the endpoint requires it. The default deny threshold is 0.85 and can be changed -with `MODELHUB_LLM_CLASSIFIER_MIN_DENY_CONFIDENCE`. For Alibaba Model Studio, -the aliases are `MODELHUB_QWEN_ENDPOINT`, `MODELHUB_QWEN_MODEL`, and -`MODELHUB_QWEN_API_KEY` (or `DASHSCOPE_API_KEY`); endpoint omission uses the -DashScope OpenAI-compatible URL. A root `.env` entry named `dashscope` is also -recognized directly, and the default model is `qwen3.7-flash`. Calls default to -one concurrent request and 20 -requests per rolling hour, configurable with `MODELHUB_LLM_MAX_CONCURRENT_REQUESTS` -and `MODELHUB_LLM_MAX_CALLS_PER_HOUR`. The LLM may only veto an -ambiguous candidate: it cannot bypass deterministic checks, introduce a new -framework, or override public success-evidence gates. Results are cached under -`.modelhub_state/llm_classifications.json`. +The online worker makes zero LLM calls. Candidate admission, GPU/framework +selection, queue cleanup, and failure feedback are deterministic and based on +repository metadata, platform capabilities, public outcomes, and explicit error +signatures. Merely storing a DashScope key in `.env` or setting a Qwen/LLM +environment variable does not activate inference. The standalone classifier +module remains only as an offline research helper for human-reviewed batches of +previously unseen errors; it is not wired into submission or outcome sync. Outcome synchronization downloads a bounded set of failure archives for submissions created by this worker (at most 40 per sync, four workers, three download attempts). Deterministic signatures classify memory, repository layout, -context-length, storage, and platform faults first. Only unresolved runtime errors -are sent to the optional LLM; a semantic result is promoted only at confidence -0.80 or higher. Signed log URLs remain in the ignored local outcome store and are -removed after classification. +context-length, storage, and platform faults. Unresolved runtime errors remain +explicitly ambiguous for later rule development instead of being sent to an LLM. +Signed log URLs remain in the ignored local outcome store and are removed after +classification. The 12-account failure study and routing rationale are recorded in `docs/failure-analysis-2026-08-10.md`. @@ -226,6 +215,9 @@ repository root `.env` key named `dashscope` without logging its value. Version `2026.08.11.1` adds account-owned cancellation of queued tasks that are deterministically over the selected GPU's ModelHub memory boundary, with a second active-state check and fail-closed handling for incomplete evidence. +Version `2026.08.11.2` removes LLM inference from every online path; credentials +alone cannot activate it, and unresolved cases remain available for offline, +human-reviewed rule development. ## Deploy diff --git a/docs/failure-analysis-2026-08-10.md b/docs/failure-analysis-2026-08-10.md index e62a59f7..c6e3e2ea 100644 --- a/docs/failure-analysis-2026-08-10.md +++ b/docs/failure-analysis-2026-08-10.md @@ -17,8 +17,8 @@ stratified sample, not the raw platform-wide frequency. | Code | Count | Sample share | Primary handling | | --- | ---: | ---: | --- | | `PREFLIGHT_OOM` | 299 | 25.1% | Deterministic model-size/GPU-memory gate | -| `MODEL_NOT_SUPPORTED` | 244 | 20.5% | Architecture history, then LLM for the long tail | -| missing structured report | 185 | 15.6% | Root-exception rules; LLM only when still ambiguous | +| `MODEL_NOT_SUPPORTED` | 244 | 20.5% | Architecture history and deterministic compatibility feedback | +| missing structured report | 185 | 15.6% | Root-exception rules; retain unknown roots for offline analysis | | `MODEL_LOAD_FAILED` | 132 | 11.1% | Repository checks, architecture/quantization review | | `EXECUTE_EMPTY_RESULT` | 126 | 10.6% | Separate platform faults from model faults first | | `MODEL_FILE_NOT_FOUND` | 102 | 8.6% | Require framework-specific root files | @@ -26,7 +26,7 @@ stratified sample, not the raw platform-wide frequency. | `CONTEXT_LENGTH_ERROR` | 20 | 1.7% | Clamp template context to the model limit | | `MISSING_OPERATOR` | 14 | 1.2% | Prefer another proven GPU/framework; semantic review | | `DEVICE_OOM` | 8 | 0.7% | Model/GPU memory-risk feedback | -| other | 9 | 0.8% | Taxonomy or LLM fallback | +| other | 9 | 0.8% | Taxonomy or explicit ambiguous classification | ## Important root causes @@ -61,18 +61,15 @@ stratified sample, not the raw platform-wide frequency. 1. Deterministic checks always run first and cannot be overridden. 2. Publicly proven GPU/framework eligibility remains mandatory. -3. Ambiguous custom architecture/remote-code cases may be sent to a configured - Qwen model. Ordinary quantization metadata alone does not justify an LLM call. -4. Only a high-confidence LLM denial blocks a candidate. `allow` cannot enable - a new framework, bypass OOM/file checks, or create exploration traffic. -5. LLM results are cached by model/profile/GPU/framework. A persisted rolling - hourly budget and single-request semaphore prevent repeated cycles from - spending unbounded inference time. -6. Outcome sync automatically inspects at most 40 locally submitted failure logs +3. The online worker never calls an LLM. Ambiguous architecture, remote-code, + and runtime cases remain explicitly unresolved instead of receiving a guess. +4. Previously unseen errors may be grouped offline with experimental tooling, + but only a human-reviewed deterministic rule can affect later submissions. +5. Outcome sync automatically inspects at most 40 locally submitted failure logs at a time with four download workers and no more than three attempts per log. - Confident semantic classifications feed the model/profile statistics; platform - failures are excluded from compatibility rates. -7. Repeated infrastructure failures still affect speed: three consecutive + Rule-classified model/profile failures feed the statistics; platform failures + are excluded from compatibility rates. +6. Repeated infrastructure failures still affect speed: three consecutive platform failures on a GPU/framework open a 30-minute circuit, while five attributable profile failures retain the 12-hour compatibility circuit. diff --git a/modelhub_submmit_api/README.md b/modelhub_submmit_api/README.md index 775397de..234bdd6b 100644 --- a/modelhub_submmit_api/README.md +++ b/modelhub_submmit_api/README.md @@ -21,9 +21,9 @@ It currently supports: - `hf_discovery.py`: ModelScope model discovery and inspection (keeps the legacy module name) - `modelhub_client.py`: ModelHub API client and token-pool routing - `history_stats.py`: online history aggregation, ranking, and warnings -- `candidate_preflight.py`: repository, memory, context, and LLM-assisted compatibility gates +- `candidate_preflight.py`: deterministic repository, memory, context, and compatibility gates - `failure_taxonomy.py`: deterministic/platform/semantic failure routing -- `llm_classifier.py`: optional cached OpenAI-compatible ambiguity classifier +- `llm_classifier.py`: offline-only experimental ambiguity-analysis helper - `template_selector.py`: template lookup and GPU normalization - `task_registry.py`: task-type and framework selection rules - `tests/`: unit tests and regression coverage @@ -165,21 +165,18 @@ The memory gate totals the complete recursive repository and applies the same verifiable GPU types have evidence-backed capacities; an incomplete repository size is deferred instead of estimated. See `../docs/gpu-memory-capacity-2026-08-10.md`. -Optional Qwen review uses `MODELHUB_QWEN_ENDPOINT`, `MODELHUB_QWEN_MODEL`, and -`MODELHUB_QWEN_API_KEY` (or `DASHSCOPE_API_KEY`). The generic -`MODELHUB_LLM_CLASSIFIER_*` names remain supported. A root `.env` key named -`dashscope` is loaded automatically, and the default model is `qwen3.7-flash`. -Qwen is called only for -unresolved architecture/remote-code semantics or ambiguous failure roots, with -a default rolling limit of 20 calls/hour and one concurrent request. Only -high-confidence denials block; an error, timeout, or abstention leaves the -already-vetted candidate eligible. +The online runner never constructs an LLM client. A DashScope key or any +`MODELHUB_QWEN_*`/`MODELHUB_LLM_CLASSIFIER_*` environment variable cannot enable +inference. `llm_classifier.py` remains available only for deliberately invoked, +offline experiments whose output is reviewed before being converted into a +deterministic rule. Outcome sync also classifies a bounded set of this worker's failed-task ZIP logs. -Hard error signatures run first; ambiguous runtime roots can use the configured -LLM. Platform faults are excluded from long-term compatibility scores and use a -short 30-minute breaker after three consecutive failures. Failed log downloads -are persisted and stop after three attempts. +Hard error signatures run first; ambiguous runtime roots remain explicitly +unclassified and are never sent to an LLM. Platform faults are excluded from +long-term compatibility scores and use a short 30-minute breaker after three +consecutive failures. Failed log downloads are persisted and stop after three +attempts. ## Output diff --git a/modelhub_submmit_api/main.py b/modelhub_submmit_api/main.py index 05dfe5b0..0be42b29 100644 --- a/modelhub_submmit_api/main.py +++ b/modelhub_submmit_api/main.py @@ -28,7 +28,6 @@ from market_intelligence import ( DEFAULT_THROUGHPUT_WINDOW_HOURS, MarketIntelligenceManager, ) -from llm_classifier import DEFAULT_LLM_CACHE_PATH, LLMAssistedClassifier from modelhub_client import ( DEFAULT_CAPACITY_STATE_PATH, ModelHubAPIError, @@ -142,7 +141,7 @@ def build_parser() -> argparse.ArgumentParser: ) parser.add_argument( "--llm-classifier-cache-path", - default=os.getenv("MODELHUB_LLM_CLASSIFIER_CACHE_PATH", str(DEFAULT_LLM_CACHE_PATH)), + default=os.getenv("MODELHUB_LLM_CLASSIFIER_CACHE_PATH", ".modelhub_state/llm_classifications.json"), help=argparse.SUPPRESS, ) parser.add_argument("--runs-dir", default=str(DEFAULT_RUNS_DIR), help=argparse.SUPPRESS) @@ -813,6 +812,9 @@ def run_submission( history_archive_path.parent.mkdir(parents=True, exist_ok=True) outcome_tracker = outcome_tracker or OutcomeTracker(Path(args.outcomes_path)) + # Online decisions are deliberately deterministic. The optional classifier + # module remains available for offline, human-reviewed analysis only. + outcome_tracker.set_failure_llm_classifier(None) submission_exclusion_store = SubmissionExclusionStore( Path(getattr(args, "submission_exclusions_path", DEFAULT_SUBMISSION_EXCLUSIONS_PATH)) ) @@ -821,31 +823,7 @@ def run_submission( "MODELHUB_DISABLE_CANDIDATE_PREFLIGHT", "" ).strip().lower() in {"1", "true", "yes"} if not disable_preflight: - llm_classifier = LLMAssistedClassifier( - endpoint=getattr(args, "llm_classifier_endpoint", None) - or os.getenv("MODELHUB_LLM_CLASSIFIER_ENDPOINT"), - model=getattr(args, "llm_classifier_model", None) - or os.getenv("MODELHUB_LLM_CLASSIFIER_MODEL"), - api_key=getattr(args, "llm_classifier_api_key", None) - or os.getenv("MODELHUB_LLM_CLASSIFIER_API_KEY"), - timeout_seconds=max( - 1, - int( - getattr(args, "llm_classifier_timeout_seconds", 0) - or os.getenv("MODELHUB_LLM_CLASSIFIER_TIMEOUT_SECONDS", "20") - ), - ), - min_deny_confidence=float( - getattr(args, "llm_classifier_min_deny_confidence", 0.0) - or os.getenv("MODELHUB_LLM_CLASSIFIER_MIN_DENY_CONFIDENCE", "0.85") - ), - cache_path=Path( - getattr(args, "llm_classifier_cache_path", None) - or os.getenv("MODELHUB_LLM_CLASSIFIER_CACHE_PATH", str(DEFAULT_LLM_CACHE_PATH)) - ), - ) - outcome_tracker.set_failure_llm_classifier(llm_classifier) - preflight_advisor = CandidatePreflightAdvisor(llm_classifier=llm_classifier) + preflight_advisor = CandidatePreflightAdvisor(llm_classifier=None) preflight_summary: dict[str, Any] = ( preflight_advisor.summary() if preflight_advisor is not None else {"enabled": False} ) diff --git a/modelhub_submmit_api/runner_common.py b/modelhub_submmit_api/runner_common.py index 6e4d2e07..594f53aa 100644 --- a/modelhub_submmit_api/runner_common.py +++ b/modelhub_submmit_api/runner_common.py @@ -89,7 +89,6 @@ def _add_token(tokens: list[str], token: str | None) -> None: def ensure_tokens(args: argparse.Namespace) -> None: - ensure_dashscope_key() primary_key_path = Path(getattr(args, "key_path", DEFAULT_KEY_PATH)) supplemental_key_path = primary_key_path.with_name(DEFAULT_KEYS_PATH.name) if not primary_key_path.exists(): diff --git a/modelhub_submmit_api/version.py b/modelhub_submmit_api/version.py index d82e8566..7471e7f4 100644 --- a/modelhub_submmit_api/version.py +++ b/modelhub_submmit_api/version.py @@ -1 +1 @@ -AGENT_VERSION = "2026.08.11.1" +AGENT_VERSION = "2026.08.11.2" diff --git a/tests/test_concurrency.py b/tests/test_concurrency.py index 06415114..c5dee79d 100644 --- a/tests/test_concurrency.py +++ b/tests/test_concurrency.py @@ -1,5 +1,6 @@ from __future__ import annotations +import os import sys import tempfile import threading @@ -7,6 +8,7 @@ import unittest from concurrent.futures import ThreadPoolExecutor from datetime import datetime, timedelta, timezone from pathlib import Path +from unittest.mock import patch PACKAGE_DIR = Path(__file__).resolve().parents[1] / "modelhub_submmit_api" @@ -208,6 +210,45 @@ def make_candidate(index: int) -> dict: class ClientPoolConcurrencyTests(unittest.TestCase): + def test_online_submission_does_not_construct_llm_even_when_key_is_present(self) -> None: + with tempfile.TemporaryDirectory() as temporary_dir: + root = Path(temporary_dir) + args = build_parser().parse_args( + [ + "--gpus", + "Iluvatar_bi-150", + "--task-types", + "text-generation", + "--limit", + "1", + "--max-scan-models", + "1", + "--skip-outcome-sync", + "--skip-history-archive", + ] + ) + args.runs_dir = str(root / "runs") + args.ledger_path = str(root / "ledger.jsonl") + args.outcomes_path = str(root / "outcomes.jsonl") + args.claims_path = str(root / "claims.jsonl") + args.history_archive_path = str(root / "history.jsonl") + + with ( + patch.dict(os.environ, {"MODELHUB_QWEN_API_KEY": "must-not-be-used"}), + patch( + "llm_classifier.LLMAssistedClassifier.__init__", + side_effect=AssertionError("online submission constructed an LLM client"), + ), + ): + summary = run_submission( + args, + now=datetime(2026, 1, 1, 12, tzinfo=timezone.utc), + hf_discovery=FakeDiscovery(1), # type: ignore[arg-type] + modelhub_client=AutoStrategyClient(available=1), # type: ignore[arg-type] + ) + + self.assertFalse(summary["candidatePreflight"]["llm"]["enabled"]) + def test_exhausted_recent_window_expands_to_older_models_in_same_run(self) -> None: with tempfile.TemporaryDirectory() as temporary_dir: root = Path(temporary_dir)