refactor: remove LLM from online decisions

README.md

@@ -139,32 +139,21 @@ The verified capacities, safe repository-size boundaries, evidence hierarchy,
 and source links are recorded in
 `docs/gpu-memory-capacity-2026-08-10.md`.
 
-Ambiguous custom architectures can optionally be reviewed by a small
+The online worker makes zero LLM calls. Candidate admission, GPU/framework
-OpenAI-compatible Qwen model. Qwen is lazy: deterministic rules handle repository
+selection, queue cleanup, and failure feedback are deterministic and based on
-layout, model size, context length, known errors, and ordinary quantization cases
+repository metadata, platform capabilities, public outcomes, and explicit error
-without an LLM call. Set
+signatures. Merely storing a DashScope key in `.env` or setting a Qwen/LLM
-`MODELHUB_LLM_CLASSIFIER_ENDPOINT` to the full chat-completions URL and
+environment variable does not activate inference. The standalone classifier
-`MODELHUB_LLM_CLASSIFIER_MODEL`; set `MODELHUB_LLM_CLASSIFIER_API_KEY` only when
+module remains only as an offline research helper for human-reviewed batches of
-the endpoint requires it. The default deny threshold is 0.85 and can be changed
+previously unseen errors; it is not wired into submission or outcome sync.
-with `MODELHUB_LLM_CLASSIFIER_MIN_DENY_CONFIDENCE`. For Alibaba Model Studio,
-the aliases are `MODELHUB_QWEN_ENDPOINT`, `MODELHUB_QWEN_MODEL`, and
-`MODELHUB_QWEN_API_KEY` (or `DASHSCOPE_API_KEY`); endpoint omission uses the
-DashScope OpenAI-compatible URL. A root `.env` entry named `dashscope` is also
-recognized directly, and the default model is `qwen3.7-flash`. Calls default to
-one concurrent request and 20
-requests per rolling hour, configurable with `MODELHUB_LLM_MAX_CONCURRENT_REQUESTS`
-and `MODELHUB_LLM_MAX_CALLS_PER_HOUR`. The LLM may only veto an
-ambiguous candidate: it cannot bypass deterministic checks, introduce a new
-framework, or override public success-evidence gates. Results are cached under
-`.modelhub_state/llm_classifications.json`.
 
 Outcome synchronization downloads a bounded set of failure archives for
 submissions created by this worker (at most 40 per sync, four workers, three
 download attempts). Deterministic signatures classify memory, repository layout,
-context-length, storage, and platform faults first. Only unresolved runtime errors
+context-length, storage, and platform faults. Unresolved runtime errors remain
-are sent to the optional LLM; a semantic result is promoted only at confidence
+explicitly ambiguous for later rule development instead of being sent to an LLM.
-0.80 or higher. Signed log URLs remain in the ignored local outcome store and are
+Signed log URLs remain in the ignored local outcome store and are removed after
-removed after classification.
+classification.
 
 The 12-account failure study and routing rationale are recorded in
 `docs/failure-analysis-2026-08-10.md`.
@@ -226,6 +215,9 @@ repository root `.env` key named `dashscope` without logging its value.
 Version `2026.08.11.1` adds account-owned cancellation of queued tasks that are
 deterministically over the selected GPU's ModelHub memory boundary, with a
 second active-state check and fail-closed handling for incomplete evidence.
+Version `2026.08.11.2` removes LLM inference from every online path; credentials
+alone cannot activate it, and unresolved cases remain available for offline,
+human-reviewed rule development.
 
 ## Deploy
 

docs/failure-analysis-2026-08-10.md

@@ -17,8 +17,8 @@ stratified sample, not the raw platform-wide frequency.
 | Code | Count | Sample share | Primary handling |
 | --- | ---: | ---: | --- |
 | `PREFLIGHT_OOM` | 299 | 25.1% | Deterministic model-size/GPU-memory gate |
-| `MODEL_NOT_SUPPORTED` | 244 | 20.5% | Architecture history, then LLM for the long tail |
+| `MODEL_NOT_SUPPORTED` | 244 | 20.5% | Architecture history and deterministic compatibility feedback |
-| missing structured report | 185 | 15.6% | Root-exception rules; LLM only when still ambiguous |
+| missing structured report | 185 | 15.6% | Root-exception rules; retain unknown roots for offline analysis |
 | `MODEL_LOAD_FAILED` | 132 | 11.1% | Repository checks, architecture/quantization review |
 | `EXECUTE_EMPTY_RESULT` | 126 | 10.6% | Separate platform faults from model faults first |
 | `MODEL_FILE_NOT_FOUND` | 102 | 8.6% | Require framework-specific root files |
@@ -26,7 +26,7 @@ stratified sample, not the raw platform-wide frequency.
 | `CONTEXT_LENGTH_ERROR` | 20 | 1.7% | Clamp template context to the model limit |
 | `MISSING_OPERATOR` | 14 | 1.2% | Prefer another proven GPU/framework; semantic review |
 | `DEVICE_OOM` | 8 | 0.7% | Model/GPU memory-risk feedback |
-| other | 9 | 0.8% | Taxonomy or LLM fallback |
+| other | 9 | 0.8% | Taxonomy or explicit ambiguous classification |
 
 ## Important root causes
 
@@ -61,18 +61,15 @@ stratified sample, not the raw platform-wide frequency.
 
 1. Deterministic checks always run first and cannot be overridden.
 2. Publicly proven GPU/framework eligibility remains mandatory.
-3. Ambiguous custom architecture/remote-code cases may be sent to a configured
+3. The online worker never calls an LLM. Ambiguous architecture, remote-code,
-   Qwen model. Ordinary quantization metadata alone does not justify an LLM call.
+   and runtime cases remain explicitly unresolved instead of receiving a guess.
-4. Only a high-confidence LLM denial blocks a candidate. `allow` cannot enable
+4. Previously unseen errors may be grouped offline with experimental tooling,
-   a new framework, bypass OOM/file checks, or create exploration traffic.
+   but only a human-reviewed deterministic rule can affect later submissions.
-5. LLM results are cached by model/profile/GPU/framework. A persisted rolling
+5. Outcome sync automatically inspects at most 40 locally submitted failure logs
-   hourly budget and single-request semaphore prevent repeated cycles from
-   spending unbounded inference time.
-6. Outcome sync automatically inspects at most 40 locally submitted failure logs
    at a time with four download workers and no more than three attempts per log.
-   Confident semantic classifications feed the model/profile statistics; platform
+   Rule-classified model/profile failures feed the statistics; platform failures
-   failures are excluded from compatibility rates.
+   are excluded from compatibility rates.
-7. Repeated infrastructure failures still affect speed: three consecutive
+6. Repeated infrastructure failures still affect speed: three consecutive
    platform failures on a GPU/framework open a 30-minute circuit, while five
    attributable profile failures retain the 12-hour compatibility circuit.
 

modelhub_submmit_api/README.md

@@ -21,9 +21,9 @@ It currently supports:
 - `hf_discovery.py`: ModelScope model discovery and inspection (keeps the legacy module name)
 - `modelhub_client.py`: ModelHub API client and token-pool routing
 - `history_stats.py`: online history aggregation, ranking, and warnings
-- `candidate_preflight.py`: repository, memory, context, and LLM-assisted compatibility gates
+- `candidate_preflight.py`: deterministic repository, memory, context, and compatibility gates
 - `failure_taxonomy.py`: deterministic/platform/semantic failure routing
-- `llm_classifier.py`: optional cached OpenAI-compatible ambiguity classifier
+- `llm_classifier.py`: offline-only experimental ambiguity-analysis helper
 - `template_selector.py`: template lookup and GPU normalization
 - `task_registry.py`: task-type and framework selection rules
 - `tests/`: unit tests and regression coverage
@@ -165,21 +165,18 @@ The memory gate totals the complete recursive repository and applies the same
 verifiable GPU types have evidence-backed capacities; an incomplete repository
 size is deferred instead of estimated. See `../docs/gpu-memory-capacity-2026-08-10.md`.
 
-Optional Qwen review uses `MODELHUB_QWEN_ENDPOINT`, `MODELHUB_QWEN_MODEL`, and
+The online runner never constructs an LLM client. A DashScope key or any
-`MODELHUB_QWEN_API_KEY` (or `DASHSCOPE_API_KEY`). The generic
+`MODELHUB_QWEN_*`/`MODELHUB_LLM_CLASSIFIER_*` environment variable cannot enable
-`MODELHUB_LLM_CLASSIFIER_*` names remain supported. A root `.env` key named
+inference. `llm_classifier.py` remains available only for deliberately invoked,
-`dashscope` is loaded automatically, and the default model is `qwen3.7-flash`.
+offline experiments whose output is reviewed before being converted into a
-Qwen is called only for
+deterministic rule.
-unresolved architecture/remote-code semantics or ambiguous failure roots, with
-a default rolling limit of 20 calls/hour and one concurrent request. Only
-high-confidence denials block; an error, timeout, or abstention leaves the
-already-vetted candidate eligible.
 
 Outcome sync also classifies a bounded set of this worker's failed-task ZIP logs.
-Hard error signatures run first; ambiguous runtime roots can use the configured
+Hard error signatures run first; ambiguous runtime roots remain explicitly
-LLM. Platform faults are excluded from long-term compatibility scores and use a
+unclassified and are never sent to an LLM. Platform faults are excluded from
-short 30-minute breaker after three consecutive failures. Failed log downloads
+long-term compatibility scores and use a short 30-minute breaker after three
-are persisted and stop after three attempts.
+consecutive failures. Failed log downloads are persisted and stop after three
+attempts.
 
 ## Output
 

modelhub_submmit_api/main.py

@@ -28,7 +28,6 @@ from market_intelligence import (
     DEFAULT_THROUGHPUT_WINDOW_HOURS,
     MarketIntelligenceManager,
 )
-from llm_classifier import DEFAULT_LLM_CACHE_PATH, LLMAssistedClassifier
 from modelhub_client import (
     DEFAULT_CAPACITY_STATE_PATH,
     ModelHubAPIError,
@@ -142,7 +141,7 @@ def build_parser() -> argparse.ArgumentParser:
     )
     parser.add_argument(
         "--llm-classifier-cache-path",
-        default=os.getenv("MODELHUB_LLM_CLASSIFIER_CACHE_PATH", str(DEFAULT_LLM_CACHE_PATH)),
+        default=os.getenv("MODELHUB_LLM_CLASSIFIER_CACHE_PATH", ".modelhub_state/llm_classifications.json"),
         help=argparse.SUPPRESS,
     )
     parser.add_argument("--runs-dir", default=str(DEFAULT_RUNS_DIR), help=argparse.SUPPRESS)
@@ -813,6 +812,9 @@ def run_submission(
     history_archive_path.parent.mkdir(parents=True, exist_ok=True)
 
     outcome_tracker = outcome_tracker or OutcomeTracker(Path(args.outcomes_path))
+    # Online decisions are deliberately deterministic. The optional classifier
+    # module remains available for offline, human-reviewed analysis only.
+    outcome_tracker.set_failure_llm_classifier(None)
     submission_exclusion_store = SubmissionExclusionStore(
         Path(getattr(args, "submission_exclusions_path", DEFAULT_SUBMISSION_EXCLUSIONS_PATH))
     )
@@ -821,31 +823,7 @@ def run_submission(
         "MODELHUB_DISABLE_CANDIDATE_PREFLIGHT", ""
     ).strip().lower() in {"1", "true", "yes"}
     if not disable_preflight:
-        llm_classifier = LLMAssistedClassifier(
+        preflight_advisor = CandidatePreflightAdvisor(llm_classifier=None)
-            endpoint=getattr(args, "llm_classifier_endpoint", None)
-            or os.getenv("MODELHUB_LLM_CLASSIFIER_ENDPOINT"),
-            model=getattr(args, "llm_classifier_model", None)
-            or os.getenv("MODELHUB_LLM_CLASSIFIER_MODEL"),
-            api_key=getattr(args, "llm_classifier_api_key", None)
-            or os.getenv("MODELHUB_LLM_CLASSIFIER_API_KEY"),
-            timeout_seconds=max(
-                1,
-                int(
-                    getattr(args, "llm_classifier_timeout_seconds", 0)
-                    or os.getenv("MODELHUB_LLM_CLASSIFIER_TIMEOUT_SECONDS", "20")
-                ),
-            ),
-            min_deny_confidence=float(
-                getattr(args, "llm_classifier_min_deny_confidence", 0.0)
-                or os.getenv("MODELHUB_LLM_CLASSIFIER_MIN_DENY_CONFIDENCE", "0.85")
-            ),
-            cache_path=Path(
-                getattr(args, "llm_classifier_cache_path", None)
-                or os.getenv("MODELHUB_LLM_CLASSIFIER_CACHE_PATH", str(DEFAULT_LLM_CACHE_PATH))
-            ),
-        )
-        outcome_tracker.set_failure_llm_classifier(llm_classifier)
-        preflight_advisor = CandidatePreflightAdvisor(llm_classifier=llm_classifier)
     preflight_summary: dict[str, Any] = (
         preflight_advisor.summary() if preflight_advisor is not None else {"enabled": False}
     )

modelhub_submmit_api/runner_common.py

@@ -89,7 +89,6 @@ def _add_token(tokens: list[str], token: str | None) -> None:
 
 
 def ensure_tokens(args: argparse.Namespace) -> None:
-    ensure_dashscope_key()
     primary_key_path = Path(getattr(args, "key_path", DEFAULT_KEY_PATH))
     supplemental_key_path = primary_key_path.with_name(DEFAULT_KEYS_PATH.name)
     if not primary_key_path.exists():

modelhub_submmit_api/version.py

@@ -1 +1 @@
-AGENT_VERSION = "2026.08.11.1"
+AGENT_VERSION = "2026.08.11.2"

tests/test_concurrency.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import os
 import sys
 import tempfile
 import threading
@@ -7,6 +8,7 @@ import unittest
 from concurrent.futures import ThreadPoolExecutor
 from datetime import datetime, timedelta, timezone
 from pathlib import Path
+from unittest.mock import patch
 
 
 PACKAGE_DIR = Path(__file__).resolve().parents[1] / "modelhub_submmit_api"
@@ -208,6 +210,45 @@ def make_candidate(index: int) -> dict:
 
 
 class ClientPoolConcurrencyTests(unittest.TestCase):
+    def test_online_submission_does_not_construct_llm_even_when_key_is_present(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary_dir:
+            root = Path(temporary_dir)
+            args = build_parser().parse_args(
+                [
+                    "--gpus",
+                    "Iluvatar_bi-150",
+                    "--task-types",
+                    "text-generation",
+                    "--limit",
+                    "1",
+                    "--max-scan-models",
+                    "1",
+                    "--skip-outcome-sync",
+                    "--skip-history-archive",
+                ]
+            )
+            args.runs_dir = str(root / "runs")
+            args.ledger_path = str(root / "ledger.jsonl")
+            args.outcomes_path = str(root / "outcomes.jsonl")
+            args.claims_path = str(root / "claims.jsonl")
+            args.history_archive_path = str(root / "history.jsonl")
+
+            with (
+                patch.dict(os.environ, {"MODELHUB_QWEN_API_KEY": "must-not-be-used"}),
+                patch(
+                    "llm_classifier.LLMAssistedClassifier.__init__",
+                    side_effect=AssertionError("online submission constructed an LLM client"),
+                ),
+            ):
+                summary = run_submission(
+                    args,
+                    now=datetime(2026, 1, 1, 12, tzinfo=timezone.utc),
+                    hf_discovery=FakeDiscovery(1),  # type: ignore[arg-type]
+                    modelhub_client=AutoStrategyClient(available=1),  # type: ignore[arg-type]
+                )
+
+            self.assertFalse(summary["candidatePreflight"]["llm"]["enabled"])
+
     def test_exhausted_recent_window_expands_to_older_models_in_same_run(self) -> None:
         with tempfile.TemporaryDirectory() as temporary_dir:
             root = Path(temporary_dir)