CoolBoy/submmit
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

refactor: remove LLM from online decisions

Browse Source
This commit is contained in:
CoolBoy
2026-08-11 00:59:40 +08:00
parent 9645973468
commit 38ab25fc3c
7 changed files with 84 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
README.md
View File

@@ -139,32 +139,21 @@ The verified capacities, safe repository-size boundaries, evidence hierarchy,
and source links are recorded in
`docs/gpu-memory-capacity-2026-08-10.md`.
Ambiguous custom architectures can optionally be reviewed by a small
OpenAI-compatible Qwen model. Qwen is lazy: deterministic rules handle repository
layout, model size, context length, known errors, and ordinary quantization cases
without an LLM call. Set
`MODELHUB_LLM_CLASSIFIER_ENDPOINT` to the full chat-completions URL and
`MODELHUB_LLM_CLASSIFIER_MODEL`; set `MODELHUB_LLM_CLASSIFIER_API_KEY` only when
the endpoint requires it. The default deny threshold is 0.85 and can be changed
with `MODELHUB_LLM_CLASSIFIER_MIN_DENY_CONFIDENCE`. For Alibaba Model Studio,
the aliases are `MODELHUB_QWEN_ENDPOINT`, `MODELHUB_QWEN_MODEL`, and
`MODELHUB_QWEN_API_KEY` (or `DASHSCOPE_API_KEY`); endpoint omission uses the
DashScope OpenAI-compatible URL. A root `.env` entry named `dashscope` is also
recognized directly, and the default model is `qwen3.7-flash`. Calls default to
one concurrent request and 20
requests per rolling hour, configurable with `MODELHUB_LLM_MAX_CONCURRENT_REQUESTS`
and `MODELHUB_LLM_MAX_CALLS_PER_HOUR`. The LLM may only veto an
ambiguous candidate: it cannot bypass deterministic checks, introduce a new
framework, or override public success-evidence gates. Results are cached under
`.modelhub_state/llm_classifications.json`.
The online worker makes zero LLM calls. Candidate admission, GPU/framework
selection, queue cleanup, and failure feedback are deterministic and based on
repository metadata, platform capabilities, public outcomes, and explicit error
signatures. Merely storing a DashScope key in `.env` or setting a Qwen/LLM
environment variable does not activate inference. The standalone classifier
module remains only as an offline research helper for human-reviewed batches of
previously unseen errors; it is not wired into submission or outcome sync.
Outcome synchronization downloads a bounded set of failure archives for
submissions created by this worker (at most 40 per sync, four workers, three
download attempts). Deterministic signatures classify memory, repository layout,
context-length, storage, and platform faults first. Only unresolved runtime errors
are sent to the optional LLM; a semantic result is promoted only at confidence
0.80 or higher. Signed log URLs remain in the ignored local outcome store and are
removed after classification.
context-length, storage, and platform faults. Unresolved runtime errors remain
explicitly ambiguous for later rule development instead of being sent to an LLM.
Signed log URLs remain in the ignored local outcome store and are removed after
classification.
The 12-account failure study and routing rationale are recorded in
`docs/failure-analysis-2026-08-10.md`.
@@ -226,6 +215,9 @@ repository root `.env` key named `dashscope` without logging its value.
Version `2026.08.11.1` adds account-owned cancellation of queued tasks that are
deterministically over the selected GPU's ModelHub memory boundary, with a
second active-state check and fail-closed handling for incomplete evidence.
Version `2026.08.11.2` removes LLM inference from every online path; credentials
alone cannot activate it, and unresolved cases remain available for offline,
human-reviewed rule development.
## Deploy

25
docs/failure-analysis-2026-08-10.md
View File

@@ -17,8 +17,8 @@ stratified sample, not the raw platform-wide frequency.
| Code | Count | Sample share | Primary handling |
| --- | ---: | ---: | --- |
| `PREFLIGHT_OOM` | 299 | 25.1% | Deterministic model-size/GPU-memory gate |
| `MODEL_NOT_SUPPORTED` | 244 | 20.5% | Architecture history, then LLM for the long tail |
| missing structured report | 185 | 15.6% | Root-exception rules; LLM only when still ambiguous |
| `MODEL_NOT_SUPPORTED` | 244 | 20.5% | Architecture history and deterministic compatibility feedback |
| missing structured report | 185 | 15.6% | Root-exception rules; retain unknown roots for offline analysis |
| `MODEL_LOAD_FAILED` | 132 | 11.1% | Repository checks, architecture/quantization review |
| `EXECUTE_EMPTY_RESULT` | 126 | 10.6% | Separate platform faults from model faults first |
| `MODEL_FILE_NOT_FOUND` | 102 | 8.6% | Require framework-specific root files |
@@ -26,7 +26,7 @@ stratified sample, not the raw platform-wide frequency.
| `CONTEXT_LENGTH_ERROR` | 20 | 1.7% | Clamp template context to the model limit |
| `MISSING_OPERATOR` | 14 | 1.2% | Prefer another proven GPU/framework; semantic review |
| `DEVICE_OOM` | 8 | 0.7% | Model/GPU memory-risk feedback |
| other | 9 | 0.8% | Taxonomy or LLM fallback |
| other | 9 | 0.8% | Taxonomy or explicit ambiguous classification |
## Important root causes
@@ -61,18 +61,15 @@ stratified sample, not the raw platform-wide frequency.
1. Deterministic checks always run first and cannot be overridden.
2. Publicly proven GPU/framework eligibility remains mandatory.
3. Ambiguous custom architecture/remote-code cases may be sent to a configured
Qwen model. Ordinary quantization metadata alone does not justify an LLM call.
4. Only a high-confidence LLM denial blocks a candidate. `allow` cannot enable
a new framework, bypass OOM/file checks, or create exploration traffic.
5. LLM results are cached by model/profile/GPU/framework. A persisted rolling
hourly budget and single-request semaphore prevent repeated cycles from
spending unbounded inference time.
6. Outcome sync automatically inspects at most 40 locally submitted failure logs
3. The online worker never calls an LLM. Ambiguous architecture, remote-code,
and runtime cases remain explicitly unresolved instead of receiving a guess.
4. Previously unseen errors may be grouped offline with experimental tooling,
but only a human-reviewed deterministic rule can affect later submissions.
5. Outcome sync automatically inspects at most 40 locally submitted failure logs
at a time with four download workers and no more than three attempts per log.
Confident semantic classifications feed the model/profile statistics; platform
failures are excluded from compatibility rates.
7. Repeated infrastructure failures still affect speed: three consecutive
Rule-classified model/profile failures feed the statistics; platform failures
are excluded from compatibility rates.
6. Repeated infrastructure failures still affect speed: three consecutive
platform failures on a GPU/framework open a 30-minute circuit, while five
attributable profile failures retain the 12-hour compatibility circuit.

27
modelhub_submmit_api/README.md
View File

@@ -21,9 +21,9 @@ It currently supports:
- `hf_discovery.py`: ModelScope model discovery and inspection (keeps the legacy module name)
- `modelhub_client.py`: ModelHub API client and token-pool routing
- `history_stats.py`: online history aggregation, ranking, and warnings
- `candidate_preflight.py`: repository, memory, context, and LLM-assisted compatibility gates
- `candidate_preflight.py`: deterministic repository, memory, context, and compatibility gates
- `failure_taxonomy.py`: deterministic/platform/semantic failure routing
- `llm_classifier.py`: optional cached OpenAI-compatible ambiguity classifier
- `llm_classifier.py`: offline-only experimental ambiguity-analysis helper
- `template_selector.py`: template lookup and GPU normalization
- `task_registry.py`: task-type and framework selection rules
- `tests/`: unit tests and regression coverage
@@ -165,21 +165,18 @@ The memory gate totals the complete recursive repository and applies the same
verifiable GPU types have evidence-backed capacities; an incomplete repository
size is deferred instead of estimated. See `../docs/gpu-memory-capacity-2026-08-10.md`.
Optional Qwen review uses `MODELHUB_QWEN_ENDPOINT`, `MODELHUB_QWEN_MODEL`, and
`MODELHUB_QWEN_API_KEY` (or `DASHSCOPE_API_KEY`). The generic
`MODELHUB_LLM_CLASSIFIER_*` names remain supported. A root `.env` key named
`dashscope` is loaded automatically, and the default model is `qwen3.7-flash`.
Qwen is called only for
unresolved architecture/remote-code semantics or ambiguous failure roots, with
a default rolling limit of 20 calls/hour and one concurrent request. Only
high-confidence denials block; an error, timeout, or abstention leaves the
already-vetted candidate eligible.
The online runner never constructs an LLM client. A DashScope key or any
`MODELHUB_QWEN_*`/`MODELHUB_LLM_CLASSIFIER_*` environment variable cannot enable
inference. `llm_classifier.py` remains available only for deliberately invoked,
offline experiments whose output is reviewed before being converted into a
deterministic rule.
Outcome sync also classifies a bounded set of this worker's failed-task ZIP logs.
Hard error signatures run first; ambiguous runtime roots can use the configured
LLM. Platform faults are excluded from long-term compatibility scores and use a
short 30-minute breaker after three consecutive failures. Failed log downloads
are persisted and stop after three attempts.
Hard error signatures run first; ambiguous runtime roots remain explicitly
unclassified and are never sent to an LLM. Platform faults are excluded from
long-term compatibility scores and use a short 30-minute breaker after three
consecutive failures. Failed log downloads are persisted and stop after three
attempts.
## Output

32
modelhub_submmit_api/main.py
View File

@@ -28,7 +28,6 @@ from market_intelligence import (
DEFAULT_THROUGHPUT_WINDOW_HOURS,
MarketIntelligenceManager,
)
from llm_classifier import DEFAULT_LLM_CACHE_PATH, LLMAssistedClassifier
from modelhub_client import (
DEFAULT_CAPACITY_STATE_PATH,
ModelHubAPIError,
@@ -142,7 +141,7 @@ def build_parser() -> argparse.ArgumentParser:
)
parser.add_argument(
"--llm-classifier-cache-path",
default=os.getenv("MODELHUB_LLM_CLASSIFIER_CACHE_PATH", str(DEFAULT_LLM_CACHE_PATH)),
default=os.getenv("MODELHUB_LLM_CLASSIFIER_CACHE_PATH", ".modelhub_state/llm_classifications.json"),
help=argparse.SUPPRESS,
)
parser.add_argument("--runs-dir", default=str(DEFAULT_RUNS_DIR), help=argparse.SUPPRESS)
@@ -813,6 +812,9 @@ def run_submission(
history_archive_path.parent.mkdir(parents=True, exist_ok=True)
outcome_tracker = outcome_tracker or OutcomeTracker(Path(args.outcomes_path))
# Online decisions are deliberately deterministic. The optional classifier
# module remains available for offline, human-reviewed analysis only.
outcome_tracker.set_failure_llm_classifier(None)
submission_exclusion_store = SubmissionExclusionStore(
Path(getattr(args, "submission_exclusions_path", DEFAULT_SUBMISSION_EXCLUSIONS_PATH))
)
@@ -821,31 +823,7 @@ def run_submission(
"MODELHUB_DISABLE_CANDIDATE_PREFLIGHT", ""
).strip().lower() in {"1", "true", "yes"}
if not disable_preflight:
llm_classifier = LLMAssistedClassifier(
endpoint=getattr(args, "llm_classifier_endpoint", None)
or os.getenv("MODELHUB_LLM_CLASSIFIER_ENDPOINT"),
model=getattr(args, "llm_classifier_model", None)
or os.getenv("MODELHUB_LLM_CLASSIFIER_MODEL"),
api_key=getattr(args, "llm_classifier_api_key", None)
or os.getenv("MODELHUB_LLM_CLASSIFIER_API_KEY"),
timeout_seconds=max(
1,
int(
getattr(args, "llm_classifier_timeout_seconds", 0)
or os.getenv("MODELHUB_LLM_CLASSIFIER_TIMEOUT_SECONDS", "20")
),
),
min_deny_confidence=float(
getattr(args, "llm_classifier_min_deny_confidence", 0.0)
or os.getenv("MODELHUB_LLM_CLASSIFIER_MIN_DENY_CONFIDENCE", "0.85")
),
cache_path=Path(
getattr(args, "llm_classifier_cache_path", None)
or os.getenv("MODELHUB_LLM_CLASSIFIER_CACHE_PATH", str(DEFAULT_LLM_CACHE_PATH))
),
)
outcome_tracker.set_failure_llm_classifier(llm_classifier)
preflight_advisor = CandidatePreflightAdvisor(llm_classifier=llm_classifier)
preflight_advisor = CandidatePreflightAdvisor(llm_classifier=None)
preflight_summary: dict[str, Any] = (
preflight_advisor.summary() if preflight_advisor is not None else {"enabled": False}
)

1
modelhub_submmit_api/runner_common.py
View File

@@ -89,7 +89,6 @@ def _add_token(tokens: list[str], token: str | None) -> None:
def ensure_tokens(args: argparse.Namespace) -> None:
ensure_dashscope_key()
primary_key_path = Path(getattr(args, "key_path", DEFAULT_KEY_PATH))
supplemental_key_path = primary_key_path.with_name(DEFAULT_KEYS_PATH.name)
if not primary_key_path.exists():

2
modelhub_submmit_api/version.py
View File

@@ -1 +1 @@
AGENT_VERSION = "2026.08.11.1"
AGENT_VERSION = "2026.08.11.2"

41
tests/test_concurrency.py
View File

@@ -1,5 +1,6 @@
from __future__ import annotations
import os
import sys
import tempfile
import threading
@@ -7,6 +8,7 @@ import unittest
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timedelta, timezone
from pathlib import Path
from unittest.mock import patch
PACKAGE_DIR = Path(__file__).resolve().parents[1] / "modelhub_submmit_api"
@@ -208,6 +210,45 @@ def make_candidate(index: int) -> dict:
class ClientPoolConcurrencyTests(unittest.TestCase):
def test_online_submission_does_not_construct_llm_even_when_key_is_present(self) -> None:
with tempfile.TemporaryDirectory() as temporary_dir:
root = Path(temporary_dir)
args = build_parser().parse_args(
[
"--gpus",
"Iluvatar_bi-150",
"--task-types",
"text-generation",
"--limit",
"1",
"--max-scan-models",
"1",
"--skip-outcome-sync",
"--skip-history-archive",
]
)
args.runs_dir = str(root / "runs")
args.ledger_path = str(root / "ledger.jsonl")
args.outcomes_path = str(root / "outcomes.jsonl")
args.claims_path = str(root / "claims.jsonl")
args.history_archive_path = str(root / "history.jsonl")
with (
patch.dict(os.environ, {"MODELHUB_QWEN_API_KEY": "must-not-be-used"}),
patch(
"llm_classifier.LLMAssistedClassifier.__init__",
side_effect=AssertionError("online submission constructed an LLM client"),
),
):
summary = run_submission(
args,
now=datetime(2026, 1, 1, 12, tzinfo=timezone.utc),
hf_discovery=FakeDiscovery(1), # type: ignore[arg-type]
modelhub_client=AutoStrategyClient(available=1), # type: ignore[arg-type]
)
self.assertFalse(summary["candidatePreflight"]["llm"]["enabled"])
def test_exhausted_recent_window_expands_to_older_models_in_same_run(self) -> None:
with tempfile.TemporaryDirectory() as temporary_dir:
root = Path(temporary_dir)