CoolBoy/submmit
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

refactor: remove LLM from online decisions

Browse Source
This commit is contained in:
CoolBoy
2026-08-11 00:59:40 +08:00
parent 9645973468
commit 38ab25fc3c
7 changed files with 84 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
README.md
View File

@@ -139,32 +139,21 @@ The verified capacities, safe repository-size boundaries, evidence hierarchy,
and source links are recorded in
`docs/gpu-memory-capacity-2026-08-10.md`.
Ambiguous custom architectures can optionally be reviewed by a small
OpenAI-compatible Qwen model. Qwen is lazy: deterministic rules handle repository
layout, model size, context length, known errors, and ordinary quantization cases
without an LLM call. Set
`MODELHUB_LLM_CLASSIFIER_ENDPOINT` to the full chat-completions URL and
`MODELHUB_LLM_CLASSIFIER_MODEL`; set `MODELHUB_LLM_CLASSIFIER_API_KEY` only when
the endpoint requires it. The default deny threshold is 0.85 and can be changed
with `MODELHUB_LLM_CLASSIFIER_MIN_DENY_CONFIDENCE`. For Alibaba Model Studio,
the aliases are `MODELHUB_QWEN_ENDPOINT`, `MODELHUB_QWEN_MODEL`, and
`MODELHUB_QWEN_API_KEY` (or `DASHSCOPE_API_KEY`); endpoint omission uses the
DashScope OpenAI-compatible URL. A root `.env` entry named `dashscope` is also
recognized directly, and the default model is `qwen3.7-flash`. Calls default to
one concurrent request and 20
requests per rolling hour, configurable with `MODELHUB_LLM_MAX_CONCURRENT_REQUESTS`
and `MODELHUB_LLM_MAX_CALLS_PER_HOUR`. The LLM may only veto an
ambiguous candidate: it cannot bypass deterministic checks, introduce a new
framework, or override public success-evidence gates. Results are cached under
`.modelhub_state/llm_classifications.json`.
The online worker makes zero LLM calls. Candidate admission, GPU/framework
selection, queue cleanup, and failure feedback are deterministic and based on
repository metadata, platform capabilities, public outcomes, and explicit error
signatures. Merely storing a DashScope key in `.env` or setting a Qwen/LLM
environment variable does not activate inference. The standalone classifier
module remains only as an offline research helper for human-reviewed batches of
previously unseen errors; it is not wired into submission or outcome sync.
Outcome synchronization downloads a bounded set of failure archives for
submissions created by this worker (at most 40 per sync, four workers, three
download attempts). Deterministic signatures classify memory, repository layout,
context-length, storage, and platform faults first. Only unresolved runtime errors
are sent to the optional LLM; a semantic result is promoted only at confidence
0.80 or higher. Signed log URLs remain in the ignored local outcome store and are
removed after classification.
context-length, storage, and platform faults. Unresolved runtime errors remain
explicitly ambiguous for later rule development instead of being sent to an LLM.
Signed log URLs remain in the ignored local outcome store and are removed after
classification.
The 12-account failure study and routing rationale are recorded in
`docs/failure-analysis-2026-08-10.md`.
@@ -226,6 +215,9 @@ repository root `.env` key named `dashscope` without logging its value.
Version `2026.08.11.1` adds account-owned cancellation of queued tasks that are
deterministically over the selected GPU's ModelHub memory boundary, with a
second active-state check and fail-closed handling for incomplete evidence.
Version `2026.08.11.2` removes LLM inference from every online path; credentials
alone cannot activate it, and unresolved cases remain available for offline,
human-reviewed rule development.
## Deploy