feat: add failure-aware preflight and Qwen review

This commit is contained in:
CoolBoy
2026-08-10 21:44:42 +08:00
parent 5e47d9e695
commit 3d15f60284
21 changed files with 2672 additions and 21 deletions

View File

@@ -97,9 +97,62 @@ progress are stored in `.modelhub_state/gpu_strategy.json`.
Five consecutive local failures open a 12-hour GPU/framework circuit breaker.
A sub-20% success rate over the latest 20 terminal tasks opens a 6-hour breaker.
Platform/infrastructure failures are excluded from long-term compatibility rates
and model/profile breakers. Three consecutive platform failures on a GPU/framework
instead open a short 30-minute breaker, so a temporary broken runner or lack of an
idle card does not permanently poison otherwise successful evidence.
Candidate shortages expand the model search window; they never unlock an
unvetted GPU or framework.
Before a candidate reaches the submit queue, failure-informed preflight checks
the actual ModelScope repository structure and file sizes. Non-GGUF text
frameworks require root-level config, weights, and tokenizer assets. The memory
gate recursively totals the entire repository—including duplicate weight formats
and nested shards—and applies ModelHub's observed 20% loading overhead. It covers
all 14 GPU types currently marked `canVerify=true`; nine capacities come directly
from structured ModelHub OOM reports and five from published specifications until
ModelHub supplies a stronger observation. If a known GPU's repository file sizes
are incomplete, the candidate is deferred rather than guessed. Template context
length is also clamped to the model's advertised limit. A newly introduced GPU
with no capacity evidence is likewise deferred. Override or extend known
capacities with
`MODELHUB_GPU_MEMORY_GIB_JSON`, for example
`{"New_gpu": 64}`.
The verified capacities, safe repository-size boundaries, evidence hierarchy,
and source links are recorded in
`docs/gpu-memory-capacity-2026-08-10.md`.
Ambiguous custom architectures can optionally be reviewed by a small
OpenAI-compatible Qwen model. Qwen is lazy: deterministic rules handle repository
layout, model size, context length, known errors, and ordinary quantization cases
without an LLM call. Set
`MODELHUB_LLM_CLASSIFIER_ENDPOINT` to the full chat-completions URL and
`MODELHUB_LLM_CLASSIFIER_MODEL`; set `MODELHUB_LLM_CLASSIFIER_API_KEY` only when
the endpoint requires it. The default deny threshold is 0.85 and can be changed
with `MODELHUB_LLM_CLASSIFIER_MIN_DENY_CONFIDENCE`. For Alibaba Model Studio,
the aliases are `MODELHUB_QWEN_ENDPOINT`, `MODELHUB_QWEN_MODEL`, and
`MODELHUB_QWEN_API_KEY` (or `DASHSCOPE_API_KEY`); endpoint omission uses the
DashScope OpenAI-compatible URL. A root `.env` entry named `dashscope` is also
recognized directly, and the default model is `qwen3.7-flash`. Calls default to
one concurrent request and 20
requests per rolling hour, configurable with `MODELHUB_LLM_MAX_CONCURRENT_REQUESTS`
and `MODELHUB_LLM_MAX_CALLS_PER_HOUR`. The LLM may only veto an
ambiguous candidate: it cannot bypass deterministic checks, introduce a new
framework, or override public success-evidence gates. Results are cached under
`.modelhub_state/llm_classifications.json`.
Outcome synchronization downloads a bounded set of failure archives for
submissions created by this worker (at most 40 per sync, four workers, three
download attempts). Deterministic signatures classify memory, repository layout,
context-length, storage, and platform faults first. Only unresolved runtime errors
are sent to the optional LLM; a semantic result is promoted only at confidence
0.80 or higher. Signed log URLs remain in the ignored local outcome store and are
removed after classification.
The 12-account failure study and routing rationale are recorded in
`docs/failure-analysis-2026-08-10.md`.
ModelScope HTTP 429 responses use exponential backoff and `Retry-After`. Successful
pages remain cached, so a later cycle retries the failed page instead of restarting
the whole pagination scan.
@@ -149,6 +202,11 @@ Version `2026.08.05.1` removes self-funded GPU exploration, switches accepted
traffic to 70/30 long-term/recent exploitation, raises the public framework gate
to 300 samples, makes success dominate queue pressure, and adds recent local
GPU/framework circuit breakers.
Version `2026.08.10.2` adds evidence-backed sizing for every currently verifiable
GPU, recursive repository-size checks, deterministic failure-aware preflight,
and rate-limited lazy Qwen review for unresolved semantic cases.
Version `2026.08.10.3` selects `qwen3.7-flash` by default and recognizes the
repository root `.env` key named `dashscope` without logging its value.
## Deploy