79 lines
4.5 KiB
Markdown
79 lines
4.5 KiB
Markdown
# ModelHub failure analysis — 2026-08-10
|
|
|
|
## Coverage
|
|
|
|
- Accounts: 12/12
|
|
- Historical tasks returned: 12,650
|
|
- Terminal failures: 10,763
|
|
- Failures with downloadable logs: 7,793
|
|
- GPU-stratified recent log sample: 1,189 (up to 120 per GPU)
|
|
- Log download failures: 0
|
|
|
|
The sample is deliberately capped per GPU. Percentages below describe the
|
|
stratified sample, not the raw platform-wide frequency.
|
|
|
|
## Structured failure codes
|
|
|
|
| Code | Count | Sample share | Primary handling |
|
|
| --- | ---: | ---: | --- |
|
|
| `PREFLIGHT_OOM` | 299 | 25.1% | Deterministic model-size/GPU-memory gate |
|
|
| `MODEL_NOT_SUPPORTED` | 244 | 20.5% | Architecture history and deterministic compatibility feedback |
|
|
| missing structured report | 185 | 15.6% | Root-exception rules; retain unknown roots for offline analysis |
|
|
| `MODEL_LOAD_FAILED` | 132 | 11.1% | Repository checks, architecture/quantization review |
|
|
| `EXECUTE_EMPTY_RESULT` | 126 | 10.6% | Separate platform faults from model faults first |
|
|
| `MODEL_FILE_NOT_FOUND` | 102 | 8.6% | Require framework-specific root files |
|
|
| `TOKENIZER_FAILED` | 50 | 4.2% | Require tokenizer assets for text frameworks |
|
|
| `CONTEXT_LENGTH_ERROR` | 20 | 1.7% | Clamp template context to the model limit |
|
|
| `MISSING_OPERATOR` | 14 | 1.2% | Prefer another proven GPU/framework; semantic review |
|
|
| `DEVICE_OOM` | 8 | 0.7% | Model/GPU memory-risk feedback |
|
|
| other | 9 | 0.8% | Taxonomy or explicit ambiguous classification |
|
|
|
|
## Important root causes
|
|
|
|
- The OOM reports expose stable allocated memory values for nine GPU types.
|
|
ModelHub's check is based on the full recursive repository size, not only the
|
|
selected weight format. The preflight therefore includes duplicate formats,
|
|
tokenizers, indexes, and nested shards, adds the same observed 20% loading
|
|
overhead, and blocks when the result exceeds an evidence-backed capacity.
|
|
- A separate replay sampled 60 real OOM combinations. Of 53 repositories still
|
|
reachable on ModelScope, 51 had both measurable selected weights and a known
|
|
GPU capacity; the new preflight identified all 51 as OOM before submission.
|
|
- `MODEL_FILE_NOT_FOUND` commonly means `/model/config.json` is absent or the
|
|
repository only contains an adapter/subdirectory checkpoint. Non-GGUF
|
|
frameworks now require root config, weights, and tokenizer assets.
|
|
- Successful-repository replay covered 61 ModelScope-reachable models. Every
|
|
successful non-GGUF model had root config, tokenizer, and weights. Successful
|
|
GGUF repositories were the valid exception and remain allowed without them.
|
|
- Context failures were caused by templates requesting 4,096 or 10,000 tokens
|
|
from models whose config advertised a smaller maximum. Rendered configs are
|
|
now clamped instead of rejected.
|
|
- Seventy `EXECUTE_EMPTY_RESULT` samples on an Iluvatar path came from a broken
|
|
launch script (`welcome.sh` missing / `data` interpreted as a command). These
|
|
are GPU/framework infrastructure failures, not evidence that the model is bad.
|
|
- Many missing reports on Biren referenced a missing platform SSH key; Sunrise
|
|
tokenizer-labelled failures also contained “no idle card” messages. The new
|
|
taxonomy checks infrastructure signatures before assigning model blame.
|
|
- Architecture failures included new/custom `model_type` values, unsupported
|
|
quantization methods, and backend operator gaps. These are version-dependent;
|
|
a permanent hard-coded architecture blacklist would become stale.
|
|
|
|
## Decision policy
|
|
|
|
1. Deterministic checks always run first and cannot be overridden.
|
|
2. Publicly proven GPU/framework eligibility remains mandatory.
|
|
3. The online worker never calls an LLM. Ambiguous architecture, remote-code,
|
|
and runtime cases remain explicitly unresolved instead of receiving a guess.
|
|
4. Previously unseen errors may be grouped offline with experimental tooling,
|
|
but only a human-reviewed deterministic rule can affect later submissions.
|
|
5. Outcome sync automatically inspects at most 40 locally submitted failure logs
|
|
at a time with four download workers and no more than three attempts per log.
|
|
Rule-classified model/profile failures feed the statistics; platform failures
|
|
are excluded from compatibility rates.
|
|
6. Repeated infrastructure failures still affect speed: three consecutive
|
|
platform failures on a GPU/framework open a 30-minute circuit, while five
|
|
attributable profile failures retain the 12-hour compatibility circuit.
|
|
|
|
Raw task and log samples are stored under the ignored local directory
|
|
`.modelhub_state/failure_analysis/`; credentials, account profiles, and signed
|
|
log URLs are not included in this document or tracked by Git.
|