submmit/docs/failure-analysis-2026-08-10.md

ModelHub failure analysis — 2026-08-10

Coverage

• Accounts: 12/12
• Historical tasks returned: 12,650
• Terminal failures: 10,763
• Failures with downloadable logs: 7,793
• GPU-stratified recent log sample: 1,189 (up to 120 per GPU)
• Log download failures: 0

The sample is deliberately capped per GPU. Percentages below describe the stratified sample, not the raw platform-wide frequency.

Structured failure codes

Code	Count	Sample share	Primary handling
PREFLIGHT_OOM	299	25.1%	Deterministic model-size/GPU-memory gate
MODEL_NOT_SUPPORTED	244	20.5%	Architecture history, then LLM for the long tail
missing structured report	185	15.6%	Root-exception rules; LLM only when still ambiguous
MODEL_LOAD_FAILED	132	11.1%	Repository checks, architecture/quantization review
EXECUTE_EMPTY_RESULT	126	10.6%	Separate platform faults from model faults first
MODEL_FILE_NOT_FOUND	102	8.6%	Require framework-specific root files
TOKENIZER_FAILED	50	4.2%	Require tokenizer assets for text frameworks
CONTEXT_LENGTH_ERROR	20	1.7%	Clamp template context to the model limit
MISSING_OPERATOR	14	1.2%	Prefer another proven GPU/framework; semantic review
DEVICE_OOM	8	0.7%	Model/GPU memory-risk feedback
other	9	0.8%	Taxonomy or LLM fallback

Important root causes

• The OOM reports expose stable allocated memory values for nine GPU types. ModelHub's check is based on the full recursive repository size, not only the selected weight format. The preflight therefore includes duplicate formats, tokenizers, indexes, and nested shards, adds the same observed 20% loading overhead, and blocks when the result exceeds an evidence-backed capacity.
• A separate replay sampled 60 real OOM combinations. Of 53 repositories still reachable on ModelScope, 51 had both measurable selected weights and a known GPU capacity; the new preflight identified all 51 as OOM before submission.
• MODEL_FILE_NOT_FOUND commonly means /model/config.json is absent or the repository only contains an adapter/subdirectory checkpoint. Non-GGUF frameworks now require root config, weights, and tokenizer assets.
• Successful-repository replay covered 61 ModelScope-reachable models. Every successful non-GGUF model had root config, tokenizer, and weights. Successful GGUF repositories were the valid exception and remain allowed without them.
• Context failures were caused by templates requesting 4,096 or 10,000 tokens from models whose config advertised a smaller maximum. Rendered configs are now clamped instead of rejected.
• Seventy EXECUTE_EMPTY_RESULT samples on an Iluvatar path came from a broken launch script (welcome.sh missing / data interpreted as a command). These are GPU/framework infrastructure failures, not evidence that the model is bad.
• Many missing reports on Biren referenced a missing platform SSH key; Sunrise tokenizer-labelled failures also contained “no idle card” messages. The new taxonomy checks infrastructure signatures before assigning model blame.
• Architecture failures included new/custom model_type values, unsupported quantization methods, and backend operator gaps. These are version-dependent; a permanent hard-coded architecture blacklist would become stale.

Decision policy

1. Deterministic checks always run first and cannot be overridden.
2. Publicly proven GPU/framework eligibility remains mandatory.
3. Ambiguous custom architecture/remote-code cases may be sent to a configured Qwen model. Ordinary quantization metadata alone does not justify an LLM call.
4. Only a high-confidence LLM denial blocks a candidate. allow cannot enable a new framework, bypass OOM/file checks, or create exploration traffic.
5. LLM results are cached by model/profile/GPU/framework. A persisted rolling hourly budget and single-request semaphore prevent repeated cycles from spending unbounded inference time.
6. Outcome sync automatically inspects at most 40 locally submitted failure logs at a time with four download workers and no more than three attempts per log. Confident semantic classifications feed the model/profile statistics; platform failures are excluded from compatibility rates.
7. Repeated infrastructure failures still affect speed: three consecutive platform failures on a GPU/framework open a 30-minute circuit, while five attributable profile failures retain the 12-hour compatibility circuit.

Raw task and log samples are stored under the ignored local directory .modelhub_state/failure_analysis/; credentials, account profiles, and signed log URLs are not included in this document or tracked by Git.