CoolBoy/submmit
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

feat: add failure-aware preflight and Qwen review

Browse Source
This commit is contained in:
CoolBoy
2026-08-10 21:44:42 +08:00
parent 5e47d9e695
commit 3d15f60284
21 changed files with 2672 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

81
docs/failure-analysis-2026-08-10.md Normal file
View File

@@ -0,0 +1,81 @@
# ModelHub failure analysis — 2026-08-10
## Coverage
- Accounts: 12/12
- Historical tasks returned: 12,650
- Terminal failures: 10,763
- Failures with downloadable logs: 7,793
- GPU-stratified recent log sample: 1,189 (up to 120 per GPU)
- Log download failures: 0
The sample is deliberately capped per GPU. Percentages below describe the
stratified sample, not the raw platform-wide frequency.
## Structured failure codes
| Code | Count | Sample share | Primary handling |
| --- | ---: | ---: | --- |
| `PREFLIGHT_OOM` | 299 | 25.1% | Deterministic model-size/GPU-memory gate |
| `MODEL_NOT_SUPPORTED` | 244 | 20.5% | Architecture history, then LLM for the long tail |
| missing structured report | 185 | 15.6% | Root-exception rules; LLM only when still ambiguous |
| `MODEL_LOAD_FAILED` | 132 | 11.1% | Repository checks, architecture/quantization review |
| `EXECUTE_EMPTY_RESULT` | 126 | 10.6% | Separate platform faults from model faults first |
| `MODEL_FILE_NOT_FOUND` | 102 | 8.6% | Require framework-specific root files |
| `TOKENIZER_FAILED` | 50 | 4.2% | Require tokenizer assets for text frameworks |
| `CONTEXT_LENGTH_ERROR` | 20 | 1.7% | Clamp template context to the model limit |
| `MISSING_OPERATOR` | 14 | 1.2% | Prefer another proven GPU/framework; semantic review |
| `DEVICE_OOM` | 8 | 0.7% | Model/GPU memory-risk feedback |
| other | 9 | 0.8% | Taxonomy or LLM fallback |
## Important root causes
- The OOM reports expose stable allocated memory values for nine GPU types.
ModelHub's check is based on the full recursive repository size, not only the
selected weight format. The preflight therefore includes duplicate formats,
tokenizers, indexes, and nested shards, adds the same observed 20% loading
overhead, and blocks when the result exceeds an evidence-backed capacity.
- A separate replay sampled 60 real OOM combinations. Of 53 repositories still
reachable on ModelScope, 51 had both measurable selected weights and a known
GPU capacity; the new preflight identified all 51 as OOM before submission.
- `MODEL_FILE_NOT_FOUND` commonly means `/model/config.json` is absent or the
repository only contains an adapter/subdirectory checkpoint. Non-GGUF
frameworks now require root config, weights, and tokenizer assets.
- Successful-repository replay covered 61 ModelScope-reachable models. Every
successful non-GGUF model had root config, tokenizer, and weights. Successful
GGUF repositories were the valid exception and remain allowed without them.
- Context failures were caused by templates requesting 4,096 or 10,000 tokens
from models whose config advertised a smaller maximum. Rendered configs are
now clamped instead of rejected.
- Seventy `EXECUTE_EMPTY_RESULT` samples on an Iluvatar path came from a broken
launch script (`welcome.sh` missing / `data` interpreted as a command). These
are GPU/framework infrastructure failures, not evidence that the model is bad.
- Many missing reports on Biren referenced a missing platform SSH key; Sunrise
tokenizer-labelled failures also contained “no idle card” messages. The new
taxonomy checks infrastructure signatures before assigning model blame.
- Architecture failures included new/custom `model_type` values, unsupported
quantization methods, and backend operator gaps. These are version-dependent;
a permanent hard-coded architecture blacklist would become stale.
## Decision policy
1. Deterministic checks always run first and cannot be overridden.
2. Publicly proven GPU/framework eligibility remains mandatory.
3. Ambiguous custom architecture/remote-code cases may be sent to a configured
Qwen model. Ordinary quantization metadata alone does not justify an LLM call.
4. Only a high-confidence LLM denial blocks a candidate. `allow` cannot enable
a new framework, bypass OOM/file checks, or create exploration traffic.
5. LLM results are cached by model/profile/GPU/framework. A persisted rolling
hourly budget and single-request semaphore prevent repeated cycles from
spending unbounded inference time.
6. Outcome sync automatically inspects at most 40 locally submitted failure logs
at a time with four download workers and no more than three attempts per log.
Confident semantic classifications feed the model/profile statistics; platform
failures are excluded from compatibility rates.
7. Repeated infrastructure failures still affect speed: three consecutive
platform failures on a GPU/framework open a 30-minute circuit, while five
attributable profile failures retain the 12-hour compatibility circuit.
Raw task and log samples are stored under the ignored local directory
`.modelhub_state/failure_analysis/`; credentials, account profiles, and signed
log URLs are not included in this document or tracked by Git.

70
docs/gpu-memory-capacity-2026-08-10.md Normal file
View File

@@ -0,0 +1,70 @@
# ModelHub GPU memory boundaries — 2026-08-10
## Scope and method
The live ModelHub machine-info endpoint returned 18 GPU types on 2026-08-10.
Four were disabled (`canVerify=false`): `Mthreads_s5000`,
`Kunlunxin_r-200-8f`, `Cambricon_mlu-590`, and `Ascend_950`. They are not current
submission targets. The remaining 14 are covered below.
ModelHub `PREFLIGHT_OOM` logs reveal the platform's actual rule:
```text
required memory = complete recursive repository size × 1.20
maximum repository size = allocated GPU memory ÷ 1.20
```
This was cross-checked against `Kwaipilot/KAT-Coder-V2.5-Dev`: the recursive
ModelScope tree was 64.5916 GiB and ModelHub reported 64.6 GiB on disk, then
required 77.5 GiB after the 20% multiplier. Consequently, parameter count or a
single selected weight format is not a valid substitute.
## Current boundaries
| ModelHub GPU type | Allocated memory | Maximum complete repository | Primary evidence |
| --- | ---: | ---: | --- |
| `Cambricon_mlu-370-x4` | 24 GiB | 20.000 GiB | 49 ModelHub OOM logs; manufacturer says 24 GB |
| `Ascend_910-b4` | 32 GiB | 26.667 GiB | 8 ModelHub OOM logs |
| `Iluvatar_bi-100` | 32 GiB | 26.667 GiB | 48 ModelHub OOM logs |
| `Iluvatar_bi-150` | 32 GiB | 26.667 GiB | 24 ModelHub OOM logs |
| `Iluvatar_mrv-100` | 32 GiB | 26.667 GiB | 50 ModelHub OOM logs |
| `Vastai_va16` | 32 GiB | 26.667 GiB | 40 ModelHub OOM logs |
| `Cambricon_mlu-370-x8` | 48 GiB | 40.000 GiB | manufacturer specification |
| `Mthreads_s4000` | 48 GiB | 40.000 GiB | manufacturer specification |
| `Ascend_910-b3` | 64 GiB | 53.333 GiB | published deployment specification |
| `Biren_166m` | 64 GiB | 53.333 GiB | manufacturer publication |
| `hygon_k100-ai` | 64 GiB | 53.333 GiB | 15 ModelHub OOM logs |
| `MetaX_c-500` | 64 GiB | 53.333 GiB | 29 ModelHub OOM logs; manufacturer says 64 GB |
| `Sunrise_pt-200-x1` | 64 GiB | 53.333 GiB | 36 ModelHub OOM logs |
| `Kunlunxin_p-800` | 96 GiB | 80.000 GiB | public procurement specification |
Published sources:
- ModelHub machine inventory: <https://modelhub.org.cn/api/computility/power/machine/list/machine-info>
- Cambricon MLU370-X4 (24 GB): <https://cambricon.com/index.php?a=lists&c=index&catid=371&m=content>
- Cambricon MLU370-X8 (48 GB): <https://cambricon.com/index.php?a=lists&c=index&catid=406&m=content>
- Iluvatar TianGai/ZhiKai series (32 GB): <https://www.iluvatar.com/serias?fullCode=cpjs-yj-xlxl>
- MetaX C500 (64 GB): <https://www.metax-tech.com/prod.html?cid=107&id=21>
- MTT S4000 (48 GB): <https://docs.mthreads.com/s4000/s4000-doc-online/product_specifications/>
- Biren 166M (64 GB): <https://www.birentech.com/news/id6rz98v3obczy77cmxzfgk3/>
- Ascend 910B3/B4 deployment capacities (64/32 GB): <https://aclanthology.org/2025.emnlp-main.1630.pdf>
- Kunlunxin P800 procurement requirement (at least 96 GB per card): <https://pms2g.shrcb.com/cms/cmscaigougg/1d93b4b8166041e096c65d739073ade1.html>
## Runtime behavior
The code uses the evidence hierarchy `local ModelHub OOM > explicit environment
override > historical ModelHub OOM > published specification`. A newly
downloaded structured OOM log records the actual allocation for that ModelHub
GPU type and replaces the published fallback on subsequent planning cycles.
For a known GPU, missing even one file size causes
`preflight_model_size_unknown`; the candidate is deferred rather than estimated.
An exact boundary is accepted, while anything larger is rejected locally as
`preflight_predicted_oom` before consuming a platform queue slot.
If ModelHub later introduces another GPU, it is deferred as
`preflight_gpu_memory_unknown` until a platform observation, published capacity,
or explicit override supplies evidence.
This boundary reproduces ModelHub's repository preflight. It does not promise
that every model below the boundary will run: framework support, operators,
quantization, context length, and runtime KV-cache memory remain separate checks.