CoolBoy/submmit
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

refactor: remove LLM from online decisions

Browse Source
This commit is contained in:
CoolBoy
2026-08-11 00:59:40 +08:00
parent 9645973468
commit 38ab25fc3c
7 changed files with 84 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
docs/failure-analysis-2026-08-10.md
View File

@@ -17,8 +17,8 @@ stratified sample, not the raw platform-wide frequency.
| Code | Count | Sample share | Primary handling |
| --- | ---: | ---: | --- |
| `PREFLIGHT_OOM` | 299 | 25.1% | Deterministic model-size/GPU-memory gate |
| `MODEL_NOT_SUPPORTED` | 244 | 20.5% | Architecture history, then LLM for the long tail |
| missing structured report | 185 | 15.6% | Root-exception rules; LLM only when still ambiguous |
| `MODEL_NOT_SUPPORTED` | 244 | 20.5% | Architecture history and deterministic compatibility feedback |
| missing structured report | 185 | 15.6% | Root-exception rules; retain unknown roots for offline analysis |
| `MODEL_LOAD_FAILED` | 132 | 11.1% | Repository checks, architecture/quantization review |
| `EXECUTE_EMPTY_RESULT` | 126 | 10.6% | Separate platform faults from model faults first |
| `MODEL_FILE_NOT_FOUND` | 102 | 8.6% | Require framework-specific root files |
@@ -26,7 +26,7 @@ stratified sample, not the raw platform-wide frequency.
| `CONTEXT_LENGTH_ERROR` | 20 | 1.7% | Clamp template context to the model limit |
| `MISSING_OPERATOR` | 14 | 1.2% | Prefer another proven GPU/framework; semantic review |
| `DEVICE_OOM` | 8 | 0.7% | Model/GPU memory-risk feedback |
| other | 9 | 0.8% | Taxonomy or LLM fallback |
| other | 9 | 0.8% | Taxonomy or explicit ambiguous classification |
## Important root causes
@@ -61,18 +61,15 @@ stratified sample, not the raw platform-wide frequency.
1. Deterministic checks always run first and cannot be overridden.
2. Publicly proven GPU/framework eligibility remains mandatory.
3. Ambiguous custom architecture/remote-code cases may be sent to a configured
Qwen model. Ordinary quantization metadata alone does not justify an LLM call.
4. Only a high-confidence LLM denial blocks a candidate. `allow` cannot enable
a new framework, bypass OOM/file checks, or create exploration traffic.
5. LLM results are cached by model/profile/GPU/framework. A persisted rolling
hourly budget and single-request semaphore prevent repeated cycles from
spending unbounded inference time.
6. Outcome sync automatically inspects at most 40 locally submitted failure logs
3. The online worker never calls an LLM. Ambiguous architecture, remote-code,
and runtime cases remain explicitly unresolved instead of receiving a guess.
4. Previously unseen errors may be grouped offline with experimental tooling,
but only a human-reviewed deterministic rule can affect later submissions.
5. Outcome sync automatically inspects at most 40 locally submitted failure logs
at a time with four download workers and no more than three attempts per log.
Confident semantic classifications feed the model/profile statistics; platform
failures are excluded from compatibility rates.
7. Repeated infrastructure failures still affect speed: three consecutive
Rule-classified model/profile failures feed the statistics; platform failures
are excluded from compatibility rates.
6. Repeated infrastructure failures still affect speed: three consecutive
platform failures on a GPU/framework open a 30-minute circuit, while five
attributable profile failures retain the 12-hour compatibility circuit.