feat: compact durable logs and refine framework routing
This commit is contained in:
83
docs/failure-analysis-2026-09-04.md
Normal file
83
docs/failure-analysis-2026-09-04.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Durable log analysis — 2026-09-04
|
||||
|
||||
This report was produced from the live `agent-state` snapshot and the monthly
|
||||
`agent-archive-2026-08` branch. Records were deduplicated by task identity before
|
||||
classification; intent rows and state-sync generations were not counted as
|
||||
independent model submissions.
|
||||
|
||||
## Data quality
|
||||
|
||||
- 15,178 raw outcome rows became 14,178 unique outcomes after deduplication.
|
||||
- 13,318 unique terminal outcomes contained 733 successes and 12,585 failures.
|
||||
- The hot intent WAL contained 9,213 rows, but only 1,299 unique model/GPU
|
||||
routes. Capacity-full responses alone appeared 5,887 times.
|
||||
- The ledger contained 1,028 platform tasks. A single model can legitimately
|
||||
produce several tasks because GPU/framework routes and retries differ.
|
||||
- Raw terminal success rate is biased downward: failures often terminate much
|
||||
faster, while successful tasks can remain waiting/running for a long time.
|
||||
Routing therefore uses attributable outcomes and confidence bounds rather
|
||||
than this raw percentage.
|
||||
|
||||
## Deduplicated failure categories
|
||||
|
||||
| Category | Count | Share |
|
||||
| --- | ---: | ---: |
|
||||
| Ambiguous runtime | 3,067 | 24.37% |
|
||||
| Parameters/template | 2,935 | 23.32% |
|
||||
| Framework/architecture unsupported | 1,428 | 11.35% |
|
||||
| Memory capacity | 1,169 | 9.29% |
|
||||
| Platform infrastructure | 911 | 7.24% |
|
||||
| Repository structure | 725 | 5.76% |
|
||||
| Missing log evidence | 719 | 5.71% |
|
||||
| Tokenizer compatibility | 576 | 4.58% |
|
||||
| Model load | 382 | 3.04% |
|
||||
| Context length | 318 | 2.53% |
|
||||
| Architecture compatibility | 212 | 1.68% |
|
||||
| Runtime memory | 55 | 0.44% |
|
||||
| Backend operator | 44 | 0.35% |
|
||||
| Generic validation failure | 43 | 0.34% |
|
||||
| Attention backend | 1 | 0.01% |
|
||||
|
||||
The most useful structured codes were `MODEL_NOT_SUPPORTED` (1,700),
|
||||
`EXECUTE_EMPTY_RESULT` (1,392), `PREFLIGHT_OOM` (1,166), `TOKENIZER_FAILED`
|
||||
(696), `MODEL_FILE_NOT_FOUND` (590), `MODEL_LOAD_FAILED` (382), and
|
||||
`CONTEXT_LENGTH_ERROR` (268).
|
||||
|
||||
## Representative evidence and algorithm changes
|
||||
|
||||
1. Capacity saturation repeatedly returned the account-limit message. This is
|
||||
platform flow control, not model/framework failure. It is now recorded as
|
||||
`capacity_deferred`, ends the current submission cycle after the pool has
|
||||
tried every account, and does not enter success statistics.
|
||||
2. The platform repeatedly required a failed non-`transformers` verification
|
||||
before accepting `transformers`. Routing now uses a vetted non-transformers
|
||||
route when one exists and treats the prerequisite response as a neutral
|
||||
deferred result. Once durable history contains a non-transformers failure
|
||||
for that model, transformers is unlocked and competes normally by success
|
||||
confidence.
|
||||
3. `PREFLIGHT_OOM` logs explicitly compare repository load size (including the
|
||||
platform's 20% loading allowance) with available device memory. These remain
|
||||
deterministic pre-submit blocks and deterministic queue cleanup evidence.
|
||||
4. Explicit “framework does not support model/architecture” messages continue
|
||||
to create exact GPU + framework + task + architecture blocks. Generic image,
|
||||
driver, storage and connection errors remain platform-neutral and do not
|
||||
poison compatibility scores.
|
||||
5. Platform history often omits `framework`. Outcome sync now joins task IDs
|
||||
with the local durable ledger and restores framework/task/profile metadata,
|
||||
improving per-framework statistics without guessing from model names.
|
||||
|
||||
## Durable-state corrections
|
||||
|
||||
- Hot intent history retains unresolved intents plus 200 recent terminal
|
||||
intents; older terminal attempts are gzip archived by month.
|
||||
- Community raw samples are capped locally at 200 and excluded from the hot Git
|
||||
snapshot. The aggregated GPU/framework statistics remain durable.
|
||||
- Model/GPU official-capability cache is bounded to the 1,500 newest entries.
|
||||
- Outcome compaction starts at 1,000 rows instead of 2,000.
|
||||
- An unchanged snapshot produces no Git commit. A failed push retries the exact
|
||||
same commit and generation instead of creating a new generation every minute.
|
||||
- Submission intent batches default to 100, reducing Git transactions while
|
||||
preserving write-ahead recovery.
|
||||
|
||||
These changes keep full forensic evidence in cold archive branches while making
|
||||
the hot branch small enough for quick restart and reliable server-side unpack.
|
||||
Reference in New Issue
Block a user