feat: clean deterministic OOM tasks on startup

This commit is contained in:
CoolBoy
2026-08-11 00:52:16 +08:00
parent 3d15f60284
commit 9645973468
7 changed files with 641 additions and 1 deletions

View File

@@ -52,6 +52,9 @@ Optional tuning:
- `MODELSCOPE_PAGE_INTERVAL_SECONDS` default `0.25`
- `MODELSCOPE_PAGE_CACHE_TTL_SECONDS` default `900`
- `MODELHUB_AGENT_VERIFY_CACHE_TTL_SECONDS` default `900`
- `MODELHUB_QUEUE_CLEANUP_INTERVAL_CYCLES` default `120`; cleanup also runs once at startup
- `MODELHUB_QUEUE_CLEANUP_READ_CONCURRENCY` default `6`
- `MODELHUB_QUEUE_CLEANUP_REPORT_PATH` default `.modelhub_state/queue_cleanup_latest.json`
## Adaptive GPU Strategy
@@ -119,6 +122,19 @@ capacities with
`MODELHUB_GPU_MEMORY_GIB_JSON`, for example
`{"New_gpu": 64}`.
At poller startup, the same deterministic memory gate is applied to existing
`waiting` and `running` tasks across every configured account. A task is stopped
through `PUT /api/async/task/stop-create-contest-task` only when its own current
recursive repository size, multiplied by ModelHub's observed `1.20` overhead,
exceeds the known capacity of its selected GPU. The task ID is fetched and
stopped with the token belonging to that account, and its active state is
rechecked immediately before the mutation. Missing file sizes, unknown GPU
capacities, listing failures, and tasks that have already changed state all fail
closed and are never stopped. This does not match against another task from the
same model or infer failure from historical similarity. The cleanup repeats
every 120 poll cycles by default and writes its full evidence report to
`.modelhub_state/queue_cleanup_latest.json`.
The verified capacities, safe repository-size boundaries, evidence hierarchy,
and source links are recorded in
`docs/gpu-memory-capacity-2026-08-10.md`.
@@ -207,6 +223,9 @@ GPU, recursive repository-size checks, deterministic failure-aware preflight,
and rate-limited lazy Qwen review for unresolved semantic cases.
Version `2026.08.10.3` selects `qwen3.7-flash` by default and recognizes the
repository root `.env` key named `dashscope` without logging its value.
Version `2026.08.11.1` adds account-owned cancellation of queued tasks that are
deterministically over the selected GPU's ModelHub memory boundary, with a
second active-state check and fail-closed handling for incomplete evidence.
## Deploy