feat: clean deterministic OOM tasks on startup

This commit is contained in:
CoolBoy
2026-08-11 00:52:16 +08:00
parent 3d15f60284
commit 9645973468
7 changed files with 641 additions and 1 deletions

View File

@@ -16,6 +16,7 @@ It currently supports:
- `main.py`: core discovery, scoring, dedup, and submission
- `daily_runner.py`: daily wave orchestration
- `poll_runner.py`: long-running queue refiller
- `queue_cleanup.py`: fail-closed cleanup for active tasks that are certain to exceed GPU memory
- `runner_common.py`: shared token / key file loading
- `hf_discovery.py`: ModelScope model discovery and inspection (keeps the legacy module name)
- `modelhub_client.py`: ModelHub API client and token-pool routing
@@ -115,6 +116,10 @@ bash run_poll.sh --dry-run
raises that account's persisted known limit; a capacity rejection enters cooldown.
- Each `[scan]` log records the discovery stage and candidate yield. The final `[daily] wave_done`
log includes `skip_reasons`, making empty candidate pools distinguishable from API failures.
- At startup and every 120 poll cycles, active tasks are checked with the same recursive-size
memory rule as new submissions. Only tasks whose own repository size times `1.20` exceeds
their selected GPU capacity are stopped, after a fresh account-scoped active-state check.
Incomplete size/capacity evidence is never used for cancellation.
## Important Flags
@@ -148,6 +153,7 @@ Common flags:
- `--submit-concurrency`: concurrent task submission calls used by each cycle (0 = auto)
- `--post-cycle-cooldown-seconds`: pause after a successful cycle before next cycle (default 2)
- `--max-cycles`: optional hard stop for testing or batch windows
- `--disable-queue-cleanup`: disable automatic deterministic OOM cleanup
Failure-informed preflight is enabled by default. It rejects deterministic
missing-file and predicted-OOM cases, clamps unsafe context-length arguments,
@@ -198,6 +204,7 @@ Persistent local scheduler state is written under `.modelhub_state/`:
- `market_intelligence.json`: cached public queue, throughput, health, and framework statistics
- `account_capacity.json`: learned per-account active-task limits
- `submission_exclusions.jsonl`: non-retryable model/GPU uniqueness rejections
- `queue_cleanup_latest.json`: latest active-task sizing evidence and cancellation result
## Verification