feat: clean deterministic OOM tasks on startup
This commit is contained in:
@@ -16,6 +16,7 @@ It currently supports:
|
||||
- `main.py`: core discovery, scoring, dedup, and submission
|
||||
- `daily_runner.py`: daily wave orchestration
|
||||
- `poll_runner.py`: long-running queue refiller
|
||||
- `queue_cleanup.py`: fail-closed cleanup for active tasks that are certain to exceed GPU memory
|
||||
- `runner_common.py`: shared token / key file loading
|
||||
- `hf_discovery.py`: ModelScope model discovery and inspection (keeps the legacy module name)
|
||||
- `modelhub_client.py`: ModelHub API client and token-pool routing
|
||||
@@ -115,6 +116,10 @@ bash run_poll.sh --dry-run
|
||||
raises that account's persisted known limit; a capacity rejection enters cooldown.
|
||||
- Each `[scan]` log records the discovery stage and candidate yield. The final `[daily] wave_done`
|
||||
log includes `skip_reasons`, making empty candidate pools distinguishable from API failures.
|
||||
- At startup and every 120 poll cycles, active tasks are checked with the same recursive-size
|
||||
memory rule as new submissions. Only tasks whose own repository size times `1.20` exceeds
|
||||
their selected GPU capacity are stopped, after a fresh account-scoped active-state check.
|
||||
Incomplete size/capacity evidence is never used for cancellation.
|
||||
|
||||
## Important Flags
|
||||
|
||||
@@ -148,6 +153,7 @@ Common flags:
|
||||
- `--submit-concurrency`: concurrent task submission calls used by each cycle (0 = auto)
|
||||
- `--post-cycle-cooldown-seconds`: pause after a successful cycle before next cycle (default 2)
|
||||
- `--max-cycles`: optional hard stop for testing or batch windows
|
||||
- `--disable-queue-cleanup`: disable automatic deterministic OOM cleanup
|
||||
|
||||
Failure-informed preflight is enabled by default. It rejects deterministic
|
||||
missing-file and predicted-OOM cases, clamps unsafe context-length arguments,
|
||||
@@ -198,6 +204,7 @@ Persistent local scheduler state is written under `.modelhub_state/`:
|
||||
- `market_intelligence.json`: cached public queue, throughput, health, and framework statistics
|
||||
- `account_capacity.json`: learned per-account active-task limits
|
||||
- `submission_exclusions.jsonl`: non-retryable model/GPU uniqueness rejections
|
||||
- `queue_cleanup_latest.json`: latest active-task sizing evidence and cancellation result
|
||||
|
||||
## Verification
|
||||
|
||||
|
||||
Reference in New Issue
Block a user