143 lines
5.3 KiB
Markdown
143 lines
5.3 KiB
Markdown
# ModelHub Submission Runner
|
|
|
|
This package automates ModelScope model discovery and ModelHub submission.
|
|
It currently supports:
|
|
|
|
- one-shot submission planning via `main.py`
|
|
- daily batch execution via `run_daily.sh`
|
|
- continuous queue refill via `run_poll.sh`
|
|
- multiple ModelHub tokens read from `KEY.md` and `KEYS.md`
|
|
- automatic task/framework/template selection across the supported GPU catalog
|
|
|
|
## Layout
|
|
|
|
- `main.py`: core discovery, scoring, dedup, and submission
|
|
- `daily_runner.py`: daily wave orchestration
|
|
- `poll_runner.py`: long-running queue refiller
|
|
- `runner_common.py`: shared token / key file loading
|
|
- `hf_discovery.py`: ModelScope model discovery and inspection (keeps the legacy module name)
|
|
- `modelhub_client.py`: ModelHub API client and token-pool routing
|
|
- `history_stats.py`: online history aggregation, ranking, and warnings
|
|
- `template_selector.py`: template lookup and GPU normalization
|
|
- `task_registry.py`: task-type and framework selection rules
|
|
- `tests/`: unit tests and regression coverage
|
|
|
|
## Key Files
|
|
|
|
- `KEY.md`: primary ModelScope and ModelHub tokens
|
|
- `KEYS.md`: optional supplemental ModelHub tokens
|
|
- `templates/public_submit/adapt_task_templates.jsonl`: public submit templates
|
|
|
|
The runner reads both files automatically. Add more accounts by appending
|
|
`XC_TOKEN3`, `XC_TOKEN4`, and so on to `KEYS.md`.
|
|
|
|
Template lookup is also relative. The selector searches from the current working
|
|
directory and the module directory. The primary project layout is:
|
|
|
|
- `templates/public_submit/adapt_task_templates.jsonl`
|
|
|
|
It still accepts the legacy fallback path below for compatibility with older
|
|
deployments:
|
|
|
|
- `model adaptation/templates/public_submit/adapt_task_templates.jsonl`
|
|
|
|
If your Space keeps templates in another location, set `MODELHUB_TEMPLATE_FILE`
|
|
to the exact JSONL path.
|
|
|
|
## Quick Start
|
|
|
|
Run a single daily batch:
|
|
|
|
```bash
|
|
cd /path/to/submmit
|
|
# testing: one run defaults to 3 targets if daily-target is not specified
|
|
bash run_daily.sh --rounds 1
|
|
```
|
|
|
|
Run the continuous queue refiller:
|
|
|
|
```bash
|
|
cd /path/to/submmit
|
|
bash run_poll.sh
|
|
```
|
|
|
|
Dry-run either entrypoint to inspect candidate selection without submitting:
|
|
|
|
```bash
|
|
cd /path/to/submmit
|
|
bash run_daily.sh --dry-run
|
|
bash run_poll.sh --dry-run
|
|
```
|
|
|
|
## Behavior
|
|
|
|
- The runner auto-discovers all safe GPU/template combinations from the public submit catalog.
|
|
- Each model can be submitted at most once per GPU.
|
|
- Multiple ModelHub tokens are pooled and used to route submissions to the account with available async capacity.
|
|
- History stats are online-only. The local ledger is used for local accounting, but platform history is only used after the local ledger reaches the configured threshold.
|
|
- The default history threshold is `500` records.
|
|
|
|
## Important Flags
|
|
|
|
Common flags:
|
|
|
|
- `--daily-target`: total target submissions for the day; `0` means unlimited
|
|
- `--min-downloads`: ModelScope download floor
|
|
- `--history-stats-threshold`: local ledger threshold before using online history stats
|
|
- `--max-scan-models`: hard cap on scanned HF models for a run (0 = auto)
|
|
- `--scan-multiplier`: multiplier used for auto scan cap derivation from quota/queue capacity
|
|
- `--read-concurrency`: concurrent HTTP reads while scanning model candidates (default 4)
|
|
- `--max-submits-per-run`: max tasks to submit per run cycle (0 = unlimited)
|
|
- `--submit-concurrency`: concurrent task submissions (default auto, uses 0)
|
|
- `--skip-outcome-sync`: skip outcome sync before scanning
|
|
- `--skip-history-archive`: skip history archive download for this run
|
|
- `--dry-run`: plan only, do not submit
|
|
- `run_daily.sh` injects `--daily-target 3` when no daily-target flag is provided. Set `SUBMIT_DAILY_TARGET` or pass `--daily-target` explicitly for a different target.
|
|
|
|
`run_poll.sh` adds:
|
|
|
|
- `--poll-interval-seconds`: sleep when all accounts are saturated
|
|
- `--idle-interval-seconds`: sleep when a cycle submits nothing
|
|
- `--max-scan-models`: hard cap on scanned HF models for this cycle (0 = auto)
|
|
- `--scan-multiplier`: multiplier used for auto scan cap derivation from quota/queue capacity
|
|
- `--max-submits-per-run`: max tasks to submit per poll cycle (0 = unlimited)
|
|
- `--skip-outcome-sync`: skip outcome sync before scanning
|
|
- `--skip-history-archive`: skip history archive download for this cycle
|
|
- `--submit-concurrency`: concurrent task submission calls used by each cycle (0 = auto)
|
|
- `--post-cycle-cooldown-seconds`: pause after a successful cycle before next cycle (default 0)
|
|
- `--max-cycles`: optional hard stop for testing or batch windows
|
|
|
|
## Output
|
|
|
|
Run artifacts are written under:
|
|
|
|
- `runs/`: one-shot submission runs
|
|
- `daily_runs/`: batch orchestration runs
|
|
- `poll_runs/`: poller cycles
|
|
|
|
Each run typically includes:
|
|
|
|
- `summary.json`
|
|
- `pre_submit_report.json`
|
|
- `candidates.jsonl`
|
|
- `submitted.jsonl`
|
|
- `skipped.jsonl`
|
|
- `failed.jsonl`
|
|
|
|
## Verification
|
|
|
|
Run the full test suite:
|
|
|
|
```bash
|
|
cd /path/to/submmit
|
|
python3 -m unittest discover -s tests -v
|
|
```
|
|
|
|
## Notes
|
|
|
|
- This is a submission automation tool, not a scheduler daemon. Use `screen`, `tmux`,
|
|
`nohup`, or `systemd` if you want it to keep running in the background.
|
|
- The platform still enforces per-account async capacity limits, so the poller can
|
|
keep the queue close to full but cannot override the platform cap.
|
|
- `bash run_poll.sh` now defaults to unlimited mode and keeps refilling until you stop the process manually.
|