one-shot submission planning via main.py
daily batch execution via run_daily.sh
continuous queue refill via run_poll.sh
multiple ModelHub tokens read from KEY.md and KEYS.md
automatic task/framework/template selection across the supported GPU catalog
adaptive long-term/exploration/recent GPU scheduling with a persistent local snapshot

Layout

main.py: core discovery, scoring, dedup, and submission
daily_runner.py: daily wave orchestration
poll_runner.py: long-running queue refiller
runner_common.py: shared token / key file loading
hf_discovery.py: ModelScope model discovery and inspection (keeps the legacy module name)
modelhub_client.py: ModelHub API client and token-pool routing
history_stats.py: online history aggregation, ranking, and warnings
template_selector.py: template lookup and GPU normalization
task_registry.py: task-type and framework selection rules
tests/: unit tests and regression coverage

Key Files

KEY.md: primary ModelScope and ModelHub tokens
KEYS.md: optional supplemental ModelHub tokens
templates/public_submit/adapt_task_templates.jsonl: public submit templates

The runner reads both files automatically. Add more accounts by appending XC_TOKEN3, XC_TOKEN4, and so on to KEYS.md.

Template lookup is also relative. The selector searches from the current working directory and the module directory. The primary project layout is:

templates/public_submit/adapt_task_templates.jsonl

It still accepts the legacy fallback path below for compatibility with older deployments:

model adaptation/templates/public_submit/adapt_task_templates.jsonl

If your Space keeps templates in another location, set MODELHUB_TEMPLATE_FILE to the exact JSONL path.

Quick Start

Run a single daily batch:

cd /path/to/submmit
# testing: one run defaults to 3 targets if daily-target is not specified
bash run_daily.sh --rounds 1

Run the continuous queue refiller:

cd /path/to/submmit
bash run_poll.sh

Dry-run either entrypoint to inspect candidate selection without submitting:

cd /path/to/submmit
bash run_daily.sh --dry-run
bash run_poll.sh --dry-run

Behavior

The runner auto-discovers all safe GPU/template combinations from the public submit catalog.
Automatic GPU selection uses smooth 50/30/20 scheduling: long-term Wilson-ranked top 3 GPUs, all supported GPUs, and the best GPU from the latest 1,000 terminal tasks.
A strategy generation lasts exactly 200 platform-accepted submissions. Rejected API calls and duplicates do not advance it. The next cycle refreshes platform history before submitting again.
Strategy state is stored in .modelhub_state/gpu_strategy.json; a generation never recalculates during candidate submission.
Candidate discovery starts with the configured recent window, then automatically expands to 7 days, 30 days, and older history (up to 3,000 models) when the recent pool is exhausted.
Model verification responses are cached across poll cycles for 15 minutes. Local model/GPU failures cool down after 24 hours instead of remaining permanently blocked.
Community deduplication is model/GPU-specific: another GPU's adaptation does not block the current GPU. Every actual submission performs a fresh uncached check for its exact GPU.
If the community lookup is unavailable, submission is deferred. A platform model-uniqueness rejection permanently excludes only that model/GPU combination from future local retries.
Each model can be submitted at most once per GPU.
Multiple ModelHub tokens are pooled and used to route submissions to the account with available async capacity.
Concurrent submissions reserve account slots locally, and an account-capacity race automatically falls through to another account.
Concurrent local processes claim model/GPU pairs in .modelhub_state/submission_claims.jsonl; shared ledger, history, and outcome files use process locks and atomic replacement.
ModelScope list pages are paced and cached for 15 minutes. HTTP 429 responses use exponential backoff and Retry-After; pages already downloaded remain usable and the failed page is retried on the next cycle.
Every third poll cycle, a full account gets one controlled capacity probe. A successful probe raises that account's persisted known limit; a capacity rejection enters cooldown.
Each [scan] log records the discovery stage and candidate yield. The final [daily] wave_done log includes skip_reasons, making empty candidate pools distinguishable from API failures.

Important Flags

Common flags:

--daily-target: total target submissions for the day; 0 means unlimited
--min-downloads: ModelScope download floor
--history-stats-threshold: local ledger threshold before using online history stats
--max-scan-models: hard cap on scanned HF models for a run (0 = auto)
--scan-multiplier: multiplier used for auto scan cap derivation from quota/queue capacity
--read-concurrency: concurrent HTTP reads while scanning model candidates (default 4)
--max-submits-per-run: max tasks to submit per run cycle (0 = unlimited)
--submit-concurrency: concurrent task submissions (default auto, uses 0)
--skip-outcome-sync: skip outcome sync before scanning
--skip-history-archive: skip history archive download for this run
--dry-run: plan only, do not submit
--gpu-strategy-refresh-submissions: accepted tasks per strategy generation (default 200)
--disable-gpu-strategy: restore legacy ordering; explicit --gpu/--gpus also bypasses adaptive selection
run_daily.sh injects --daily-target 3 when no daily-target flag is provided. Set SUBMIT_DAILY_TARGET or pass --daily-target explicitly for a different target.

run_poll.sh adds:

--poll-interval-seconds: sleep when all accounts are saturated (default 15)
--idle-interval-seconds: sleep when a cycle submits nothing (default 60)
--max-scan-models: hard cap on scanned HF models for this cycle (0 = auto)
--scan-multiplier: multiplier used for auto scan cap derivation from quota/queue capacity
--max-submits-per-run: max tasks to submit per poll cycle (0 = unlimited)
--skip-outcome-sync: skip outcome sync before scanning
--skip-history-archive: skip history archive download for this cycle
--submit-concurrency: concurrent task submission calls used by each cycle (0 = auto)
--post-cycle-cooldown-seconds: pause after a successful cycle before next cycle (default 2)
--max-cycles: optional hard stop for testing or batch windows

Output

Run artifacts are written under:

runs/: one-shot submission runs
daily_runs/: batch orchestration runs
poll_runs/: poller cycles

Each run typically includes:

summary.json
pre_submit_report.json
candidates.jsonl
submitted.jsonl
skipped.jsonl
failed.jsonl

Persistent local scheduler state is written under .modelhub_state/:

gpu_strategy.json: GPU ranks, generation progress, and 50/30/20 accepted counters
account_capacity.json: learned per-account active-task limits
submission_exclusions.jsonl: non-retryable model/GPU uniqueness rejections

Verification

Run the full test suite:

cd /path/to/submmit
python3 -m unittest discover -s tests -v

Notes

This is a submission automation tool, not a scheduler daemon. Use screen, tmux, nohup, or systemd if you want it to keep running in the background.
The platform still enforces per-account async capacity limits, so the poller can keep the queue close to full but cannot override the platform cap.
bash run_poll.sh now defaults to unlimited mode and keeps refilling until you stop the process manually.
Queue polling defaults to 15 seconds, successful-cycle cooldown to 2 seconds, and per-cycle submissions to all available slots. Override these values when the platform requires a lower request rate.