submmit/modelhub_submmit_api/README.md

ModelHub Submission Runner

This package automates ModelScope model discovery and ModelHub submission. It currently supports:

• one-shot submission planning via main.py
• daily batch execution via run_daily.sh
• continuous queue refill via run_poll.sh
• multiple ModelHub tokens read from KEY.md and KEYS.md
• automatic task/framework/template selection across the supported GPU catalog
• adaptive long-term/recent GPU exploitation with a persistent local snapshot
• live queue/throughput-aware GPU weighting and confidence-ranked framework selection

Layout

• main.py: core discovery, scoring, dedup, and submission
• daily_runner.py: daily wave orchestration
• poll_runner.py: long-running queue refiller
• runner_common.py: shared token / key file loading
• hf_discovery.py: ModelScope model discovery and inspection (keeps the legacy module name)
• modelhub_client.py: ModelHub API client and token-pool routing
• history_stats.py: online history aggregation, ranking, and warnings
• candidate_preflight.py: repository, memory, context, and LLM-assisted compatibility gates
• failure_taxonomy.py: deterministic/platform/semantic failure routing
• llm_classifier.py: optional cached OpenAI-compatible ambiguity classifier
• template_selector.py: template lookup and GPU normalization
• task_registry.py: task-type and framework selection rules
• tests/: unit tests and regression coverage

Key Files

• KEY.md: primary ModelScope and ModelHub tokens
• KEYS.md: optional supplemental ModelHub tokens
• templates/public_submit/adapt_task_templates.jsonl: public submit templates

The runner reads both files automatically. Add more accounts by appending XC_TOKEN3, XC_TOKEN4, and so on to KEYS.md.

Template lookup is also relative. The selector searches from the current working directory and the module directory. The primary project layout is:

• templates/public_submit/adapt_task_templates.jsonl

It still accepts the legacy fallback path below for compatibility with older deployments:

• model adaptation/templates/public_submit/adapt_task_templates.jsonl

If your Space keeps templates in another location, set MODELHUB_TEMPLATE_FILE to the exact JSONL path.

Quick Start

Run a single daily batch:

cd /path/to/submmit
# testing: one run defaults to 3 targets if daily-target is not specified
bash run_daily.sh --rounds 1

Run the continuous queue refiller:

cd /path/to/submmit
bash run_poll.sh

Dry-run either entrypoint to inspect candidate selection without submitting:

cd /path/to/submmit
bash run_daily.sh --dry-run
bash run_poll.sh --dry-run

Behavior

• The runner auto-discovers all safe GPU/template combinations from the public submit catalog.
• Automatic GPU selection uses exact 70/30 accepted-task scheduling: long-term Wilson-ranked top 3 GPUs and the top GPUs from the latest 1,000 terminal tasks. There is no all-GPU exploration category.
• Within each category, weighted-fair scheduling uses estimated queue backlog hours, recent public throughput/success, machine availability, and worker concurrency. Unavailable or stalled GPU pools are circuit-broken instead of continuing to absorb work.
• Compatible frameworks are ranked by ModelHub public aggregate success statistics plus capped local GPU+framework evidence, with a 300-sample public minimum and Wilson confidence bounds. Missing or undersized public evidence receives zero traffic rather than falling back to exploration.
• New frameworks are discovered from the live catalog but get no novelty bonus. They are eligible only with a complete official build config that passes local validation and a confidence score at least 10% above the best incumbent.
• Five consecutive local failures pause a GPU/framework pair for 12 hours; a sub-20% rate over the latest 20 terminal tasks pauses it for 6 hours.
• A strategy generation lasts exactly 200 platform-accepted submissions. Rejected API calls and duplicates do not advance it. The next cycle refreshes platform history before submitting again.
• Strategy state is stored in .modelhub_state/gpu_strategy.json; a generation never recalculates during candidate submission.
• Candidate discovery starts with the configured recent window, then automatically expands to 7 days, 30 days, and older history (up to 3,000 models) when the recent pool is exhausted.
• Model verification responses are cached across poll cycles for 15 minutes. Local model/GPU failures cool down after 24 hours instead of remaining permanently blocked.
• Community deduplication is model/GPU-specific: another GPU's adaptation does not block the current GPU. Every actual submission performs a fresh uncached check for its exact GPU.
• If the community lookup is unavailable, submission is deferred. A platform model-uniqueness rejection permanently excludes only that model/GPU combination from future local retries.
• Each model can be submitted at most once per GPU.
• Multiple ModelHub tokens are pooled and used to route submissions to the account with available async capacity.
• Concurrent submissions reserve account slots locally, and an account-capacity race automatically falls through to another account.
• Concurrent local processes claim model/GPU pairs in .modelhub_state/submission_claims.jsonl; shared ledger, history, and outcome files use process locks and atomic replacement.
• ModelScope list pages are paced and cached for 15 minutes. HTTP 429 responses use exponential backoff and Retry-After; pages already downloaded remain usable and the failed page is retried on the next cycle.
• Every third poll cycle, a full account gets one controlled capacity probe. A successful probe raises that account's persisted known limit; a capacity rejection enters cooldown.
• Each [scan] log records the discovery stage and candidate yield. The final [daily] wave_done log includes skip_reasons, making empty candidate pools distinguishable from API failures.

Important Flags

Common flags:

• --daily-target: total target submissions for the day; 0 means unlimited
• --min-downloads: ModelScope download floor
• --history-stats-threshold: local ledger threshold before using online history stats
• --max-scan-models: hard cap on scanned HF models for a run (0 = auto)
• --scan-multiplier: multiplier used for auto scan cap derivation from quota/queue capacity
• --read-concurrency: concurrent HTTP reads while scanning model candidates (default 4)
• --max-submits-per-run: max tasks to submit per run cycle (0 = unlimited)
• --submit-concurrency: concurrent task submissions (default auto, uses 0)
• --skip-outcome-sync: skip outcome sync before scanning
• --skip-history-archive: skip history archive download for this run
• --dry-run: plan only, do not submit
• --gpu-strategy-refresh-submissions: accepted tasks per strategy generation (default 200)
• --disable-gpu-strategy: restore legacy ordering; explicit --gpu/--gpus also bypasses adaptive selection
• --disable-market-intelligence: disable live queue/throughput and framework-stat weighting
• run_daily.sh injects --daily-target 3 when no daily-target flag is provided. Set SUBMIT_DAILY_TARGET or pass --daily-target explicitly for a different target.

run_poll.sh adds:

• --poll-interval-seconds: sleep when all accounts are saturated (default 15)
• --idle-interval-seconds: sleep when a cycle submits nothing (default 60)
• --max-scan-models: hard cap on scanned HF models for this cycle (0 = auto)
• --scan-multiplier: multiplier used for auto scan cap derivation from quota/queue capacity
• --max-submits-per-run: max tasks to submit per poll cycle (0 = unlimited)
• --skip-outcome-sync: skip outcome sync before scanning
• --skip-history-archive: skip history archive download for this cycle
• --submit-concurrency: concurrent task submission calls used by each cycle (0 = auto)
• --post-cycle-cooldown-seconds: pause after a successful cycle before next cycle (default 2)
• --max-cycles: optional hard stop for testing or batch windows

Failure-informed preflight is enabled by default. It rejects deterministic missing-file and predicted-OOM cases, clamps unsafe context-length arguments, and records its decisions in candidatePreflight and each candidate's preflightMetadata. Use --disable-candidate-preflight only for diagnosis.

The memory gate totals the complete recursive repository and applies the same 20% overhead observed in ModelHub PREFLIGHT_OOM reports. All 14 currently verifiable GPU types have evidence-backed capacities; an incomplete repository size is deferred instead of estimated. See ../docs/gpu-memory-capacity-2026-08-10.md.

Optional Qwen review uses MODELHUB_QWEN_ENDPOINT, MODELHUB_QWEN_MODEL, and MODELHUB_QWEN_API_KEY (or DASHSCOPE_API_KEY). The generic MODELHUB_LLM_CLASSIFIER_* names remain supported. A root .env key named dashscope is loaded automatically, and the default model is qwen3.7-flash. Qwen is called only for unresolved architecture/remote-code semantics or ambiguous failure roots, with a default rolling limit of 20 calls/hour and one concurrent request. Only high-confidence denials block; an error, timeout, or abstention leaves the already-vetted candidate eligible.

Outcome sync also classifies a bounded set of this worker's failed-task ZIP logs. Hard error signatures run first; ambiguous runtime roots can use the configured LLM. Platform faults are excluded from long-term compatibility scores and use a short 30-minute breaker after three consecutive failures. Failed log downloads are persisted and stop after three attempts.

Output

Run artifacts are written under:

• runs/: one-shot submission runs
• daily_runs/: batch orchestration runs
• poll_runs/: poller cycles

Each run typically includes:

• summary.json
• pre_submit_report.json
• candidates.jsonl
• submitted.jsonl
• skipped.jsonl
• failed.jsonl

Persistent local scheduler state is written under .modelhub_state/:

• gpu_strategy.json: GPU ranks, generation progress, and 70/30 accepted counters
• market_intelligence.json: cached public queue, throughput, health, and framework statistics
• account_capacity.json: learned per-account active-task limits
• submission_exclusions.jsonl: non-retryable model/GPU uniqueness rejections

Verification

Run the full test suite:

cd /path/to/submmit
python3 -m unittest discover -s tests -v

Notes

• This is a submission automation tool, not a scheduler daemon. Use screen, tmux, nohup, or systemd if you want it to keep running in the background.
• The platform still enforces per-account async capacity limits, so the poller can keep the queue close to full but cannot override the platform cap.
• bash run_poll.sh now defaults to unlimited mode and keeps refilling until you stop the process manually.
• Queue polling defaults to 15 seconds, successful-cycle cooldown to 2 seconds, and per-cycle submissions to all available slots. Override these values when the platform requires a lower request rate.