CoolBoy/submmit
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

perf: complete history bootstrap in one streaming phase

Browse Source
This commit is contained in:
CoolBoy
2026-09-04 11:35:43 +08:00
parent 54ed4351a0
commit 2c748dcec2
6 changed files with 183 additions and 298 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
README.md
View File

@@ -59,15 +59,11 @@ Optional tuning:
- `MODELHUB_QUEUE_CLEANUP_REPORT_PATH` default `.modelhub_state/queue_cleanup_latest.json`
- `MODELHUB_ARCHITECTURE_BLACKLIST_PATH` default `.modelhub_state/architecture_compatibility_blacklist.json`
- `MODELHUB_ARCHITECTURE_BLOCK_TTL_DAYS` default `30`
- `MODELHUB_ARCHITECTURE_COMMUNITY_PROBE_SIZE` default `50`
- `MODELHUB_ARCHITECTURE_COMMUNITY_LOOKBACK_DAYS` default `30`
- `MODELHUB_ARCHITECTURE_COMMUNITY_LATEST_LIMIT` default `100` (synchronous seed only)
- `MODELHUB_ARCHITECTURE_BOOTSTRAP_WORKERS` default `8`
- `MODELHUB_ARCHITECTURE_BOOTSTRAP_TASKS_PER_ACCOUNT` default `10` (synchronous seed only)
- `MODELHUB_ARCHITECTURE_BOOTSTRAP_MAX_LOGS` default `120` (synchronous seed only)
- `MODELHUB_ARCHITECTURE_BACKFILL_PAGE_SIZE` default `50`
- `MODELHUB_ARCHITECTURE_BACKFILL_PAGES_PER_CYCLE` default `2`
- `MODELHUB_ARCHITECTURE_BACKFILL_LOGS_PER_CYCLE` default `100`
- `MODELHUB_ARCHITECTURE_BACKFILL_PAGE_SIZE` default `100`
- `MODELHUB_ARCHITECTURE_BACKFILL_PAGES_PER_BATCH` default `10` (about 1,000 task rows)
- `MODELHUB_ARCHITECTURE_BACKFILL_LOG_BATCH_SIZE` default `200`
- `MODELHUB_ARCHITECTURE_BACKFILL_CHECKPOINT_RECORDS` default `2000`
- `MODELHUB_RECENT_MODEL_RESERVE_SLOTS` default `5` per account
- `MODELHUB_RECENT_MODEL_DAYS` default `7`
- `MODELHUB_STATE_SYNC_REMOTE` default `https://dev.modelhub.org.cn/CoolBoy/submmit.git`
@@ -173,17 +169,18 @@ change that window. The stats report exposes `architectureCompatibilityBlocks`
and per-GPU/framework block counts. The live snapshot is written to
`.modelhub_state/architecture_compatibility_blacklist.json`.
Before the first cleanup/submission cycle, the poller imports only a small recent
seed so cold start cannot be dominated by historical API and log downloads. The
submission pass then runs first on every cycle. Afterwards a resumable backfill
walks every configured account's complete history in bounded pages, classifies
each available failure log, and immediately reduces the page to cumulative
Cold start does not scan historical tasks before submission. The first normal
submission pass runs with official/public live routing evidence, then a one-time
resumable initialization walks every configured account's complete history in
bounded batches of about 1,000 rows. It classifies every available failure log
in internal batches of 200 and immediately reduces each metadata batch to cumulative
success/failure statistics, memory observations, safe configurations, and exact
compatibility blocks. Only the cursor, temporary deduplication task IDs, and
derived decision checkpoint are synchronized; raw task rows and logs are not.
The task-ID set is deleted when backfill completes. A restart resumes the cursor
instead of rescanning prior pages. Empty public evidence is a fallback signal,
never proof that no incompatibility exists.
instead of rescanning prior pages. Checkpoints are pushed about every 2,000 rows,
not after each API page. Once complete, full-history work is permanently disabled
and normal outcome synchronization processes only newer task changes.
Before a candidate reaches the submit queue, failure-informed preflight checks
the actual ModelScope repository structure and file sizes. Non-GGUF text
@@ -439,11 +436,19 @@ processed, and each page is compacted immediately into decision state. It also
fixes large-batch classification so checkpoint compaction cannot discard an
uninspected failure row.
Version `2026.09.04.4` removes the long-lived per-cycle backfill. With no durable
checkpoint, the worker first fills the queue and then completes the entire
historical initialization as one streaming phase: 1,000 task rows per bounded
metadata batch, 200 failure logs per internal classification batch, and one
durable Git checkpoint per roughly 2,000 scanned rows. Raw rows are discarded
after every aggregate merge. Transient API failure pauses at the current cursor
for the next poll cycle; successful completion disables the scan permanently.
## Deploy
Create a tag and submit the repository URL plus tag in "我的适配智能体".
```bash
git tag -a agent-v32 -m "ModelHub agent 2026.09.04.3"
git push origin main agent-v32
git tag -a agent-v33 -m "ModelHub agent 2026.09.04.4"
git push origin main agent-v33
```