4 Commits

Author SHA1 Message Date
Victor Cao
0b14a4d1ee live: 6 submits per hour (cycle cap 6, attempts 30, prechecks 60) 2026-10-18 00:13:14 +08:00
Victor Cao
c95b66bb40 live: bake ALL_TASKS_FOOTHOLD=1 for v0.2.9 2026-10-17 23:10:16 +08:00
Victor Cao
9f265595e7 live: all-tasks foothold path (ALL_TASKS_FOOTHOLD, default off)
Wire onlyMine=false pass/fail/running slices into candidate merge;
exclude waiting; demote queue-only pending text; keep onlyMine queue gate.
2026-10-17 22:55:28 +08:00
Victor Cao
b0d66da881 live: ban gguf/openchat/peft-lora; bake verify_fail marks from 2026-10-17 analysis 2026-10-17 19:23:51 +08:00
13 changed files with 562 additions and 28 deletions

View File

@@ -18,6 +18,9 @@ ENV PYTHONUNBUFFERED=1
ENV EXTERNAL_SERVICE_TOKEN=4c44b92223c64bca9ed67701ceb2f4e8
ENV XC_TOKEN=4c44b92223c64bca9ed67701ceb2f4e8
ENV XC_TOKEN_FILE=/app/secrets/xc_token
ENV ALL_TASKS_FOOTHOLD=1
ENV LIVE_MAX_SUBMITS_PER_CYCLE=6
ENV LIVE_MAX_ATTEMPTS_PER_CYCLE=30
EXPOSE 8080

View File

@@ -19,12 +19,12 @@ Tag `v0.2.3-live-token` 内置个人凭证,风险与停交说明见 [token bak
Victor 的确认已内置,`LIVE_CONFIRM` 仍接受但不再必需。每次 task/add 前,
通过 Xc-Token 全量分页读取 onlyMine task/page;非终态计入队列,读取失败停交。
`LIVE_QUEUE_THRESHOLD=95`(严格小于),`LIVE_MAX_SUBMITS_PER_CYCLE=3`,
`LIVE_QUEUE_THRESHOLD=95`(严格小于),`LIVE_MAX_SUBMITS_PER_CYCLE=6`(每小时 6 个;`LIVE_MAX_ATTEMPTS_PER_CYCLE=30`,60027/60028 及 code≠0 不计名额),
`LIVE_CYCLE_SECONDS=3600`(最小 60 秒)。环境队列提示仅可额外阻塞。
候选来自 `PENDING_SUBMIT_MD` / `PENDING_SUBMIT_TXT`,否则使用
`src/data/candidates.json`(随镜像内置);A verified_reuse → 其他 A → B。
禁提、重复、GPU 未映射、占位符候选跳过。按 try_next_gpus 顺序选择首个已映射且未提交的模型/GPU 对。去重合并远端全部状态的 model/GPU、内置 `src/data/submitted-ledger.json` 和进程内提交记录;永久拒绝对内置于 `src/data/reject-marks.json`,不依赖容器文件。A 允许同模型换 GPU;do-not-resubmit 模型跨 GPU 全禁。服务启动记录 remote/baked/marks 数量;每次提交前刷新远端,失败则 hold `ledger_rebuild_failed`。
只读平台预检默认开启(`LIVE_PRECHECK=0` 关闭),每周期最多 `LIVE_MAX_PRECHECKS_PER_CYCLE=30` 次;预检失败仍由 task/add 判定。用 `python3 scripts/sync_reject_marks.py <log.txt>` 合并永久拒绝日志到 baked marks,重建镜像后保留;支持 `--dry-run`、`--marks` 和 `--date`。
只读平台预检默认开启(`LIVE_PRECHECK=0` 关闭),每周期最多 `LIVE_MAX_PRECHECKS_PER_CYCLE=60` 次;预检失败仍由 task/add 判定。用 `python3 scripts/sync_reject_marks.py <log.txt>` 合并永久拒绝日志到 baked marks,重建镜像后保留;支持 `--dry-run`、`--marks` 和 `--date`。
凭证优先级:`EXTERNAL_SERVICE_TOKEN` → `XC_TOKEN_FILE` → `XC_TOKEN` → `MODELHUB_XC_TOKEN`;跳过空值和占位符。每周期只记录 source 环境变量名,使用 `Xc-Token` 请求头,不记录凭证值。
离线再生成:`python3 scripts/bake_candidates.py`、`python3 scripts/bake_ledger.py`,读取 `/workspace/modelhub-adapt` 本地文件;候选 342(A 46、B 296),ledger 206 对及 28 个模型禁提项。
配置优先 build-config,失败使用 sample 并记录来源及原因。token 缺失只 hold。
@@ -87,3 +87,5 @@ tests/ # stdlib unittest
成功指标统一为 `verify_pass`(`verifyResult==1`,验证通过);任务 status 为 `success` 但 `verifyResult==-1` 属于验证失败,积分为 0。
每轮 live 在凭证解析后记录一次 `[live] stats total=N queue=N verify_pass=N verify_fail=N failed=N unknown=N points=X`;读取失败则记录 `[live] stats unavailable reason=<安全原因>`,提交仍由原队列闸门决定。`GET /status` 保留凭证存在布尔值,新增 `success_metric: "verify_pass"` 和 `stats`:最近成功读取的上述数字及 `updated_at`(epoch 秒),尚未读取时为 null。queue 按未终结状态计数;积分仅累计 verify_pass 的有效数字。
`ALL_TASKS_FOOTHOLD=1` 可启用全部任务落点(默认 `0`):只读取通过、失败、running 三桶,硬顶 3/3/2 页、每页 ≤100、单请求 ≤5 秒、总读取 ≤20 秒。任一桶错误整轮回退原候选,stderr 输出固定原因码与分页/截断计数。目标卡只扣除切片中已观察的卡;切片不保证全站覆盖、最新排序、已下载或零重复排队,仍保留原队列门控、marks 与 precheck。上限可用 `ALL_TASKS_MAX_PAGES_PASS/FAIL/RUNNING`、`ALL_TASKS_PAGE_SIZE` 调低,不能调高硬顶。

View File

@@ -60,11 +60,11 @@ Victor 确认已内置,`LIVE_CONFIRM` 仍接受但不必需。
`LIVE_QUEUE_THRESHOLD` 默认 95,只有完整分页的 onlyMine task/page 非终态数严格小于阈值才提交。
缺凭证、HTTP/平台错误、JSON 错误、分页不完整或无效配置均停交。
ASYNC_QUEUE_DEPTH / QUEUE_SNAPSHOT_JSON 仅为额外阻塞提示。
每周期最多 `LIVE_MAX_SUBMITS_PER_CYCLE`(默认 3)次;周期
每周期最多 `LIVE_MAX_SUBMITS_PER_CYCLE`(默认 6,即每小时 6 个;`LIVE_MAX_ATTEMPTS_PER_CYCLE` 默认 30)次;周期
`LIVE_CYCLE_SECONDS` 默认 3600、最小 60 秒,通过 shutdown Event 等待。
候选 A verified_reuse 优先,禁提过滤、GPU enum、占位符检查及远端、内置 ledger 和进程内去重。
永久拒绝对保存在 `src/data/reject-marks.json`,随镜像重建保留;不写容器状态文件。
`LIVE_PRECHECK` 默认 1,只读预检已验证/验证中 GPU;`LIVE_MAX_PRECHECKS_PER_CYCLE` 默认 30。
`LIVE_PRECHECK` 默认 1,只读预检已验证/验证中 GPU;`LIVE_MAX_PRECHECKS_PER_CYCLE` 默认 60。
build-config 失败回退 sample 并记录原因;日志不记录响应正文或凭证。
统计成功口径:成功指标统一为 `verify_pass`(`verifyResult==1`,验证通过);任务 status 为 `success` 但 `verifyResult==-1` 属于验证失败,积分为 0。 每轮日志格式为 `[live] stats total=N queue=N verify_pass=N verify_fail=N failed=N unknown=N points=X`;失败日志为 `[live] stats unavailable reason=<安全原因>`。

213
src/all_tasks_foothold.py Normal file
View File

@@ -0,0 +1,213 @@
"""Bounded all-scope evidence slices, isolated from my-queue state."""
from dataclasses import dataclass, field
import json
import os
import re
import sys
import time
import urllib.parse
import urllib.request
from .config_params import GPU_ENUMS
from .live_queue import model_key, valid_total
from .pending_pool import classify_foothold
from .task_add_client import resolve_token
TARGET_GPUS = ('Biren_166m', 'Sunrise_pt-200-x1', 'Iluvatar_bi-150', 'Iluvatar_mrv-100')
RANK = {'verified_reuse': 0, 'failed_foothold': 1, 'in_flight': 2}
REASONS = {'ok', 'missing/invalid_token', 'transport_error', 'http_error',
'platform_error', 'malformed_data', 'filter_mismatch', 'deadline',
'conflicting_task', 'reader_error'}
@dataclass
class FootholdSnapshot:
candidates: list = field(default_factory=list)
reason: str = 'ok'
buckets: dict = field(default_factory=dict)
def _limit(env, default):
try:
value = int(os.environ.get(env, str(default)))
return min(value, default) if value > 0 else default
except ValueError:
return default
def classify_api_record(record, bucket):
"""Positive whitelist. A mismatch invalidates the complete snapshot."""
if not isinstance(record, dict):
return None
status = record.get('status')
status = status.strip().lower() if isinstance(status, str) else None
result = record.get('verifyResult')
if type(result) is str and result.strip() in ('1', '-1', '0'):
result = int(result.strip())
if result is not None and type(result) is not int:
return None
expected = {'pass': ('success', 1), 'fail': ('success', -1)}
if bucket in expected:
if (status, result) != expected[bucket]:
return None
elif bucket != 'running' or status != 'running' or result not in (None, 0):
return None
model, gpu = record.get('modelId'), record.get('gpuType')
if not isinstance(model, str) or not re.fullmatch(r'[\w.-]+/[\w./-]+', model_key(model)):
return None
if not isinstance(gpu, str) or not re.fullmatch(r'[\w .-]+', gpu.strip()):
return None
gpu = GPU_ENUMS.get(gpu.strip(), gpu.strip())
return (model.strip(), gpu, {'pass': 'verified_reuse', 'fail': 'failed_foothold', 'running': 'in_flight'}[bucket])
def build_candidates(evidence, *, target_gpus=TARGET_GPUS):
groups = {}
targets = list(dict.fromkeys(GPU_ENUMS[g] for g in target_gpus if g in GPU_ENUMS))
for model, gpu, kind in evidence:
key = model_key(model)
row = groups.setdefault(key, dict(model_address=model, section='A',
foothold_class=kind, evidence_source='all-tasks', no_download=False, seen_gpus=[]))
if gpu not in row['seen_gpus']:
row['seen_gpus'].append(gpu)
if RANK[kind] < RANK[row['foothold_class']]:
row['foothold_class'] = kind
for row in groups.values():
row['try_next_gpus'] = [g for g in targets if g not in row['seen_gpus']]
return sorted(groups.values(), key=candidate_priority)
def _kind(row):
text = row.get('foothold_text', '')
if text and row.get('evidence_source') != 'all-tasks':
return classify_foothold(text)
kind = row.get('foothold_class', 'unknown')
if kind == 'in_flight' and row.get('evidence_source') != 'all-tasks':
return 'unknown'
return kind
def candidate_priority(row):
return (RANK.get(_kind(row), 3), row.get('section', 'B') != 'A')
def _choices(row):
values = row['try_next_gpus'] if 'try_next_gpus' in row else [row.get('targetGpu')]
return list(dict.fromkeys(GPU_ENUMS[g] for g in values if isinstance(g, str) and g in GPU_ENUMS))
def merge_candidates(base, dynamic):
"""Keep base constraints; intersect explicit targets and subtract seen cards."""
rows = {}
constrained = set()
for incoming in [*base, *dynamic]:
key = model_key(incoming.get('model_address') or incoming.get('modelAddress'))
row = dict(incoming)
row['foothold_class'] = _kind(row)
explicit_targets = 'try_next_gpus' in row or 'targetGpu' in row
row['try_next_gpus'] = _choices(row)
if key not in rows:
rows[key] = row
if explicit_targets:
constrained.add(key)
continue
existing = rows[key]
seen = set(existing.get('seen_gpus', [])) | set(row.get('seen_gpus', []))
existing['seen_gpus'] = list(seen)
if key not in constrained and explicit_targets:
existing['try_next_gpus'] = row['try_next_gpus']
constrained.add(key)
incoming_choices = row['try_next_gpus'] if explicit_targets else existing['try_next_gpus']
existing['try_next_gpus'] = [g for g in existing['try_next_gpus'] if g in incoming_choices and g not in seen]
if candidate_priority(row)[0] < candidate_priority(existing)[0]:
existing['foothold_class'] = row['foothold_class']
existing['evidence_source'] = row.get('evidence_source')
if row.get('skip_reason') and not existing.get('skip_reason'):
existing['skip_reason'] = row['skip_reason']
return sorted(rows.values(), key=candidate_priority)
def load_all_tasks_foothold(*, opener=None, token_resolver=None, clock=None):
opener = opener or urllib.request.urlopen
clock = clock or time.monotonic
buckets = {}
def fail(reason):
return FootholdSnapshot(reason=reason, buckets=buckets)
try:
token = (token_resolver or resolve_token)()
except Exception:
return fail('missing/invalid_token')
deadline = clock() + 20
size = _limit('ALL_TASKS_PAGE_SIZE', 100)
evidence, tasks = [], {}
for bucket, cap, filters in (
('pass', 3, {'verifyResult': 1, 'status': 'success'}),
('fail', 3, {'verifyResult': -1, 'status': 'success'}),
('running', 2, {'status': 'running'}),
):
info = buckets[bucket] = dict(pages=0, valid=0, unique=0, truncated=False)
fingerprints = set()
raw_count = 0
for page in range(1, _limit('ALL_TASKS_MAX_PAGES_' + bucket.upper(), cap) + 1):
remaining = deadline - clock()
if remaining <= 0:
return fail('deadline')
query = urllib.parse.urlencode(dict(onlyMine='false', pageSize=size, current=page, **filters))
request = urllib.request.Request(os.environ.get('MODELHUB_API_BASE', 'https://modelhub.org.cn').rstrip('/') + '/api/adapt/task/page?' + query,
headers={'Xc-Token': token, 'Accept': 'application/json'})
try:
with opener(request, timeout=min(5, remaining)) as response:
if not 200 <= response.status < 300:
return fail('http_error')
body = json.loads(response.read())
except (ValueError, UnicodeError):
return fail('malformed_data')
except Exception:
return fail('transport_error')
if clock() >= deadline:
return fail('deadline')
if not isinstance(body, dict) or type(body.get('code')) not in (int, str) or str(body['code']) != '0':
return fail('platform_error')
data = body.get('data')
if not isinstance(data, dict) or not valid_total(data.get('total')):
return fail('malformed_data')
records = data.get('records')
if not isinstance(records, list) or len(records) > size:
return fail('malformed_data')
info['pages'] += 1
page_evidence = []
for record in records:
item = classify_api_record(record, bucket)
if item is None or any(token.lower() in part.lower() for part in item):
return fail('filter_mismatch')
task = record.get('taskId')
if task is not None and (type(task) not in (str, int) or not str(task).strip()):
return fail('malformed_data')
identity = ('task', str(task)) if task is not None else ('evidence', model_key(item[0]), item[1], item[2])
normalized = (model_key(item[0]), *item[1:])
if identity in tasks and tasks[identity] != normalized:
return fail('conflicting_task')
info['valid'] += 1
page_evidence.append(normalized)
if identity not in tasks:
tasks[identity] = normalized
evidence.append(item)
info['unique'] += 1
fingerprint = tuple(page_evidence)
raw_count += len(records)
info['truncated'] = bool(records) and raw_count < int(data['total'])
if fingerprint in fingerprints:
info['truncated'] = True
break
fingerprints.add(fingerprint)
if not records or not info['truncated']:
break
return FootholdSnapshot(build_candidates(evidence), buckets=buckets)
def report_snapshot(snapshot):
reason = snapshot.reason if snapshot.reason in REASONS else 'reader_error'
print('[foothold] reason=' + reason, file=sys.stderr)
for bucket in ('pass', 'fail', 'running'):
if bucket in snapshot.buckets:
info = snapshot.buckets[bucket]
print('[foothold] bucket=' + bucket + ' ' + ' '.join(f'{k}={int(info[k])}' for k in ('pages', 'valid', 'unique', 'truncated')), file=sys.stderr)

View File

@@ -2,6 +2,10 @@
import re
RULES = (
('gguf', r'gguf|\.gguf|ggml'),
('openchat', r'openchat'),
# Full-weight exports override adapter hints anywhere in the identifier.
('peft-lora', r'^(?!.*(?:merged|merge|fullft|full-ft)).*(?:\blora\b|adapter_model|adapter-config|peft)'),
('gpt2', r'gpt-?2|dialogpt|distilgpt2'),
('phi', r'phi[-_. ]?(?:3(?:\.5)?|4)(?:\D|$)'),
('nemotron-h', r'nemotron-h'), ('olmo2', r'olmo-?2'),

View File

@@ -56,6 +56,62 @@
"code": 60028,
"reason": "CREATE_CONTEST_TASK_MODEL_VERIFY_RUNNING",
"date": "2026-10-17"
},
{
"model": "Qurtana/Qwen-2.5-0.5B-MathInstruct.rev.2",
"gpu": "Biren_166m",
"code": 60029,
"reason": "VERIFY_FAIL:SCORE_GAP",
"date": "2026-10-17"
},
{
"model": "prithivMLmods/Reasoning-SmolLM2-135M",
"gpu": "Biren_166m",
"code": 60029,
"reason": "VERIFY_FAIL:SCORE_GAP",
"date": "2026-10-17"
},
{
"model": "Qwen/Qwen2.5-Coder-7B-Instruct",
"gpu": "Sunrise_pt-200-x1",
"code": 60029,
"reason": "VERIFY_FAIL:SCORE_GAP",
"date": "2026-10-17"
},
{
"model": "FlofloB/100k_fineweb_continued_pretraining_Qwen2.5-0.5B-Instruct_Unsloth_merged_16bit",
"gpu": "Biren_166m",
"code": 60029,
"reason": "VERIFY_FAIL:SCORE_GAP",
"date": "2026-10-17"
},
{
"model": "prithivMLmods/Bellatrix-Tiny-0.5B",
"gpu": "Biren_166m",
"code": 60029,
"reason": "VERIFY_FAIL:SCORE_GAP",
"date": "2026-10-17"
},
{
"model": "sunmengjie/DeepSeek-R1-Distill-Qwen-1.5B-GRPO",
"gpu": "Biren_166m",
"code": 60029,
"reason": "VERIFY_FAIL:PEFT_UNMERGED",
"date": "2026-10-17"
},
{
"model": "DevQuasar/efficientscaling.Z1-7B-GGUF",
"gpu": "Mthreads_s4000",
"code": 60029,
"reason": "VERIFY_FAIL:NO_IDLE_DEVICE",
"date": "2026-10-17"
},
{
"model": "abaryan/CyberXP_Agent_Llama_3.2_1B",
"gpu": "Biren_166m",
"code": 60029,
"reason": "VERIFY_FAIL:REF_CACHE_MISS",
"date": "2026-10-17"
}
]
}

View File

@@ -9,6 +9,8 @@ from .config_params import GPU_ENUMS, build_config_params_with_meta
from .gate import evaluate_gate, queue_threshold
from .live_queue import read_queue, read_queue_stats, model_key
from .pending_pool import build_ordered_candidates
from .all_tasks_foothold import (load_all_tasks_foothold, merge_candidates, candidate_priority,
FootholdSnapshot, report_snapshot)
from .task_add_client import TaskAddClient, build_payload, resolve_token_with_source, token_probe_status
from .ledger import load_baked
from .marks import load_marks, REJECT_CODES, PERMANENT_PAIR_CODES
@@ -19,12 +21,12 @@ _marked = {}
_cycle_lock = threading.Lock()
def run_cycle(candidates=None, client=None, queue_reader=None, now=None, stats_reader=None, precheck=None):
def run_cycle(candidates=None, client=None, queue_reader=None, now=None, stats_reader=None, precheck=None, foothold_reader=None):
with _cycle_lock:
return _run_cycle(candidates, client, queue_reader, now, stats_reader, precheck)
return _run_cycle(candidates, client, queue_reader, now, stats_reader, precheck, foothold_reader)
def _run_cycle(candidates, client, queue_reader, now, stats_reader, precheck):
def _run_cycle(candidates, client, queue_reader, now, stats_reader, precheck, foothold_reader):
decisions = []
def emit(action, reason, **fields):
@@ -69,18 +71,18 @@ def _run_cycle(candidates, client, queue_reader, now, stats_reader, precheck):
line = 'unavailable reason=' + (reason if isinstance(reason, str) and reason in safe_reasons else 'read_failed')
print('[live] stats ' + line, file=sys.stderr)
try:
cap = int(os.environ.get('LIVE_MAX_SUBMITS_PER_CYCLE', '3'))
cap = int(os.environ.get('LIVE_MAX_SUBMITS_PER_CYCLE', '6'))
if cap < 1:
raise ValueError
except ValueError:
emit('hold', 'invalid_cycle_cap')
return decisions
try:
attempt_cap = int(os.environ.get('LIVE_MAX_ATTEMPTS_PER_CYCLE', '15'))
attempt_cap = int(os.environ.get('LIVE_MAX_ATTEMPTS_PER_CYCLE', '30'))
if attempt_cap < 1:
raise ValueError
except ValueError:
attempt_cap = 15
attempt_cap = 30
if candidates is None:
try:
if os.environ.get('PENDING_SUBMIT_MD') or os.environ.get('PENDING_SUBMIT_TXT'):
@@ -90,7 +92,21 @@ def _run_cycle(candidates, client, queue_reader, now, stats_reader, precheck):
except (OSError, ValueError, IndexError):
emit('hold', 'candidate_read_failed')
return decisions
candidates = sorted(candidates, key=lambda c: (c.get('section', 'B') != 'A', c.get('foothold_class') != 'verified_reuse'))
if os.environ.get('ALL_TASKS_FOOTHOLD', '0') == '1' and (foothold_reader is not None or queue_reader is None):
try:
snapshot = (foothold_reader or load_all_tasks_foothold)()
if not isinstance(snapshot, FootholdSnapshot):
raise ValueError
except Exception:
snapshot = FootholdSnapshot(reason='reader_error')
report_snapshot(snapshot)
dynamic = snapshot.candidates if snapshot.reason == 'ok' else []
if os.environ.get('PENDING_SUBMIT_MD') or os.environ.get('PENDING_SUBMIT_TXT'):
section = os.environ.get('PENDING_SECTION', '').strip().upper()
if section:
dynamic = [row for row in dynamic if row.get('section') == section]
candidates = merge_candidates(candidates, dynamic)
candidates = sorted(candidates, key=candidate_priority)
if not candidates:
emit('hold', 'no_candidates')
return decisions
@@ -106,11 +122,11 @@ def _run_cycle(candidates, client, queue_reader, now, stats_reader, precheck):
emit('hold', 'state_read_failed')
return decisions
try:
precheck_cap = int(os.environ.get('LIVE_MAX_PRECHECKS_PER_CYCLE', '30'))
precheck_cap = int(os.environ.get('LIVE_MAX_PRECHECKS_PER_CYCLE', '60'))
if precheck_cap < 1:
raise ValueError
except ValueError:
precheck_cap = 30
precheck_cap = 60
prechecks = 0
emitted_marks = set()
@@ -163,7 +179,7 @@ def _run_cycle(candidates, client, queue_reader, now, stats_reader, precheck):
emit('hold', 'cycle_cap')
break
mapping = {}
choices = candidate.get('try_next_gpus') or [candidate.get('targetGpu')]
choices = candidate['try_next_gpus'] if 'try_next_gpus' in candidate else [candidate.get('targetGpu')]
gpu = choose(choices, key, model, mapping)
if gpu is None:
emit('skip', 'all_gpus_done_or_unmapped', model=model)

View File

@@ -46,12 +46,12 @@ def classify_foothold(text):
"""Classify literal pending evidence; mixed markers prefer verified."""
for label, markers in (
('verified_reuse', ('已验证成功', '验证通过')),
('in_flight', ('正在验证', '验证排队中', '验证进行中')),
('in_flight', ('正在验证', '验证进行中')),
('failed_foothold', ('验证失败',)),
):
if any(marker in text for marker in markers):
return label
return 'unknown'
return 'queued_only' if '验证排队中' in text else 'unknown'
def has_no_download_marker(text):

View File

@@ -0,0 +1,125 @@
import io
import json
import os
import unittest
from unittest.mock import Mock, patch
from urllib.parse import parse_qs, urlparse
from src.all_tasks_foothold import (load_all_tasks_foothold, classify_api_record,
build_candidates, merge_candidates, candidate_priority, TARGET_GPUS, FootholdSnapshot)
from src import live_queue
def record(status='success', result=1, model='org/model', gpu='壁砺', **extra):
return dict(modelId=model, gpuType=gpu, status=status, verifyResult=result, **extra)
def response(records=(), total=None, **body):
result = Mock(status=200)
result.read.return_value = json.dumps(body or dict(code=0, data=dict(records=list(records), total=len(records) if total is None else total))).encode()
result.__enter__ = Mock(return_value=result)
result.__exit__ = Mock(return_value=False)
return result
class TestFoothold(unittest.TestCase):
def setUp(self):
self.env = patch.dict(os.environ, {}, clear=True)
self.env.start()
self.addCleanup(self.env.stop)
self.network = patch('urllib.request.urlopen', side_effect=AssertionError('network forbidden'))
self.network.start()
self.addCleanup(self.network.stop)
def load(self, responses, **kw):
opener = Mock(side_effect=responses)
snapshot = load_all_tasks_foothold(opener=opener, token_resolver=lambda: 'offline-credential', **kw)
return snapshot, opener
def test_whitelist_rejects_waiting_conflicts_and_bad_types(self):
for status, result, bucket in [('waiting', 1, 'pass'), ('failed', -1, 'fail'),
('running', 1, 'running'), ('success', True, 'pass'),
('success', {}, 'pass'), ('other', 1, 'pass'), ('success', 1.0, 'pass')]:
self.assertIsNone(classify_api_record(record(status, result), bucket))
for result in (1, '1'):
self.assertIsNotNone(classify_api_record(record(result=result), 'pass'))
for result in (None, 0, '0'):
self.assertIsNotNone(classify_api_record(record(' RUNNING ', result), 'running'))
for fields in (dict(modelId=True), dict(gpuType=''), dict(modelId='')):
self.assertIsNone(classify_api_record(dict(record(), **fields), 'pass'))
def test_scope_buckets_aliases_order_and_queue_isolation(self):
before = live_queue.last_stats()
snap, opener = self.load([response([record()]), response([record(result=-1, model='org/fail')]),
response([record('running', None, model='org/run')])])
self.assertEqual(snap.reason, 'ok')
self.assertEqual([r['foothold_class'] for r in snap.candidates], ['verified_reuse', 'failed_foothold', 'in_flight'])
self.assertEqual(snap.candidates[0]['try_next_gpus'], list(TARGET_GPUS[1:]))
for call in opener.call_args_list:
query = parse_qs(urlparse(call.args[0].full_url).query)
self.assertEqual(query['onlyMine'], ['false'])
self.assertNotIn('waiting', query.get('status', []))
self.assertLessEqual(call.kwargs['timeout'], 5)
self.assertEqual(before, live_queue.last_stats())
self.assertEqual(len(opener.call_args_list), 3)
def test_any_bucket_error_discards_whole_snapshot(self):
for bad in (response([record('waiting', 1)]), response(data=None, code=0),
response(code=403), OSError('private-response')):
snap, _ = self.load([response([record()]), bad])
self.assertNotEqual(snap.reason, 'ok')
self.assertEqual(snap.candidates, [])
snap, _ = self.load([response(), response(), response()])
self.assertEqual(snap.reason, 'ok')
self.assertEqual(snap.candidates, [])
def test_hard_caps_total_drift_and_invalid_env(self):
with patch.dict(os.environ, {'ALL_TASKS_MAX_PAGES_PASS': '999', 'ALL_TASKS_MAX_PAGES_FAIL': 'bad',
'ALL_TASKS_MAX_PAGES_RUNNING': '-1', 'ALL_TASKS_PAGE_SIZE': '999'}):
pages = [response([record(model='org/p' + str(n))], total=1000+n) for n in range(3)]
pages += [response([record(result=-1, model='org/f' + str(n))], total=1000+n) for n in range(3)]
pages += [response([record('running', None, model='org/r' + str(n))], total=1000+n) for n in range(2)]
snap, opener = self.load(pages)
self.assertEqual(opener.call_count, 8)
self.assertTrue(all(b['truncated'] for b in snap.buckets.values()))
self.assertTrue(all('pageSize=100' in c.args[0].full_url for c in opener.call_args_list))
snap, opener = self.load([], clock=Mock(side_effect=[0, 21]))
self.assertEqual(snap.reason, 'deadline')
opener.assert_not_called()
def test_repeated_tasks_pages_and_conflicts(self):
r = record(taskId=1)
snap, opener = self.load([response([r], total=999), response([r], total=1000), response(), response()])
self.assertEqual(snap.buckets['pass']['unique'], 1)
self.assertTrue(snap.buckets['pass']['truncated'])
self.assertEqual(opener.call_count, 4)
snap, _ = self.load([response([r]), response([record(result=-1, taskId=1)])])
self.assertEqual(snap.reason, 'conflicting_task')
self.assertEqual(snap.candidates, [])
def test_missing_token_never_opens(self):
opener = Mock()
snap = load_all_tasks_foothold(opener=opener, token_resolver=Mock(side_effect=ValueError()))
self.assertEqual(snap.reason, 'missing/invalid_token')
opener.assert_not_called()
def test_merge_constraints_empty_targets_and_uniform_priority(self):
base = [dict(model_address='https://huggingface.co/ORG/Model/', framework='custom',
skip_reason='manual', section='B', foothold_class='verified_reuse',
try_next_gpus=['壁砺'], targetGpu='天垓150'),
dict(model_address='org/static', section='A', foothold_class='verified_reuse', try_next_gpus=['天垓150'])]
dynamic = build_candidates([('org/model', 'Biren_166m', 'in_flight'), ('org/run', 'MetaX_c-500', 'in_flight')])
merged = merge_candidates(base, dynamic)
row = next(r for r in merged if 'huggingface' in r['model_address'])
self.assertEqual(row['framework'], 'custom')
self.assertEqual(row['skip_reason'], 'manual')
self.assertEqual(row['try_next_gpus'], [])
self.assertEqual(row['foothold_class'], 'verified_reuse')
self.assertEqual(len(merged), 3)
self.assertEqual(merged[-1]['model_address'], 'org/run')
self.assertEqual(merged[-1]['try_next_gpus'], list(TARGET_GPUS))
self.assertEqual(candidate_priority(dict(foothold_class='in_flight'))[0], 3)
def test_pending_queue_only_text(self):
from src.pending_pool import classify_foothold
self.assertEqual(classify_foothold('验证排队中'), 'queued_only')
self.assertEqual(classify_foothold('验证排队中; 验证通过'), 'verified_reuse')

36
tests/test_banned.py Normal file
View File

@@ -0,0 +1,36 @@
import unittest
from src.banned import banned_rule
class TestBanned(unittest.TestCase):
def test_default_engine_and_openchat(self):
for model, rule in [('org/Model-GGUF', 'gguf'), ('org/model.gguf', 'gguf'),
('org/GGML-model', 'gguf'), ('OpenChat/model', 'openchat')]:
with self.subTest(model=model):
self.assertEqual(banned_rule(model), rule)
def test_unmerged_adapter_hints(self):
for model in ('org/model-lora', 'org/LORA', 'org/adapter_model',
'org/model-adapter-config', 'org/Model-PEFT'):
with self.subTest(model=model):
self.assertEqual(banned_rule(model), 'peft-lora')
def test_full_weights_and_unrelated_names(self):
for model in ('org/lora-merged', 'org/merge-lora', 'org/PEFT-FULLFT',
'org/full-ft-adapter_model', 'org/adapter-config-MERGED',
'org/flora', 'org/lorax', 'org/model_lora',
'org/model-GRPO', 'abaryan/CyberXP_Agent_Llama_3.2_1B'):
with self.subTest(model=model):
self.assertIsNone(banned_rule(model))
def test_full_weight_hint_does_not_override_other_bans(self):
self.assertEqual(banned_rule('org/lora-merged-GGUF'), 'gguf')
self.assertEqual(banned_rule('org/openchat-fullft'), 'openchat')
def test_existing_rules(self):
for model in ('gpt2', 'org/Phi-3-mini', 'nvidia/Nemotron-H-4B',
'allenai/OLMo-2-7B', 'org/LFM2', 'org/Spark-X2.5',
'org/Breeze', 'org/gemma2', 'org/mistral-v0.1', 'org/tinyllama'):
with self.subTest(model=model):
self.assertIsNotNone(banned_rule(model))

View File

@@ -60,7 +60,7 @@ class TestLiveCycle(unittest.TestCase):
self.assertEqual(self.submitted_pairs(), [])
def test_attempt_cap_and_invalid_fallback(self):
for value, expected in ((None, 15), ('bad', 15), ('0', 15), ('-1', 15), ('2', 2)):
for value, expected in ((None, 30), ('bad', 30), ('0', 30), ('-1', 30), ('2', 2)):
with self.subTest(value=value):
_marked.clear()
self.client.reset_mock()
@@ -71,10 +71,10 @@ class TestLiveCycle(unittest.TestCase):
self.assertEqual(decisions[-1]['reason'], 'attempt_cap')
def test_submit_cap_counts_successes_after_rejections(self):
self.client.submit.side_effect = [dict(http_status=200, platform_code=c) for c in (60020, 0, 60028, 0, 0)]
decisions = self.cycle([candidate(str(i)) for i in range(7)], (1,) * 5)
self.assertEqual(self.client.submit.call_count, 5)
self.assertEqual(sum(d['action'] == 'submitted' for d in decisions), 3)
self.client.submit.side_effect = [dict(http_status=200, platform_code=c) for c in (60020, 0, 60028, 0, 0, 0, 0, 0)]
decisions = self.cycle([candidate(str(i)) for i in range(10)], (1,) * 8)
self.assertEqual(self.client.submit.call_count, 8)
self.assertEqual(sum(d['action'] == 'submitted' for d in decisions), 6)
self.assertEqual(decisions[-1]['reason'], 'cycle_cap')
def test_task_limit_stops_cycle(self):
@@ -110,9 +110,10 @@ class TestLiveCycle(unittest.TestCase):
self.assertIn('[live] hold', self.logs.getvalue())
def test_order_and_cap(self):
rows = [candidate('b'), candidate('a', 'A'), candidate('verified', 'A', 'verified_reuse'), candidate('c'), candidate('d')]
self.cycle(rows, (90, 90, 90))
self.assertEqual([c.args[0]['modelAddress'] for c in self.client.submit.call_args_list], ['org/verified', 'org/a', 'org/b'])
rows = [candidate('b'), candidate('a', 'A'), candidate('verified', 'A', 'verified_reuse'), candidate('c'), candidate('d'), candidate('e'), candidate('f'), candidate('g')]
self.cycle(rows, (90,) * 6)
self.assertEqual([c.args[0]['modelAddress'] for c in self.client.submit.call_args_list],
['org/verified', 'org/a', 'org/b', 'org/c', 'org/d', 'org/e'])
self.assertIn('cycle_cap', self.logs.getvalue())
def test_midcycle(self):
@@ -215,3 +216,63 @@ class TestLiveCycle(unittest.TestCase):
def test_injected_queue_skips_stats_network(self):
self.cycle()
self.assertEqual(self.logs.getvalue().count('[live] stats unavailable reason=not_read'), 1)
def test_foothold_explicit_disabled_and_injected_queue_stay_offline(self):
from src.all_tasks_foothold import FootholdSnapshot
dynamic = Mock(return_value=FootholdSnapshot())
with patch.dict(os.environ, {'ALL_TASKS_FOOTHOLD': '1'}):
run_cycle([candidate('explicit')], self.client, lambda: (0, 'ok', set()), foothold_reader=dynamic)
dynamic.assert_not_called()
with patch('src.live_cycle.build_ordered_candidates', return_value=[candidate('base')]), patch.dict(os.environ, {'PENDING_SUBMIT_MD': 'offline'}):
run_cycle(None, self.client, lambda: (0, 'ok', set()), foothold_reader=dynamic)
with patch.dict(os.environ, {'ALL_TASKS_FOOTHOLD': '1'}):
run_cycle(None, self.client, lambda: (0, 'ok', set()))
dynamic.assert_not_called()
def test_foothold_merge_order_constraints_pending_and_fallback(self):
from src.all_tasks_foothold import FootholdSnapshot, build_candidates
base = [candidate('base', 'A', 'verified_reuse')]
snapshot = FootholdSnapshot(build_candidates([('org/dynamic', 'Biren_166m', 'in_flight')]))
with patch('src.live_cycle.build_ordered_candidates', return_value=base) as pending, patch.dict(os.environ, {'PENDING_SUBMIT_MD': 'offline', 'PENDING_SECTION': 'A', 'ALL_TASKS_FOOTHOLD': '1'}):
run_cycle(None, self.client, lambda: (0, 'ok', set()), foothold_reader=lambda: snapshot)
pending.assert_called_once_with()
self.assertEqual([c.args[0]['modelAddress'] for c in self.client.submit.call_args_list], ['org/base', 'org/dynamic'])
_submitted.clear()
self.client.reset_mock()
run_cycle(None, self.client, lambda: (0, 'ok', set()), foothold_reader=Mock(side_effect=OSError('private-response')))
self.assertEqual(self.client.submit.call_args.args[0]['modelAddress'], 'org/base')
self.assertIn('[foothold] reason=reader_error', self.logs.getvalue())
self.assertNotIn('private-response', self.logs.getvalue())
def test_foothold_empty_intersection_never_revives_target(self):
from src.all_tasks_foothold import FootholdSnapshot, build_candidates
base = [dict(candidate('empty'), targetGpu='曦望')]
snapshot = FootholdSnapshot(build_candidates([('org/empty', 'Biren_166m', 'verified_reuse')]))
with patch('src.live_cycle.build_ordered_candidates', return_value=base), patch.dict(os.environ, {'PENDING_SUBMIT_MD': 'offline', 'ALL_TASKS_FOOTHOLD': '1'}):
decisions = run_cycle(None, self.client, lambda: (0, 'ok', set()), foothold_reader=lambda: snapshot)
self.client.submit.assert_not_called()
self.assertEqual(decisions[-1]['reason'], 'all_gpus_done_or_unmapped')
def test_dynamic_still_respects_queue_marks_ledger_precheck(self):
from src.all_tasks_foothold import FootholdSnapshot
snapshot = FootholdSnapshot([dict(candidate('guarded'), evidence_source='all-tasks', foothold_class='verified_reuse')])
with patch('src.live_cycle.build_ordered_candidates', return_value=[]), patch.dict(os.environ, {'PENDING_SUBMIT_MD': 'offline', 'ALL_TASKS_FOOTHOLD': '1'}):
for count in (95, None):
decisions = run_cycle(None, self.client, lambda: (count, 'ok', set()), foothold_reader=lambda: snapshot)
self.assertEqual(decisions[-1]['action'], 'hold')
with patch('src.live_cycle.load_baked', return_value=({('org/guarded', 'Biren_166m')}, set())):
run_cycle(None, self.client, lambda: (0, 'ok', set()), foothold_reader=lambda: snapshot)
_marked[('org/guarded', 'Biren_166m')] = 60028
run_cycle(None, self.client, lambda: (0, 'ok', set()), foothold_reader=lambda: snapshot)
_marked.clear()
with patch.dict(os.environ, {'LIVE_PRECHECK': '1'}):
run_cycle(None, self.client, lambda: (0, 'ok', set()), foothold_reader=lambda: snapshot,
precheck=lambda _: ('ok', {'Biren_166m': 60027}))
self.client.submit.assert_not_called()
def test_pending_section_b_does_not_gain_dynamic_section_a(self):
from src.all_tasks_foothold import FootholdSnapshot
dynamic = FootholdSnapshot([candidate('dynamic', 'A', 'verified_reuse')])
with patch('src.live_cycle.build_ordered_candidates', return_value=[candidate('base')]), patch.dict(os.environ, {'PENDING_SUBMIT_MD': 'offline', 'PENDING_SECTION': 'B', 'ALL_TASKS_FOOTHOLD': '1'}):
run_cycle(None, self.client, lambda: (0, 'ok', set()), foothold_reader=lambda: dynamic)
self.assertEqual([c.args[0]['modelAddress'] for c in self.client.submit.call_args_list], ['org/base'])

View File

@@ -59,7 +59,7 @@ class TestPendingPool(unittest.TestCase):
'org/fail', 'org/queue', 'org/progress', 'org/b'])
self.assertEqual([r['foothold_class'] for r in rows],
['verified_reuse', 'verified_reuse', 'unknown', 'in_flight',
'failed_foothold', 'in_flight', 'in_flight', 'unknown'])
'failed_foothold', 'queued_only', 'in_flight', 'unknown'])
for row in rows:
self.assertEqual(row['evidence_source'], 'GAP' if row['foothold_class'] == 'unknown' else 'pending-foothold')
self.assertEqual(rows[0]['foothold_text'], '他人:已验证成功; 正在验证')

View File

@@ -18,11 +18,29 @@ from src.precheck import precheck_model
class TestMarks(unittest.TestCase):
def test_baked_marks(self):
marks = load_marks()
self.assertEqual(len(marks), 8)
self.assertEqual(len(marks), 16)
self.assertTrue(all(g in GPU_ENUMS.values() for m, g in marks))
self.assertEqual(marks['madeagents/hammer2.1-0.5b', 'Biren_166m'], 60027)
self.assertEqual(PERMANENT_PAIR_CODES, {60012, *range(60019, 60030)})
def test_verify_fail_marks(self):
rows = json.loads(Path('src/data/reject-marks.json').read_text())['marks']
keys = [(row['model'].lower(), row['gpu']) for row in rows]
self.assertEqual(len(keys), len(set(keys)))
failures = [row for row in rows if row['reason'].startswith('VERIFY_FAIL:')]
self.assertEqual(len(failures), 8)
self.assertEqual(sorted(row['reason'] for row in failures),
sorted(['VERIFY_FAIL:SCORE_GAP'] * 5 +
['VERIFY_FAIL:PEFT_UNMERGED', 'VERIFY_FAIL:NO_IDLE_DEVICE',
'VERIFY_FAIL:REF_CACHE_MISS']))
marks = load_marks()
for row in failures:
self.assertEqual(row['code'], 60029)
self.assertIn(row['code'], PERMANENT_PAIR_CODES)
self.assertEqual(marks[row['model'].lower(), row['gpu']], 60029)
self.assertEqual(marks['abaryan/cyberxp_agent_llama_3.2_1b', 'Biren_166m'], 60029)
self.assertNotIn(('abaryan/cyberxp_agent_llama_3.2_1b', 'Iluvatar_bi-100'), marks)
def test_invalid_marks(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / 'marks.json'
@@ -186,8 +204,8 @@ class TestLiveMarks(unittest.TestCase):
def test_precheck_cap(self):
self.precheck.return_value = 'ok', {'Biren_166m': 60027}
rows = [dict(model_address=f'unit/m{i}', targetGpu='壁砺') for i in range(32)]
for value, count in (('2', 2), ('bad', 30), ('0', 30)):
rows = [dict(model_address=f'unit/m{i}', targetGpu='壁砺') for i in range(62)]
for value, count in (('2', 2), ('bad', 60), ('0', 60)):
_marked.clear()
self.precheck.reset_mock()
with patch.dict(os.environ, {'LIVE_MAX_PRECHECKS_PER_CYCLE': value, 'LIVE_MAX_ATTEMPTS_PER_CYCLE': '1'}):