rpant/iolai26-solve
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: rpant/iolai26-solve
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-28 09:36:12 +08:00
commit 5b016c1af1
45 changed files with 461120 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
.gitattributes vendored Normal file
View File

@@ -0,0 +1,38 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
*.awq filter=lfs diff=lfs merge=lfs -text
*.gptq filter=lfs diff=lfs merge=lfs -text
*.gguf filter=lfs diff=lfs merge=lfs -text

32
.gitignore vendored Normal file
View File

@@ -0,0 +1,32 @@
data/dev/linguini.csv
.venv/
__pycache__/
*.pyc
*.pyo
.pytest_cache/
.DS_Store
*.egg-info/
.ipynb_checkpoints/
fable.md
story.md
research_report.md
# local outputs, not build artifacts to ship
submission.csv
/tmp/
# never commit credentials
.env
*.token
# local Claude Code session state, not project source
.claude/
# huggingface_hub local_dir download metadata (created by prepare_weights.py)
.cache/
# dev-only: evaluation harness, tests, dev data — not part of the HF submission
# (the sandbox runs script.py only; runtime metrics live in solver/metrics.py)
eval/
tests/
data/

120
AGENTS.md Normal file
View File

@@ -0,0 +1,120 @@
# IOL-AI 2026 Solver — Implementation Seed
Seed context and build plan for implementing a solver for International Linguistics Olympiad (IOL) style problems. Read this fully before writing code. Design principle: the structure of language does the reasoning; the LLM proposes hypotheses that a deterministic engine executes and a symbolic verifier selects. Do not outsource rule application to the LLM.
## 1. The task
Solve self-contained linguistic-reasoning puzzles over unseen, mostly low-resource languages. Each puzzle gives a small amount of bilingual/parallel data plus hints, and asks the solver to translate, fill blanks, match forms to meanings, or convert numerals. No prior knowledge of the language is needed or expected; everything must be induced from the given data.
### Data (Linguini-style CSV)
Columns: `id`, `context`, `query`, `work_lang`, `task_lang`, `task_type`, `eval_type`.
- One row = one complete problem. `context` holds the parallel data + hints. `query` holds numbered items sharing that context.
- Read test set from `/tmp/data/test.csv`. Write `submission.csv` with a `pred` column: one **JSON list** of answers per row, one entry per numbered item, in order.
### Task types (route on `task_type`)
- `translation` (work->task and task->work): analysis one way, generation the other.
- `fill_blanks`: complete a paradigm cell.
- `match_letters`: match forms to meanings (assignment problem).
- `text_to_num` / `num_to_text`: recover and apply a numeral system.
- Others may appear (transliteration, glossing, error-correction, phonological rule application) — handle gracefully, fall back rather than crash.
### Scoring (design to this)
Final score = geometric mean `sqrt(EMw * chrFw)`, both **points-weighted** by official IOL point values.
- Geometric mean => neither factor can be near zero. **Never emit an empty or wildly-off answer**; always produce a plausible string (chrF floor).
- Need genuine exact hits (symbolic core) AND fuzzy overlap (fallback).
- Generation *into* the unknown language is the hardest direction; invest there.
### Submission and runtime constraints (hard)
- Submit a public Hugging Face repo containing `script.py` + model weights. Load models from `.` (repo root). **No internet in the eval sandbox** — vendor every dependency and weight.
- **30 minutes per run for the entire test set.** Throughput is the real budget. Symbolic-first; batch all LLM calls; spend sampling on hard/high-point items only.
- Greedy decoding for reproducibility. Up to 3 submissions/day; 2 count for the private leaderboard.
- Compute at inference: single **T4, 16 GB** (Turing: fp16 yes, bf16 no). Use 4-bit AWQ/GPTQ + LoRA via vLLM.
- Full repo is allowed: ship YAML configs, prompt libraries, DSL grammar files, multiple LoRA adapters, finetuned weights.
- Finetuning (SFT or RL) is allowed; train **offline** on other hardware and upload weights. Only inference is T4-bound.
## 2. Core insight (why this architecture)
1. LLMs (small ones especially) can *propose* linguistic rules but reliably fail to *apply* them consistently. => LLM proposes a grammar; a **deterministic interpreter applies it**.
2. The given attested pairs are a **free, exact verifier**. Leave-one-out fit on them (points-weighted EM + chrF = the competition metric) selects among candidate grammars. No LLM judge needed for correctness. This makes the domain verifiable, which is what makes test-time scaling and RL-with-verifiable-rewards work.
3. Explicit morpheme segmentation before anything else is a large, cheap win.
4. Analogical exemplars (from a multilingual model) help the LLM *propose* better grammars, never to deduce.
## 3. Architecture
```
Router/typology scan -> Preprocess (normalize, segment, align, paradigm table)
-> Symbolic-first solvers: numeral CSP | matching (Hungarian) | concatenative morphology
-> Program-synthesis subagent: LLM proposes grammar in DSL -> interpreter runs it
-> CEGIS refine <= R rounds (feed failing pairs back)
-> Verifier: leave-one-out EM+chrF on attested pairs + MDL penalty; self-consistency on ties
-> Select grammar (never empty; chrF-floor fallback = closest attested form / analogy)
-> Apply selected grammar deterministically to query items -> answers
-> (phase 5) Verbalizer LoRA: execution trace -> human explanation
```
Principles: segment first; symbolic-solved items cost ~0 LLM tokens; adaptive compute; guard against repetition loops (token cap + repetition penalty + loop detection).
### The DSL (small, executable, verifiable)
Primitives for a puzzle grammar. Programs are tiny so execution/verification is effectively free.
- `LEXICON`: morpheme -> gloss/feature
- `AFFIX(position, form, trigger)`
- `REWRITE(pattern -> repl / context)` (morphophonology: harmony, sandhi, elision)
- `ORDER(slot permutation)` (source constituent order -> target)
- `AGREE(feature copy)`
- `REDUP(template)`
- `NUMERAL(base, digitmap, combine_op)`
Interpreter must run **both directions** (analyze and generate). Generation uses the grammar + proportional analogy for unseen stems; round-trip verify generated forms.
### Verifier (single object, reused everywhere)
`score(grammar, held_out_pairs) -> points_weighted_EM, chrF`. Used as: test-time selector, CEGIS signal, and (later) RL reward. Add MDL penalty `L(grammar)+L(data|grammar)` to prefer the simplest adequate grammar.
## 4. Suggested repo layout
```
script.py # entrypoint: read /tmp/data/test.csv -> write submission.csv
solver/
router.py # task typing + typology scan + budget tier
preprocess.py # unicode NFC (keep tone/diacritics), tokenize, align, segment
align.py # minimal-pair set-difference aligner (pure python)
segment.py # MDL-guided segmentation conditioned on alignment
analogy.py # proportional analogy (a:b::c:d), analogical grids
numerals.py # base detection + CSP/algebraic solve
matching.py # scipy Hungarian over alignment-consistency scores
dsl/
grammar.py # DSL datatypes
interpreter.py # analyze() and generate(), both directions
synth.py # program-synthesis subagent + CEGIS loop
verifier.py # leave-one-out EM+chrF + MDL; selection + self-consistency
llm.py # vLLM wrapper, 4-bit base + LoRA adapter swap, batching, guards
fallback.py # chrF-floor: never return empty
budget.py # adaptive compute allocation under 30-min cap
prompts/ # YAML prompt library (proposer, analogical generator, critic)
weights/ # 4-bit base + LoRA adapters (proposer/deducer/verbalizer/critic)
eval/
scorer.py # reimplement official geomean scorer
dev_harness.py # run + score on public dev puzzles
data/dev/ # public IOL/Linguini/PuzzLing/modeLing/LINGOLY (validation ONLY)
```
## 5. Build order (do in this sequence; each has a gate)
0. **Harness first.** `eval/scorer.py` (points-weighted EM + chrF geomean) and `verifier.py` leave-one-out. Assemble `data/dev/` from public puzzles (respect CC-BY-SA / IOL copyright; validation only). Baselines: analogy-only floor, small-LLM few-shot. Gate: scorer matches hand-computed values.
1. **Cheap symbolic points.** `numerals.py`, `matching.py`, `align.py`, `segment.py`, `analogy.py`. Gate: numerals/matching beat LLM baseline; segmentation shows a positive downstream delta (replicate the known morpheme-boundary gain). If segmentation does not help, the aligner is mis-conditioned.
2. **Synthesis core.** DSL + interpreter + `synth.py` CEGIS with proposer LLM. Ablate: direct-LLM vs synthesis+verifier vs +segmentation vs +analogical exemplars; vary CEGIS rounds. Gate: synthesis+verifier clearly beats direct-LLM on translation/fill_blanks at fixed budget.
3. **Verifier-gated test-time scaling.** Best-of-N grammar sampling ranked by verifier; self-consistency; adaptive budget. Gate: monotone accuracy gain that still fits 30 min; pick operating N from the compute-accuracy curve.
4. **Finetuning (offline, highest ceiling).** Procedurally generate synthetic puzzles with gold grammar/segmentation/derivation. SFT the proposer/deducer for DSL emission + reliable rule application; then GRPO/RLVR with the verifier as reward. Validate on **real** public puzzles (guard the synthetic-real gap). Gate: finetuned small model >= best prompt-only config on real held-out puzzles at equal/lower cost.
5. **Explanation track (later).** Verbalizer LoRA: execution trace -> prose; narrow LLM critic checks faithfulness to the trace. The automated solver's trace is the explanation substrate.
## 6. Stack and risks
Models (T4, 4-bit + LoRA, vLLM): base = Qwen2.5-7B-Instruct; try a Coder variant for DSL emission; optional Aya-Expanse-8B as multilingual analogical generator. Role specialization via hot-swapped LoRA adapters over one base (stay in 16 GB).
Libraries (vendor all, offline): `unicodedata`/`regex`; pure-python aligner + MDL segmenter (avoid native `pynini`/`hfst`; implement rewrite-rule + analogy engine in python); `scipy` (Hungarian); hand-rolled CSP for numerals (spaces are tiny) or `z3-solver` if redistributable; `sacrebleu` (chrF); `vllm`; `peft`.
Risks: throughput vs 30 min (symbolic-first, batch, adaptive budget, cap N); small-model deduction leakage (always execute rules deterministically); DSL coverage of hard phenomena — tone, reduplication, circumfixes, harmony, ergativity, suppletion (grow primitives from failure analysis; keep an LLM fallback so nothing is empty); synthetic-real gap (validate on real puzzles only); repetition collapse (token caps + penalties); quantization hurting multilingual quality (compare AWQ/GPTQ/fp16 on dev before committing).
## 7. Non-negotiables
- Read `/tmp/data/test.csv`; write `submission.csv` with `pred` as a JSON list per row. Load all models/deps from repo root; assume no network.
- Never return empty for any item. Stay within 30 minutes for the whole set. Greedy decoding.
- LLM proposes; symbolic engine applies and verifies. Segment before reasoning. Keep tone/diacritics.

220
DESIGN.md Normal file
View File

@@ -0,0 +1,220 @@
# IOL-AI 2026 Solver — System Design
**Paradigm: single-shot LLM reasoning with deterministic tools.** One model
call per puzzle. Before the call, exact algorithms analyze the puzzle and
inject what they found into the prompt; after the call, verified symbolic
answers override the model where the verifier has real evidence, and format
guards clean up the rest. The LLM does what only it can (compositional
semantics, novel-rule reasoning); the tools do what they do perfectly
(segmentation, alignment, arithmetic, verification against attested data);
neither is asked to do the other's job.
This document describes the final v1 architecture: what runs, in what order,
why each component exists, and the evidence behind the design choices.
---
## 1. Constraints the design answers to
| Constraint | Design consequence |
|---|---|
| Single T4 (16 GB), 30-minute wall clock, offline | 7–14B AWQ model max; one batched generation pass; symbolic work is ~free (seconds); hard budget cutoff with graceful degradation |
| Score = √(EMw · chrFw), points-weighted | Never emit an empty answer (geometric mean → one zero factor craters the row); chrF partial credit makes a plausible-but-wrong answer far better than nothing |
| EM = `strip().lower()` equality only | Punctuation and spacing are significant; answers must match the gold's surface conventions |
| Test set = fresh IOL 2026 problems (Linguini format) | Zero contamination help — the model must genuinely deduce; scaffolding is most valuable exactly here |
| Optional human-judged explanation track | Emit an `explanation` column; jury wants short structured bullets, not raw traces |
Key empirical facts driving the architecture (from the literature survey and
our own 160-puzzle Linguini baseline):
- **All models score < 25% EM on Linguini; open T4-sized models are in the
low single digits.** The model is not a magic bullet at this scale.
- **Explicit morpheme segmentation in the prompt is the best-evidenced
intervention in the field** (documented gains up to +60 points/puzzle).
- **GPT-5 Pro scored 8/98 on the live IOL 2025 without scaffolding** — fresh
problems are brutal for pure single-shot reasoning at any scale.
- Our symbolic stack alone: EM 2.4% / chrF 20 on all 160 Linguini puzzles in
~9 seconds — comparable to what a naive T4 baseline can do, at ~0 cost.
## 2. Runtime flow
```
/tmp/data/test.csv
│
▼
┌────────────────────────────────────────────────────────────┐
│ 1. SYMBOLIC PASS (~seconds, no GPU) │
│ parse → route by task type → │
│ numeral CSP · table completion · template translation │
│ · Hungarian matching · analogy · fallback ladder │
│ every item gets: answer + verifier confidence + method │
└────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────┐
│ 2. SCAFFOLDED LLM PASS (the 30-min budget) │
│ for each puzzle with an unverified item, ONE call: │
│ context + query │
│ + morpheme segmentation hypotheses │
│ + word-alignment hypotheses │
│ + verified numeral values (when induced) │
│ + symbolic candidate answers (conf ≥ 0.4 only) │
│ → model reasons compositionally, emits │
│ FINAL ANSWERS: … then EXPLANATION: bullets │
│ batched (4/batch) · greedy · weakest-puzzles-first · │
│ hard budget cutoff │
└────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────┐
│ 3. MERGE + GUARDS │
│ verified symbolic (conf ≥ 0.5) > LLM answer > │
│ never-empty fallback │
│ per-task-type cleanup: digits for text_to_num, letter │
│ extraction for match_letters, preamble/quote stripping │
└────────────────────────────────────────────────────────────┘
│
▼
submission.csv (id, pred = JSON list, explanation = bullets)
```
The merge rule is the heart of the design: **an answer is kept symbolic only
when the verifier showed the producing rule actually generalizes** (leave-
one-out fit on the attested pairs, or a numeral system that round-trips every
attested example). The LLM cannot verify itself; the tools can — so verified
tool output outranks the model, and unverified tool output is demoted to a
*labeled hint inside the prompt* instead of an override.
## 3. Components
### Entry & orchestration
| File | Purpose |
|---|---|
| `script.py` | Competition entrypoint. Sets offline env vars, reads the test CSV, calls the pipeline, writes `submission.csv` (`id`, `pred`, `explanation`). ~60 lines; all logic lives below. |
| `solver/pipeline.py` | `run_pipeline()` — the three-stage flow above, **shared verbatim by `script.py` and the dev harness**, so the locally tested path is exactly the submitted path. Owns the merge policy (`CONF_KEEP = 0.5`), the weakest-first LLM ordering, the budget cutoff, per-task format guards, and explanation assembly. |
| `solver/budget.py` | Wall-clock accounting: elapsed/remaining with a safety margin; the pipeline consults it before each LLM batch. |
### The LLM layer
| File | Purpose |
|---|---|
| `solver/llm.py` | Client abstraction. `HFTransformersClient` (production: fp16, `device_map=auto`, greedy, LEFT-padded batching, version-drift-safe chat templating — the sandbox runs an older transformers than Colab); `NullClient` (no weights → symbolic-only pipeline still runs, used locally and as the crash floor); `CallableClient` (tests inject scripted models); `VLLMClient` (optional experiments only — never required). `load_client()` finds weights at the repo root (`config.json`), per the submission convention `MODEL_ID = "."`. |
| `solver/direct.py` | The single-shot solver. `SYSTEM` prompt (v1): derive rules *only from the given data*, don't assume English-like grammar, work compositionally (morphemes → order → sound changes), per-task-type answer formats, match the examples' punctuation; then `FINAL ANSWERS:` (one bare answer per line) then `EXPLANATION:` (2–4 bullets). `parse_output()` is a superset of the official baseline parser — a model that ignores the explanation instruction still parses. Explanations therefore cost **zero extra forward passes**. |
| `solver/scaffold.py` | Builds the tool-analysis prompt block: segmentation hypotheses (multi-morph words only), alignment hypotheses (top-2, score-filtered), verified numeral values, and symbolic candidates labeled with source + fit. Size-capped (≤ 40 seg lines, ≤ 30 align lines, 60-word vocab) — measured max prompt ≈ 1.5k tokens over the 160 real puzzles. Low-confidence candidates are *excluded* so garbage can't anchor the model. |
### The symbolic toolbox (all pure Python, no dependencies, runs anywhere)
| File | Purpose |
|---|---|
| `solver/preprocess.py` | Parsing reality. Normalization (NFC, keeps diacritics/tone/punctuation), context parsing (pipe tables 2–5 cols with header detection, numbered/lettered lists, `=`/dash pairs, prose hints), query parsing (numbered items that continue context numbering, bare-line items, `(k)` blank markers in any column, wrapped-line continuation, instruction detection that survives task-language colons), context-derived items for instruction-only queries. Item counts match gold on **158/160** real Linguini puzzles. |
| `solver/align.py` | Word alignment from tiny corpora: minimal-pair set-difference links (exact, high-precision), Dice co-occurrence, morphological back-off (single-word glosses project into inflected forms containing them), and competition/explaining-away (a target already claimed by a stronger suitor is demoted — breaks the pervasive ties of 10-sentence corpora). |
| `solver/segment.py` | MDL morpheme segmentation conditioned on alignment (shared-gloss groups get boundary bonuses). Feeds the scaffold — the field's best-evidenced intervention. |
| `solver/template.py` | Template translation: find the attested sentence closest to the query (bag distance, length-mismatch tie-break), swap the differing tokens through alignments; context-aware substitution (candidate must exist in the template's target) and affix-matched replacement (kupu:nakupu :: moko:namoko). Both directions. Abstains rather than fabricates. |
| `solver/tables.py` | Paradigm-table completion: match query-row cells to context-table columns (positional when same width, char-overlap otherwise; damaged/merged-marker rows handled), then learn the source→answer column mapping and pick the most learnable source column by LOO fit. Chain: template → char-level analogy voting → echo. |
| `solver/numerals.py` | Numeral CSP: induce morpheme values from attested (phrase, value) pairs under a multiplicative-additive convention, with constraint propagation and node caps (abstains fast instead of hanging on morphophonologically complex systems). Generation ranks candidate phrasings by attested-style consistency. Induced systems are **round-trip verified against every attested example** — the strongest confidence in the system (0.9). |
| `solver/matching.py` | match_letters: pure-Python Hungarian over lexical-alignment + structural-similarity scores. Real-data EM is weak (~5%), so its confidence (0.45) deliberately sits *below* the keep-threshold: the assignment reaches the model as a hint, never an override. |
| `solver/analogy.py` | Proportional analogy a:b::c:x (prefix/suffix/infix edits), with multi-exemplar voting. Used inside tables and fallback. |
| `solver/fallback.py` | The chrF floor: analogy transfer from the closest attested pair → echo of the closest attested target → word-by-word alignment gloss → the query itself. Guarantees no empty answer ever. |
| `solver/router.py` | Task-type dispatch producing (answers, confidences, methods) per item. Confidence semantics: LOO/eval fit of the producing solver; 0.9 for round-trip-verified numerals; 0.0 for fallback — "0.0" is precisely the signal "worth LLM budget". |
| `solver/verifier.py` | The single verification object: EM+chrF fit on attested pairs, with two honest regimes — fixed programs are evaluated directly (they must reproduce the data), fit-from-data predictors are scored **leave-one-out** (else memorization always beats generalization). Fold-capped at 12 for the time budget. |
### Research-track components (not in the runtime path)
| File | Purpose |
|---|---|
| `solver/dsl/` + `solver/synth.py` + `prompts/` | The CEGIS grammar-synthesis loop: LLM proposes a grammar in a small DSL (lexicon/affixes/rewrites/order), a deterministic interpreter executes it both directions, the verifier scores it, failures feed back. Fully built and tested, but **not called by the pipeline**: the literature says the proposer is the bottleneck and T4-sized models are weak proposers of formal artifacts. Kept as the ablation/ceiling-raising track (no published DSL/CEGIS solver exists for this domain — it's a genuine contribution if it pans out). |
### Metrics (runtime) & evaluation (dev-only, not in the submission repo)
| File | Purpose |
|---|---|
| `solver/metrics.py` | Official semantics, dependency-free, needed AT RUNTIME by the verifier and fallback: EM = `strip().lower()` with per-item gold alternatives; chrF replicates sacrebleu exactly (0.0 delta over 500 fuzz cases); geomean aggregate. |
| `eval/`, `tests/`, `data/` | **Local-only (gitignored)** — the sandbox runs `script.py` alone, so the dev harness (runs `run_pipeline` over gold-labeled CSVs), the 5 test suites (50+ tests), and the Linguini dev-set generator stay out of the public HF repo. They live on disk in the working copy. |
## 4. The prompt (v1 rationale)
Decisions, given the "don't ablate CoT now, pick a best v1" directive:
- **Brief reasoning retained, but re-aimed.** The literature shows generic
step-by-step reasoning can *hurt* here (models anchor on English-like
rules). The v1 prompt keeps concise reasoning — small models need scaffolding
to organize composition — but adds the explicit counter-instruction:
*"Derive the rules ONLY from that data — do not assume the language works
like English or any language you know,"* and directs the reasoning into a
compositional procedure (morphemes → meanings → order → sound changes).
- **Tool output is framed as hypotheses** ("may contain errors — the attested
data always wins"), and candidates carry their source and fit score, so the
model can calibrate trust per hint.
- **Output contract is baseline-compatible plus explanation.** `FINAL
ANSWERS:` parsing is a superset of the official notebook's parser;
`EXPLANATION:` bullets ride in the same generation. Parser handles the
reversed order too.
- **Format guards, not format hopes.** text_to_num answers are reduced to
digits; match_letters answers to a single valid option letter; preambles
("The answer is…") and quotes stripped. EM's strict semantics make these
worth real points.
## 5. Explanation track (opted in)
Every row ships an `explanation`:
- **LLM-solved puzzles**: the model's own `EXPLANATION:` bullets (rules found
+ key evidence) — the short structured form the jury asks for.
- **Symbolically-solved puzzles**: a generated trace sentence per method
(e.g. numeral: "induced each morpheme's value … verified the system
reproduces every given example"). These are *true* descriptions of what the
solver did — arguably more faithful than any post-hoc model prose.
Coverage measured at 160/160 on the dev set (requirement: ≥ 50%).
## 6. Time budget
Measured: symbolic pass + scaffold building ≈ 11 s for 160 puzzles; prompts
median ~800 tokens, max ~1.5k. Generation dominates. The shipped model is
**Qwen2.5-7B-Instruct-AWQ** (~5.6 GB, Apache-2.0): same organizer-validated
family as the workshop baseline, but 2–3× faster than the 14B on the T4 and
leaving ~9 GB of KV headroom for batch 6 — under the 30-minute cap, more
scaffolded puzzles per minute beats a smarter-but-slower model whose runs get
cut off. (The 14B also physically cannot be prepared on the current dev
machine: ~20 GB local footprint with git-lfs against ~13 GB free.) A *full
160-puzzle* set at 1536 new tokens would still exceed 30 minutes — the
weakest-first ordering + cutoff degrades the tail to verified symbolic/
fallback answers rather than failing. The actual hidden test set is the IOL
2026 individual contest (typically ~5 problems reformatted → far fewer rows),
which fits comfortably. Colab-tunable knobs, in order: `MAX_NEW_TOKENS`
(1536 → 1024), `LLM_BATCH` (6 → 8), model swap via `scripts/prepare_weights.py`.
## 7. What was deliberately left out
- **Agentic tool-calling loops** — multiple forward passes per problem;
7–14B models are unreliable tool-callers; budget-hostile. The tools run
*before* the call instead.
- **Multi-agent orchestration** — no evidence it helps at T4 scale; the one
positive result used a frontier model on an adjacent benchmark.
- **CEGIS in the runtime path** — see research-track note above.
- **Self-consistency / best-of-N** — greedy decoding is mandated-adjacent
(reproducibility) and no linguistics-specific evidence supports the spend.
- **CoT A/B testing** — explicitly deferred per direction; the v1 prompt
takes the middle path described in §4.
## 8. Submission checklist
- [x] `script.py` at repo root; reads `/tmp/data/test.csv`, writes
`submission.csv` with `id`, `pred` (JSON list), `explanation`
- [x] Offline: `HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1` set in-script;
weights load from `.`
- [x] Never-empty answers; per-row crash containment; budget cutoff
- [x] **OOM containment** (learned from the first live submission, which
crashed): prefill logits on the sandbox's transformers are
batch-total-tokens × 152k vocab × fp32 — so generation batches are packed
by a 3,500-token budget, OOM retries drop to single prompts then abstain,
`logits_to_keep=1` is passed when supported, and `submission.csv` is
checkpoint-written after the symbolic pass and every LLM batch so ANY
later crash still leaves a complete submission
- [x] All 5 test suites green; dev harness runs the identical pipeline
- [x] **Weights in repo**: Qwen2.5-7B-Instruct-AWQ at the repo root
(git-lfs via `.gitattributes`; `LICENSE` retained; Apache-2.0 satisfies
the competition's open-licensing/redistribution rule; README carries the
HF model-card metadata)
- [ ] One Colab T4 validation run via the workshop notebook (timing + real
model output through `parse_output`) before the first real submission

202
LICENSE Normal file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2024 Alibaba Cloud
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

41
README.md Normal file
View File

@@ -0,0 +1,41 @@
---
license: apache-2.0
base_model: Qwen/Qwen2.5-14B-Instruct-AWQ
library_name: transformers
pipeline_tag: text-generation
language:
- en
tags:
- iol-ai-2026
- linguistic-reasoning
---
# IOL-AI 2026 Solver
Solves IOL-style linguistics puzzles (Linguini CSV) with a single-shot LLM.
Ships **Qwen/Qwen2.5-14B-Instruct-AWQ** (4-bit AWQ, Apache-2.0) at the repo
root, loaded from `MODEL_ID = "."`; runs on a T4 within the 30-minute budget.
## Architecture
- **`script.py`** — entrypoint: reads `/tmp/data/test.csv`, writes
`submission.csv` (`id`, `pred` JSON list, `explanation`). Config flags at the
top; submission written before the model loads and after every step.
- **`solver/pipeline.py`** — orchestration. The model answers every puzzle from
a minimal prompt (no scaffold, no chain-of-thought); output is parsed into one
answer per item and aligned by position. A light greedy-anchored
self-consistency vote refines answers while the clock allows. A deterministic
symbolic layer (`solver/`) is a last-resort fallback only.
- **`solver/llm.py`** — batched transformers/AWQ generation, greedy,
`repetition_penalty=1.0`, per-token deadline.
- **`solver/direct.py`** — prompt and answer parsing.
## Run
```bash
python3 script.py [test.csv] [submission.csv] # defaults: /tmp/data/test.csv, submission.csv
```
Weights are the unmodified
[Qwen/Qwen2.5-14B-Instruct-AWQ](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct-AWQ)
release (Apache-2.0; `LICENSE` retained).

35
config.json Normal file
View File

@@ -0,0 +1,35 @@
{
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151645,
"hidden_act": "silu",
"hidden_size": 5120,
"initializer_range": 0.02,
"intermediate_size": 13824,
"max_position_embeddings": 32768,
"max_window_layers": 70,
"model_type": "qwen2",
"num_attention_heads": 40,
"num_hidden_layers": 48,
"num_key_value_heads": 8,
"quantization_config": {
"bits": 4,
"group_size": 128,
"modules_to_not_convert": [],
"quant_method": "awq",
"version": "gemm",
"zero_point": true
},
"rms_norm_eps": 1e-06,
"rope_theta": 1000000.0,
"sliding_window": 131072,
"tie_word_embeddings": false,
"torch_dtype": "float16",
"transformers_version": "4.41.1",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 152064
}

218
fable.md Normal file
View File

@@ -0,0 +1,218 @@
# fable.md — research-direction notes from implementation
Notes written while building the IOL-AI solver seed (see AGENTS.md for the plan,
README.md for repo state). Each item is something the implementation surfaced
that the research plan should absorb.
**Update (real-data phase):** items 4, 8, 9 are now RESOLVED — the official
eval notebook (rita-berrada/iolai-2026-workshop) fixed the semantics, and
facebook/linguini gave 160 real dev puzzles. New findings start at §11.
## 11. Real-data baseline resets expectations (and the plan's emphasis)
Symbolic-only on all 160 Linguini puzzles: **EM 2.4%, chrF 20, in ~9 s.**
(The synthetic dev set scored 1.0 — constructed puzzles flatter substitution
methods.) Real translation items are compositional, real paradigms involve
metathesis/harmony/infixation, real numeral systems have morphophonology that
breaks token-level CSP. Consequences:
- The LLM direct path (notebook format) is the main score carrier for now;
the symbolic stack's near-term value is (a) exact hits where structure is
clean, (b) the chrF floor, (c) **verified confidences that route the LLM
budget** (157/160 puzzles flagged low-confidence — correctly).
- The DSL/CEGIS path is the ceiling-raiser, not the floor: its job is to
convert LLM linguistic insight into verified exact hits. Ablate it against
direct-LLM on the T4 before investing in finetuning.
## 12. Official semantics differ from my assumptions in scoring-relevant ways
- EM is `strip().lower()` ONLY — final punctuation and internal spacing are
significant. Answers must carry the gold's punctuation conventions; a
format-inducer pass (§3) is now demonstrably worth EM points.
- Gold items can be a LIST OF ALTERNATIVES (14/160 rows); scorer takes max.
- chrF is sacrebleu with effective-order smoothing; my reimplementation now
matches to 0.0 delta over 500 fuzz cases (tests/test_scorer.py).
## 13. Linguini formats: what the parser must survive (now does, 158/160)
Pipe tables 2-5 columns with header rows; numbered example sentences whose
numbering the query CONTINUES (item "17." refers to nothing in the query);
bare-line items with no numbering; (k)-blank markers in any column, both
directions in one puzzle, including damaged rows ("(5) to tie" merged cell);
items living in the CONTEXT while the query is instruction-only
("Determine the correct correspondences", "Fill in the blanks (1–14)");
task-language strings that end in ':' (vowel length) — never use trailing
colon alone to detect instructions. Two rows remain unparseable-by-count: one
has misaligned gold (7 answers, 6 items), one enumerates payloads inline in
prose. Position-aligned scoring makes over/under-parsing cost only the
misaligned tail — worth a guard that pads rather than truncates.
## 14. match_letters needs morpheme-level CSP, not surface matching
16 puzzles / 223 items (25% of all items) are unordered form↔meaning
matching with usually ZERO attested pairs. Surface similarity carries no
signal (5.4% EM ≈ barely above random). The real structure: recurring
morphemes across forms must map consistently to recurring words across
meanings (Zuni doko:ko ↔ 'chicken', mo:chikwa ↔ 'peach'). That is a small
constraint-satisfaction / bilingual-lexicon-induction problem over sets —
highly verifiable (a candidate assignment implies a consistent lexicon or it
doesn't) and a perfect fit for the propose-verify architecture: LLM proposes
morpheme↔word hypotheses, a solver checks global consistency, Hungarian
finishes. My soft quadratic-assignment attempt (power iteration over
similarity graphs) was neutral; the discrete version is the right next try.
## 15. Throughput reality on the T4 (from the notebook)
The baseline runs ~1 min/problem at 1536 new tokens on the 14B AWQ — a full
160-puzzle set would need ~2.7 h sequential. The 30-minute budget therefore
REQUIRES the two-pass design: symbolic first (~9 s), then batched direct-LLM
on the low-confidence subset, weakest-first, with a hard budget cutoff
(script.py implements this). Concrete knobs to tune on Colab: batch size
(KV-cache limited on 16 GB with a 14B model — try 2-4), MAX_NEW_TOKENS
(1536 baseline; shorter for translation-only puzzles), and whether a 7B
model with bigger batches beats the 14B with tiny batches at fixed wall
clock. That ablation needs the actual T4.
## 16a. v1 migration to single-shot-with-tools (done 2026-07-21)
The architecture inversion recommended by the landscape survey is
implemented: the tools now feed the answering model (scaffold in the prompt:
segmentation, alignment, verified numeral values, labeled symbolic
candidates), verified symbolic output overrides the model, unverified output
is demoted to hints, and CEGIS left the runtime path. Explanations ride in
the same generation (`EXPLANATION:` after `FINAL ANSWERS:`) — the
human-judged track costs zero extra passes. Open empirical questions for the
T4: does the scaffold help a 14B AWQ model the way it helped GPT-4-class
models (the segmentation evidence says gains concentrate above ~15% baseline
— a 14B may be below it); and the deferred CoT-vs-IO prompt A/B (§ "Probing
LLMs", arXiv:2502.00817).
## 16b. First live submission: the T4 constraint is prefill logits, not KV
The first real submission OOMed: on the sandbox's transformers, `generate`
computes LM-head logits over EVERY prompt position and casts to float32 —
batch 6 × ~2.9k tokens × 152k vocab × 4B = the exact 9.82 GiB failed
allocation. KV cache (what batch sizing usually optimizes) is negligible
under GQA; **total prompt tokens per batch** is the real T4 memory bound at
~0.9 MB/token. Fixes: token-budget batch packing (3,500), OOM → retry
singly → abstain, `logits_to_keep=1` where supported. The deeper lesson:
the crash killed the script before submission.csv existed — "never empty"
must hold at the FILE level, not the item level. submission.csv is now
checkpoint-written from ~15 s in and atomically replaced as results improve.
## 16. The sandbox runs OLDER transformers than Colab
The notebook comments that `apply_chat_template(..., return_dict=True)`
returns a dict on Colab but a bare tensor in the sandbox. llm.py sidesteps
the whole class of drift by using `tokenize=False` + a normal tokenizer
call, which is stable across versions. Keep any future transformers usage on
the oldest-common-API path; there is no way to pin the sandbox's version.
## 1. The symbolic template translator may be the real workhorse, not the DSL
The plan positions LLM→DSL synthesis as the core translation engine. But the
minimal-pair **substitution translator** (`solver/template.py`: find the
attested sentence closest to the query, swap the differing tokens through
alignment links) solved every synthetic translation item exactly, with zero
LLM tokens, once three refinements landed:
- **competition / explaining-away in alignment** (`align.py`): demote a target
token already strongly claimed by another source token. Tiny corpora make
co-occurrence ties pervasive; this one change flipped several wrong answers.
- **morphological back-off**: single-word glosses (`moko = dog`) project into
inflected forms containing them (`namoko`).
- **affix-matched substitution**: when replacing a form, prefer the candidate
sharing an affix with the form replaced (kupu:nakupu :: moko:namoko) — a
proportional analogy inside the template frame.
Implication: budget the DSL/CEGIS path for the residual — items where the
query is far (bag-distance > ~len/2) from every attested sentence, long
compositional sentences, and phenomena the template can't reach (reordering,
harmony). Measure on real dev data what fraction that residual actually is;
it directly sets the LLM compute budget and the finetuning priority.
## 2. Verifier honesty requires two scoring regimes (found a real trap)
Memorizing predictors (template translator, fallback) score EM=1.0 on attested
pairs by construction, so scoring every candidate with plain `evaluate()` lets
memorization beat a generalizing grammar on ties — with MDL penalty it beats
it *always*. The fix in `router._pick_direction_solver`: fixed programs
(grammars) are scored directly (they must reproduce the data), while
fit-from-data predictors are scored **leave-one-out**. Any future candidate
added to the selection pool must declare which regime it belongs to. This also
matters for RL later: the reward must be LOO-style for anything with data
access, or the policy learns to memorize.
## 3. Answer-format induction is a first-class subproblem the plan ignores
The synthetic num_to_text case exposed it: for 23, `hun ox` (20+3) and
`kan hun ox` (1×20+3) are both arithmetically valid; the gold followed the
attested *style* (explicit unit multiplier). The fix was a style model scoring
candidates by consistency with attested phrasing. This generalizes: whenever
multiple surface answers are semantically correct, the scorer only rewards the
one matching the dataset's convention (capitalization, articles, hyphenation
of morph boundaries, multiplier explicitness). Recommend: an explicit
"format inducer" pass that learns per-puzzle answer conventions from context
examples, applied as the last stage of every task type. Cheap, and it converts
chrF-close answers into EM hits — which the geometric mean doubly rewards.
## 4. Context parsing is still the top empirical risk
Everything downstream consumes `extract_pairs`. The current parser
(separator voting + per-line fallback) handles the formats I could construct,
but real Linguini CSVs were not available in this environment. **Before any
model work, pull the actual Linguini/PuzzLing dev CSVs and fuzz the parser
against every context in them**; count rows where zero pairs are extracted —
each such row is guaranteed near-zero score. A cheap LLM repair path (ask the
model to emit the pair list as JSON when the symbolic parser yields < 2 pairs)
would cap this risk at one batched call per malformed puzzle.
## 5. Direction detection deserves attested-data validation, not just wording
`detect_direction` keys off phrases like "into English". Real queries may
say "What does X mean?", "Give the form meaning ...", or nothing explicit.
A robust cross-check: try both directions and see which side of the attested
pairs the query string is *script-similar* to (character n-gram overlap with
task-language vs work-language material). If the query looks like the unknown
language, it's an analysis item. Cheap and language-agnostic.
## 6. CEGIS should get verifier-guided repair hints, not just failures
The refine prompt currently shows failing (input, expected, got) triples. The
symbolic layer knows more: which morpheme boundary the output diverged at,
which affix went unused, whether the failure is pure reordering. Feeding a
one-line diagnosis per failure ("output differs only in word order",
"expected form contains attested morph 'na-' that your grammar never
attaches") should cut CEGIS rounds — worth an ablation axis in phase 2/3.
## 7. Self-consistency has a free implementation via round-trip
The interpreter runs both directions, so `analyze(generate(x)) == x` is a
zero-cost consistency check usable as (a) a tie-breaker in grammar selection
(already anticipated in AGENTS.md), and (b) a confidence signal for budget
allocation — spend refinement rounds on items whose round-trip fails.
## 8. Points weighting has no data column — resolve early
The scorer supports per-item weights, but the Linguini schema
(id/context/query/...) carries no points column. If official IOL point values
weight the leaderboard metric, they must be embedded somewhere (query text?
separate mapping by id?). Resolve this with a probe submission early — it
changes budget allocation (high-point items deserve the CEGIS rounds).
## 9. Local scorer vs sacrebleu chrF: verify parity once, in the eval image
`eval/scorer.py` implements chrF2 (n≤6, β=2, whitespace stripped) by hand to
stay dependency-free. sacrebleu differs in epsilon smoothing on zero-match
orders. Before trusting local ablations, run both on a few hundred string
pairs in the submission container and confirm the delta is < 1e-3; otherwise
model selection could silently optimize the wrong metric.
## 10. Prompt library format drifted from the plan (deliberately)
AGENTS.md says YAML prompts; the repo uses plain `.md` templates with
`str.format` placeholders (`prompts/*.md`). Rationale: zero dependencies, no
YAML-escaping pain with multiline linguistic data. If config-driven prompt
variants are needed for best-of-N diversity (the greedy-decoding constraint
means diversity must come from prompts, not sampling temperature — see
`synth.synthesize_best_of_n`), add a small variants list per template rather
than reintroducing YAML.

14
generation_config.json Normal file
View File

@@ -0,0 +1,14 @@
{
"bos_token_id": 151643,
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151643,
"repetition_penalty": 1.05,
"temperature": 0.7,
"top_k": 20,
"top_p": 0.8,
"transformers_version": "4.41.1"
}

151387
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model-00001-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4e874dc3b1febb5b22fe74a8793066ae430d90cdbc51765d0a4eb44a82a1fbbd
size 3988804408

3
model-00002-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b3b25da74cc854cdc726956f8152f1dda8519c7bb7d4724ac12c1312463e61c8
size 3968309440

3
model-00003-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:25b97bf28033ed4560387293ce76bd3dc55a22882fea005a10c137b6478dae85
size 2023056736

1258
model.safetensors.index.json Normal file
View File

File diff suppressed because it is too large Load Diff

32
prompts/proposer.md Normal file
View File

@@ -0,0 +1,32 @@
You are a linguist solving a puzzle about {task_lang}, an unfamiliar language. You are given attested sentence pairs. Induce the grammar and output it as JSON — the grammar will be EXECUTED BY A MACHINE, so it must be complete and mechanical.
## Attested data ({task_lang} = {work_lang})
{pairs_block}
## Hints from the puzzle
{hints_block}
## Automatic analysis (may contain errors — trust the data over this)
Morpheme segmentation of {task_lang} words:
{segmentation_block}
Word alignments (task-language token -> likely meaning):
{alignment_block}
## Output format
Emit ONLY a JSON object:
{{
"lexicon": [{{"morph": "<task-lang morpheme>", "gloss": "<{work_lang} word(s)>", "pos": "<N|V|ADJ|...>"}}],
"affixes": [{{"position": "prefix|suffix|circumfix", "form": "<morph>", "form2": "<circumfix 2nd part>", "feature": "<the {work_lang} word or marker this realizes, e.g. 'the', 'plural'>", "trigger": "<pos it attaches to, or empty>"}}],
"rewrites": [{{"pattern": "<regex>", "repl": "<replacement>", "context": ""}}],
"order": ["<pos tags in {task_lang} constituent order, e.g. V, N>"],
"agree": [],
"redup": []
}}
Rules:
- Every {task_lang} morpheme in the data must be accounted for (lexicon or affix).
- Prefer affix `feature` values that are actual {work_lang} function words ("the", "will", "not") — they are emitted verbatim when translating into {work_lang}.
- rewrites capture sound changes at morpheme boundaries (e.g. "aa" -> "a").
- The grammar must reproduce EVERY attested pair exactly when executed.
- Simplest grammar that fits wins. No prose, JSON only.

10
prompts/refine.md Normal file
View File

@@ -0,0 +1,10 @@
Your previous grammar for {task_lang} was executed by the machine and FAILED on these attested pairs:
{failures_block}
("got" is what your grammar produced under mechanical execution; "expected" is the attested truth.)
Previous grammar:
{previous_grammar}
Diagnose the failures (wrong lexicon gloss? missing affix? missing rewrite rule at a morpheme boundary? wrong constituent order?) and emit a CORRECTED complete JSON grammar in the same format. The corrected grammar must reproduce every attested pair, including the ones above, when executed mechanically. JSON only.

15
requirements.txt Normal file
View File

@@ -0,0 +1,15 @@
# Inferred from the official eval notebook (iolai-2026-workshop, cell 2):
# pip install -q -U transformers accelerate datasets sacrebleu gptqmodel "numpy==2.2.6"
# The submission sandbox provides the runtime (older transformers preinstalled);
# these pins are for reproducing the Colab test environment.
transformers
accelerate
gptqmodel # loader for AWQ/GPTQ checkpoints (harmless otherwise)
numpy==2.2.6
# --- notebook-eval only (not needed by script.py at inference) ---
datasets
sacrebleu
# --- local dev only ---
pandas
huggingface_hub
pyarrow

130
script.py Normal file
View File

@@ -0,0 +1,130 @@
"""Entrypoint: read /tmp/data/test.csv, write submission.csv (id, pred, explanation)."""
from __future__ import annotations
import os
from pathlib import Path
# ===== CHANGE HERE — your model (must fit the T4's ~15 GB) =====
# "." for the real submission (this repo ships Qwen2.5-14B-Instruct-AWQ at the
# root); a Hub name (e.g. "Qwen/Qwen2.5-14B-Instruct-AWQ") while testing on
# Colab.
MODEL_ID = "."
# None => the pipeline picks per test size (deep mode 2048 for a small test set,
# coverage mode 1024 for a large one). Set an int to force it.
MAX_NEW_TOKENS = None
LLM_BATCH = 6 # puzzles gathered per checkpoint cycle (the client then
# sub-batches by token budget to fit the T4)
# Skip the LLM and emit the symbolic-only baseline. Diagnostic; leave False.
SYMBOLIC_ONLY = False
# LLM pass uses a minimal prompt: no scaffold injection, no chain-of-thought.
# Set IOL_LEAN=0 for the scaffolded path.
LEAN_MODE = os.environ.get("IOL_LEAN", "1") == "1"
# Answer match_letters via the free-form LLM pass. Set IOL_MATCH_ASSIGN=1 to use
# the assignment solver instead.
MATCH_ASSIGNMENT = os.environ.get("IOL_MATCH_ASSIGN", "0") == "1"
# Generation batch size. 1 = one prompt at a time, no padding. Larger batches
# are faster but pad to the longest prompt. Set IOL_GEN_BATCH to change.
GEN_BATCH_SIZE = int(os.environ.get("IOL_GEN_BATCH", "1"))
# Light greedy-anchored self-consistency: N sampled passes that can only
# displace the greedy answer on genuine agreement. Budget-gated. 0 disables.
VOTE_SAMPLES = int(os.environ.get("IOL_VOTE_SAMPLES", "2"))
VOTE_TEMP = float(os.environ.get("IOL_VOTE_TEMP", "0.5"))
# Optional segmentation hint in the prompt. Off by default. Set IOL_HINT=1.
HINT = os.environ.get("IOL_HINT", "0") == "1"
# The eval sandbox has no internet; only go offline when loading local
# weights so Colab testing with a Hub MODEL_ID still downloads normally.
if MODEL_ID == "." or Path(MODEL_ID).exists():
os.environ.setdefault("HF_HUB_OFFLINE", "1")
os.environ.setdefault("TRANSFORMERS_OFFLINE", "1")
# reduce CUDA fragmentation on the T4 (must be set before torch initializes)
os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
import csv
import json
import sys
sys.path.insert(0, str(Path(__file__).resolve().parent))
from solver.budget import Budget
from solver.llm import load_client
from solver.pipeline import run_pipeline
TEST_CSV = "/tmp/data/test.csv"
OUT_CSV = "submission.csv"
csv.field_size_limit(min(sys.maxsize, 2 ** 31 - 1))
def read_rows(path: str):
with open(path, newline="", encoding="utf-8") as f:
return [{k: (v or "") for k, v in row.items()} for row in csv.DictReader(f)]
def write_submission(results, out_path: str) -> None:
"""Atomic write (tmp + rename) so a crash mid-write never leaves a
truncated submission.csv."""
tmp = out_path + ".tmp"
with open(tmp, "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["id", "pred", "explanation"])
writer.writeheader()
for r in results:
writer.writerow({
"id": r.row_id,
"pred": json.dumps([str(a).strip() or "?" for a in r.answers],
ensure_ascii=False),
"explanation": r.explanation,
})
os.replace(tmp, out_path)
def main(test_path: str = TEST_CSV, out_path: str = OUT_CSV) -> None:
budget = Budget()
rows = read_rows(test_path)
try:
if SYMBOLIC_ONLY:
from solver.llm import NullClient
client = NullClient()
print("SYMBOLIC_ONLY: skipping the LLM; submitting the symbolic "
"baseline", flush=True)
else:
client = load_client(MODEL_ID)
if hasattr(client, "batch_size"):
client.batch_size = GEN_BATCH_SIZE
# checkpoint after the symbolic pass and every LLM batch: a crash at
# any later point still leaves a complete submission on disk
results = run_pipeline(rows, client, budget,
llm_batch=LLM_BATCH, max_new_tokens=MAX_NEW_TOKENS,
checkpoint=lambda rs: write_submission(rs, out_path),
lean=LEAN_MODE,
use_match_assignment=MATCH_ASSIGNMENT,
vote_samples=VOTE_SAMPLES, vote_temp=VOTE_TEMP,
hint=HINT)
write_submission(results, out_path)
print(f"wrote {out_path}: {len(results)} rows in {budget.elapsed():.1f}s",
flush=True)
except BaseException as e:
# last resort: if the pipeline itself died before the first
# checkpoint, emit query echoes — an empty pred is a zero row
if not Path(out_path).exists():
from solver.pipeline import PuzzleResult
fallback = [PuzzleResult(str(r.get("id", i)),
[str(r.get("query", "?")).strip() or "?"],
"- fallback")
for i, r in enumerate(rows)]
write_submission(fallback, out_path)
print(f"pipeline failed ({type(e).__name__}); wrote fallback "
f"{out_path}", flush=True)
raise
if __name__ == "__main__":
args = sys.argv[1:]
main(args[0] if args else TEST_CSV, args[1] if len(args) > 1 else OUT_CSV)

37
scripts/prepare_weights.py Normal file
View File

@@ -0,0 +1,37 @@
"""Download the chosen model directly into the repo root, per the submission
convention (MODEL_ID = "." — weights ship inside the HF repo and load from
the working directory).
Downloads with local_dir (no separate HF cache copy — the weights exist once
on disk, plus git-lfs objects after committing). Weight files are picked up
by git-lfs via .gitattributes (*.safetensors etc.).
Usage:
python scripts/prepare_weights.py [model_id] [dest]
Default model: Qwen/Qwen2.5-7B-Instruct-AWQ (~5.6 GB, Apache-2.0 —
redistribution-safe per the competition's licensing rule; fits the T4 with
headroom for batched generation). Swap to Qwen/Qwen2.5-14B-Instruct-AWQ
(~10 GB) on a machine with ~25 GB free disk.
"""
import sys
from pathlib import Path
from huggingface_hub import snapshot_download
DEFAULT_MODEL = "Qwen/Qwen2.5-14B-Instruct-AWQ"
def main() -> None:
model_id = sys.argv[1] if len(sys.argv) > 1 else DEFAULT_MODEL
dest = Path(sys.argv[2] if len(sys.argv) > 2 else ".").resolve()
print(f"downloading {model_id} -> {dest}/ (direct, no cache copy)")
snapshot_download(repo_id=model_id, local_dir=str(dest))
print(f"done. Commit with git (LFS tracks the weight files), or verify "
f"with: python -c \"from transformers import AutoConfig; "
f"AutoConfig.from_pretrained('.')\"")
if __name__ == "__main__":
main()

0
solver/__init__.py Normal file
View File

135
solver/align.py Normal file
View File

@@ -0,0 +1,135 @@
"""Word alignment from tiny parallel corpora, pure python.
Two complementary signals:
1. Minimal-pair set difference: if two sentence pairs differ in exactly one
token on each side, those tokens correspond. Exact and high-precision;
these puzzles are constructed to contain such pairs.
2. Dice co-occurrence over the whole pair set: soft alignment for everything
the minimal pairs don't cover.
"""
from __future__ import annotations
from collections import Counter, defaultdict
from itertools import combinations
from typing import Dict, List, Tuple
from .preprocess import Pair, strip_punct, tokenize
def _toks(s: str) -> List[str]:
return [strip_punct(t).casefold() for t in tokenize(s) if strip_punct(t)]
def minimal_pair_links(pairs: List[Pair]) -> Counter:
"""Set-difference alignment: for every pair of examples whose source sides
differ by exactly one token multiset element and likewise on target, link
the differing tokens. Returns Counter[(src_tok, tgt_tok)] link strengths."""
links: Counter = Counter()
toks = [(Counter(_toks(p.src)), Counter(_toks(p.tgt))) for p in pairs]
for (s1, t1), (s2, t2) in combinations(toks, 2):
ds1, ds2 = s1 - s2, s2 - s1
dt1, dt2 = t1 - t2, t2 - t1
# exactly one differing token on each side, in both examples
if sum(ds1.values()) == 1 and sum(ds2.values()) == 1 \
and sum(dt1.values()) == 1 and sum(dt2.values()) == 1:
a1, a2 = next(iter(ds1)), next(iter(ds2))
b1, b2 = next(iter(dt1)), next(iter(dt2))
links[(a1, b1)] += 2 # strong: attested by contrast
links[(a2, b2)] += 2
# shared residue: tokens present in both examples also co-align weakly
return links
def dice_scores(pairs: List[Pair]) -> Dict[Tuple[str, str], float]:
"""Dice coefficient between source and target tokens across examples."""
src_count: Counter = Counter()
tgt_count: Counter = Counter()
co: Counter = Counter()
for p in pairs:
st, tt = set(_toks(p.src)), set(_toks(p.tgt))
for a in st:
src_count[a] += 1
for b in tt:
tgt_count[b] += 1
for a in st:
for b in tt:
co[(a, b)] += 1
return {
(a, b): 2 * c / (src_count[a] + tgt_count[b])
for (a, b), c in co.items()
}
def _morph_backoff(pairs: List[Pair], scores) -> None:
"""Substring evidence from single-word glosses: if (moko = dog) is
attested and token `namoko` co-occurs with `dog`, boost (namoko, dog) —
inflected forms inherit their stem's translation. Applied in place."""
word_pairs = [
(_toks(p.src)[0], _toks(p.tgt)[0])
for p in pairs
if len(_toks(p.src)) == 1 and len(_toks(p.tgt)) == 1
]
for (a, b) in list(scores.keys()):
for w, x in word_pairs:
if x == b and len(w) >= 3 and w in a and w != a:
scores[(a, b)] += 2.0 # inflected src contains attested stem
if w == a and len(x) >= 3 and x in b and x != b:
scores[(a, b)] += 2.0 # inflected tgt contains attested stem
def align(pairs: List[Pair]) -> Dict[str, List[Tuple[str, float]]]:
"""Combined alignment: src token -> ranked [(tgt token, score)].
Minimal-pair links dominate (score offset +1.0 per link unit); Dice fills
in the rest; single-word glosses back off into inflected forms containing
them. Scores are comparable only within one puzzle.
"""
links = minimal_pair_links(pairs)
dice = dice_scores(pairs)
scores: Dict[Tuple[str, str], float] = defaultdict(float)
for k, v in dice.items():
scores[k] += v
for k, v in links.items():
scores[k] += 1.0 * v
_morph_backoff(pairs, scores)
# competition ("explaining away"): a target token strongly claimed by
# some other source is a worse candidate — demote it proportionally to
# its best competing suitor. Breaks the pervasive co-occurrence ties of
# 10-sentence corpora in favor of unclaimed targets.
best_suitor: Dict[str, float] = defaultdict(float)
second_suitor: Dict[str, float] = defaultdict(float)
for (a, b), s in scores.items():
if s > best_suitor[b]:
second_suitor[b] = best_suitor[b]
best_suitor[b] = s
elif s > second_suitor[b]:
second_suitor[b] = s
out: Dict[str, List[Tuple[str, float]]] = defaultdict(list)
for (a, b), s in scores.items():
rival = second_suitor[b] if s >= best_suitor[b] else best_suitor[b]
out[a].append((b, s - 0.3 * rival))
for a in out:
out[a].sort(key=lambda x: -x[1])
return dict(out)
def one_to_one(pairs: List[Pair]) -> Dict[str, str]:
"""Greedy 1:1 token alignment: highest-scoring links assigned first, each
token used once. Sharper than independent argmax when several tokens tie
on co-occurrence (small corpora make ties common)."""
amap = align(pairs)
edges = [(s, a, b) for a, cands in amap.items() for b, s in cands]
edges.sort(key=lambda e: (-e[0], e[1], e[2]))
taken_a, taken_b, out = set(), set(), {}
for s, a, b in edges:
if a not in taken_a and b not in taken_b:
out[a] = b
taken_a.add(a)
taken_b.add(b)
return out
def best_translation(align_map: Dict[str, List[Tuple[str, float]]], tok: str) -> str:
cands = align_map.get(tok.casefold(), [])
return cands[0][0] if cands else ""

98
solver/analogy.py Normal file
View File

@@ -0,0 +1,98 @@
"""Proportional analogy: solve a : b :: c : x at the string level.
Used for (1) generating unseen inflected forms from paradigm neighbors and
(2) the chrF-floor fallback. Transformation model: a -> b is a prefix and/or
suffix replacement around a shared core, which covers concatenative
morphology. The same edit is applied to c. All consistent answers are
returned, ranked by preserved stem material.
"""
from __future__ import annotations
from collections import Counter
from typing import List, Optional, Tuple
# A rule is (a_pre, b_pre, a_suf, b_suf): replace prefix a_pre with b_pre and
# suffix a_suf with b_suf.
Rule = Tuple[str, str, str, str]
def _lcp(a: str, b: str) -> int:
i = 0
while i < min(len(a), len(b)) and a[i] == b[i]:
i += 1
return i
def _lcsuf(a: str, b: str) -> int:
i = 0
while i < min(len(a), len(b)) and a[-1 - i] == b[-1 - i]:
i += 1
return i
def edit_rules(a: str, b: str) -> List[Rule]:
"""Candidate decompositions of the transformation a -> b."""
rules: List[Rule] = []
p = _lcp(a, b)
s = _lcsuf(a, b)
if p > 0:
rules.append(("", "", a[p:], b[p:])) # keep shared prefix, swap suffix
if s > 0:
rules.append((a[: len(a) - s], b[: len(b) - s], "", "")) # swap prefix
if p > 0 and s > 0 and p + s <= min(len(a), len(b)):
# circumfix-ish: shared prefix AND suffix, swap the middle — model as
# suffix swap on the part after the shared prefix
rules.append(("", "", a[p : len(a) - s], b[p : len(b) - s]))
if not rules:
rules.append((a, b, "", "")) # suppletion: whole-string replacement
return rules
def apply_rule(rule: Rule, c: str) -> Optional[str]:
a_pre, b_pre, a_suf, b_suf = rule
out = c
if a_pre and not out.startswith(a_pre):
return None
out = b_pre + out[len(a_pre):]
if a_suf:
if not out.endswith(a_suf):
return None
out = out[: len(out) - len(a_suf)] + b_suf
else:
out = out + b_suf
return out
def apply_rule_mid(rule: Rule, c: str) -> Optional[str]:
"""Apply a middle-swap rule (encoded as suffix-swap) as an infix
substitution when the plain application fails: replace the last
occurrence of a_suf inside c."""
_, _, a_mid, b_mid = rule
if not a_mid or a_mid not in c:
return None
i = c.rfind(a_mid)
return c[:i] + b_mid + c[i + len(a_mid):]
def solve(a: str, b: str, c: str) -> List[str]:
"""Candidate solutions x to a : b :: c : x, best first."""
scored: Counter = Counter()
for rule in edit_rules(a, b):
x = apply_rule(rule, c)
if x is None:
x = apply_rule_mid(rule, c)
if x:
stem_kept = len(c) - len(rule[0]) - len(rule[2])
scored[x] = max(scored[x], stem_kept)
return [w for w, _ in scored.most_common()]
def solve_from_pairs(pairs: List[Tuple[str, str]], c: str) -> List[str]:
"""Given attested (form_a, form_b) pairs exhibiting one transformation,
vote for the best x completing c : x under that transformation."""
votes: Counter = Counter()
for a, b in pairs:
for rank, x in enumerate(solve(a, b, c)):
votes[x] += 1.0 / (1 + rank)
return [w for w, _ in votes.most_common()]

40
solver/budget.py Normal file
View File

@@ -0,0 +1,40 @@
"""Adaptive compute allocation under the 30-minute wall clock.
Symbolic solvers cost ~0; the budget really governs LLM calls (CEGIS rounds
and best-of-N). Strategy: reserve a safety margin, spread the rest over the
LLM-needing puzzles, and degrade rounds/N as the clock runs down — never let
the tail of the test set hit the fallback because the head overspent.
"""
from __future__ import annotations
import time
class Budget:
def __init__(self, total_seconds: float = 1620.0, safety_margin: float = 120.0):
self.start = time.monotonic()
self.total = total_seconds
self.safety = safety_margin
def elapsed(self) -> float:
return time.monotonic() - self.start
def remaining(self) -> float:
return self.total - self.safety - self.elapsed()
def exhausted(self) -> bool:
return self.remaining() <= 0
def cegis_rounds(self, puzzles_left: int, seconds_per_round: float = 12.0) -> int:
"""How many CEGIS refinement rounds this puzzle can afford, assuming
every remaining puzzle needs at least one proposal."""
if puzzles_left <= 0:
puzzles_left = 1
per_puzzle = self.remaining() / puzzles_left
rounds = int(per_puzzle / seconds_per_round) - 1 # -1 for the initial proposal
return max(0, min(rounds, 3))
def allow_llm(self, puzzles_left: int, seconds_per_call: float = 12.0) -> bool:
"""False once only fallback-speed work fits for the remaining set."""
return self.remaining() > puzzles_left * 0.2 + seconds_per_call

280
solver/direct.py Normal file
View File

@@ -0,0 +1,280 @@
"""LLM answering: prompts, generation, and output parsing (scaffolded and lean)."""
from __future__ import annotations
import re
from typing import List, Optional, Sequence, Tuple
from .llm import LLMClient
from .preprocess import Puzzle
SYSTEM = (
"You solve International Linguistics Olympiad problems. Every problem is "
"about a language you have never seen; ALL the evidence you need is in the "
"given data. Derive the rules ONLY from that data — do not assume the "
"language works like English or any language you know. "
"A 'Mechanical analysis' section may provide segmentation, alignment, and "
"candidate answers computed by exact algorithms: use them as hypotheses, "
"adopt candidates that fit the data, and correct them when the data "
"disagrees. "
"First, reason briefly about the linguistic patterns — keep it to a few "
"lines, not an essay: line up the given examples, segment the words into "
"morphemes, note what each recurring morpheme means and the order they "
"combine in, and any sound changes; check that the pattern holds across the "
"examples, then apply it to each query item. "
"Answer format by task type -- translation: the translated form only, in "
"the language asked for; fill_blanks: only the missing form for each "
"blank; match_letters: only the option letter (for example A, B, C); "
"text_to_num: the number in digits; num_to_text: the number written out "
"in the puzzle language; anything else: exactly what the instruction "
"asks, nothing more. Match the punctuation style of the given examples. "
"Then write a line that says exactly "
"FINAL ANSWERS: and, below it, one answer per line in the order the items "
"are asked -- the bare answer only, no numbering, no quotes, no markdown, "
"no extra text. If you are unsure, still give your single best guess for "
"every item -- never leave an item blank. After the answers, write a line "
"that says exactly EXPLANATION: followed by 2-4 short bullet points "
"stating the rules you found (word order, affixes and their functions, "
"sound changes, numeral bases) and the key evidence for them. Do not "
"repeat your reasoning.\n\n"
"Output shape (follow it exactly; do not bold or number the markers):\n"
"<a few lines of pattern reasoning>\n"
"FINAL ANSWERS:\n"
"first answer\n"
"second answer\n"
"EXPLANATION:\n"
"- rule and evidence\n"
"- rule and evidence"
)
# Minimal prompt: no scaffold, no chain-of-thought. Selected by LEAN_MODE.
LEAN_SYSTEM = (
"You solve International Linguistics Olympiad problems. Answer every "
"numbered item. Put each answer on its own line, in order, with no "
"numbering and no extra text. Give your best guess for every item; never "
"leave one blank."
)
MAX_NEW_TOKENS = 1536
# Markers tolerate leading markdown/quote decoration (#, *, >, -, spaces) and
# trailing decoration/colon, and capture any inline content after the marker
# ("FINAL ANSWERS: foo" -> "foo" is the first answer). Qwen habitually bolds
# these headers; the strict "^marker$" form silently dropped every such output.
# trailing class excludes newline ([^\S\n]) so the marker never swallows the
# line break and misreads the next line as inline content
_FINAL_RX = re.compile(r"(?im)^[ \t#>*_`-]*final[ \t]*answers?\b[^\S\n]*[:*_`]?[^\S\n]*(.*)$")
_EXPL_RX = re.compile(r"(?im)^[ \t#>*_`-]*explanation\b[^\S\n]*[:*_`]?[^\S\n]*(.*)$")
_NUMBERING_RX = re.compile(r"^\s*\(?\d{1,3}[.)]\s*")
# leading markdown decoration on an answer line: bullets, bold, backticks
_ANSWER_DECOR_RX = re.compile(r"^[\s>*_`•·–-]+")
def build_prompt(puzzle: Puzzle, scaffold: str = "") -> str:
base = f"{puzzle.context.strip()}\n\n{puzzle.query.strip()}"
if scaffold:
base += f"\n\n{scaffold}"
return base
def _clean_answer_line(line: str) -> str:
line = _NUMBERING_RX.sub("", line.strip())
line = _ANSWER_DECOR_RX.sub("", line)
# strip trailing markdown emphasis but keep sentence punctuation (EM-significant)
line = re.sub(r"[*_`]+$", "", line)
return line.strip().strip("'\"“”").strip()
def parse_output(text: str, n_items: Optional[int] = None
) -> Tuple[List[str], str, bool]:
"""Returns (answers, explanation, found_marker).
`found_marker` is True only when an explicit FINAL ANSWERS marker was
present. On a parse failure (no marker) we return ([], expl_if_any, False)
rather than treating the reasoning prose as answers -- the caller then
keeps its symbolic answers instead of overwriting them with garbage.
When `n_items` is given it is used only as a sanity cap on how many
answer lines to accept (guards against a runaway list)."""
markers = list(_FINAL_RX.finditer(text))
found = bool(markers)
if markers:
m = markers[-1]
# a marker may carry its first answer inline: "FINAL ANSWERS: foo"
inline = m.group(1).strip() if m.lastindex else ""
tail = (inline + "\n" if inline else "") + text[m.end():]
else:
tail = ""
expl = ""
em = _EXPL_RX.search(tail) if found else None
if em:
inline_e = em.group(1).strip() if em.lastindex else ""
rest_e = tail[em.end():].strip()
expl = (inline_e + ("\n" + rest_e if rest_e else "")) if inline_e else rest_e
tail = tail[: em.start()]
answers: List[str] = []
cap = (2 * n_items + 4) if n_items else None
for line in tail.splitlines():
cleaned = _clean_answer_line(line)
if cleaned:
answers.append(cleaned)
if cap and len(answers) >= cap:
break
# models sometimes put EXPLANATION before FINAL ANSWERS despite the prompt
if not expl:
em2 = _EXPL_RX.search(text)
if em2 and (not markers or em2.start() < markers[-1].start()):
seg = text[em2.end():]
stop = _FINAL_RX.search(seg)
expl = (seg[: stop.start()] if stop else seg).strip()
return answers, expl, found
# lines the model may prepend/append around a bare answer list
_CHATTY_RX = re.compile(
r"^(?:here (?:are|is)\b|answers?\s*:?\s*$|the answers?\b|translations?\s*:?\s*$|"
r"note\b|okay\b|sure\b|solution\b|let me\b)", re.IGNORECASE)
def parse_output_lean(text: str, n_items: Optional[int] = None
) -> Tuple[List[str], str, bool]:
"""Lean parse: the minimal prompt asks for bare answers, one per line, with
no FINAL ANSWERS marker — so every non-empty line IS an answer. We only
strip numbering/markdown decoration and drop obvious chatty preamble lines.
When more than n_items lines survive, keep the LAST n_items (any stray
preamble sits at the top). No marker requirement, no salvage."""
if not text:
return [], "", False
answers: List[str] = []
for line in text.splitlines():
cleaned = _clean_answer_line(line)
if cleaned and not _CHATTY_RX.match(cleaned):
answers.append(cleaned)
if n_items and len(answers) > n_items:
answers = answers[-n_items:]
return answers, "", bool(answers)
_LEAN_LABEL_RX = re.compile(r"^\s*\(?(\d{1,3})\)?[.):\]]\s")
def align_lean(text: str, puzzle: Puzzle) -> List[Optional[str]]:
"""Turn a lean (bare-lines) model output into exactly len(puzzle.items)
answers, blending two placement methods that stack:
1. LABEL-AWARE placement — if the model numbered its answer lines and those
numbers cover most of the item labels, place each answer under its own
label. This is robust to the model reordering items or skipping one (a
single skipped line would otherwise shift every later answer and zero the
whole block on both metrics).
2. POSITIONAL last-N fallback — when the output isn't reliably numbered, take
the cleaned, non-chatty lines in order (dropping any preamble at the top).
Both share our line hygiene (numbering/markdown stripping via
`_clean_answer_line`, preamble drop via `_CHATTY_RX`). Returns a length-N
list; None marks items the model did not answer, which the caller leaves to
the fallback."""
items = puzzle.items
n = len(items)
if n == 0:
return []
labeled: dict = {}
for line in (text or "").splitlines():
s = line.strip()
if not s:
continue
m = _LEAN_LABEL_RX.match(s)
cleaned = _clean_answer_line(s)
if not cleaned or _CHATTY_RX.match(cleaned):
continue
if m:
labeled[m.group(1)] = cleaned # last write wins (models restate)
item_labels = [it.number for it in items]
if labeled and all(lbl for lbl in item_labels):
covered = sum(1 for lbl in item_labels if lbl in labeled)
if covered >= max(1, (2 * n + 2) // 3): # ~2/3 labelled -> trust labels
return [labeled.get(lbl) for lbl in item_labels]
# positional fallback: our bare-line parse, aligned by position
bare, _e, _f = parse_output_lean(text, n)
out: List[Optional[str]] = [None] * n
for i in range(min(n, len(bare))):
out[i] = bare[i]
return out
def build_salvage_prompt(base_prompt: str, raw_reasoning: str,
max_chars: int = 2400) -> str:
"""A short follow-up prompt that reuses reasoning the model already
produced (but never terminated with a FINAL ANSWERS block, e.g. it hit the
token cap). We feed the reasoning back as context and ask ONLY for the
answer block -- cheaper and more reliable than a bare retry."""
reasoning = raw_reasoning.strip()
if len(reasoning) > max_chars:
reasoning = reasoning[-max_chars:] # keep the most recent reasoning
return (
f"{base_prompt}\n\n"
"You already worked through this problem:\n"
"-----\n"
f"{reasoning}\n"
"-----\n"
"Now output ONLY the answer block, nothing else. Write the line "
"FINAL ANSWERS: then one bare answer per line in item order (your best "
"guess for every item, never blank), then a line EXPLANATION: with 2-4 "
"short bullets."
)
def solve_direct(puzzles: Sequence[Puzzle], client: LLMClient,
scaffolds: Optional[Sequence[str]] = None,
max_new_tokens: int = MAX_NEW_TOKENS,
salvage: bool = True,
system: str = SYSTEM,
lean: bool = False,
sample: bool = False,
temperature: float = 0.5
) -> List[Tuple[List[str], str, str, bool]]:
"""Batched single-shot answering. Returns per puzzle
(answers, explanation, raw_text, found_marker). `sample`/`temperature` drive
the sampled passes used by self-consistency voting; salvage is skipped when
sampling. In `lean` mode every non-empty line is an answer (no marker)."""
if not client.available or not puzzles:
return [([], "", "", False) for _ in puzzles]
prompts = [
build_prompt(p, scaffolds[i] if scaffolds else "")
for i, p in enumerate(puzzles)
]
raws = client.generate(prompts, max_new_tokens=max_new_tokens, system=system,
sample=sample, temperature=temperature)
out: List[Tuple[List[str], str, str, bool]] = []
salvage_idx: List[int] = []
for i, r in enumerate(raws):
if lean:
aligned = align_lean(r, puzzles[i])
answers, expl, found = aligned, "", any(a for a in aligned)
else:
answers, expl, found = parse_output(r, len(puzzles[i].items))
out.append((answers, expl, r, found))
if salvage and not lean and not sample and not found and r.strip():
salvage_idx.append(i)
if salvage_idx:
sp = [build_salvage_prompt(prompts[i], raws[i]) for i in salvage_idx]
# answers only — a small cap keeps the salvage pass cheap
cap = min(max_new_tokens, 384)
sraws = client.generate(sp, max_new_tokens=cap, system=system)
for i, sr in zip(salvage_idx, sraws):
a2, e2, f2 = parse_output(sr, len(puzzles[i].items))
if f2 and a2:
prev = out[i]
out[i] = (a2, e2 or prev[1], prev[2], True)
return out
def align_answers(direct: List[str], n_items: int) -> List[Optional[str]]:
"""Position-align a FINAL ANSWERS block to the expected item count."""
out: List[Optional[str]] = [None] * n_items
for i in range(min(n_items, len(direct))):
out[i] = direct[i]
return out

0
solver/dsl/__init__.py Normal file
View File

188
solver/dsl/grammar.py Normal file
View File

@@ -0,0 +1,188 @@
"""DSL datatypes for puzzle grammars.
A Grammar is a small, fully-executable description of one puzzle language:
lexicon + affixes + rewrite rules + constituent order (+ numerals). The LLM
proposes grammars as JSON; `from_json` parses defensively (a malformed rule
is dropped, never fatal). `mdl` gives the description length used by the
verifier's simplicity penalty.
JSON shape the proposer LLM emits:
{
"lexicon": [{"morph": "kupu", "gloss": "bird", "pos": "N"}, ...],
"affixes": [{"position": "prefix"|"suffix"|"circumfix", "form": "na",
"form2": "", "feature": "DEF", "trigger": "N"}, ...],
"rewrites": [{"pattern": "a+a", "repl": "aa", "context": ""}, ...] # regex
"order": ["V", "S", "O"], # target constituent order
"agree": [{"src_slot": "S", "dst_slot": "V", "feature": "NUM"}, ...],
"redup": [{"scope": "first_syllable", "feature": "PL"}, ...]
}
"""
from __future__ import annotations
import json
import re
from dataclasses import dataclass, field
from typing import Dict, List, Optional
@dataclass
class LexEntry:
morph: str
gloss: str
pos: str = ""
features: Dict[str, str] = field(default_factory=dict)
@dataclass
class Affix:
position: str # prefix | suffix | infix | circumfix
form: str
feature: str = "" # what it marks, e.g. "PL", "PST", "DEF"
trigger: str = "" # pos or feature it attaches to; "" = any
form2: str = "" # second part for circumfix
@dataclass
class Rewrite:
pattern: str # regex over the surface string
repl: str
context: str = "" # optional regex that must match for rule to fire
def apply(self, s: str) -> str:
try:
if self.context and not re.search(self.context, s):
return s
return re.sub(self.pattern, self.repl, s)
except re.error:
return s
@dataclass
class Agree:
src_slot: str
dst_slot: str
feature: str
@dataclass
class Redup:
scope: str = "first_syllable" # or "full", "first_cv"
feature: str = "PL"
@dataclass
class Grammar:
lexicon: List[LexEntry] = field(default_factory=list)
affixes: List[Affix] = field(default_factory=list)
rewrites: List[Rewrite] = field(default_factory=list)
order: List[str] = field(default_factory=list)
agree: List[Agree] = field(default_factory=list)
redup: List[Redup] = field(default_factory=list)
notes: str = ""
# ---- lookup helpers ----
def by_gloss(self) -> Dict[str, LexEntry]:
return {e.gloss.casefold(): e for e in self.lexicon}
def by_morph(self) -> Dict[str, LexEntry]:
return {e.morph: e for e in self.lexicon}
def mdl(self) -> float:
"""Description length: total symbols in the grammar. Lightly weighted
by the verifier; only breaks ties between equally-fitting grammars."""
n = 0
for e in self.lexicon:
n += len(e.morph) + len(e.gloss) + 2
for a in self.affixes:
n += len(a.form) + len(a.form2) + len(a.feature) + 3
for r in self.rewrites:
n += len(r.pattern) + len(r.repl) + len(r.context) + 3
n += 2 * len(self.order) + 4 * len(self.agree) + 4 * len(self.redup)
return float(n)
def to_json(self) -> str:
return json.dumps(
{
"lexicon": [
{"morph": e.morph, "gloss": e.gloss, "pos": e.pos, "features": e.features}
for e in self.lexicon
],
"affixes": [
{"position": a.position, "form": a.form, "form2": a.form2,
"feature": a.feature, "trigger": a.trigger}
for a in self.affixes
],
"rewrites": [
{"pattern": r.pattern, "repl": r.repl, "context": r.context}
for r in self.rewrites
],
"order": self.order,
"agree": [
{"src_slot": g.src_slot, "dst_slot": g.dst_slot, "feature": g.feature}
for g in self.agree
],
"redup": [{"scope": d.scope, "feature": d.feature} for d in self.redup],
},
ensure_ascii=False,
)
def _get(d: dict, *keys: str, default: str = "") -> str:
for k in keys:
if k in d and d[k] is not None:
return str(d[k])
return default
def from_json(text: str) -> Optional[Grammar]:
"""Parse an LLM-emitted grammar. Tolerates surrounding prose/code fences
and drops malformed entries instead of failing."""
m = re.search(r"\{.*\}", text, re.DOTALL)
if not m:
return None
try:
data = json.loads(m.group(0))
except json.JSONDecodeError:
# common LLM damage: trailing commas
try:
data = json.loads(re.sub(r",\s*([}\]])", r"\1", m.group(0)))
except json.JSONDecodeError:
return None
if not isinstance(data, dict):
return None
g = Grammar()
for e in data.get("lexicon") or []:
if isinstance(e, dict):
morph, gloss = _get(e, "morph", "form", "word"), _get(e, "gloss", "meaning")
if morph and gloss:
feats = e.get("features") if isinstance(e.get("features"), dict) else {}
g.lexicon.append(LexEntry(morph, gloss, _get(e, "pos"), {str(k): str(v) for k, v in (feats or {}).items()}))
for a in data.get("affixes") or []:
if isinstance(a, dict):
form = _get(a, "form")
pos = _get(a, "position", default="suffix").lower()
if form and pos in ("prefix", "suffix", "infix", "circumfix"):
g.affixes.append(Affix(pos, form, _get(a, "feature", "gloss"), _get(a, "trigger"), _get(a, "form2")))
for r in data.get("rewrites") or []:
if isinstance(r, dict) and _get(r, "pattern"):
try:
re.compile(_get(r, "pattern"))
if _get(r, "context"):
re.compile(_get(r, "context"))
except re.error:
continue
g.rewrites.append(Rewrite(_get(r, "pattern"), _get(r, "repl", "replacement"), _get(r, "context")))
order = data.get("order") or []
if isinstance(order, list):
g.order = [str(x) for x in order]
for ag in data.get("agree") or []:
if isinstance(ag, dict) and _get(ag, "feature"):
g.agree.append(Agree(_get(ag, "src_slot", "src"), _get(ag, "dst_slot", "dst"), _get(ag, "feature")))
for rd in data.get("redup") or []:
if isinstance(rd, dict):
g.redup.append(Redup(_get(rd, "scope", default="first_syllable"), _get(rd, "feature", default="PL")))
if not g.lexicon and not g.affixes and not g.rewrites:
return None
return g

198
solver/dsl/interpreter.py Normal file
View File

@@ -0,0 +1,198 @@
"""Deterministic interpreter for DSL grammars: generate() (work -> task
language) and analyze() (task -> work language).
The interpreter is intentionally strict and simple: it executes exactly what
the grammar says. If a grammar needs cleverness, the proposer must encode it
(e.g. list 'birds' as its own lexicon entry instead of relying on affix
machinery). The verifier then selects grammars that this interpreter executes
into correct outputs — that closed loop is the whole design.
Conventions the proposer prompt establishes:
- lexicon glosses are work-language words/phrases (may be multiword);
- affix `feature` is the work-language cue it realizes: a function word
("the", "will", "not") or a marker name ("plural") — during generation a
feature fires when its cue appears in the work sentence next to the stem;
- `order` is a list of pos tags giving target-language constituent order;
- rewrites are surface regex applied after morph concatenation (word-level).
"""
from __future__ import annotations
import re
from typing import Dict, List, Optional, Tuple
from .grammar import Affix, Grammar, LexEntry
def _words(s: str) -> List[str]:
return [w for w in re.findall(r"[^\s]+", s.strip()) if w]
def _clean(w: str) -> str:
return w.strip(",;.!?()[]\"'«»").casefold()
class Interpreter:
def __init__(self, grammar: Grammar):
self.g = grammar
# gloss index: multiword glosses first (longest match wins)
self._gloss_entries: List[Tuple[List[str], LexEntry]] = sorted(
(( [_clean(w) for w in _words(e.gloss)], e) for e in grammar.lexicon if e.gloss),
key=lambda t: -len(t[0]),
)
self._morphs: Dict[str, LexEntry] = {e.morph: e for e in grammar.lexicon}
self._affix_by_cue: Dict[str, List[Affix]] = {}
for a in grammar.affixes:
self._affix_by_cue.setdefault(_clean(a.feature), []).append(a)
# ------------------------------------------------------------------
# generation: work-language sentence -> task-language string
# ------------------------------------------------------------------
def generate(self, work_sentence: str) -> Optional[str]:
toks = [_clean(w) for w in _words(work_sentence)]
if not toks:
return None
n = len(toks)
used = [False] * n
stems: List[Tuple[int, LexEntry]] = [] # (position of first gloss word, entry)
# 1. cover with lexicon glosses, longest first
for gloss_words, entry in self._gloss_entries:
L = len(gloss_words)
i = 0
while i + L <= n:
if not any(used[i : i + L]) and toks[i : i + L] == gloss_words:
for j in range(i, i + L):
used[j] = True
stems.append((i, entry))
i += L
else:
i += 1
if not stems:
return None
stems.sort(key=lambda t: t[0])
# 2. leftover tokens fire affixes on the nearest eligible stem
pending: Dict[int, List[Affix]] = {k: [] for k in range(len(stems))}
uncovered = [i for i in range(n) if not used[i]]
for i in uncovered:
cue = toks[i]
for a in self._affix_by_cue.get(cue, []):
k = self._nearest_stem(stems, i, a)
if k is not None:
pending[k].append(a)
break
# 3. order stems by target constituent order if pos info available
idx = list(range(len(stems)))
if self.g.order and all(e.pos for _, e in stems):
rank = {pos: r for r, pos in enumerate(self.g.order)}
idx.sort(key=lambda k: (rank.get(stems[k][1].pos, len(rank)), stems[k][0]))
# 4. build surface words: affix attachment then rewrites
out_words = []
for k in idx:
_, entry = stems[k]
w = entry.morph
for a in pending[k]:
w = self._attach(w, a)
w = self._apply_rewrites(w)
out_words.append(w)
surface = " ".join(out_words)
return self._apply_rewrites_sentence(surface)
def _nearest_stem(self, stems: List[Tuple[int, LexEntry]], cue_pos: int, affix: Affix) -> Optional[int]:
best_k, best_d = None, 10 ** 9
for k, (pos, entry) in enumerate(stems):
if affix.trigger and affix.trigger not in (entry.pos, entry.gloss):
continue
d = abs(pos - cue_pos)
if d < best_d:
best_k, best_d = k, d
return best_k
def _attach(self, w: str, a: Affix) -> str:
if a.position == "prefix":
return a.form + w
if a.position == "suffix":
return w + a.form
if a.position == "circumfix":
return a.form + w + (a.form2 or a.form)
if a.position == "infix":
# after the first vowel-less onset (common infix site: after first C)
m = re.match(r"^([^aeiouAEIOU]*)(.*)$", w)
return (m.group(1) + a.form + m.group(2)) if m else w + a.form
return w
def _apply_rewrites(self, w: str) -> str:
for r in self.g.rewrites:
w = r.apply(w)
return w
def _apply_rewrites_sentence(self, s: str) -> str:
# rewrites with explicit spaces/anchors act at sentence level too
for r in self.g.rewrites:
if " " in r.pattern or r.pattern.startswith("^") or r.pattern.endswith("$"):
s = r.apply(s)
return s
# ------------------------------------------------------------------
# analysis: task-language sentence -> work-language string
# ------------------------------------------------------------------
def analyze(self, task_sentence: str) -> Optional[str]:
words = [_clean(w) for w in _words(task_sentence)]
if not words:
return None
rendered: List[Tuple[str, str, str]] = [] # (prefix cues, gloss, suffix cues)
any_known = False
for w in words:
pre, gloss, post, known = self._analyze_word(w)
any_known = any_known or known
rendered.append((pre, gloss, post))
if not any_known:
return None
out: List[str] = []
for pre, gloss, post in rendered:
for c in pre.split():
out.append(c)
out.append(gloss)
for c in post.split():
out.append(c)
return " ".join(x for x in out if x)
def _analyze_word(self, w: str) -> Tuple[str, str, str, bool]:
"""Decompose one surface word into (prefix cues, stem gloss, suffix
cues, matched?). Tries direct lexicon hit, then affix stripping
(longest affix first), then returns the word untouched."""
if w in self._morphs:
return "", self._morphs[w].gloss, "", True
affixes = sorted(self.g.affixes, key=lambda a: -len(a.form))
for a in affixes:
if a.position == "prefix" and w.startswith(a.form):
pre, gloss, post, ok = self._analyze_word(w[len(a.form):])
if ok:
return (self._cue(a) + " " + pre).strip(), gloss, post, True
if a.position == "suffix" and w.endswith(a.form):
pre, gloss, post, ok = self._analyze_word(w[: len(w) - len(a.form)])
if ok:
return pre, gloss, (post + " " + self._cue(a)).strip(), True
if a.position == "circumfix" and w.startswith(a.form) and w.endswith(a.form2 or a.form):
inner = w[len(a.form): len(w) - len(a.form2 or a.form)]
pre, gloss, post, ok = self._analyze_word(inner)
if ok:
return (self._cue(a) + " " + pre).strip(), gloss, post, True
# last resort: greedy stem containment (rewrite rules may have altered edges)
for morph, entry in sorted(self._morphs.items(), key=lambda kv: -len(kv[0])):
if len(morph) >= 3 and morph in w:
return "", entry.gloss, "", True
return "", w, "", False
@staticmethod
def _cue(a: Affix) -> str:
"""How an affix surfaces in the work-language output: function-word
cues are emitted verbatim; abstract markers (PL, PST) are dropped —
the proposer should prefer word cues for translatable material."""
cue = a.feature.strip()
if cue and cue.isalpha() and cue.casefold() == cue:
return cue
return ""

74
solver/fallback.py Normal file
View File

@@ -0,0 +1,74 @@
"""chrF-floor fallback: never return an empty or wildly-off answer.
The geometric-mean metric means one empty answer costs far more than a wrong
but plausible one. Fallback ladder (best available wins):
1. analogy from the closest attested source (transfers its target with the
observed source->query edit applied),
2. the attested target of the most chrF-similar attested source,
3. echo of query content words mapped through alignment,
4. the raw query text itself (last resort: shares characters with gold more
often than an empty string does).
"""
from __future__ import annotations
from typing import List, Optional, Tuple
from . import analogy
from .metrics import chrf
from .align import align as build_align, best_translation
from .preprocess import Pair, strip_punct, tokenize
def closest_attested(query: str, sources: List[str]) -> Tuple[int, float]:
"""Index and similarity of the attested source closest to the query."""
best_i, best_s = -1, -1.0
for i, s in enumerate(sources):
sc = chrf(query, s)
if sc > best_s:
best_i, best_s = i, sc
return best_i, best_s
def fallback_answer(query: str, pairs: List[Pair], direction: str = "to_work") -> str:
"""direction: 'to_work' = translate task->work (analysis);
'to_task' = work->task (generation). Pairs are (task, work)."""
if direction == "to_task":
srcs = [p.tgt for p in pairs]
tgts = [p.src for p in pairs]
flipped = [Pair(src=p.tgt, tgt=p.src) for p in pairs]
else:
srcs = [p.src for p in pairs]
tgts = [p.tgt for p in pairs]
flipped = pairs
query = query.strip()
if not query:
return tgts[0] if tgts else "?"
if srcs:
i, sim = closest_attested(query, srcs)
if i >= 0:
# 1. analogy transfer: apply the srcs[i]->query edit to tgts[i]
transfer = analogy.solve(srcs[i], query, tgts[i])
if transfer and sim > 0.3:
return transfer[0]
# 2. echo the closest attested target
if sim > 0.15 and tgts[i]:
return tgts[i]
# 3. word-by-word through alignment
amap = build_align(flipped)
words = [strip_punct(t) for t in tokenize(query)]
mapped = [best_translation(amap, w) or w for w in words if w]
if mapped:
return " ".join(mapped)
# 4. absolute floor
return query
def ensure_nonempty(ans: Optional[str], query: str, pairs: List[Pair], direction: str = "to_work") -> str:
if ans and str(ans).strip():
return str(ans).strip()
return fallback_answer(query, pairs, direction) or "?"

346
solver/llm.py Normal file
View File

@@ -0,0 +1,346 @@
"""LLM clients: HFTransformersClient (transformers/AWQ, T4, greedy) plus
NullClient/CallableClient for dev and an optional VLLMClient."""
from __future__ import annotations
import re
from pathlib import Path
from typing import Callable, List, Optional, Sequence, Tuple
MAX_NEW_TOKENS = 1024
DEFAULT_MODEL_DIR = "." # submission: weights ship at the repo root
class LLMClient:
"""Interface: generate(prompts) -> one completion per prompt."""
available: bool = False
# can this backend score candidate next-tokens (needed for the match_letters
# assignment solver)? Only the real transformers backend can.
can_score: bool = False
deadline: Optional[float] = None # monotonic wall-clock abort (armed by caller)
def generate(self, prompts: Sequence[str], max_new_tokens: int = MAX_NEW_TOKENS,
system: Optional[str] = None, sample: bool = False,
temperature: float = 0.5) -> List[str]:
raise NotImplementedError
class NullClient(LLMClient):
available = False
def generate(self, prompts, max_new_tokens=MAX_NEW_TOKENS, system=None,
sample=False, temperature=0.5):
return ["" for _ in prompts]
class CallableClient(LLMClient):
available = True
def __init__(self, fn: Callable[[str], str]):
self.fn = fn
def generate(self, prompts, max_new_tokens=MAX_NEW_TOKENS, system=None,
sample=False, temperature=0.5):
return [self.fn(p) for p in prompts]
def pack_by_tokens(lengths: Sequence[int], token_budget: int, max_batch: int
) -> List[List[int]]:
"""Group prompt indices into batches whose TOTAL token count stays under
`token_budget`. On the sandbox's transformers version, prefill computes
float32 logits over every prompt position (batch x seq x 152k vocab), so
total-tokens-per-batch — not batch count — is what bounds T4 memory.
An oversized single prompt still gets its own batch (handled by the OOM
retry path)."""
batches: List[List[int]] = []
cur: List[int] = []
cur_tokens = 0
for i, n in enumerate(lengths):
if cur and (cur_tokens + n > token_budget or len(cur) >= max_batch):
batches.append(cur)
cur, cur_tokens = [], 0
cur.append(i)
cur_tokens += n
if cur:
batches.append(cur)
return batches
class HFTransformersClient(LLMClient):
"""transformers backend, mirroring the official notebook's loading code
(fp16, device_map=auto, greedy) plus token-budget batching and OOM
recovery. Import is lazy."""
available = True
can_score = True # supports score_next_logprobs (match_letters assignment)
# total prompt tokens per generation batch. On the sandbox's transformers,
# prefill computes fp32 logits over every prompt position (batch x seq x
# 152k vocab), ~0.9 MB/token; total-tokens-per-batch — not batch count — is
# the T4 memory bound. 3500 is the safe fallback used when we cannot
# measure free VRAM; __init__ raises it to fit whatever headroom the loaded
# model actually leaves (≈6500 for a 7B-AWQ, ≈3500 for a 14B-AWQ).
TOKEN_BUDGET = 3500
MB_PER_TOKEN = 0.9 # fp32 prefill logits at Qwen's 152k vocab
VRAM_RESERVE_GB = 1.8 # KV cache + activations + fragmentation slack
def __init__(self, model_dir: str = DEFAULT_MODEL_DIR, batch_size: int = 8):
import inspect
import time as _time
import torch
from transformers import (AutoModelForCausalLM, AutoTokenizer,
StoppingCriteria, StoppingCriteriaList)
self.torch = torch
self.tok = AutoTokenizer.from_pretrained(model_dir)
# Pin to GPU 0: device_map="auto" can silently offload layers to CPU on a
# tight T4 (~100x slower). Fall back to "auto" if the pinned load fails.
try:
self.model = AutoModelForCausalLM.from_pretrained(
model_dir, torch_dtype=torch.float16,
device_map={"": 0} if torch.cuda.is_available() else "auto",
).eval()
except Exception:
self.model = AutoModelForCausalLM.from_pretrained(
model_dir, torch_dtype=torch.float16, device_map="auto",
).eval()
if self.tok.pad_token_id is None:
self.tok.pad_token = self.tok.eos_token
self.batch_size = batch_size
# newer transformers can skip full-sequence prefill logits entirely
fwd_params = inspect.signature(self.model.forward).parameters
self._logits_kwarg = next(
(k for k in ("logits_to_keep", "num_logits_to_keep") if k in fwd_params),
None)
self._tune_token_budget()
self.last_truncated: List[bool] = []
# per-token wall-clock abort: a batch started near the deadline can't
# overrun and get the process killed (the budget is otherwise only
# checked between batches). set self.deadline (monotonic ts) to arm it.
self.deadline: Optional[float] = None
class _Deadline(StoppingCriteria):
def __call__(self, input_ids, scores, **kw):
return _time.monotonic() > deadline_holder[0]
deadline_holder = [float("inf")]
self._deadline_holder = deadline_holder
self._deadline_crit = StoppingCriteriaList([_Deadline()])
def _tune_token_budget(self) -> None:
"""Size the per-batch token budget to the VRAM the loaded weights
actually leave free. Falls back to the safe class default if the GPU
can't be queried (CPU dev, older CUDA)."""
torch = self.torch
try:
if not torch.cuda.is_available():
return
free_bytes, _ = torch.cuda.mem_get_info()
free_gb = free_bytes / 1e9
budget = int((free_gb - self.VRAM_RESERVE_GB) * 1000 / self.MB_PER_TOKEN)
# the logits_to_keep fast path removes the big allocation entirely,
# but stay conservative regardless; clamp to a sane window
self.TOKEN_BUDGET = max(2500, min(7000, budget))
except Exception:
pass
def _chat_texts(self, prompts: Sequence[str], system: Optional[str]) -> List[str]:
texts = []
for p in prompts:
messages = ([{"role": "system", "content": system}] if system else []) \
+ [{"role": "user", "content": p}]
texts.append(self.tok.apply_chat_template(
messages, add_generation_prompt=True, tokenize=False))
return texts
def _generate_chunk(self, chunk: List[str], max_new_tokens: int,
sample: bool = False, temperature: float = 0.5
) -> Tuple[List[str], List[bool]]:
torch = self.torch
# pad only when batching more than one prompt (padding can perturb a
# quantized model's greedy outputs)
enc = self.tok(chunk, return_tensors="pt", padding=len(chunk) > 1,
add_special_tokens=False).to(self.model.device)
kwargs = {self._logits_kwarg: 1} if self._logits_kwarg else {}
# repetition_penalty=1.0 explicitly: the shipped generation_config sets
# 1.05, which is applied even under greedy and biases against the
# repeated characters common in these answers.
kwargs["repetition_penalty"] = 1.0
if sample:
kwargs.update(do_sample=True, temperature=temperature, top_p=0.95)
else:
kwargs["do_sample"] = False
if self.deadline is not None:
self._deadline_holder[0] = self.deadline
kwargs["stopping_criteria"] = self._deadline_crit
with torch.no_grad():
gen = self.model.generate(
**enc, max_new_tokens=max_new_tokens,
pad_token_id=self.tok.pad_token_id, **kwargs,
)
new_tokens = gen[:, enc["input_ids"].shape[1]:]
eos = self.tok.eos_token_id
texts, truncated = [], []
for row in new_tokens:
texts.append(_guard(self.tok.decode(row, skip_special_tokens=True)).strip())
# no EOS in the generated span => generation was cut at the cap
truncated.append(eos is None or int((row == eos).sum()) == 0)
return texts, truncated
def generate(self, prompts, max_new_tokens=MAX_NEW_TOKENS, system=None,
sample=False, temperature=0.5):
if not prompts:
self.last_truncated = []
return []
torch = self.torch
texts = self._chat_texts(prompts, system)
lengths = [len(self.tok(t, add_special_tokens=False)["input_ids"])
for t in texts]
out: List[str] = [""] * len(texts)
trunc: List[bool] = [False] * len(texts)
prev_side = self.tok.padding_side
self.tok.padding_side = "left" # sequences must end at the gen position
try:
for batch in pack_by_tokens(lengths, self.TOKEN_BUDGET, self.batch_size):
chunk = [texts[i] for i in batch]
try:
results, tflags = self._generate_chunk(
chunk, max_new_tokens, sample, temperature)
except torch.cuda.OutOfMemoryError:
# halve pressure: clear cache, retry one prompt at a time;
# a prompt that OOMs alone yields "" (symbolic answer stands)
torch.cuda.empty_cache()
results, tflags = [], []
for t in chunk:
try:
r1, f1 = self._generate_chunk(
[t], max_new_tokens, sample, temperature)
results.append(r1[0])
tflags.append(f1[0])
except torch.cuda.OutOfMemoryError:
torch.cuda.empty_cache()
results.append("")
tflags.append(False)
for i, r, f in zip(batch, results, tflags):
out[i] = r
trunc[i] = f
finally:
self.tok.padding_side = prev_side
self.last_truncated = trunc
return out
def score_next_logprobs(self, prompts, cand_token_ids, system=None,
batch_size=4):
"""For each prompt, return the max next-token log-prob over each
candidate group. `cand_token_ids` is a list of token-id groups (one per
option), shared across prompts. Returns List[List[float]]
(prompt x option). One forward pass per batch; used by the match_letters
assignment solver. Missing/OOM/timed-out prompts get all-zero rows so
the caller can fall back."""
import time
if not prompts:
return []
torch = self.torch
n_opt = len(cand_token_ids)
texts = self._chat_texts(prompts, system)
out: List[List[float]] = []
prev_side = self.tok.padding_side
self.tok.padding_side = "left"
bs = batch_size
i = 0
try:
while i < len(texts):
if self.deadline is not None and time.monotonic() > self.deadline:
out.extend([[0.0] * n_opt for _ in range(len(texts) - i)])
break
chunk = texts[i:i + bs]
try:
enc = self.tok(chunk, return_tensors="pt", padding=True,
add_special_tokens=False, truncation=True,
max_length=6144).to(self.model.device)
fwd = {self._logits_kwarg: 1} if self._logits_kwarg else {}
with torch.no_grad():
logits = self.model(**enc, **fwd).logits[:, -1, :].float()
logprobs = torch.log_softmax(logits, dim=-1)
for b in range(len(chunk)):
row = [max((logprobs[b, t].item() for t in group),
default=-1e9) if group else -1e9
for group in cand_token_ids]
out.append(row)
i += bs
except torch.cuda.OutOfMemoryError:
torch.cuda.empty_cache()
if bs == 1:
out.append([0.0] * n_opt)
i += 1
else:
bs = max(1, bs // 2)
finally:
self.tok.padding_side = prev_side
return out
class VLLMClient(LLMClient):
"""Optional vLLM backend for throughput experiments. Never required."""
available = True
def __init__(self, model_dir: str = DEFAULT_MODEL_DIR, max_model_len: int = 4096):
from vllm import LLM
self.llm = LLM(model=model_dir, dtype="half", max_model_len=max_model_len,
gpu_memory_utilization=0.90)
def generate(self, prompts, max_new_tokens=MAX_NEW_TOKENS, system=None):
from vllm import SamplingParams
tok = self.llm.get_tokenizer()
texts = []
for p in prompts:
messages = ([{"role": "system", "content": system}] if system else []) \
+ [{"role": "user", "content": p}]
texts.append(tok.apply_chat_template(messages, add_generation_prompt=True,
tokenize=False))
params = SamplingParams(temperature=0.0, max_tokens=max_new_tokens)
outs = self.llm.generate(texts, params)
return [_guard(o.outputs[0].text if o.outputs else "").strip() for o in outs]
def _guard(text: str, max_repeat: int = 4) -> str:
"""Loop-collapse guard: truncate at the point where a line repeats more
than `max_repeat` times consecutively."""
lines = text.splitlines()
out, streak = [], 0
for i, l in enumerate(lines):
if i > 0 and l.strip() and l == lines[i - 1]:
streak += 1
if streak >= max_repeat:
break
else:
streak = 0
out.append(l)
return "\n".join(out)
def load_client(model_id: Optional[str] = None) -> LLMClient:
"""Best available client for `model_id` (script.py's MODEL_ID):
- a local path ("." in the submission, weights/base in dev) is loaded
when its config.json exists;
- a Hub name (contains "/" and is not a local dir) is passed straight to
transformers — the Colab-testing path, mirroring the workshop notebook;
- anything unloadable degrades to NullClient (symbolic-only pipeline)."""
if model_id and "/" in model_id and not Path(model_id).exists():
try:
return HFTransformersClient(model_dir=model_id)
except Exception:
return NullClient()
for d in ([model_id] if model_id else []) + [DEFAULT_MODEL_DIR, "weights/base"]:
if d and Path(d, "config.json").exists():
try:
return HFTransformersClient(model_dir=d)
except Exception:
continue
return NullClient()

266
solver/matching.py Normal file
View File

@@ -0,0 +1,266 @@
"""match_letters: optimal one-to-one assignment (pure-python Hungarian) over
either surface-similarity scores or the model's next-token log-probs."""
from __future__ import annotations
from typing import Dict, List, Optional, Sequence, Tuple
from .align import align as build_align
from .preprocess import Pair, Puzzle, strip_punct, tokenize
def hungarian(cost: List[List[float]]) -> List[int]:
"""Minimum-cost perfect matching on a square cost matrix.
Returns assignment: row i -> column result[i]. Jonker-style O(n^3)
shortest augmenting path implementation."""
n = len(cost)
if n == 0:
return []
INF = float("inf")
u = [0.0] * (n + 1)
v = [0.0] * (n + 1)
p = [0] * (n + 1) # p[j] = row matched to column j (1-indexed)
way = [0] * (n + 1)
for i in range(1, n + 1):
p[0] = i
j0 = 0
minv = [INF] * (n + 1)
used = [False] * (n + 1)
while True:
used[j0] = True
i0, delta, j1 = p[j0], INF, 0
for j in range(1, n + 1):
if not used[j]:
cur = cost[i0 - 1][j - 1] - u[i0] - v[j]
if cur < minv[j]:
minv[j] = cur
way[j] = j0
if minv[j] < delta:
delta = minv[j]
j1 = j
for j in range(n + 1):
if used[j]:
u[p[j]] += delta
v[j] -= delta
else:
minv[j] -= delta
j0 = j1
if p[j0] == 0:
break
while j0:
j1 = way[j0]
p[j0] = p[j1]
j0 = j1
ans = [0] * n
for j in range(1, n + 1):
if p[j]:
ans[p[j] - 1] = j - 1
return ans
def _char_ngrams(s: str, nmin: int = 2, nmax: int = 4) -> set:
s = s.casefold().replace(" ", "")
return {s[i : i + n] for n in range(nmin, nmax + 1) for i in range(len(s) - n + 1)}
def _sim(a: str, b: str) -> float:
ga, gb = _char_ngrams(a), _char_ngrams(b)
if not ga or not gb:
return 0.0
return len(ga & gb) / max(len(ga | gb), 1)
def score_matrix(
forms: Sequence[str], meanings: Sequence[str], pairs: List[Pair]
) -> List[List[float]]:
"""Higher = better match. Combines token-level alignment evidence from the
attested pairs with surface similarity to attested forms sharing meaning
words."""
amap = build_align(pairs) if pairs else {}
# index attested: meaning word -> attested source strings
attested_by_word: Dict[str, List[str]] = {}
for p in pairs:
for w in tokenize(p.tgt):
w = strip_punct(w).casefold()
if w:
attested_by_word.setdefault(w, []).append(p.src)
S = []
for f in forms:
f_toks = [strip_punct(t).casefold() for t in tokenize(f)]
row = []
for m in meanings:
m_words = [strip_punct(w).casefold() for w in tokenize(m)]
score = 0.0
# alignment evidence: form tokens aligned to meaning words
for ft in f_toks:
for tgt, s in amap.get(ft, []):
if tgt in m_words:
score += s
# surface similarity to attested sources of these meaning words
for w in m_words:
for src in attested_by_word.get(w, []):
score += 0.5 * _sim(f, src)
row.append(score)
S.append(row)
return S
def _self_sim(texts: Sequence[str], char_level: bool) -> List[List[float]]:
"""Pairwise similarity within one side: shared char n-grams for unknown
forms, shared content words for meanings."""
n = len(texts)
feats = []
for t in texts:
if char_level:
feats.append(_char_ngrams(t, 3, 5))
else:
stop = {"the", "a", "an", "of", "is", "it", "he", "she", "they",
"are", "in", "to", "for", "with", "and", "or"}
feats.append({w for w in (strip_punct(x).casefold() for x in tokenize(t))
if w and w not in stop})
S = [[0.0] * n for _ in range(n)]
for i in range(n):
for j in range(i + 1, n):
inter = len(feats[i] & feats[j])
union = len(feats[i] | feats[j]) or 1
S[i][j] = S[j][i] = inter / union
return S
def structural_scores(forms: Sequence[str], meanings: Sequence[str],
iters: int = 4) -> List[List[float]]:
"""Structure-matching signal for zero-lexical-evidence matching: forms
sharing morphemes should map to meanings sharing words. Soft assignment
power iteration X <- Sf @ X @ Sm (a light quadratic-assignment relaxation)
starting from uniform. Returns an (n_forms x n_meanings) score matrix."""
nf, nm = len(forms), len(meanings)
if nf == 0 or nm == 0:
return [[0.0] * nm for _ in range(nf)]
Sf = _self_sim(forms, char_level=True)
Sm = _self_sim(meanings, char_level=False)
# seed with degree-profile agreement: a form clustered with k others
# should map to a meaning clustered with ~k others. (A uniform seed is a
# degenerate fixed point — every row converges to the same profile.)
def profile(S, i):
return sorted((v for v in S[i] if v > 0.05), reverse=True)[:6]
X = []
for i in range(nf):
pf = profile(Sf, i)
row = []
for j in range(nm):
pm = profile(Sm, j)
d = sum(abs(a - b) for a, b in zip(pf, pm)) + abs(len(pf) - len(pm))
row.append(1.0 / (1.0 + d))
X.append(row)
for _ in range(iters):
# Y = Sf @ X @ Sm (tiny n: pure-python is fine)
T = [[sum(Sf[i][k] * X[k][j] for k in range(nf)) for j in range(nm)]
for i in range(nf)]
Y = [[sum(T[i][k] * Sm[k][j] for k in range(nm)) for j in range(nm)]
for i in range(nf)]
# row-normalize to keep the iteration bounded
X = []
for row in Y:
z = sum(row) or 1.0
X.append([v / z for v in row])
return X
def solve_matching(
forms: Sequence[str], meanings: Sequence[str], pairs: List[Pair]
) -> List[Tuple[str, str]]:
"""Optimal assignment of forms to meanings. Combines lexical/alignment
evidence (when attested pairs exist) with the structural signal (always).
Pads to square with zero scores when lengths differ."""
n = max(len(forms), len(meanings))
S = score_matrix(forms, meanings, pairs)
S2 = structural_scores(forms, meanings)
cost = [[0.0] * n for _ in range(n)]
for i in range(n):
for j in range(n):
s = 0.0
if i < len(forms) and j < len(meanings):
s = S[i][j] + 3.0 * len(meanings) * S2[i][j]
cost[i][j] = -s
assign = hungarian(cost)
out = []
for i, f in enumerate(forms):
j = assign[i]
out.append((f, meanings[j] if j < len(meanings) else ""))
return out
# --------------------------------------------------------------------------
# LLM-scored assignment. Free-form generation tends to answer match_letters with
# the identity permutation (A, B, C, ...) — a valid permutation that scores ~0.
# Instead we score each (item, option-letter) pair from the model's next-token
# log-probs and take the optimal one-to-one assignment, so the bijection is
# enforced exactly rather than hoped for. The assignment engine is our existing
# Hungarian; only the score source changes (surface features -> the model's own
# distribution).
# --------------------------------------------------------------------------
_MATCH_SYSTEM = (
"You match items to their correct counterparts in a linguistics problem. "
"Reply with one option letter only."
)
def letters_from_scores(scores: List[List[float]], letters: Sequence[str]
) -> List[str]:
"""Given a per-item score over option letters (item x option), return one
letter per item. When items and options are equinumerous (the IOL matching
shape) the assignment is a strict bijection via Hungarian; otherwise it
degrades to an independent per-item argmax."""
n_items, n_opt = len(scores), len(letters)
if n_items == 0 or n_opt == 0:
return []
if n_items == n_opt:
cost = [[-scores[i][j] for j in range(n_opt)] for i in range(n_items)]
assign = hungarian(cost)
return [letters[assign[i]] for i in range(n_items)]
return [letters[max(range(n_opt), key=lambda j: scores[i][j])]
for i in range(n_items)]
def solve_match_letters_llm(puzzle: Puzzle, client) -> Optional[List[str]]:
"""Assign each numbered item to an option letter using the model's
next-token log-probs, then Hungarian. Returns one letter per puzzle item
(in item order), or None if it declines: no scoring backend, not a
well-formed lettered-matching shape, or fewer than 3 items/options."""
if not getattr(client, "can_score", False):
return None
items = puzzle.items
if not puzzle.lettered or len(items) < 3:
return None
letters = sorted(puzzle.lettered)
if len(letters) < 3:
return None
tok = client.tok
cand_ids: List[List[int]] = []
for L in letters:
ids = set()
for form in (L, " " + L):
t = tok.encode(form, add_special_tokens=False)
if t:
ids.add(t[0])
cand_ids.append(sorted(ids))
if any(not g for g in cand_ids):
return None
ctx = (puzzle.context or "").strip()
prompts = []
for it in items:
num = it.number or "?"
form = (it.text or "").strip()
prompts.append(
f"{ctx}\n\nWhich lettered option corresponds to item {num} "
f"({form})? Reply with the option letter only.")
scores = client.score_next_logprobs(prompts, cand_ids, system=_MATCH_SYSTEM)
if not scores or len(scores) != len(items):
return None
return letters_from_scores(scores, letters)

132
solver/metrics.py Normal file
View File

@@ -0,0 +1,132 @@
"""Metrics matching the official eval notebook (iolai-2026-workshop).
Lives in solver/ (not eval/) because the RUNTIME needs it: the verifier and
the fallback ladder score candidate answers with chrF/EM at inference time.
The dev-only eval/ package re-exports from here.
Semantics:
- EM: case-insensitive exact string equality after .strip() ONLY — punctuation
and internal whitespace are significant. Gold items may carry alternatives
(list of accepted strings); a hit on any alternative counts.
- chrF: sacrebleu CHRF() defaults (char n-grams 1..6, beta=2, whitespace not
in n-grams, epsilon-smoothed per order), 0..1 here (notebook prints 0..100).
Max over alternatives.
- Aggregate: per-item average of each metric; geometric mean reported as the
headline (the competition combines EM and chrF; the notebook prints both).
chrF is implemented in pure python replicating sacrebleu's algorithm so the
eval sandbox needs no dependency; tests/test_scorer.py checks parity against
sacrebleu when it is installed.
"""
from __future__ import annotations
import math
import re
import unicodedata
from collections import Counter
from typing import Dict, List, Optional, Sequence, Union
CHRF_NGRAM_ORDER = 6
CHRF_BETA = 2.0
_EPS = 1e-16
Gold = Union[str, Sequence[str]] # a gold item: one string or alternatives
def _alts(gold: Gold) -> List[str]:
if isinstance(gold, str):
return [gold]
return [str(a) for a in gold] or [""]
def normalize_answer(s: str) -> str:
"""Official EM normalization: strip + lowercase. Nothing else — final
punctuation and internal spacing are significant."""
return str(s).strip().lower()
def exact_match(pred: str, gold: Gold) -> float:
p = normalize_answer(pred)
return 1.0 if any(p == normalize_answer(a) for a in _alts(gold)) else 0.0
def _char_ngrams(s: str, n: int) -> Counter:
return Counter(s[i : i + n] for i in range(len(s) - n + 1))
def _chrf_single(pred: str, gold: str, n_order: int = CHRF_NGRAM_ORDER,
beta: float = CHRF_BETA) -> float:
"""Exact replication of sacrebleu CHRF defaults (whitespace stripped,
effective-order smoothing): precision/recall averaged over orders where
BOTH sides have n-grams; hypothesis counts are zeroed for orders the
reference lacks. Returns [0, 1] (sacrebleu reports x100)."""
pred_s = "".join(str(pred).split())
gold_s = "".join(str(gold).split())
avg_prec = avg_rec = 0.0
effective = 0
for n in range(1, n_order + 1):
gn = _char_ngrams(gold_s, n)
pn = _char_ngrams(pred_s, n)
n_ref = sum(gn.values())
n_hyp = sum(pn.values()) if gn else 0 # sacrebleu: no ref => no hyp hits
if n_hyp > 0 and n_ref > 0:
overlap = sum((pn & gn).values())
avg_prec += overlap / n_hyp
avg_rec += overlap / n_ref
effective += 1
if effective == 0:
return 0.0
avg_prec /= effective
avg_rec /= effective
if avg_prec + avg_rec == 0:
return 0.0
b2 = beta * beta
return (1 + b2) * avg_prec * avg_rec / (b2 * avg_prec + avg_rec)
def chrf(pred: str, gold: Gold) -> float:
return max(_chrf_single(pred, a) for a in _alts(gold))
def item_scores(pred: str, gold: Gold) -> Dict[str, float]:
return {"em": exact_match(pred, gold), "chrf": chrf(pred, gold)}
def score_submission(
preds: Sequence[Sequence[str]],
golds: Sequence[Sequence[Gold]],
weights: Optional[Sequence[Sequence[float]]] = None,
) -> Dict[str, float]:
"""Score a full submission.
preds/golds: per row, a list of items; each gold item is a string or a
list of accepted alternatives. weights: optional per-item point values;
uniform if None. Length mismatches within a row are penalized: missing
preds score 0, extra preds are ignored (notebook lines preds up by
position exactly the same way).
"""
total_w = 0.0
em_w = 0.0
chrf_w = 0.0
n_items = 0
for ri, (prow, grow) in enumerate(zip(preds, golds)):
wrow = list(weights[ri]) if weights is not None else [1.0] * len(grow)
for ii, gold in enumerate(grow):
w = wrow[ii] if ii < len(wrow) else 1.0
pred = prow[ii] if ii < len(prow) else ""
total_w += w
em_w += w * exact_match(pred, gold)
chrf_w += w * chrf(pred, gold)
n_items += 1
if total_w == 0:
return {"em": 0.0, "chrf": 0.0, "score": 0.0, "n_items": 0}
em_avg = em_w / total_w
chrf_avg = chrf_w / total_w
return {
"em": em_avg,
"chrf": chrf_avg,
"score": math.sqrt(em_avg * chrf_avg),
"n_items": n_items,
}

247
solver/numerals.py Normal file
View File

@@ -0,0 +1,247 @@
"""Numeral system induction: recover morpheme values + combination structure
from attested (numeral phrase, integer) pairs, then convert both directions.
Model (covers the large majority of IOL numeral systems):
value(phrase) = fold over tokens, where adjacent (multiplier, base-power)
groups combine multiplicatively and groups combine additively — i.e. the
standard "mixed-radix polynomial" reading: [2] [20] [3] -> 2*20 + 3.
Some systems are subtractive or overcounting; a signed-additive variant is
also searched. Token values are solved by constraint search: each distinct
token gets an unknown integer value; attested equations constrain them.
Search is tiny: numeral puzzles use ~5-15 morpheme types with values drawn
from {1..9, base, base^2, ...}. We enumerate candidate value sets per token
from divisors/residues of the attested numbers, then DFS with propagation.
"""
from __future__ import annotations
import re
from collections import Counter
from typing import Dict, List, Optional, Sequence, Tuple
from .preprocess import strip_punct, tokenize
BASES = (10, 20, 5, 12, 60, 4, 6, 8, 15)
MAX_TOKEN_VALUE = 10_000
def _norm_tokens(phrase: str) -> List[str]:
toks = []
for t in tokenize(phrase.casefold()):
t = strip_punct(t)
# split on hyphens: numeral compounds are often hyphenated
toks.extend([p for p in re.split(r"[-‑]", t) if p])
return toks
def _eval(vals: Sequence[int]) -> int:
"""Evaluate token values with the multiplicative-additive convention: a
smaller value directly before a larger one multiplies it; otherwise
values add. E.g. [2,20,3] -> 2*20+3 = 43; [3,100,20,7] -> 327."""
total = 0
cur = vals[0]
for prev, v in zip(vals, vals[1:]):
if v > prev:
cur = cur * v # e.g. 2 then 20 -> 40
else:
total += cur
cur = v
return total + cur
class NumeralSystem:
def __init__(self, values: Dict[str, int]):
self.values = dict(values)
def text_to_num(self, phrase: str) -> Optional[int]:
toks = _norm_tokens(phrase)
if not toks or any(t not in self.values for t in toks):
return None
return _eval([self.values[t] for t in toks])
def num_to_text(self, n: int, attested_phrases: Sequence[str]) -> Optional[str]:
"""Generate the phrase for n: enumerate token sequences (up to length
6) whose evaluation equals n, then pick the one most consistent with
the attested phrasing style (e.g. do multi-token phrases always give
a base its explicit multiplier, even 'one'?)."""
toks = sorted(self.values, key=lambda t: -self.values[t])
found: List[List[str]] = []
self._search(n, toks, [], 6, found, limit=16, budget=[100_000])
if not found:
return None
style = _StyleModel(self.values, attested_phrases)
found.sort(key=lambda seq: (-style.score(seq), len(seq)))
return " ".join(found[0])
def _search(self, target: int, toks: List[str], acc: List[str], depth: int,
found: List[List[str]], limit: int, budget: List[int]) -> None:
if len(found) >= limit or budget[0] <= 0:
return
budget[0] -= 1
if target == 0 and acc:
found.append(list(acc))
return
if depth == 0 or target <= 0:
return
for t in toks:
v = self.values[t]
if v > target:
continue
# multiplicative: k * v <= target with k attested as token
for m in toks:
mv = self.values[m]
if 1 <= mv < v and mv * v <= target:
self._search(target - mv * v, toks, acc + [m, t], depth - 2,
found, limit, budget)
self._search(target - v, toks, acc + [t], depth - 1, found, limit, budget)
class _StyleModel:
"""Scores a candidate numeral phrase by consistency with attested style:
(a) are base tokens (value >= 10) given an explicit smaller multiplier in
attested multi-token phrases? (b) reuse of attested token bigrams."""
def __init__(self, values: Dict[str, int], phrases: Sequence[str]):
self.values = values
self.bigrams = set()
obs: List[bool] = []
for ph in phrases:
toks = _norm_tokens(ph)
if not toks or any(t not in values for t in toks):
continue
self.bigrams.update(zip(toks, toks[1:]))
if len(toks) < 2:
continue
for i, t in enumerate(toks):
if values[t] >= 10:
obs.append(i > 0 and values[toks[i - 1]] < values[t])
self.prefer_explicit = sum(obs) > len(obs) / 2 if obs else False
def score(self, seq: Sequence[str]) -> float:
s = 0.0
s += 0.5 * sum(1 for bg in zip(seq, seq[1:]) if bg in self.bigrams)
base_seen: Counter = Counter()
if len(seq) >= 2:
for i, t in enumerate(seq):
if self.values[t] >= 10:
base_seen[t] += 1
explicit = i > 0 and self.values[seq[i - 1]] < self.values[t]
s += 1.0 if explicit == self.prefer_explicit else -1.0
# positional systems use each base power once; repeats are degenerate
s -= 2.0 * sum(c - 1 for c in base_seen.values())
return s
def induce(attested: List[Tuple[str, int]], max_candidates: int = 8) -> Optional[NumeralSystem]:
"""Induce token values from attested (phrase, value) pairs by DFS with
forward checking. Candidate values per token come from structural
positions: divisors of attested values, small digits, and base powers."""
eqs: List[Tuple[List[str], int]] = []
vocab: List[str] = []
for phrase, val in attested:
toks = _norm_tokens(phrase)
if not toks:
continue
eqs.append((toks, val))
for t in toks:
if t not in vocab:
vocab.append(t)
if not eqs:
return None
# Candidate values per token.
digits = set(range(1, 10))
base_powers = {b ** k for b in BASES for k in (1, 2, 3) if b ** k <= MAX_TOKEN_VALUE}
cands: Dict[str, List[int]] = {}
for t in vocab:
cs = set(digits) | base_powers
# a token appearing alone in an equation must equal that value
for toks, val in eqs:
if toks == [t]:
cs = {val}
break
if t in toks:
cs |= {val} | {d for d in _divisors(val) if d <= MAX_TOKEN_VALUE}
cands[t] = sorted(cs)
# Constraint propagation on short equations before search: a 1-token
# equation pins its token; a 2-token equation with one token pinned
# constrains the other to {V-a, V/a}.
changed = True
while changed:
changed = False
for toks, val in eqs:
unknown = [t for t in set(toks) if len(cands[t]) > 1]
if len(set(toks)) == 1:
t = toks[0]
if len(toks) == 1 and cands[t] != [val]:
cands[t] = [val]
changed = True
elif len(toks) == 2 and len(unknown) == 1:
t = unknown[0]
other = toks[0] if toks[1] == t else toks[1]
if len(cands[other]) == 1:
a = cands[other][0]
allowed = {val - a}
if a and val % a == 0:
allowed.add(val // a)
new = [v for v in cands[t] if v in allowed]
if new and new != cands[t]:
cands[t] = new
changed = True
# Order: most-constrained tokens first.
order = sorted(vocab, key=lambda t: len(cands[t]))
assignment: Dict[str, int] = {}
budget = {"nodes": 200_000}
def consistent() -> bool:
for toks, val in eqs:
if all(t in assignment for t in toks):
if _eval([assignment[t] for t in toks]) != val:
return False
return True
def dfs(i: int) -> bool:
if budget["nodes"] <= 0:
return False # search space too big — abstain, don't hang
if i == len(order):
return True
t = order[i]
for v in cands[t]:
budget["nodes"] -= 1
assignment[t] = v
if consistent() and dfs(i + 1):
return True
assignment.pop(t, None)
return False
if dfs(0) and budget["nodes"] > 0:
sys_ = NumeralSystem(assignment)
# verify every attested equation round-trips
if all(sys_.text_to_num(p) == v for p, v in attested):
return sys_
return None
def _divisors(n: int) -> List[int]:
n = abs(n)
out = []
for d in range(1, int(n ** 0.5) + 1):
if n % d == 0:
out += [d, n // d]
return sorted(set(out))
def extract_attested(pairs) -> List[Tuple[str, int]]:
"""From preprocess Pairs, pull (phrase, int) where one side is a number."""
out = []
for p in pairs:
for a, b in ((p.src, p.tgt), (p.tgt, p.src)):
bs = b.strip().replace(",", "").replace(" ", "")
if re.fullmatch(r"\d+", bs):
out.append((a, int(bs)))
break
return out

382
solver/pipeline.py Normal file
View File

@@ -0,0 +1,382 @@
"""Pipeline (shared by script.py and eval/dev_harness.py): symbolic pass, then
LLM answering (lean or scaffolded), merge, never-empty guarantee."""
from __future__ import annotations
import re
from dataclasses import dataclass, field
from typing import Callable, List, Optional, Sequence
from .budget import Budget
from .direct import align_answers, solve_direct
from .llm import LLMClient, NullClient
from .matching import solve_match_letters_llm
from .preprocess import Puzzle, parse_puzzle
from .router import solve_puzzle_ex
from .scaffold import build_scaffold, light_hint
CONF_KEEP = 0.5 # symbolic answers at/above this verifier fit override the LLM
LLM_BATCH = 4 # puzzles per generation batch (T4 KV-cache friendly)
DEEP_MODE_MAX_ROWS = 30 # at/below this many puzzles, spend more per puzzle
_DIGITS_RX = re.compile(r"\d[\d,. ]*")
_LETTER_RX = re.compile(r"\b([A-Z])\b")
_PREAMBLE_RX = re.compile(
r"^(?:the\s+)?(?:answer|translation|result)\s*(?:is|:)\s*", re.IGNORECASE)
_TERMINAL_PUNCT = (".", "!", "?")
@dataclass
class PuzzleResult:
row_id: str
answers: List[str]
explanation: str = ""
confs: List[float] = field(default_factory=list)
methods: List[str] = field(default_factory=list)
llm_used: bool = False
raw: str = ""
def clean_llm_answer(ans: str, task_type: str, option_labels: Sequence[str] = ()) -> str:
"""Per-task-type format guard on a parsed LLM answer line. Conservative:
only rewrites when the expected shape is unambiguous."""
a = _PREAMBLE_RX.sub("", ans.strip()).strip().strip("'\"“”")
if task_type == "text_to_num":
m = _DIGITS_RX.search(a)
if m:
digits = re.sub(r"[,. ]", "", m.group(0))
if digits.isdigit():
return digits
elif task_type == "match_letters":
if len(a) > 2: # "B. the bird sleeps" or "option B" -> "B"
candidates = _LETTER_RX.findall(a)
wanted = [c for c in candidates if not option_labels or c in option_labels]
if len(set(wanted)) == 1:
return wanted[0]
return a
def _vote(cands: Sequence[Optional[str]], anchor: Optional[str]) -> Optional[str]:
"""Greedy-anchored vote: keep `anchor` (the greedy answer) unless at least
two sampled candidates agree on the same normalised form AND that form
outnumbers the anchor's support. Monotone — it can only fire on genuine
agreement, so it never replaces greedy with a lone sample."""
from collections import Counter
cands = [c for c in cands if c and str(c).strip()]
if anchor is None:
anchor = cands[0] if cands else None
if len(cands) < 3:
return anchor
norm = lambda s: re.sub(r"\s+", " ", str(s).strip().lower())
groups: dict = {}
for c in cands:
groups.setdefault(norm(c), []).append(c)
anchor_support = len(groups.get(norm(anchor), [])) if anchor else 0
best_key = max(groups, key=lambda k: len(groups[k]))
if len(groups[best_key]) >= 2 and len(groups[best_key]) > anchor_support:
return Counter(groups[best_key]).most_common(1)[0][0]
return anchor
def induce_format(answer: str, puzzle: Puzzle, item_idx: int) -> str:
"""Nudge an answer toward the dataset's surface convention (fable §3):
if the attested answers on this item's side overwhelmingly end in a
terminal punctuation mark or start with a capital, mirror that. Converts
chrF-close answers into EM hits, which the geometric-mean scoring rewards
twice. Conservative: only fires on a near-unanimous (>=85%) convention,
only ADDS a missing terminal mark or leading capital, never strips."""
if puzzle.task_type not in ("translation", "fill_blanks"):
return answer
a = answer.strip()
if not a:
return answer
it = puzzle.items[item_idx] if item_idx < len(puzzle.items) else None
direction = getattr(it, "direction", None)
# answer side: to_work -> work-language (tgt); else task-language (src)
side = [p.tgt for p in puzzle.pairs] if direction == "to_work" \
else [p.src for p in puzzle.pairs]
side = [s.strip() for s in side if s and s.strip()]
if len(side) < 4:
return answer
n = len(side)
# terminal punctuation: only if a single mark dominates
for mark in _TERMINAL_PUNCT:
if sum(1 for s in side if s.endswith(mark)) / n >= 0.85:
if not a.endswith(_TERMINAL_PUNCT):
a = a + mark
break
# leading capitalization
if sum(1 for s in side if s[:1].isupper()) / n >= 0.85:
if a[:1].islower():
a = a[:1].upper() + a[1:]
return a
def _symbolic_explanation(puzzle: Puzzle, methods: Sequence[str]) -> str:
used = [m for m in dict.fromkeys(methods) if m not in ("none", "fallback")]
if not used:
return ("- Answered by nearest-attested analogy over the given examples "
"(no reliable rule could be verified).")
tmpl = {
"numeral system (verified)": (
"- Induced each morpheme's numeric value from the attested numerals "
"and verified the system reproduces every given example; applied it "
"to each query item (smaller-before-larger multiplies, otherwise "
"values add)."),
"table completion": (
"- Learned the mapping between the paradigm-table columns from the "
"attested rows (leave-one-out verified) and applied it to each "
"incomplete row."),
"template substitution": (
"- For each query, took the closest attested sentence and swapped "
"the differing words through morpheme alignments induced from "
"minimal pairs in the data."),
"induced grammar": (
"- Induced a lexicon and affix rules that reproduce the attested "
"pairs exactly, then applied them mechanically to the query items."),
"optimal matching": (
"- Scored every form-meaning pair by shared-morpheme/shared-word "
"consistency and picked the globally optimal assignment."),
}
return "\n".join(tmpl.get(m, f"- Solved by {m}.") for m in used)
def run_pipeline(rows: Sequence[dict], client: Optional[LLMClient] = None,
budget: Optional[Budget] = None, verbose: bool = True,
conf_keep: float = CONF_KEEP, llm_batch: int = LLM_BATCH,
max_new_tokens: Optional[int] = None,
checkpoint: Optional[Callable[[List["PuzzleResult"]], None]] = None,
lean: bool = False,
use_match_assignment: bool = True,
vote_samples: int = 0,
vote_temp: float = 0.5,
hint: bool = False
) -> List[PuzzleResult]:
"""`checkpoint`, when given, is called with the (complete, valid) results
after the symbolic pass and after every LLM batch — so a crash at ANY
later point still leaves a full submission on disk."""
client = client or NullClient()
budget = budget or Budget()
# adaptive mode: the hidden test is small (an IOL contest reformatted into
# a handful of multi-item rows), so default to spending more per puzzle.
# A large row count flips us to coverage mode (shorter generations, serve
# the most puzzles).
deep_mode = len(rows) <= DEEP_MODE_MAX_ROWS
if max_new_tokens is None:
max_new_tokens = 2048 if deep_mode else 1024
def log(msg: str) -> None:
if verbose:
print(msg, flush=True)
def save() -> None:
if checkpoint is not None:
try:
checkpoint(results)
except Exception:
pass
# ---- 1. symbolic pass ----
results: List[PuzzleResult] = []
puzzles: List[Optional[Puzzle]] = []
for i, row in enumerate(rows):
rid = str(row.get("id", i))
try:
p = parse_puzzle(row)
answers, confs, methods = solve_puzzle_ex(p, NullClient(), budget,
puzzles_left=len(rows) - i)
except Exception:
p = None
answers, confs, methods = [str(row.get("query", "?")).strip() or "?"], [0.0], ["fallback"]
puzzles.append(p)
results.append(PuzzleResult(rid, answers, "", confs, methods))
# make every result submission-valid NOW (explanations + non-empty), so
# each checkpoint from here on is a complete fallback submission
for i, r in enumerate(results):
r.answers = [str(a).strip() or "?" for a in r.answers]
r.explanation = (_symbolic_explanation(puzzles[i], r.methods)
if puzzles[i] is not None else
"- No parseable structure found; answered by "
"closest-example analogy.")
log(f"symbolic pass done in {budget.elapsed():.1f}s")
save()
# arm the per-token wall-clock abort so generation can't overrun the budget
try:
client.deadline = budget.start + budget.total - budget.safety
except Exception:
pass
# ---- 1b. match_letters assignment pass (model logprobs -> Hungarian) ----
# Free-form generation answers match_letters with the identity permutation
# (scores ~0). Solve it as an assignment from the model's own distribution
# instead. Solved puzzles get high confidence so the free-form LLM pass
# skips them; a declined puzzle falls through to that pass unchanged.
# Gated by use_match_assignment: when off, match_letters puzzles go through
# the normal free-form LLM pass.
if not use_match_assignment:
log("match_letters assignment pass disabled; using free-form LLM path")
if use_match_assignment and getattr(client, "can_score", False):
n_assigned = 0
for i, p in enumerate(puzzles):
if p is None or p.task_type != "match_letters" or budget.exhausted():
continue
try:
letters = solve_match_letters_llm(p, client)
except Exception as e:
log(f" match_letters solver failed on {results[i].row_id}: "
f"{type(e).__name__}: {e}")
letters = None
r = results[i]
if letters and len(letters) == len(r.answers):
r.answers = [str(x).strip() or "?" for x in letters]
r.confs = [0.9] * len(letters)
r.methods = ["llm-assignment"] * len(letters)
r.llm_used = True
n_assigned += 1
if n_assigned:
log(f"match_letters assignment solver used on {n_assigned} puzzle(s)")
save()
# ---- 2. scaffolded LLM pass ----
# In lean mode symbolic is a pure fallback: the model answers EVERY puzzle
# and its answer wins wherever it produced one (symbolic stands only for the
# items it left blank). Otherwise the model runs only on low-confidence
# puzzles and verified symbolic answers override it.
if lean:
need = [i for i in range(len(results)) if puzzles[i] is not None]
else:
need = [i for i, r in enumerate(results)
if puzzles[i] is not None and any(c < conf_keep for c in r.confs)]
if deep_mode:
# weakest-first: a budget cutoff then drops the puzzles we could help least
need.sort(key=lambda i: sum(min(c, conf_keep) for c in results[i].confs)
/ max(len(results[i].confs), 1))
else:
# coverage: shortest prompt first maximizes puzzles served per second
need.sort(key=lambda i: len(puzzles[i].context) + len(puzzles[i].query))
from .direct import LEAN_SYSTEM, SYSTEM
sys_prompt = LEAN_SYSTEM if lean else SYSTEM
log(f"LLM pass ({'lean' if lean else 'scaffold'}, "
f"{'deep' if deep_mode else 'coverage'}, "
f"max_new_tokens={max_new_tokens}): {len(need)}/{len(rows)} puzzles "
f"need the model; client={'yes' if client.available else 'no'}")
parse_ok = parse_fail = 0
if client.available and need:
done = 0
for start in range(0, len(need), llm_batch):
if budget.exhausted():
log(f"budget cutoff after {done} LLM puzzles")
break
batch = need[start : start + llm_batch]
if lean:
# lean: no scaffold, unless the optional light hint is enabled
scaffolds = [light_hint(puzzles[i]) if hint else "" for i in batch]
else:
scaffolds = []
for i in batch:
p, r = puzzles[i], results[i]
try:
scaffolds.append(build_scaffold(p, r.answers, r.confs, r.methods))
except Exception:
scaffolds.append("")
try:
outs = solve_direct([puzzles[i] for i in batch], client, scaffolds,
max_new_tokens=max_new_tokens, system=sys_prompt,
lean=lean)
except Exception as e:
# systemic generation failure (bad load, driver, etc.) — the
# symbolic answers already on every item are the submission
log(f"LLM batch failed ({type(e).__name__}: {e}); "
f"keeping symbolic answers for the rest")
break
for i, (direct, expl, raw, found) in zip(batch, outs):
r = results[i]
p = puzzles[i]
r.llm_used = True
r.raw = raw
if found:
parse_ok += 1
else:
# no answer block parsed even after salvage: do NOT let
# reasoning prose overwrite the symbolic answers
parse_fail += 1
continue
labels = sorted(p.lettered) if p.lettered else ()
aligned = align_answers(direct, len(r.answers))
for j, d in enumerate(aligned):
# lean: the model's answer wins wherever it gave one;
# scaffold: only where symbolic isn't confident
if d and (lean or r.confs[j] < conf_keep):
cleaned = clean_llm_answer(d, p.task_type, labels)
if cleaned:
# lean mode ships the model's answer as-is (no
# punctuation/casing induction)
if not lean:
cleaned = induce_format(cleaned, p, j)
r.answers[j] = cleaned.strip() or r.answers[j]
r.methods[j] = "llm"
if expl:
r.explanation = expl
done += len(batch)
log(f" llm {done}/{len(need)} ok={parse_ok} fail={parse_fail} "
f"t={budget.elapsed():.0f}s")
save()
log(f"LLM pass done: {parse_ok} parsed, {parse_fail} unparsable "
f"(kept symbolic); {budget.elapsed():.0f}s elapsed")
# ---- 2b. light greedy-anchored self-consistency voting (lean only) ----
# The greedy answers are already checkpointed; sampled passes can only
# displace an item on genuine agreement (see _vote), so this is monotone and
# budget-gated — if the clock runs out we simply keep the greedy answers.
if lean and vote_samples > 0 and client.available and need and not budget.exhausted():
ballots = {i: [list(results[i].answers)] for i in need} # greedy = ballot 0
done_votes = 0
for _s in range(vote_samples):
if budget.exhausted():
break
failed = False
for start in range(0, len(need), llm_batch):
if budget.exhausted():
break
batch = need[start : start + llm_batch]
try:
outs = solve_direct([puzzles[i] for i in batch], client,
["" for _ in batch],
max_new_tokens=max_new_tokens,
system=sys_prompt, lean=True,
sample=True, temperature=vote_temp)
except Exception as e:
log(f"vote pass failed ({type(e).__name__}: {e}); "
f"keeping greedy answers")
failed = True
break
for i, (direct, _e, _raw, _f) in zip(batch, outs):
p = puzzles[i]
labels = sorted(p.lettered) if p.lettered else ()
ballot = []
for j in range(len(results[i].answers)):
d = direct[j] if j < len(direct) else None
ballot.append(clean_llm_answer(d, p.task_type, labels)
if d else None)
ballots[i].append(ballot)
if failed:
break
done_votes += 1
for i in need: # re-vote (greedy-anchored) after each sample pass
r = results[i]
anchor = ballots[i][0]
for j in range(len(r.answers)):
v = _vote([b[j] for b in ballots[i] if j < len(b)],
anchor[j] if j < len(anchor) else None)
if v and str(v).strip():
r.answers[j] = str(v).strip()
save()
log(f"voting: {done_votes}/{vote_samples} sample pass(es); "
f"t={budget.elapsed():.0f}s")
# ---- 3. final never-empty guarantee ----
for r in results:
r.answers = [str(a).strip() or "?" for a in r.answers]
return results

384
solver/preprocess.py Normal file
View File

@@ -0,0 +1,384 @@
"""Parse Linguini puzzles: normalize text, parse the context (pipe tables,
numbered/lettered lists, pairs) and the query into answerable items."""
from __future__ import annotations
import re
import unicodedata
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Tuple
_ITEM_PREFIX = re.compile(r"^\s*\(?(\d{1,3})[\.\)]\s+")
_LETTER_PREFIX = re.compile(r"^\s*\(?([A-Z])[\.\)]\s+")
_BLANK_MARK = re.compile(r"\((\d{1,3})\)")
_BLANK_LINE = re.compile(r"_{2,}|…|\.{4,}")
# non-pipe two-side separators, tried in order on non-table lines
_SEPARATORS = [
("tab", re.compile(r"\t+")),
("equals", re.compile(r"\s+=\s+")),
("emdash", re.compile(r"\s+—\s+")),
("endash", re.compile(r"\s+–\s+")),
("arrow", re.compile(r"\s*(?:->|→)\s*")),
("hyphen", re.compile(r"\s+-\s+")),
("means", re.compile(r"\s+means\s+", re.IGNORECASE)),
]
# common work-language names (queries say "Translate into English:")
_WORK_LANG_NAMES = {
"eng": "english", "fra": "french", "spa": "spanish", "por": "portuguese",
"rus": "russian", "deu": "german",
}
def normalize(text: str) -> str:
"""NFC-normalize, unify exotic whitespace/quotes. Keeps diacritics, tone
marks, case, and punctuation (EM comparison is punctuation-sensitive)."""
if text is None:
return ""
t = unicodedata.normalize("NFC", str(text))
t = t.replace(" ", " ")
t = re.sub(r"[ \t]+", " ", t)
return t.strip()
def tokenize(s: str) -> List[str]:
"""Unicode word tokenization. Keeps combining marks, word-internal
apostrophes/hyphens, and subscript/superscript markers (tone letters,
person markers like you_{sg})."""
s = normalize(s)
return re.findall(r"[^\s,;.!?()\[\]\"«»|]+", s)
def strip_punct(tok: str) -> str:
return tok.strip(",;.!?()[]\"«»").strip()
@dataclass
class Pair:
src: str # task-language side by convention
tgt: str # work-language side (gloss/translation)
sep: str = ""
line_no: int = -1
label: str = "" # numbered prefix if the line carried one
@dataclass
class QueryItem:
number: str # label as it appeared ("17", "3", "") — "" for bare lines
text: str
direction: Optional[str] = None # "to_task" | "to_work" | None
has_blank: bool = False
row: Optional[List[str]] = None # for table-blank items: full row cells
blank_col: Optional[int] = None # which cell holds this item's (k) marker
@dataclass
class Puzzle:
id: str
context: str
query: str
work_lang: str = ""
task_lang: str = ""
task_type: str = ""
eval_type: str = ""
pairs: List[Pair] = field(default_factory=list)
items: List[QueryItem] = field(default_factory=list)
hints: List[str] = field(default_factory=list)
tables: List[List[List[str]]] = field(default_factory=list) # blocks of rows of cells
numbered: Dict[str, str] = field(default_factory=dict) # "1" -> form (list contexts)
lettered: Dict[str, str] = field(default_factory=dict) # "A" -> meaning
def _split_cells(line: str) -> List[str]:
return [c.strip() for c in line.split("|")]
def _strip_item_prefix(line: str) -> Tuple[str, str]:
"""Returns (label, rest). Label may be a number or capital letter."""
m = _ITEM_PREFIX.match(line)
if m:
return m.group(1), line[m.end():].strip()
m = _LETTER_PREFIX.match(line)
if m:
return m.group(1), line[m.end():].strip()
return "", line.strip()
def _looks_header(cells: List[str]) -> bool:
"""A table header names languages/columns: 'Proto-Chamic | Tsat | meaning'."""
if len(cells) < 2:
return False
tail = cells[-1].lower()
if tail in ("meaning", "meanings", "translation", "translations", "english",
"value", "values", "gloss"):
return True
# all cells capitalized single-ish words with no digits — likely names
ok = 0
for c in cells:
if c and not any(ch.isdigit() for ch in c) and c[0].isupper() and len(c.split()) <= 3:
ok += 1
return ok == len(cells) and len(cells) >= 3
def parse_context(ctx: str) -> Tuple[List[Pair], List[str], List[List[List[str]]], Dict[str, str], Dict[str, str]]:
"""Parse context into (pairs, hints, tables, numbered, lettered)."""
pairs: List[Pair] = []
hints: List[str] = []
tables: List[List[List[str]]] = []
numbered: Dict[str, str] = {}
lettered: Dict[str, str] = {}
cur_table: List[List[str]] = []
for i, raw in enumerate(str(ctx).splitlines()):
line = normalize(raw)
if not line:
if cur_table:
tables.append(cur_table)
cur_table = []
continue
label, body = _strip_item_prefix(line)
if "|" in body:
cells = _split_cells(body)
if _looks_header(cells) and not cur_table:
hints.append(line)
continue
cur_table.append(cells)
has_blank = bool(_BLANK_MARK.search(body))
if len(cells) >= 2 and cells[0] and cells[-1] and not has_blank:
pairs.append(Pair(src=cells[0], tgt=cells[-1], sep="pipe",
line_no=i, label=label))
if label and not has_blank:
numbered[label] = cells[0]
continue
if cur_table:
tables.append(cur_table)
cur_table = []
# non-pipe separators (= , — , tab ...)
matched = False
for name, rx in _SEPARATORS:
parts = rx.split(body, maxsplit=1)
if len(parts) == 2 and parts[0].strip() and parts[1].strip():
pairs.append(Pair(src=parts[0].strip(), tgt=parts[1].strip(),
sep=name, line_no=i, label=label))
if label:
# the full line is the referable entry ("equalities (1-9)")
numbered[label] = body
matched = True
break
if matched:
continue
# single-column list entries (match_letters forms/meanings)
if label:
if label.isdigit():
numbered[label] = body
else:
lettered[label] = body
continue
hints.append(line)
if cur_table:
tables.append(cur_table)
return pairs, hints, tables, numbered, lettered
_INSTRUCTION_VERBS = (
r"(translate|fill|write|spell|determine|give|complete|convert|match|answer|"
r"say|pair|transcribe|provide|express|render|decipher|find|identify|"
r"choose|select|here|below|these|the following)"
)
_INSTRUCTION_RX = re.compile(r"^" + _INSTRUCTION_VERBS + r"\b", re.IGNORECASE)
_INSTRUCTION_ANY_RX = re.compile(r"\b" + _INSTRUCTION_VERBS + r"\b", re.IGNORECASE)
def _is_instruction(line: str) -> bool:
"""Instruction lines are work-language imperatives ("Translate into X:").
Matching is verb-anchored — a bare trailing colon is NOT enough, because
task-language forms can end in ':' (length marks: "si teŋku bugdiŋi:").
A line that ends with ':' AND contains an instruction verb anywhere is
also an instruction ("In Drehu tusi is 'book'. Translate from Drehu:")."""
line = (line or "").strip()
if not line:
return False
if _BLANK_MARK.search(line) or "|" in line:
return False
if _INSTRUCTION_RX.match(line):
return True
return line.endswith(":") and bool(_INSTRUCTION_ANY_RX.search(line))
def parse_query(query: str) -> Tuple[List[QueryItem], List[str]]:
"""Split query into answerable items + instruction lines.
Item sources, in the order encountered:
- (k)-markers inside lines (usually pipe rows): one item per marker, with
the row cells and blank column recorded;
- numbered lines "17. ..." (numbering may continue the context's);
- bare non-instruction lines: one item per line.
"""
text = str(query or "")
items: List[QueryItem] = []
instructions: List[str] = []
_TERMINAL = (".", "!", "?", ":", ";", '"', "”", "’")
for raw in text.splitlines():
line = normalize(raw)
if not line:
continue
marks = _BLANK_MARK.findall(line)
if marks:
cells = _split_cells(line) if "|" in line else [line]
for k in marks:
blank_col = next(
(ci for ci, c in enumerate(cells) if f"({k})" in c), None)
items.append(QueryItem(
number=k, text=line, has_blank=True,
row=cells if len(cells) > 1 else None, blank_col=blank_col))
continue
if "|" in line:
label, body = _strip_item_prefix(line)
if label:
# numbered table row = one item; the answer fills whichever
# column the context table has that this row lacks
items.append(QueryItem(number=label, text=body,
row=_split_cells(body)))
else:
instructions.append(line) # header/echo row
continue
label, body = _strip_item_prefix(line)
if label:
items.append(QueryItem(number=label, text=body,
has_blank=bool(_BLANK_LINE.search(body))))
continue
if _is_instruction(line):
instructions.append(line)
continue
if items and items[-1].number and not items[-1].text.rstrip().endswith(_TERMINAL):
items[-1].text += " " + line # wrapped continuation of a numbered item
continue
items.append(QueryItem(number="", text=line,
has_blank=bool(_BLANK_LINE.search(line))))
# when the query has numbered items, stray unnumbered lines around them
# are notes ("spoken on Bvuŋkaden"), not answerable items
if any(it.number for it in items):
items = [it for it in items if it.number]
# items with numeric labels answer in label order when labels are complete
if items and all(it.number.isdigit() for it in items):
items.sort(key=lambda it: int(it.number))
return items, instructions
def detect_direction(item_text: str, task_material: str, work_material: str,
instructions: List[str], work_lang: str) -> str:
"""Per-item direction: does the answer belong to the task language
('to_task') or the work language ('to_work')?
1. Explicit instruction: "into English" (work-lang name) vs "into X".
2. Script similarity: if the item text overlaps the task-language material
character-wise, it is task-language text needing analysis (to_work).
"""
joined = (" ".join(instructions) + " " + item_text).lower()
wl_name = _WORK_LANG_NAMES.get(work_lang.split("_")[0][:3].lower(), "")
m = re.search(r"(?:into|in|to)\s+(?:the\s+)?([A-Za-zÀ-ž’' -]{2,30}?)\s*(?:language)?\s*[:.]", joined + ":")
if m:
named = m.group(1).strip().lower()
if wl_name and wl_name in named:
return "to_work"
if named and not any(w in named for w in ("digit", "numeral", "number", "blank")):
return "to_task"
sim_task = _char_overlap(item_text, task_material)
sim_work = _char_overlap(item_text, work_material)
return "to_work" if sim_task >= sim_work else "to_task"
def _char_overlap(s: str, material: str, n: int = 3) -> float:
s_ = "".join(s.lower().split())
m_ = "".join(material.lower().split())
if len(s_) < n or len(m_) < n:
return 0.0
grams = {s_[i : i + n] for i in range(len(s_) - n + 1)}
hits = sum(1 for g in grams if g in m_)
return hits / len(grams)
_RANGE_RX = re.compile(r"\((\d{1,3})\s*[–—-]\s*(\d{1,3})\)")
def _items_from_context(p: Puzzle) -> List[QueryItem]:
"""When the query is instruction-only ("Fill in the blanks (1–14)",
"Determine the correct correspondences", "Write the equalities (1–9) in
numerals"), the answerable items live in the CONTEXT: (k) blank markers,
or the numbered list entries. Last resort: the query itself is one item."""
rng = _RANGE_RX.search(p.query or "")
lo, hi = (int(rng.group(1)), int(rng.group(2))) if rng else (None, None)
def in_range(k: str) -> bool:
return lo is None or (k.isdigit() and lo <= int(k) <= hi)
ctx_blanks: List[QueryItem] = []
for raw in str(p.context).splitlines():
line = normalize(raw)
for k in _BLANK_MARK.findall(line):
if not in_range(k):
continue
cells = _split_cells(line) if "|" in line else [line]
blank_col = next((ci for ci, c in enumerate(cells) if f"({k})" in c), None)
ctx_blanks.append(QueryItem(number=k, text=line, has_blank=True,
row=cells if len(cells) > 1 else None,
blank_col=blank_col))
if ctx_blanks:
ctx_blanks.sort(key=lambda it: int(it.number))
return ctx_blanks
if p.numbered and (p.task_type == "match_letters" or rng or p.lettered):
keys = sorted((k for k in p.numbered if in_range(k)), key=int)
if keys:
return [QueryItem(number=k, text=p.numbered[k]) for k in keys]
q = normalize(p.query)
return [QueryItem(number="", text=q)] if q else []
def parse_puzzle(row: dict) -> Puzzle:
"""Build a Puzzle from a CSV/dataset row (id, context, query, work_lang,
task_lang, task_type, eval_type)."""
ctx = str(row.get("context", "") or "")
p = Puzzle(
id=str(row.get("id", "")),
context=ctx,
query=str(row.get("query", "") or ""),
work_lang=str(row.get("work_lang", "") or ""),
task_lang=str(row.get("task_lang", "") or ""),
task_type=str(row.get("task_type", "") or "").strip().lower(),
eval_type=str(row.get("eval_type", "") or ""),
)
p.pairs, p.hints, p.tables, p.numbered, p.lettered = parse_context(ctx)
p.items, instructions = parse_query(p.query)
p.hints.extend(instructions)
# letter-labelled query entries are answer OPTIONS when digit-labelled
# items coexist (match tasks list both: "19. form ... S. meaning")
digit_items = [it for it in p.items if it.number.isdigit()]
letter_items = [it for it in p.items if it.number and not it.number.isdigit()]
if digit_items and letter_items:
for it in letter_items:
p.lettered[it.number] = it.text
p.items = digit_items
if not p.items:
p.items = _items_from_context(p)
task_material = " ".join(x.src for x in p.pairs) + " " + " ".join(p.numbered.values())
work_material = " ".join(x.tgt for x in p.pairs) + " " + " ".join(p.lettered.values())
for it in p.items:
if it.row is not None:
continue # table-blank items get direction from their row in the router
it.direction = detect_direction(it.text, task_material, work_material,
instructions, p.work_lang)
return p

226
solver/router.py Normal file
View File

@@ -0,0 +1,226 @@
"""Router: dispatch each puzzle to its symbolic solver (numerals, matching,
tables, translation) and return one answer per item. Never raises or empties."""
from __future__ import annotations
import re
from typing import Callable, List, Optional, Tuple
from .budget import Budget
from .fallback import ensure_nonempty, fallback_answer
from .llm import LLMClient, NullClient
from .matching import solve_matching
from .numerals import extract_attested, induce
from .preprocess import Pair, Puzzle, QueryItem, normalize
from .synth import synthesize
from .tables import TableSolver
from .template import TemplateTranslator
from .verifier import evaluate, leave_one_out
_NUM_RX = re.compile(r"\d+")
_QUOTED = re.compile(r"[\"«]([^\"»]+)[\"»]|‘([^’]{2,})’|'([^']{2,})'")
def _payload(text: str) -> str:
"""Payload of a whole-query item: quoted material, else text after a
colon, else the final word of an instruction-like sentence."""
t = normalize(text)
m = _QUOTED.search(t)
if m:
return next(g for g in m.groups() if g).strip()
if ":" in t:
tail = t.split(":", 1)[1].strip()
if tail:
return tail
m2 = re.match(r"^(give|translate|write|say|transcribe)\b.*\b(?:word|numeral|phrase|form)\s+(\S+)\s*$",
t, re.IGNORECASE)
if m2:
return m2.group(2).strip(".?!")
return t
def solve_puzzle(puzzle: Puzzle, client: Optional[LLMClient] = None,
budget: Optional[Budget] = None, puzzles_left: int = 1) -> List[str]:
return solve_puzzle_ex(puzzle, client, budget, puzzles_left)[0]
def solve_puzzle_ex(puzzle: Puzzle, client: Optional[LLMClient] = None,
budget: Optional[Budget] = None, puzzles_left: int = 1
) -> Tuple[List[str], List[float], List[str]]:
"""Returns (answers, confidences, methods). Confidence is the verifier
evidence behind each answer (LOO/eval fit of the solver that produced it,
or ~1.0 for round-trip-verified numeral systems); 0.0 marks answers that
came from the never-empty fallback ladder — those are the items worth LLM
budget. Methods name the producing solver, for prompt candidate blocks
and explanation-track traces."""
client = client or NullClient()
budget = budget or Budget()
items = puzzle.items or [QueryItem(number="", text=puzzle.query or "")]
try:
answers, confs, methods = _dispatch(puzzle, items, client, budget, puzzles_left)
except Exception:
answers, confs, methods = None, None, None
if answers is None:
answers = [None] * len(items)
if confs is None:
confs = [0.0] * len(items)
if methods is None:
methods = ["none"] * len(items)
answers = (list(answers) + [None] * len(items))[: len(items)]
confs = (list(confs) + [0.0] * len(items))[: len(items)]
methods = (list(methods) + ["none"] * len(items))[: len(items)]
out = []
for i, (item, ans) in enumerate(zip(items, answers)):
if ans is None or not str(ans).strip():
confs[i] = 0.0
methods[i] = "fallback"
direction = item.direction or "to_work"
out.append(ensure_nonempty(ans, _payload(item.text), puzzle.pairs, direction))
return out, confs, methods
def _dispatch(puzzle: Puzzle, items: List[QueryItem], client: LLMClient,
budget: Budget, left: int
) -> Tuple[List[Optional[str]], List[float], List[str]]:
tt = puzzle.task_type
if tt in ("text_to_num", "num_to_text"):
return _solve_numerals(puzzle, items)
if tt == "match_letters":
return _solve_matching(puzzle, items)
table = TableSolver(puzzle)
answers: List[Optional[str]] = [None] * len(items)
confs: List[float] = [0.0] * len(items)
methods: List[str] = ["none"] * len(items)
plain_idx = []
for i, it in enumerate(items):
if it.row is not None and table.usable:
ans_conf = table.solve(it)
if ans_conf is not None:
answers[i], confs[i] = ans_conf
methods[i] = "table completion"
if answers[i] is None:
plain_idx.append(i)
if plain_idx:
translated, t_confs, t_methods = _solve_translation(
puzzle, [items[i] for i in plain_idx], client, budget, left)
for i, ans, c, m in zip(plain_idx, translated, t_confs, t_methods):
answers[i], confs[i], methods[i] = ans, c, m
return answers, confs, methods
# ---------------------------------------------------------------- numerals
def _solve_numerals(puzzle: Puzzle, items: List[QueryItem]
) -> Tuple[List[Optional[str]], List[float]]:
attested = extract_attested(puzzle.pairs)
system = induce(attested) if attested else None
out: List[Optional[str]] = []
for item in items:
text = _payload(item.text)
if puzzle.task_type == "text_to_num":
val = system.text_to_num(text) if system else None
out.append(str(val) if val is not None else None)
else:
m = _NUM_RX.search(text)
if system and m:
out.append(system.num_to_text(int(m.group(0)), [p for p, _ in attested]))
else:
out.append(None)
# an induced system is round-trip verified on every attested equation
confs = [0.9 if a is not None else 0.0 for a in out]
methods = ["numeral system (verified)" if a is not None else "none" for a in out]
return out, confs, methods
# ---------------------------------------------------------------- matching
def _solve_matching(puzzle: Puzzle, items: List[QueryItem]
) -> Tuple[List[Optional[str]], List[float]]:
"""Items are forms; options are the lettered meanings (from context or
query). Answers are option letters when options exist, else the matched
meaning text."""
forms = [_payload(it.text) for it in items]
if puzzle.lettered:
labels = sorted(puzzle.lettered)
meanings = [puzzle.lettered[l] for l in labels]
else:
labels = None
meanings = [p.tgt for p in puzzle.pairs]
if not meanings:
return [None] * len(forms), [0.0] * len(forms), ["none"] * len(forms)
matched = dict(solve_matching(forms, meanings, puzzle.pairs))
out: List[Optional[str]] = []
for f in forms:
m = matched.get(f)
if m and labels:
out.append(labels[meanings.index(m)])
else:
out.append(m)
# Hungarian is optimal for its score matrix, but the matrix itself is only
# as good as the alignment evidence behind it — real-data EM is low, so
# this stays BELOW the pipeline's keep-threshold: it surfaces as a
# candidate hint in the LLM prompt rather than overriding the LLM
conf = 0.45 if puzzle.pairs else 0.25
return (out, [conf if a else 0.0 for a in out],
["optimal matching" if a else "none" for a in out])
# ------------------------------------------------------------- translation
def _solve_translation(puzzle: Puzzle, items: List[QueryItem], client: LLMClient,
budget: Budget, left: int
) -> Tuple[List[Optional[str]], List[float]]:
directions = {it.direction or "to_work" for it in items}
primary = "to_task" if "to_task" in directions else "to_work"
rounds = budget.cegis_rounds(left) if budget.allow_llm(left) else -1
synth_res = synthesize(puzzle, client, primary, rounds) if rounds >= 0 else None
solvers = {d: _pick_direction_solver(puzzle, synth_res, d) for d in directions}
answers, confs, methods = [], [], []
for it in items:
ans, conf, method = solvers[it.direction or "to_work"](_payload(it.text))
answers.append(ans)
confs.append(conf)
methods.append(method)
return answers, confs, methods
def _pick_direction_solver(puzzle: Puzzle, synth_res, d: str) -> Callable[[str], Optional[str]]:
"""Rank candidate solvers honestly and chain them (first non-None answer
wins). The grammar is a fixed program so it is evaluated directly on the
attested pairs (it must reproduce them); the template translator and the
fallback are *fit from* those pairs (they memorize them), so they are
scored leave-one-out — otherwise memorization would always beat a
generalizing grammar."""
attested = [(p.tgt, p.src) if d == "to_task" else (p.src, p.tgt) for p in puzzle.pairs]
def _subset(held_in_pairs) -> List[Pair]:
keep = set(held_in_pairs)
return [p for p in puzzle.pairs
if ((p.tgt, p.src) if d == "to_task" else (p.src, p.tgt)) in keep]
ranked: List[Tuple[float, int, str, Callable[[str], Optional[str]]]] = []
if synth_res and synth_res.interpreter:
fn = synth_res.interpreter.generate if d == "to_task" else synth_res.interpreter.analyze
v = evaluate(fn, attested, synth_res.grammar.mdl())
ranked.append((v.score, 0, "induced grammar", fn))
tmpl = TemplateTranslator(puzzle.pairs, d)
v_tmpl = leave_one_out(lambda held: TemplateTranslator(_subset(held), d).translate, attested)
ranked.append((v_tmpl.score, 1, "template substitution", tmpl.translate))
v_fb = leave_one_out(lambda held: (lambda q, kept=_subset(held): fallback_answer(q, kept, d)), attested)
ranked.append((v_fb.score, 2, "nearest attested", lambda q: fallback_answer(q, puzzle.pairs, d)))
ranked.sort(key=lambda t: (-t[0], t[1]))
def solve(q: str) -> Tuple[Optional[str], float, str]:
for score, _, name, fn in ranked:
ans = fn(q)
if ans:
return ans, max(score, 0.0), name
return None, 0.0, "none"
return solve

113
solver/scaffold.py Normal file
View File

@@ -0,0 +1,113 @@
"""Deterministic analysis blocks (segmentation, alignment, numerals) for the
prompt: the full scaffold and the optional light hint."""
from __future__ import annotations
from typing import Dict, List, Optional, Sequence, Tuple
from .align import align as build_align
from .numerals import extract_attested, induce
from .preprocess import Puzzle, strip_punct, tokenize
from .segment import Segmenter
MAX_SEG_LINES = 40
MAX_ALIGN_LINES = 30
MAX_VOCAB = 60
def analysis_blocks(puzzle: Puzzle) -> Tuple[str, str]:
"""(segmentation block, alignment block) for the prompt. Also used by the
CEGIS proposer (synth.py)."""
amap = build_align(puzzle.pairs)
align_lines = []
for tok, cands in sorted(amap.items()):
top = [f"{t} ({s:.1f})" for t, s in cands[:2] if s > 0.2]
if top:
align_lines.append(f" {tok} ~ {', '.join(top)}")
vocab, groups = [], {}
for p in puzzle.pairs:
for t in tokenize(p.src):
t = strip_punct(t).casefold()
if t and t not in vocab:
vocab.append(t)
vocab = vocab[:MAX_VOCAB]
for tok, cands in amap.items():
if cands:
groups.setdefault(cands[0][0], set()).add(tok)
seg = Segmenter().fit(vocab, share_groups=[g for g in groups.values() if len(g) > 1])
seg_lines = []
for w in vocab:
parts = seg.segment(w)
if len(parts) > 1:
seg_lines.append(f" {w} = {'-'.join(parts)}")
return ("\n".join(seg_lines[:MAX_SEG_LINES]) or " (none found)",
"\n".join(align_lines[:MAX_ALIGN_LINES]) or " (none found)")
def light_hint(puzzle: Puzzle, max_lines: int = 10) -> str:
"""A minimal, optional hint for the lean prompt: a few morpheme segmentation
guesses, framed as fallible. Off by default; enabled via a toggle."""
try:
seg_block, _align = analysis_blocks(puzzle)
except Exception:
return ""
lines = [l for l in seg_block.splitlines() if l.strip() and "none found" not in l]
if not lines:
return ""
body = "\n".join(lines[:max_lines])
return ("Optional hint (an automatic guess at word parts; it may be wrong, "
"so rely on the data itself):\n" + body)
def numeral_block(puzzle: Puzzle) -> str:
"""Induced numeral-system values, when the CSP solved and round-trip
verified them — the strongest kind of hint we can give."""
if puzzle.task_type not in ("text_to_num", "num_to_text"):
return ""
attested = extract_attested(puzzle.pairs)
system = induce(attested) if attested else None
if system is None:
return ""
vals = ", ".join(f"{t}={v}" for t, v in sorted(system.values.items(), key=lambda kv: kv[1]))
return (f"Numeral analysis (verified against every attested example):\n {vals}\n"
f" combination rule: a smaller value directly before a larger one multiplies it; "
f"otherwise values add.")
def candidate_block(items_answers: Sequence[Tuple[str, Optional[str], float, str]]) -> str:
"""Symbolic candidate answers per item: (item label, answer, confidence,
method). Only candidates with real evidence are shown — a low-confidence
echo would anchor the model on garbage."""
lines = []
for label, ans, conf, method in items_answers:
if ans and conf >= 0.4:
lines.append(f" item {label}: '{ans}' (source: {method}, fit {conf:.2f})")
if not lines:
return ""
return ("Candidate answers from mechanical analysis (adopt if consistent with "
"the data, correct if not):\n" + "\n".join(lines))
def build_scaffold(puzzle: Puzzle,
answers: Optional[Sequence[Optional[str]]] = None,
confs: Optional[Sequence[float]] = None,
methods: Optional[Sequence[str]] = None) -> str:
"""Full scaffold block for one puzzle's prompt."""
seg_block, align_block = analysis_blocks(puzzle)
parts = [
"## Mechanical analysis (computed from the data above; may contain errors — "
"the attested data always wins)",
f"Morpheme segmentation hypotheses:\n{seg_block}",
f"Word alignment hypotheses (task-language token ~ likely meaning):\n{align_block}",
]
nb = numeral_block(puzzle)
if nb:
parts.append(nb)
if answers is not None and confs is not None:
labels = [it.number or str(i + 1) for i, it in enumerate(puzzle.items)]
meths = list(methods) if methods else ["symbolic"] * len(labels)
cb = candidate_block(list(zip(labels, answers, confs, meths)))
if cb:
parts.append(cb)
return "\n\n".join(parts)

149
solver/segment.py Normal file
View File

@@ -0,0 +1,149 @@
"""MDL-guided morpheme segmentation for tiny vocabularies, pure python.
Greedy Morfessor-flavored search: start with whole words as morphs, repeatedly
apply the single split that most reduces description length
L(lexicon) + L(corpus | lexicon). Vocabularies here are tiny (10-100 word
types), so an O(V * maxlen) sweep per iteration is instant.
Alignment conditioning: tokens known (from align.py) to share a gloss get a
bonus for splits that expose their shared substring — this is the
"segmentation conditioned on alignment" step from the plan, and is what keeps
MDL from over-segmenting on 20-word corpora.
"""
from __future__ import annotations
import math
from collections import Counter
from typing import Dict, Iterable, List, Optional, Sequence, Set, Tuple
_MIN_MORPH = 1
def _lex_cost(morphs: Iterable[str]) -> float:
# ~1 char = a few bits; +1 per morph for the boundary/index overhead
return sum(len(m) + 1 for m in set(morphs)) * 4.0
def _corpus_cost(usage: Counter) -> float:
total = sum(usage.values())
if total == 0:
return 0.0
return -sum(c * math.log2(c / total) for c in usage.values())
class Segmenter:
def __init__(self, share_bonus: float = 8.0):
self.share_bonus = share_bonus
self.seg: Dict[str, List[str]] = {}
def fit(
self,
words: Sequence[str],
counts: Optional[Counter] = None,
share_groups: Optional[List[Set[str]]] = None,
max_iters: int = 200,
) -> "Segmenter":
"""words: vocabulary (task-language word types).
counts: token frequencies (defaults to 1 each).
share_groups: sets of words believed to share a morpheme (same gloss
alignment); splits exposing a shared prefix/suffix get a bonus."""
counts = counts or Counter({w: 1 for w in words})
self.seg = {w: [w] for w in dict.fromkeys(words) if w}
shared_subs = self._shared_substrings(share_groups or [])
for _ in range(max_iters):
best = self._best_split(counts, shared_subs)
if best is None:
break
word, mi, cut = best
m = self.seg[word][mi]
self.seg[word][mi : mi + 1] = [m[:cut], m[cut:]]
return self
def _shared_substrings(self, groups: List[Set[str]]) -> Set[str]:
subs: Set[str] = set()
for g in groups:
g = [w for w in g if w]
if len(g) < 2:
continue
# longest common prefix and suffix over the group
pre = g[0]
suf = g[0]
for w in g[1:]:
while pre and not w.startswith(pre):
pre = pre[:-1]
while suf and not w.endswith(suf):
suf = suf[1:]
if len(pre) >= 2:
subs.add(pre)
if len(suf) >= 2:
subs.add(suf)
return subs
def _cost(self, counts: Counter, shared_subs: Set[str]) -> float:
usage: Counter = Counter()
for w, morphs in self.seg.items():
for m in morphs:
usage[m] += counts[w]
cost = _lex_cost(usage.keys()) + _corpus_cost(usage)
cost -= self.share_bonus * sum(1 for m in usage if m in shared_subs)
return cost
def _best_split(self, counts: Counter, shared_subs: Set[str]):
base = self._cost(counts, shared_subs)
best_gain, best = 1e-6, None
for w, morphs in self.seg.items():
for mi, m in enumerate(morphs):
if len(m) < 2 * _MIN_MORPH:
continue
for cut in range(_MIN_MORPH, len(m) - _MIN_MORPH + 1):
morphs[mi : mi + 1] = [m[:cut], m[cut:]]
gain = base - self._cost(counts, shared_subs)
morphs[mi : mi + 2] = [m]
if gain > best_gain:
best_gain, best = gain, (w, mi, cut)
return best
def segment(self, word: str) -> List[str]:
"""Segment a word; unseen words are matched greedily against the
learned morph inventory (longest-match, both ends first)."""
if word in self.seg:
return list(self.seg[word])
morphs = {m for parts in self.seg.values() for m in parts}
return _greedy_decompose(word, morphs)
@property
def morphs(self) -> Set[str]:
return {m for parts in self.seg.values() for m in parts}
def _greedy_decompose(word: str, morphs: Set[str]) -> List[str]:
"""Best-effort decomposition of an unseen word over a morph set: dynamic
programming for fewest chunks, unknown spans kept as single chunks."""
n = len(word)
INF = float("inf")
# cost[i] = (num chunks, num unknown chars) to segment word[:i]
cost = [(INF, INF)] * (n + 1)
back: List[Optional[Tuple[int, str]]] = [None] * (n + 1)
cost[0] = (0, 0)
for i in range(n):
if cost[i][0] == INF:
continue
for j in range(i + 1, n + 1):
piece = word[i:j]
known = piece in morphs
c = (cost[i][0] + 1, cost[i][1] + (0 if known else len(piece)))
# prefer fewer unknown chars, then fewer chunks
key = (c[1], c[0])
if key < (cost[j][1], cost[j][0]):
cost[j] = c
back[j] = (i, piece)
out: List[str] = []
i = n
while i > 0 and back[i]:
prev, piece = back[i]
out.append(piece)
i = prev
out.reverse()
return out or [word]

126
solver/synth.py Normal file
View File

@@ -0,0 +1,126 @@
"""Program-synthesis subagent: LLM proposes grammars in the DSL, the
interpreter executes them, the verifier scores them, and failing pairs are
fed back for refinement (CEGIS), up to R rounds.
The LLM never applies rules — it only emits grammar JSON. All execution is
Interpreter; all selection is verifier.evaluate on the attested pairs.
"""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import List, Optional, Sequence, Tuple
from .dsl.grammar import Grammar, from_json
from .dsl.interpreter import Interpreter
from .llm import LLMClient
from .preprocess import Pair, Puzzle
from .scaffold import analysis_blocks
from .verifier import Verdict, evaluate
PROMPT_DIR = Path(__file__).resolve().parents[1] / "prompts"
MAX_FAILURES_SHOWN = 8
@dataclass
class SynthResult:
grammar: Optional[Grammar]
interpreter: Optional[Interpreter]
verdict: Optional[Verdict]
rounds_used: int = 0
def _load_prompt(name: str) -> str:
return (PROMPT_DIR / f"{name}.md").read_text(encoding="utf-8")
def proposer_prompt(puzzle: Puzzle) -> str:
seg_block, align_block = analysis_blocks(puzzle)
pairs_block = "\n".join(f" {p.src} = {p.tgt}" for p in puzzle.pairs) or " (none)"
hints_block = "\n".join(f" {h}" for h in puzzle.hints) or " (none)"
return _load_prompt("proposer").format(
task_lang=puzzle.task_lang or "the unknown language",
work_lang=puzzle.work_lang or "English",
pairs_block=pairs_block,
hints_block=hints_block,
segmentation_block=seg_block,
alignment_block=align_block,
)
def refine_prompt(puzzle: Puzzle, grammar: Grammar, verdict: Verdict) -> str:
fails = verdict.failures[:MAX_FAILURES_SHOWN]
failures_block = "\n".join(
f" input: {src}\n expected: {gold}\n got: {pred or '(nothing)'}"
for src, gold, pred in fails
)
return _load_prompt("refine").format(
task_lang=puzzle.task_lang or "the unknown language",
failures_block=failures_block,
previous_grammar=grammar.to_json(),
)
def _attested_for_direction(pairs: Sequence[Pair], direction: str) -> List[Tuple[str, str]]:
if direction == "to_task":
return [(p.tgt, p.src) for p in pairs] # work -> task (generation)
return [(p.src, p.tgt) for p in pairs] # task -> work (analysis)
def _predictor(interp: Interpreter, direction: str):
return interp.generate if direction == "to_task" else interp.analyze
def score_grammar(g: Grammar, pairs: Sequence[Pair], direction: str) -> Tuple[Interpreter, Verdict]:
interp = Interpreter(g)
attested = _attested_for_direction(pairs, direction)
return interp, evaluate(_predictor(interp, direction), attested, g.mdl())
def synthesize(
puzzle: Puzzle,
client: LLMClient,
direction: str = "to_task",
rounds: int = 2,
) -> SynthResult:
"""CEGIS loop: propose -> execute -> verify -> refine on failures.
Returns the best grammar seen across rounds (never a later-worse one)."""
if not client.available or not puzzle.pairs:
return SynthResult(None, None, None, 0)
best: SynthResult = SynthResult(None, None, None, 0)
prompt = proposer_prompt(puzzle)
for r in range(rounds + 1):
text = client.generate([prompt])[0]
g = from_json(text)
if g is None:
break
interp, verdict = score_grammar(g, puzzle.pairs, direction)
if best.verdict is None or verdict.score > best.verdict.score:
best = SynthResult(g, interp, verdict, r + 1)
if verdict.em >= 1.0 or r == rounds:
break
prompt = refine_prompt(puzzle, g, verdict)
return best
def synthesize_best_of_n(
puzzle: Puzzle,
client: LLMClient,
direction: str,
n: int,
rounds: int = 1,
) -> SynthResult:
"""Phase-3 test-time scaling hook: N independent proposals (greedy base is
deterministic, so diversity must come from prompt variants), each with a
short CEGIS budget; verifier picks. With greedy decoding, n>1 only helps
once prompt variants or sampling adapters exist — the plumbing is here."""
best = SynthResult(None, None, None, 0)
for _ in range(max(1, n)):
r = synthesize(puzzle, client, direction, rounds)
if r.verdict and (best.verdict is None or r.verdict.score > best.verdict.score):
best = r
if best.verdict and best.verdict.em >= 1.0:
break
return best

181
solver/tables.py Normal file
View File

@@ -0,0 +1,181 @@
"""Table-completion solver: answer items that are rows of a paradigm table
with one or more cells missing.
Covers the recurring Linguini patterns:
- fill_blanks with (k) markers in any column ("netkayʼ | (1) | push"),
including damaged rows where a marker merged with text ("*ʔikat | (4) | (5) to tie");
- numbered query rows lacking one column the context table has
("12. gsnqo'qon | foolishness" against context "word | [IPA] | gloss");
- multi-language columns (Proto-Chamic | Phan Rang Cham | Tsat | meaning).
Method per item:
1. strip (k) markers; the remaining non-empty cell texts are the knowns;
2. map knowns to context-table columns by character overlap (greedy);
3. assign the row's markers, in order, to the free columns, preferring the
free column matching the marker's position in the row (first/last);
4. build (known-column -> answer-column) pairs from the table and pick the
most learnable source column by leave-one-out template-translator fit;
5. predict; abstention returns None (router falls through to LLM/fallback).
"""
from __future__ import annotations
from collections import Counter
from typing import Callable, List, Optional, Sequence, Tuple
from .analogy import solve_from_pairs as analogy_vote
from .fallback import fallback_answer
from .preprocess import _BLANK_MARK, Pair, Puzzle, QueryItem
from .template import TemplateTranslator
from .verifier import leave_one_out
def _col_overlap(cell: str, column: Sequence[str], n: int = 3) -> float:
s = "".join(cell.lower().split())
material = " ".join(c.lower() for c in column)
if len(s) < n:
return 1.0 if any(cell.strip() == c.strip() for c in column) else 0.0
grams = {s[i : i + n] for i in range(len(s) - n + 1)}
return sum(1 for g in grams if g in material) / max(len(grams), 1)
def _clean_rows(table: List[List[str]]) -> List[List[str]]:
if not table:
return []
width = Counter(len(r) for r in table).most_common(1)[0][0]
return [r for r in table
if len(r) == width and not any(_BLANK_MARK.search(c) for c in r)]
def pick_table(puzzle: Puzzle, min_width: int = 2) -> List[List[str]]:
best: List[List[str]] = []
for t in puzzle.tables:
rows = _clean_rows(t)
if rows and len(rows[0]) >= min_width and len(rows) > len(best):
best = rows
return best
def _column_pairs(table: List[List[str]], src_col: int, ans_col: int) -> List[Pair]:
out = []
for r in table:
s, t = r[src_col].strip(), r[ans_col].strip()
if s and t and s != "-" and t != "-":
out.append(Pair(src=s, tgt=t, sep="table"))
return out
def _predictor_for(table: List[List[str]], src_col: int, ans_col: int
) -> Tuple[float, Callable[[str], Optional[str]]]:
"""LOO-scored predictor mapping src_col text to ans_col text. Chain:
template translation (multi-word rows), char-level analogy voting
(single-word paradigm columns: a:b :: query:x over all column pairs),
then the echo fallback."""
pairs = _column_pairs(table, src_col, ans_col)
if len(pairs) < 2:
return -1.0, lambda q: None
attested = [(p.src, p.tgt) for p in pairs]
def make(pair_list: List[Pair]) -> Callable[[str], Optional[str]]:
tmpl = TemplateTranslator(pair_list, "to_work")
ana = [(p.src, p.tgt) for p in pair_list]
def predict(q: str) -> Optional[str]:
ans = tmpl.translate(q)
if ans:
return ans
votes = analogy_vote(ana, q)
if votes:
return votes[0]
return fallback_answer(q, pair_list, "to_work")
return predict
def fit(held_in):
keep = set(held_in)
return make([p for p in pairs if (p.src, p.tgt) in keep])
v = leave_one_out(fit, attested)
return v.score, make(pairs)
class TableSolver:
"""Per-puzzle: caches the context table and column predictors."""
def __init__(self, puzzle: Puzzle):
self.table = pick_table(puzzle)
self._pred_cache: dict = {}
@property
def usable(self) -> bool:
return len(self.table) >= 2
def _predictor(self, src_col: int, ans_col: int):
key = (src_col, ans_col)
if key not in self._pred_cache:
self._pred_cache[key] = _predictor_for(self.table, src_col, ans_col)
return self._pred_cache[key]
def solve(self, item: QueryItem) -> Optional[Tuple[str, float]]:
"""Returns (answer, confidence) or None. Confidence is the LOO fit of
the chosen column predictor."""
if not self.usable or not item.row:
return None
n_cols = len(self.table[0])
cols = [[r[c] for r in self.table] for c in range(n_cols)]
# knowns: marker-stripped non-empty cells, with their row position
knowns: List[Tuple[int, str]] = []
marker_pos: List[Tuple[int, str]] = [] # (cell index, marker number)
for ci, cell in enumerate(item.row):
for k in _BLANK_MARK.findall(cell):
marker_pos.append((ci, k))
text = _BLANK_MARK.sub("", cell).strip()
if text and text != "-":
knowns.append((ci, text))
if not marker_pos and item.blank_col is None:
return None
if len(item.row) == n_cols and item.blank_col is not None:
# same shape as the context table: mapping is positional and the
# marker's own column is the answer column
known_col = {ki: ci for ki, (ci, _) in enumerate(knowns)}
ans_col = item.blank_col
else:
# width mismatch (damaged/narrow rows): map knowns to table
# columns by char overlap with a positional prior as tie-break
scored = sorted(
((_col_overlap(t, cols[c]) + 0.01 / (1 + abs(ci - c)), ki, c)
for ki, (ci, t) in enumerate(knowns) for c in range(n_cols)),
reverse=True,
)
known_col = {}
used = set()
for s, ki, c in scored:
if ki in known_col or c in used:
continue
known_col[ki] = c
used.add(c)
free = [c for c in range(n_cols) if c not in used]
if not free:
return None
# markers in row order take free columns in order; a single
# marker at the row's edge prefers the matching edge column
my_marker_idx = next(
(mi for mi, (_, k) in enumerate(marker_pos) if k == item.number), 0)
if len(marker_pos) <= 1 and item.row and len(free) > 1:
cell_idx = marker_pos[0][0] if marker_pos else (item.blank_col or 0)
ans_col = free[-1] if cell_idx >= len(item.row) - 1 else free[0]
else:
ans_col = free[min(my_marker_idx, len(free) - 1)]
# pick the most learnable known column as the source
best_score, best_predict, best_ki = -1.0, None, None
for ki, c in known_col.items():
score, predict = self._predictor(c, ans_col)
if score > best_score:
best_score, best_predict, best_ki = score, predict, ki
if best_predict is None or best_ki is None:
return None
ans = best_predict(knowns[best_ki][1])
return (ans, max(best_score, 0.0)) if ans else None

146
solver/template.py Normal file
View File

@@ -0,0 +1,146 @@
"""Template translation by minimal-pair substitution — the strongest
zero-LLM translation baseline for constructed puzzles.
Idea: puzzles are built so query sentences differ from attested ones by a
small substitution. Find the attested pair whose source is closest to the
query (token-level), then replace the differing tokens in its *target* using
alignment links (align.py). Works in both directions. Also supports
morph-level substitution for single-word queries (paradigm cells).
"""
from __future__ import annotations
from collections import Counter
from typing import Dict, List, Optional, Tuple
from .align import align as build_align, one_to_one
from .preprocess import Pair, strip_punct, tokenize
def _toks(s: str) -> List[str]:
return [strip_punct(t).casefold() for t in tokenize(s) if strip_punct(t)]
def _flip(pairs: List[Pair]) -> List[Pair]:
return [Pair(src=p.tgt, tgt=p.src) for p in pairs]
class TemplateTranslator:
"""direction 'to_work': translate task->work; 'to_task': work->task."""
def __init__(self, pairs: List[Pair], direction: str = "to_work"):
self.pairs = pairs if direction == "to_work" else _flip(pairs)
self.amap = build_align(self.pairs) # src tok -> ranked [(tgt, score)]
def _sub(self, src_tok: str) -> Optional[str]:
cands = self.amap.get(src_tok.casefold())
return cands[0][0] if cands else None
def _sub_in(self, src_tok: str, pool: List[str]) -> Optional[str]:
"""Best candidate for src_tok that is present in pool (context-aware:
a token may have both a bare and an inflected realization; the one
actually in the template target is the right one)."""
for c, _ in self.amap.get(src_tok.casefold(), []):
if c in pool:
return c
return None
def _sub_like(self, src_tok: str, model: str) -> Optional[str]:
"""Best candidate for src_tok, preferring one that shares an affix
(prefix/suffix >= 2 chars) with `model` — the form it will replace.
kupu:nakupu :: moko:namoko."""
cands = self.amap.get(src_tok.casefold(), [])
for c, _ in cands:
if len(c) >= 2 and len(model) >= 2 and (c[:2] == model[:2] or c[-2:] == model[-2:]):
return c
return cands[0][0] if cands else None
def translate(self, query: str) -> Optional[str]:
q = _toks(query)
if not q:
return None
# rank templates by token-bag distance, then by length mismatch: a
# same-length template is a substitution frame; a much shorter one
# (e.g. a single-word gloss) would force fabricating structure
ranked = sorted(
((_bag_distance(q, _toks(p.src)), abs(len(_toks(p.src)) - len(q)),
_toks(p.src), _toks(p.tgt)) for p in self.pairs),
key=lambda x: (x[0], x[1]),
)
max_dist = max(2, len(q) // 2)
for dist, _, s, t in ranked:
if dist == 0:
return " ".join(t)
if dist > max_dist:
break
out = self._substitute(q, s, t)
if out:
return out
return None
def _substitute(self, q: List[str], s: List[str], t: List[str]) -> Optional[str]:
"""Swap the tokens where query and template source differ, mapping
both sides through the alignment. Abstains (None) when any needed
link is missing — a wrong-but-confident answer is worse than letting
the next template or the fallback ladder take over."""
q_extra = list((Counter(q) - Counter(s)).elements())
s_extra = list((Counter(s) - Counter(q)).elements())
out = list(t)
used: set = set()
for s_tok in s_extra:
s_tgt = self._sub_in(s_tok, out)
if s_tgt is None:
return None
repl = None
for qi, q_tok in enumerate(q_extra):
if qi in used:
continue
q_tgt = self._sub_like(q_tok, s_tgt)
if q_tgt:
repl = q_tgt
used.add(qi)
break
if repl is None:
return None
out[out.index(s_tgt)] = repl
for qi, q_tok in enumerate(q_extra):
if qi not in used:
q_tgt = self._sub(q_tok)
if q_tgt and q_tgt not in out:
out.append(q_tgt)
return " ".join(out) if out else None
def _bag_distance(a: List[str], b: List[str]) -> int:
ca, cb = Counter(a), Counter(b)
return sum((ca - cb).values()) + sum((cb - ca).values())
def paradigm_complete(stem: str, pairs: List[Pair], cue: str = "") -> Optional[str]:
"""Complete a paradigm cell: find attested form-pairs (a, b) sharing a
stem, group them by their string edit, and apply the dominant edit to
`stem`. `cue` (e.g. 'plural') restricts to pairs whose gloss relation
mentions the cue when glosses are available."""
from .analogy import edit_rules, apply_rule
vocab: Dict[str, str] = {} # form -> gloss
for p in pairs:
if " " not in p.src.strip():
vocab[p.src.strip().casefold()] = p.tgt.strip().casefold()
rules: Counter = Counter()
for a in vocab:
for b in vocab:
if a != b and len(b) > len(a) and b.startswith(a[: max(2, len(a) - 1)]):
for r in edit_rules(a, b):
if cue:
ga, gb = vocab.get(a, ""), vocab.get(b, "")
# cue must relate the two glosses (e.g. 'houses' vs 'house')
if not (ga and gb and (ga in gb or gb in ga)):
continue
rules[r] += 1
for r, _ in rules.most_common(3):
out = apply_rule(r, stem.casefold())
if out and out != stem:
return out
return None

121
solver/verifier.py Normal file
View File

@@ -0,0 +1,121 @@
"""The verifier: leave-one-out fit of a candidate solver on attested pairs.
One object, reused everywhere: grammar selection, CEGIS failure feedback, and
(offline) RL reward. score = sqrt(EM * chrF) on held-out attested pairs,
minus an MDL penalty so the simplest adequate grammar wins ties.
A "candidate" is anything with `predict(src: str) -> str` for the relevant
direction; grammars, analogy baselines, and raw LLM outputs all fit.
"""
from __future__ import annotations
import math
from dataclasses import dataclass, field
from typing import Callable, List, Optional, Sequence, Tuple
from .metrics import chrf, exact_match
Predictor = Callable[[str], Optional[str]]
@dataclass
class Verdict:
em: float
chrf: float
mdl: float
failures: List[Tuple[str, str, str]] = field(default_factory=list) # (src, gold, pred)
@property
def fit(self) -> float:
return math.sqrt(max(self.em, 0.0) * max(self.chrf, 0.0))
@property
def score(self) -> float:
return self.fit - self.mdl
def __repr__(self) -> str:
return f"Verdict(em={self.em:.3f}, chrf={self.chrf:.3f}, mdl={self.mdl:.4f}, n_fail={len(self.failures)})"
def evaluate(
predict: Predictor,
pairs: Sequence[Tuple[str, str]],
mdl_cost: float = 0.0,
mdl_weight: float = 0.002,
) -> Verdict:
"""Score `predict` on attested (src, gold) pairs. A None/empty prediction
scores 0 on both metrics for that pair. mdl_cost is the grammar's
description length (see dsl.grammar.Grammar.mdl); weighted lightly so it
only breaks ties."""
if not pairs:
return Verdict(0.0, 0.0, mdl_cost * mdl_weight)
em_sum = chrf_sum = 0.0
failures = []
for src, gold in pairs:
pred = predict(src) or ""
e, c = exact_match(pred, gold), chrf(pred, gold)
em_sum += e
chrf_sum += c
if e < 1.0:
failures.append((src, gold, pred))
n = len(pairs)
return Verdict(em_sum / n, chrf_sum / n, mdl_cost * mdl_weight, failures)
LOO_MAX_FOLDS = 12
def leave_one_out(
fit_predict: Callable[[Sequence[Tuple[str, str]]], Predictor],
pairs: Sequence[Tuple[str, str]],
mdl_cost: float = 0.0,
max_folds: int = LOO_MAX_FOLDS,
) -> Verdict:
"""True LOO for candidates that are *fit* from pairs (analogy, alignment
baselines): refit without pair i, predict pair i. For a fixed grammar
(already synthesized), use `evaluate` directly — the LLM saw the pairs,
but the grammar either reproduces them or it doesn't.
Refitting is O(pairs^2)+ per fold (alignment rebuild), so folds are capped
at `max_folds` evenly-spaced held-out pairs — an unbiased estimate is all
the selection needs, and the 30-minute budget cannot afford exact LOO on
40-pair puzzles."""
if not pairs:
return Verdict(0.0, 0.0, 0.0)
n = len(pairs)
if n <= max_folds:
fold_idx = range(n)
else:
step = n / max_folds
fold_idx = sorted({int(k * step) for k in range(max_folds)})
em_sum = chrf_sum = 0.0
failures = []
for i in fold_idx:
src, gold = pairs[i]
held_in = [p for j, p in enumerate(pairs) if j != i]
pred = fit_predict(held_in)(src) or ""
e, c = exact_match(pred, gold), chrf(pred, gold)
em_sum += e
chrf_sum += c
if e < 1.0:
failures.append((src, gold, pred))
k = len(list(fold_idx))
return Verdict(em_sum / k, chrf_sum / k, mdl_cost * 0.002, failures)
def select_best(
candidates: Sequence[Tuple[str, Predictor, float]],
pairs: Sequence[Tuple[str, str]],
) -> Tuple[Optional[str], Optional[Predictor], Verdict]:
"""Pick the best (name, predictor, mdl_cost) by verifier score.
Ties broken by lower MDL (already in score), then earlier order
(candidates should be ordered by prior preference: symbolic first)."""
best: Tuple[Optional[str], Optional[Predictor], Verdict] = (None, None, Verdict(0, 0, 0))
best_score = -1e9
for name, pred, mdl in candidates:
v = evaluate(pred, pairs, mdl)
if v.score > best_score + 1e-9:
best_score = v.score
best = (name, pred, v)
return best

303282
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

207
tokenizer_config.json Normal file
View File

@@ -0,0 +1,207 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "<|object_ref_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<|object_ref_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151648": {
"content": "<|box_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151649": {
"content": "<|box_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151650": {
"content": "<|quad_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151651": {
"content": "<|quad_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151652": {
"content": "<|vision_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151653": {
"content": "<|vision_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151654": {
"content": "<|vision_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151655": {
"content": "<|image_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151656": {
"content": "<|video_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151659": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151660": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151661": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151662": {
"content": "<|fim_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151663": {
"content": "<|repo_name|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151664": {
"content": "<|file_sep|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"bos_token": null,
"chat_template": "{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0]['role'] == 'system' %}\n {{- messages[0]['content'] }}\n {%- else %}\n {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n {%- endif %}\n {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0]['role'] == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n {%- else %}\n {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {{- '<|im_start|>' + message.role }}\n {%- if message.content %}\n {{- '\\n' + message.content }}\n {%- endif %}\n {%- for tool_call in message.tool_calls %}\n {%- if tool_call.function is defined %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n<tool_call>\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- '}\\n</tool_call>' }}\n {%- endfor %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n<tool_response>\\n' }}\n {{- message.content }}\n {{- '\\n</tool_response>' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"model_max_length": 131072,
"pad_token": "<|endoftext|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long

0
weights/.gitkeep Normal file
View File