# IOL-AI 2026 Solver — Implementation Seed

Seed context and build plan for implementing a solver for International Linguistics Olympiad (IOL) style problems. Read this fully before writing code. Design principle: the structure of language does the reasoning; the LLM proposes hypotheses that a deterministic engine executes and a symbolic verifier selects. Do not outsource rule application to the LLM.

## 1. The task

Solve self-contained linguistic-reasoning puzzles over unseen, mostly low-resource languages. Each puzzle gives a small amount of bilingual/parallel data plus hints, and asks the solver to translate, fill blanks, match forms to meanings, or convert numerals. No prior knowledge of the language is needed or expected; everything must be induced from the given data.

### Data (Linguini-style CSV)
Columns: `id`, `context`, `query`, `work_lang`, `task_lang`, `task_type`, `eval_type`.
- One row = one complete problem. `context` holds the parallel data + hints. `query` holds numbered items sharing that context.
- Read test set from `/tmp/data/test.csv`. Write `submission.csv` with a `pred` column: one **JSON list** of answers per row, one entry per numbered item, in order.

### Task types (route on `task_type`)
- `translation` (work->task and task->work): analysis one way, generation the other.
- `fill_blanks`: complete a paradigm cell.
- `match_letters`: match forms to meanings (assignment problem).
- `text_to_num` / `num_to_text`: recover and apply a numeral system.
- Others may appear (transliteration, glossing, error-correction, phonological rule application) — handle gracefully, fall back rather than crash.

### Scoring (design to this)
Final score = geometric mean `sqrt(EMw * chrFw)`, both **points-weighted** by official IOL point values.
- Geometric mean => neither factor can be near zero. **Never emit an empty or wildly-off answer**; always produce a plausible string (chrF floor).
- Need genuine exact hits (symbolic core) AND fuzzy overlap (fallback).
- Generation *into* the unknown language is the hardest direction; invest there.

### Submission and runtime constraints (hard)
- Submit a public Hugging Face repo containing `script.py` + model weights. Load models from `.` (repo root). **No internet in the eval sandbox** — vendor every dependency and weight.
- **30 minutes per run for the entire test set.** Throughput is the real budget. Symbolic-first; batch all LLM calls; spend sampling on hard/high-point items only.
- Greedy decoding for reproducibility. Up to 3 submissions/day; 2 count for the private leaderboard.
- Compute at inference: single **T4, 16 GB** (Turing: fp16 yes, bf16 no). Use 4-bit AWQ/GPTQ + LoRA via vLLM.
- Full repo is allowed: ship YAML configs, prompt libraries, DSL grammar files, multiple LoRA adapters, finetuned weights.
- Finetuning (SFT or RL) is allowed; train **offline** on other hardware and upload weights. Only inference is T4-bound.

## 2. Core insight (why this architecture)

1. LLMs (small ones especially) can *propose* linguistic rules but reliably fail to *apply* them consistently. => LLM proposes a grammar; a **deterministic interpreter applies it**.
2. The given attested pairs are a **free, exact verifier**. Leave-one-out fit on them (points-weighted EM + chrF = the competition metric) selects among candidate grammars. No LLM judge needed for correctness. This makes the domain verifiable, which is what makes test-time scaling and RL-with-verifiable-rewards work.
3. Explicit morpheme segmentation before anything else is a large, cheap win.
4. Analogical exemplars (from a multilingual model) help the LLM *propose* better grammars, never to deduce.

## 3. Architecture

```
Router/typology scan  ->  Preprocess (normalize, segment, align, paradigm table)
   -> Symbolic-first solvers:  numeral CSP | matching (Hungarian) | concatenative morphology
   -> Program-synthesis subagent:  LLM proposes grammar in DSL  ->  interpreter runs it
                                    ->  CEGIS refine <= R rounds  (feed failing pairs back)
   -> Verifier: leave-one-out EM+chrF on attested pairs + MDL penalty; self-consistency on ties
   -> Select grammar (never empty; chrF-floor fallback = closest attested form / analogy)
   -> Apply selected grammar deterministically to query items  ->  answers
   -> (phase 5) Verbalizer LoRA: execution trace -> human explanation
```

Principles: segment first; symbolic-solved items cost ~0 LLM tokens; adaptive compute; guard against repetition loops (token cap + repetition penalty + loop detection).

### The DSL (small, executable, verifiable)
Primitives for a puzzle grammar. Programs are tiny so execution/verification is effectively free.
- `LEXICON`: morpheme -> gloss/feature
- `AFFIX(position, form, trigger)`
- `REWRITE(pattern -> repl / context)`  (morphophonology: harmony, sandhi, elision)
- `ORDER(slot permutation)`  (source constituent order -> target)
- `AGREE(feature copy)`
- `REDUP(template)`
- `NUMERAL(base, digitmap, combine_op)`
Interpreter must run **both directions** (analyze and generate). Generation uses the grammar + proportional analogy for unseen stems; round-trip verify generated forms.

### Verifier (single object, reused everywhere)
`score(grammar, held_out_pairs) -> points_weighted_EM, chrF`. Used as: test-time selector, CEGIS signal, and (later) RL reward. Add MDL penalty `L(grammar)+L(data|grammar)` to prefer the simplest adequate grammar.

## 4. Suggested repo layout

```
script.py                 # entrypoint: read /tmp/data/test.csv -> write submission.csv
solver/
  router.py               # task typing + typology scan + budget tier
  preprocess.py           # unicode NFC (keep tone/diacritics), tokenize, align, segment
  align.py                # minimal-pair set-difference aligner (pure python)
  segment.py              # MDL-guided segmentation conditioned on alignment
  analogy.py              # proportional analogy (a:b::c:d), analogical grids
  numerals.py             # base detection + CSP/algebraic solve
  matching.py             # scipy Hungarian over alignment-consistency scores
  dsl/
    grammar.py            # DSL datatypes
    interpreter.py        # analyze() and generate(), both directions
  synth.py                # program-synthesis subagent + CEGIS loop
  verifier.py             # leave-one-out EM+chrF + MDL; selection + self-consistency
  llm.py                  # vLLM wrapper, 4-bit base + LoRA adapter swap, batching, guards
  fallback.py             # chrF-floor: never return empty
  budget.py               # adaptive compute allocation under 30-min cap
prompts/                  # YAML prompt library (proposer, analogical generator, critic)
weights/                  # 4-bit base + LoRA adapters (proposer/deducer/verbalizer/critic)
eval/
  scorer.py               # reimplement official geomean scorer
  dev_harness.py          # run + score on public dev puzzles
data/dev/                 # public IOL/Linguini/PuzzLing/modeLing/LINGOLY (validation ONLY)
```

## 5. Build order (do in this sequence; each has a gate)

0. **Harness first.** `eval/scorer.py` (points-weighted EM + chrF geomean) and `verifier.py` leave-one-out. Assemble `data/dev/` from public puzzles (respect CC-BY-SA / IOL copyright; validation only). Baselines: analogy-only floor, small-LLM few-shot. Gate: scorer matches hand-computed values.
1. **Cheap symbolic points.** `numerals.py`, `matching.py`, `align.py`, `segment.py`, `analogy.py`. Gate: numerals/matching beat LLM baseline; segmentation shows a positive downstream delta (replicate the known morpheme-boundary gain). If segmentation does not help, the aligner is mis-conditioned.
2. **Synthesis core.** DSL + interpreter + `synth.py` CEGIS with proposer LLM. Ablate: direct-LLM vs synthesis+verifier vs +segmentation vs +analogical exemplars; vary CEGIS rounds. Gate: synthesis+verifier clearly beats direct-LLM on translation/fill_blanks at fixed budget.
3. **Verifier-gated test-time scaling.** Best-of-N grammar sampling ranked by verifier; self-consistency; adaptive budget. Gate: monotone accuracy gain that still fits 30 min; pick operating N from the compute-accuracy curve.
4. **Finetuning (offline, highest ceiling).** Procedurally generate synthetic puzzles with gold grammar/segmentation/derivation. SFT the proposer/deducer for DSL emission + reliable rule application; then GRPO/RLVR with the verifier as reward. Validate on **real** public puzzles (guard the synthetic-real gap). Gate: finetuned small model >= best prompt-only config on real held-out puzzles at equal/lower cost.
5. **Explanation track (later).** Verbalizer LoRA: execution trace -> prose; narrow LLM critic checks faithfulness to the trace. The automated solver's trace is the explanation substrate.

## 6. Stack and risks

Models (T4, 4-bit + LoRA, vLLM): base = Qwen2.5-7B-Instruct; try a Coder variant for DSL emission; optional Aya-Expanse-8B as multilingual analogical generator. Role specialization via hot-swapped LoRA adapters over one base (stay in 16 GB).

Libraries (vendor all, offline): `unicodedata`/`regex`; pure-python aligner + MDL segmenter (avoid native `pynini`/`hfst`; implement rewrite-rule + analogy engine in python); `scipy` (Hungarian); hand-rolled CSP for numerals (spaces are tiny) or `z3-solver` if redistributable; `sacrebleu` (chrF); `vllm`; `peft`.

Risks: throughput vs 30 min (symbolic-first, batch, adaptive budget, cap N); small-model deduction leakage (always execute rules deterministically); DSL coverage of hard phenomena — tone, reduplication, circumfixes, harmony, ergativity, suppletion (grow primitives from failure analysis; keep an LLM fallback so nothing is empty); synthetic-real gap (validate on real puzzles only); repetition collapse (token caps + penalties); quantization hurting multilingual quality (compare AWQ/GPTQ/fp16 on dev before committing).

## 7. Non-negotiables

- Read `/tmp/data/test.csv`; write `submission.csv` with `pred` as a JSON list per row. Load all models/deps from repo root; assume no network.
- Never return empty for any item. Stay within 30 minutes for the whole set. Greedy decoding.
- LLM proposes; symbolic engine applies and verifies. Segment before reasoning. Keep tone/diacritics.