iolai26-solve/fable.md

# fable.md — research-direction notes from implementation

Notes written while building the IOL-AI solver seed (see AGENTS.md for the plan,
README.md for repo state). Each item is something the implementation surfaced
that the research plan should absorb.

**Update (real-data phase):** items 4, 8, 9 are now RESOLVED — the official
eval notebook (rita-berrada/iolai-2026-workshop) fixed the semantics, and
facebook/linguini gave 160 real dev puzzles. New findings start at §11.

## 11. Real-data baseline resets expectations (and the plan's emphasis)

Symbolic-only on all 160 Linguini puzzles: **EM 2.4%, chrF 20, in ~9 s.**
(The synthetic dev set scored 1.0 — constructed puzzles flatter substitution
methods.) Real translation items are compositional, real paradigms involve
metathesis/harmony/infixation, real numeral systems have morphophonology that
breaks token-level CSP. Consequences:
- The LLM direct path (notebook format) is the main score carrier for now;
  the symbolic stack's near-term value is (a) exact hits where structure is
  clean, (b) the chrF floor, (c) **verified confidences that route the LLM
  budget** (157/160 puzzles flagged low-confidence — correctly).
- The DSL/CEGIS path is the ceiling-raiser, not the floor: its job is to
  convert LLM linguistic insight into verified exact hits. Ablate it against
  direct-LLM on the T4 before investing in finetuning.

## 12. Official semantics differ from my assumptions in scoring-relevant ways

- EM is `strip().lower()` ONLY — final punctuation and internal spacing are
  significant. Answers must carry the gold's punctuation conventions; a
  format-inducer pass (§3) is now demonstrably worth EM points.
- Gold items can be a LIST OF ALTERNATIVES (14/160 rows); scorer takes max.
- chrF is sacrebleu with effective-order smoothing; my reimplementation now
  matches to 0.0 delta over 500 fuzz cases (tests/test_scorer.py).

## 13. Linguini formats: what the parser must survive (now does, 158/160)

Pipe tables 2-5 columns with header rows; numbered example sentences whose
numbering the query CONTINUES (item "17." refers to nothing in the query);
bare-line items with no numbering; (k)-blank markers in any column, both
directions in one puzzle, including damaged rows ("(5) to tie" merged cell);
items living in the CONTEXT while the query is instruction-only
("Determine the correct correspondences", "Fill in the blanks (1–14)");
task-language strings that end in ':' (vowel length) — never use trailing
colon alone to detect instructions. Two rows remain unparseable-by-count: one
has misaligned gold (7 answers, 6 items), one enumerates payloads inline in
prose. Position-aligned scoring makes over/under-parsing cost only the
misaligned tail — worth a guard that pads rather than truncates.

## 14. match_letters needs morpheme-level CSP, not surface matching

16 puzzles / 223 items (25% of all items) are unordered form↔meaning
matching with usually ZERO attested pairs. Surface similarity carries no
signal (5.4% EM ≈ barely above random). The real structure: recurring
morphemes across forms must map consistently to recurring words across
meanings (Zuni doko:ko ↔ 'chicken', mo:chikwa ↔ 'peach'). That is a small
constraint-satisfaction / bilingual-lexicon-induction problem over sets — 
highly verifiable (a candidate assignment implies a consistent lexicon or it
doesn't) and a perfect fit for the propose-verify architecture: LLM proposes
morpheme↔word hypotheses, a solver checks global consistency, Hungarian
finishes. My soft quadratic-assignment attempt (power iteration over
similarity graphs) was neutral; the discrete version is the right next try.

## 15. Throughput reality on the T4 (from the notebook)

The baseline runs ~1 min/problem at 1536 new tokens on the 14B AWQ — a full
160-puzzle set would need ~2.7 h sequential. The 30-minute budget therefore
REQUIRES the two-pass design: symbolic first (~9 s), then batched direct-LLM
on the low-confidence subset, weakest-first, with a hard budget cutoff
(script.py implements this). Concrete knobs to tune on Colab: batch size
(KV-cache limited on 16 GB with a 14B model — try 2-4), MAX_NEW_TOKENS
(1536 baseline; shorter for translation-only puzzles), and whether a 7B
model with bigger batches beats the 14B with tiny batches at fixed wall
clock. That ablation needs the actual T4.

## 16a. v1 migration to single-shot-with-tools (done 2026-07-21)

The architecture inversion recommended by the landscape survey is
implemented: the tools now feed the answering model (scaffold in the prompt:
segmentation, alignment, verified numeral values, labeled symbolic
candidates), verified symbolic output overrides the model, unverified output
is demoted to hints, and CEGIS left the runtime path. Explanations ride in
the same generation (`EXPLANATION:` after `FINAL ANSWERS:`) — the
human-judged track costs zero extra passes. Open empirical questions for the
T4: does the scaffold help a 14B AWQ model the way it helped GPT-4-class
models (the segmentation evidence says gains concentrate above ~15% baseline
— a 14B may be below it); and the deferred CoT-vs-IO prompt A/B (§ "Probing
LLMs", arXiv:2502.00817).

## 16b. First live submission: the T4 constraint is prefill logits, not KV

The first real submission OOMed: on the sandbox's transformers, `generate`
computes LM-head logits over EVERY prompt position and casts to float32 —
batch 6 × ~2.9k tokens × 152k vocab × 4B = the exact 9.82 GiB failed
allocation. KV cache (what batch sizing usually optimizes) is negligible
under GQA; **total prompt tokens per batch** is the real T4 memory bound at
~0.9 MB/token. Fixes: token-budget batch packing (3,500), OOM → retry
singly → abstain, `logits_to_keep=1` where supported. The deeper lesson:
the crash killed the script before submission.csv existed — "never empty"
must hold at the FILE level, not the item level. submission.csv is now
checkpoint-written from ~15 s in and atomically replaced as results improve.

## 16. The sandbox runs OLDER transformers than Colab

The notebook comments that `apply_chat_template(..., return_dict=True)`
returns a dict on Colab but a bare tensor in the sandbox. llm.py sidesteps
the whole class of drift by using `tokenize=False` + a normal tokenizer
call, which is stable across versions. Keep any future transformers usage on
the oldest-common-API path; there is no way to pin the sandbox's version.

## 1. The symbolic template translator may be the real workhorse, not the DSL

The plan positions LLM→DSL synthesis as the core translation engine. But the
minimal-pair **substitution translator** (`solver/template.py`: find the
attested sentence closest to the query, swap the differing tokens through
alignment links) solved every synthetic translation item exactly, with zero
LLM tokens, once three refinements landed:

- **competition / explaining-away in alignment** (`align.py`): demote a target
  token already strongly claimed by another source token. Tiny corpora make
  co-occurrence ties pervasive; this one change flipped several wrong answers.
- **morphological back-off**: single-word glosses (`moko = dog`) project into
  inflected forms containing them (`namoko`).
- **affix-matched substitution**: when replacing a form, prefer the candidate
  sharing an affix with the form replaced (kupu:nakupu :: moko:namoko) — a
  proportional analogy inside the template frame.

Implication: budget the DSL/CEGIS path for the residual — items where the
query is far (bag-distance > ~len/2) from every attested sentence, long
compositional sentences, and phenomena the template can't reach (reordering,
harmony). Measure on real dev data what fraction that residual actually is;
it directly sets the LLM compute budget and the finetuning priority.

## 2. Verifier honesty requires two scoring regimes (found a real trap)

Memorizing predictors (template translator, fallback) score EM=1.0 on attested
pairs by construction, so scoring every candidate with plain `evaluate()` lets
memorization beat a generalizing grammar on ties — with MDL penalty it beats
it *always*. The fix in `router._pick_direction_solver`: fixed programs
(grammars) are scored directly (they must reproduce the data), while
fit-from-data predictors are scored **leave-one-out**. Any future candidate
added to the selection pool must declare which regime it belongs to. This also
matters for RL later: the reward must be LOO-style for anything with data
access, or the policy learns to memorize.

## 3. Answer-format induction is a first-class subproblem the plan ignores

The synthetic num_to_text case exposed it: for 23, `hun ox` (20+3) and
`kan hun ox` (1×20+3) are both arithmetically valid; the gold followed the
attested *style* (explicit unit multiplier). The fix was a style model scoring
candidates by consistency with attested phrasing. This generalizes: whenever
multiple surface answers are semantically correct, the scorer only rewards the
one matching the dataset's convention (capitalization, articles, hyphenation
of morph boundaries, multiplier explicitness). Recommend: an explicit
"format inducer" pass that learns per-puzzle answer conventions from context
examples, applied as the last stage of every task type. Cheap, and it converts
chrF-close answers into EM hits — which the geometric mean doubly rewards.

## 4. Context parsing is still the top empirical risk

Everything downstream consumes `extract_pairs`. The current parser
(separator voting + per-line fallback) handles the formats I could construct,
but real Linguini CSVs were not available in this environment. **Before any
model work, pull the actual Linguini/PuzzLing dev CSVs and fuzz the parser
against every context in them**; count rows where zero pairs are extracted —
each such row is guaranteed near-zero score. A cheap LLM repair path (ask the
model to emit the pair list as JSON when the symbolic parser yields < 2 pairs)
would cap this risk at one batched call per malformed puzzle.

## 5. Direction detection deserves attested-data validation, not just wording

`detect_direction` keys off phrases like "into English". Real queries may
say "What does X mean?", "Give the form meaning ...", or nothing explicit.
A robust cross-check: try both directions and see which side of the attested
pairs the query string is *script-similar* to (character n-gram overlap with
task-language vs work-language material). If the query looks like the unknown
language, it's an analysis item. Cheap and language-agnostic.

## 6. CEGIS should get verifier-guided repair hints, not just failures

The refine prompt currently shows failing (input, expected, got) triples. The
symbolic layer knows more: which morpheme boundary the output diverged at,
which affix went unused, whether the failure is pure reordering. Feeding a
one-line diagnosis per failure ("output differs only in word order",
"expected form contains attested morph 'na-' that your grammar never
attaches") should cut CEGIS rounds — worth an ablation axis in phase 2/3.

## 7. Self-consistency has a free implementation via round-trip

The interpreter runs both directions, so `analyze(generate(x)) == x` is a
zero-cost consistency check usable as (a) a tie-breaker in grammar selection
(already anticipated in AGENTS.md), and (b) a confidence signal for budget
allocation — spend refinement rounds on items whose round-trip fails.

## 8. Points weighting has no data column — resolve early

The scorer supports per-item weights, but the Linguini schema
(id/context/query/...) carries no points column. If official IOL point values
weight the leaderboard metric, they must be embedded somewhere (query text?
separate mapping by id?). Resolve this with a probe submission early — it
changes budget allocation (high-point items deserve the CEGIS rounds).

## 9. Local scorer vs sacrebleu chrF: verify parity once, in the eval image

`eval/scorer.py` implements chrF2 (n≤6, β=2, whitespace stripped) by hand to
stay dependency-free. sacrebleu differs in epsilon smoothing on zero-match
orders. Before trusting local ablations, run both on a few hundred string
pairs in the submission container and confirm the delta is < 1e-3; otherwise
model selection could silently optimize the wrong metric.

## 10. Prompt library format drifted from the plan (deliberately)

AGENTS.md says YAML prompts; the repo uses plain `.md` templates with
`str.format` placeholders (`prompts/*.md`). Rationale: zero dependencies, no
YAML-escaping pain with multiline linguistic data. If config-driven prompt
variants are needed for best-of-N diversity (the greedy-decoding constraint
means diversity must come from prompts, not sampling temperature — see
`synth.synthesize_best_of_n`), add a small variants list per template rather
than reintroducing YAML.