# Evaluation & reproduction — block-distilled Qwen3 + landmark router

All numbers below are produced by the scripts in `scripts/`. `examples_dump.json` contains the actual
per-example inputs (documents + question + gold) and outputs (dense vs sparse) for the accuracy test.

## Headline results (Qwen3-14B, last-8 summary router)

| check | script | result |
|---|---|---|
| **Accuracy — no drop** | `acc_regress.py` | per-example on 80 real LongBench QA: dense 32/80, sparse 33/80, **0 regressions**, +1 gain |
| **Quality (NLL)** | `bench_real.py` | 50 real samples: router k=3 within ~3% NLL of dense, k=2 within ~12% |
| **No data leakage** | `audit_sparse.py` | canary secret in a masked block **cannot** be reproduced (model hallucinates); with block active it can |
| **Router — no train leakage** | `router_leak_check.py` | on unseen docs, coverage 0.92@k=2 ≈ training eval 0.96 |
| **FlexAttention stable** | `flex_attn_test.py` | flex block-attn == eager (max\|Δ\|=0.004), deterministic, T=16384 at 0.2 GB |
| **Decode savings (weights excluded)** | `decode_savings.py` | KV-read HBM & attention-FLOPs each −4× (2k ctx) … ~25× (long ctx); cap = block/summary |

## How to run

```bash
pip install torch transformers peft accelerate
MODEL=hxia7/qwen3-14b-blockdist            # or a local path to the merged model
ROUTER=router.pt                            # ships in the repo (summary_tokens=8)

python scripts/demo_infer.py       --model $MODEL --router $ROUTER --k 2        # generation demo
python scripts/audit_sparse.py     --model $MODEL --router $ROUTER              # leakage + savings
python scripts/acc_regress.py      --model $MODEL --router $ROUTER --n 80 --k 2 # accuracy regression
python scripts/bench_real.py       --model $MODEL --router $ROUTER --k 2        # NLL + savings
python scripts/router_leak_check.py --model $MODEL --router $ROUTER             # router generalization
python scripts/flex_attn_test.py                                               # flex kernel validation
python scripts/decode_savings.py                                               # HBM/FLOP accounting
```

Datasets (LongBench-Seg etc.) load from paths hard-coded in the scripts — adjust to your HF cache.

## Honest caveats

- Absolute LongBench accuracy (~40%) is a base-model + strict-string-match artifact; dense and sparse
  are near-identical — the point is **0 regressions**, not the absolute number.
- Block-sparse decode is a **KV-bandwidth (HBM) optimization**; FLOP savings only matter at very long
  context. Big wins are in the long-context × batch serving regime.
- Cross-block **aggregation** queries ("list all X") are a limit of block attention itself (even dense
  struggles) → serve those via dense fallback.
- Eager `output_attentions` is O(T²) and OOMs past ~2k tokens; use FlexAttention (validated here) to
  scale the block mask in a real serving kernel.