45 lines
2.8 KiB
Markdown
45 lines
2.8 KiB
Markdown
# Evaluation & reproduction — block-distilled Qwen3 + landmark router
|
||
|
||
All numbers below are produced by the scripts in `scripts/`. `examples_dump.json` contains the actual
|
||
per-example inputs (documents + question + gold) and outputs (dense vs sparse) for the accuracy test.
|
||
|
||
## Headline results (Qwen3-14B, last-8 summary router)
|
||
|
||
| check | script | result |
|
||
|---|---|---|
|
||
| **Accuracy — no drop** | `acc_regress.py` | per-example on 80 real LongBench QA: dense 32/80, sparse 33/80, **0 regressions**, +1 gain |
|
||
| **Quality (NLL)** | `bench_real.py` | 50 real samples: router k=3 within ~3% NLL of dense, k=2 within ~12% |
|
||
| **No data leakage** | `audit_sparse.py` | canary secret in a masked block **cannot** be reproduced (model hallucinates); with block active it can |
|
||
| **Router — no train leakage** | `router_leak_check.py` | on unseen docs, coverage 0.92@k=2 ≈ training eval 0.96 |
|
||
| **FlexAttention stable** | `flex_attn_test.py` | flex block-attn == eager (max\|Δ\|=0.004), deterministic, T=16384 at 0.2 GB |
|
||
| **Decode savings (weights excluded)** | `decode_savings.py` | KV-read HBM & attention-FLOPs each −4× (2k ctx) … ~25× (long ctx); cap = block/summary |
|
||
|
||
## How to run
|
||
|
||
```bash
|
||
pip install torch transformers peft accelerate
|
||
MODEL=hxia7/qwen3-14b-blockdist # or a local path to the merged model
|
||
ROUTER=router.pt # ships in the repo (summary_tokens=8)
|
||
|
||
python scripts/demo_infer.py --model $MODEL --router $ROUTER --k 2 # generation demo
|
||
python scripts/audit_sparse.py --model $MODEL --router $ROUTER # leakage + savings
|
||
python scripts/acc_regress.py --model $MODEL --router $ROUTER --n 80 --k 2 # accuracy regression
|
||
python scripts/bench_real.py --model $MODEL --router $ROUTER --k 2 # NLL + savings
|
||
python scripts/router_leak_check.py --model $MODEL --router $ROUTER # router generalization
|
||
python scripts/flex_attn_test.py # flex kernel validation
|
||
python scripts/decode_savings.py # HBM/FLOP accounting
|
||
```
|
||
|
||
Datasets (LongBench-Seg etc.) load from paths hard-coded in the scripts — adjust to your HF cache.
|
||
|
||
## Honest caveats
|
||
|
||
- Absolute LongBench accuracy (~40%) is a base-model + strict-string-match artifact; dense and sparse
|
||
are near-identical — the point is **0 regressions**, not the absolute number.
|
||
- Block-sparse decode is a **KV-bandwidth (HBM) optimization**; FLOP savings only matter at very long
|
||
context. Big wins are in the long-context × batch serving regime.
|
||
- Cross-block **aggregation** queries ("list all X") are a limit of block attention itself (even dense
|
||
struggles) → serve those via dense fallback.
|
||
- Eager `output_attentions` is O(T²) and OOMs past ~2k tokens; use FlexAttention (validated here) to
|
||
scale the block mask in a real serving kernel.
|