qwen3-4b-blockdist/eval/EVAL.md

Evaluation & reproduction — block-distilled Qwen3 + landmark router

All numbers below are produced by the scripts in scripts/. examples_dump.json contains the actual per-example inputs (documents + question + gold) and outputs (dense vs sparse) for the accuracy test.

Headline results (Qwen3-14B, last-8 summary router)

check	script	result
Accuracy — no drop	acc_regress.py	per-example on 80 real LongBench QA: dense 32/80, sparse 33/80, 0 regressions, +1 gain
Quality (NLL)	bench_real.py	50 real samples: router k=3 within ~3% NLL of dense, k=2 within ~12%
No data leakage	audit_sparse.py	canary secret in a masked block cannot be reproduced (model hallucinates); with block active it can
Router — no train leakage	router_leak_check.py	on unseen docs, coverage 0.92@k=2 ≈ training eval 0.96
FlexAttention stable	flex_attn_test.py	flex block-attn == eager (max|Δ|=0.004), deterministic, T=16384 at 0.2 GB
Decode savings (weights excluded)	decode_savings.py	KV-read HBM & attention-FLOPs each −4× (2k ctx) … ~25× (long ctx); cap = block/summary

How to run

pip install torch transformers peft accelerate
MODEL=hxia7/qwen3-14b-blockdist            # or a local path to the merged model
ROUTER=router.pt                            # ships in the repo (summary_tokens=8)

python scripts/demo_infer.py       --model $MODEL --router $ROUTER --k 2        # generation demo
python scripts/audit_sparse.py     --model $MODEL --router $ROUTER              # leakage + savings
python scripts/acc_regress.py      --model $MODEL --router $ROUTER --n 80 --k 2 # accuracy regression
python scripts/bench_real.py       --model $MODEL --router $ROUTER --k 2        # NLL + savings
python scripts/router_leak_check.py --model $MODEL --router $ROUTER             # router generalization
python scripts/flex_attn_test.py                                               # flex kernel validation
python scripts/decode_savings.py                                               # HBM/FLOP accounting

Datasets (LongBench-Seg etc.) load from paths hard-coded in the scripts — adjust to your HF cache.

Honest caveats

• Absolute LongBench accuracy (~40%) is a base-model + strict-string-match artifact; dense and sparse are near-identical — the point is 0 regressions, not the absolute number.
• Block-sparse decode is a KV-bandwidth (HBM) optimization; FLOP savings only matter at very long context. Big wins are in the long-context × batch serving regime.
• Cross-block aggregation queries ("list all X") are a limit of block attention itself (even dense struggles) → serve those via dense fallback.
• Eager output_attentions is O(T²) and OOMs past ~2k tokens; use FlexAttention (validated here) to scale the block mask in a real serving kernel.