Quintus/docs/benchmarks.md

Benchmarks

The release scoreboard compares Qwen3-1.7B-Base, Qwen3-1.7B-Instruct, and Quintus-1.7B. Evaluations use a mixture of EvalPlus and lm-evaluation-harness style benchmarks, with greedy or deterministic settings where applicable.

For the detailed benchmark-control rules, see Evaluation Methodology.

Final Scoreboard

Benchmark	Qwen3-1.7B-Base	Qwen3-1.7B-Instruct	Quintus-1.7B
HumanEval pass@1	67.1%	70.7%	67.7%
MBPP pass@1	67.2%	58.2%	64.8%
GSM8K, 10-shot flexible	69.98%	69.75%	74.30%
ARC-Challenge acc_norm	55.72%	52.99%	58.36%
WinoGrande, 5-shot	65.67%	61.01%	66.38%
PIQA acc_norm	75.63%	72.09%	75.57%

Full Checkpoint Matrix

The compact scoreboard above is the headline comparison. The full matrix below records the broader evaluation suite across four checkpoints:

• Base: Qwen/Qwen3-1.7B-Base
• Instruct: Qwen/Qwen3-1.7B-Instruct
• Pre-SFT: online KD checkpoint before targeted SFT
• Quintus SFT: final public Quintus checkpoint

\Delta vs Instruct is computed as Quintus SFT minus Qwen/Qwen3-1.7B-Instruct, in percentage points. GSM8K strict and flexible scores are listed separately because parser behavior and EOS handling can change the measured result.

Area	Benchmark	Base	Instruct	Pre-SFT	Quintus SFT	\Delta vs Instruct
Coding	HumanEval pass@1	67.1%	70.7%	68.3%	67.7%	-3.0 pp
Coding	HumanEval+	60.4%	64.0%	62.8%	60.4%	-3.6 pp
Coding	MBPP pass@1	67.2%	58.2%	63.0%	64.8%	+6.6 pp
Coding	MBPP+	58.2%	50.0%	54.5%	56.3%	+6.3 pp
Math	GSM8K flexible	70.0%	69.8%	74.4%	74.3%	+4.5 pp
Math	GSM8K strict	69.6%	69.8%	74.1%	60.9%	-8.9 pp
Reasoning/commonsense	WinoGrande, 5-shot	65.7%	61.0%	66.0%	66.4%	+5.4 pp
Reasoning/commonsense	ARC-Challenge acc	51.5%	49.5%	51.9%	54.8%	+5.3 pp
Reasoning/commonsense	ARC-Challenge acc_norm	55.7%	53.0%	55.6%	58.4%	+5.4 pp
Reasoning/commonsense	BoolQ	79.0%	77.5%	77.3%	71.6%	-5.9 pp
Reasoning/commonsense	PIQA acc	75.6%	72.9%	75.8%	75.2%	+2.3 pp
Reasoning/commonsense	PIQA acc_norm	75.6%	72.1%	75.7%	75.6%	+3.5 pp

Interpretation

The strongest result is the reasoning crossover: Quintus beats both the base and the official 1.7B instruct model on GSM8K, ARC-Challenge, and WinoGrande, despite remaining at the same parameter scale.

The coding picture is mixed but useful:

• HumanEval remains slightly below Qwen3-1.7B-Instruct.
• MBPP is substantially above Qwen3-1.7B-Instruct, though still below the base model.

This suggests the model gained useful instruction-following and reasoning behavior without fully matching larger or more heavily aligned code-specialized models.

What The Benchmarks Support

These results support four claims:

1. Online KD transferred reasoning capability into a compact student.
2. The final model did not merely memorize assistant formatting; it improved several reasoning and commonsense metrics.
3. SFT helped expose the distilled capability in an assistant setting.
4. The model still has capacity limits typical of the 1.7B scale, especially on code execution reliability and long multi-step algorithm generation.

Evaluation Caveats

Benchmark comparisons are sensitive to prompt format. Raw completion, chat-template generation, and log-likelihood multiple-choice scoring can produce different rankings. For fair interpretation:

• Compare raw models against raw models when measuring base reasoning.
• Compare chat-wrapped models against chat-wrapped models when measuring format alignment.
• Treat open-ended qualitative prompts as alignment tests, not as a replacement for standardized benchmarks.

Important implementation caveats:

• GSM8K extraction can differ between strict #### parsing and flexible number extraction.
• Multiple-choice log-likelihood tasks can be distorted by chat templates.
• acc_norm is preferred when answer-option length bias can change the ranking.
• Metric extraction scripts must reject stderr and alias fields when looking for the actual score.
• Runtime versions should be recorded with benchmark outputs because harness behavior can change across releases.

Earlier Development Signals

Before the final Qwen3 8B -> 1.7B run, earlier experiments showed that sparse offline top-k KD could not consistently outperform strong baselines. Those runs were useful because they identified the bottleneck: sparse cached teacher logits were not dense enough to transfer deeper reasoning pathways.

The final move to online full-vocabulary KD is the key methodological change behind the stronger final results.