初始化项目，由ModelHub XC社区提供模型

Model: iamrahulreddy/Quintus
Source: Original Platform

docs/benchmarks.md

@@ -0,0 +1,88 @@
+# Benchmarks
+
+The release scoreboard compares Qwen3-1.7B-Base, Qwen3-1.7B-Instruct, and Quintus-1.7B. Evaluations use a mixture of EvalPlus and lm-evaluation-harness style benchmarks, with greedy or deterministic settings where applicable.
+
+For the detailed benchmark-control rules, see [Evaluation Methodology](evaluation_methodology.md).
+
+## Final Scoreboard
+
+| Benchmark | Qwen3-1.7B-Base | Qwen3-1.7B-Instruct | Quintus-1.7B |
+| :--- | :---: | :---: | :---: |
+| HumanEval pass@1 | 67.1% | 70.7% | 67.7% |
+| MBPP pass@1 | 67.2% | 58.2% | 64.8% |
+| GSM8K, 10-shot flexible | 69.98% | 69.75% | 74.30% |
+| ARC-Challenge acc_norm | 55.72% | 52.99% | 58.36% |
+| WinoGrande, 5-shot | 65.67% | 61.01% | 66.38% |
+| PIQA acc_norm | 75.63% | 72.09% | 75.57% |
+
+## Full Checkpoint Matrix
+
+The compact scoreboard above is the headline comparison. The full matrix below
+records the broader evaluation suite across four checkpoints:
+
+- `Base`: `Qwen/Qwen3-1.7B-Base`
+- `Instruct`: `Qwen/Qwen3-1.7B-Instruct`
+- `Pre-SFT`: online KD checkpoint before targeted SFT
+- `Quintus SFT`: final public Quintus checkpoint
+
+$\Delta$ vs Instruct is computed as Quintus SFT minus
+`Qwen/Qwen3-1.7B-Instruct`, in percentage points. GSM8K strict and flexible
+scores are listed separately because parser behavior and EOS handling can
+change the measured result.
+
+| Area | Benchmark | Base | Instruct | Pre-SFT | Quintus SFT | $\Delta$ vs Instruct |
+| :--- | :--- | :---: | :---: | :---: | :---: | :---: |
+| Coding | HumanEval pass@1 | 67.1% | 70.7% | 68.3% | 67.7% | -3.0 pp |
+| Coding | HumanEval+ | 60.4% | 64.0% | 62.8% | 60.4% | -3.6 pp |
+| Coding | MBPP pass@1 | 67.2% | 58.2% | 63.0% | 64.8% | +6.6 pp |
+| Coding | MBPP+ | 58.2% | 50.0% | 54.5% | 56.3% | +6.3 pp |
+| Math | GSM8K flexible | 70.0% | 69.8% | 74.4% | 74.3% | +4.5 pp |
+| Math | GSM8K strict | 69.6% | 69.8% | 74.1% | 60.9% | -8.9 pp |
+| Reasoning/commonsense | WinoGrande, 5-shot | 65.7% | 61.0% | 66.0% | 66.4% | +5.4 pp |
+| Reasoning/commonsense | ARC-Challenge acc | 51.5% | 49.5% | 51.9% | 54.8% | +5.3 pp |
+| Reasoning/commonsense | ARC-Challenge acc_norm | 55.7% | 53.0% | 55.6% | 58.4% | +5.4 pp |
+| Reasoning/commonsense | BoolQ | 79.0% | 77.5% | 77.3% | 71.6% | -5.9 pp |
+| Reasoning/commonsense | PIQA acc | 75.6% | 72.9% | 75.8% | 75.2% | +2.3 pp |
+| Reasoning/commonsense | PIQA acc_norm | 75.6% | 72.1% | 75.7% | 75.6% | +3.5 pp |
+
+## Interpretation
+
+The strongest result is the reasoning crossover: Quintus beats both the base and the official 1.7B instruct model on GSM8K, ARC-Challenge, and WinoGrande, despite remaining at the same parameter scale.
+
+The coding picture is mixed but useful:
+
+- HumanEval remains slightly below Qwen3-1.7B-Instruct.
+- MBPP is substantially above Qwen3-1.7B-Instruct, though still below the base model.
+
+This suggests the model gained useful instruction-following and reasoning behavior without fully matching larger or more heavily aligned code-specialized models.
+
+## What The Benchmarks Support
+
+These results support four claims:
+
+1. Online KD transferred reasoning capability into a compact student.
+2. The final model did not merely memorize assistant formatting; it improved several reasoning and commonsense metrics.
+3. SFT helped expose the distilled capability in an assistant setting.
+4. The model still has capacity limits typical of the 1.7B scale, especially on code execution reliability and long multi-step algorithm generation.
+
+## Evaluation Caveats
+
+Benchmark comparisons are sensitive to prompt format. Raw completion, chat-template generation, and log-likelihood multiple-choice scoring can produce different rankings. For fair interpretation:
+
+- Compare raw models against raw models when measuring base reasoning.
+- Compare chat-wrapped models against chat-wrapped models when measuring format alignment.
+- Treat open-ended qualitative prompts as alignment tests, not as a replacement for standardized benchmarks.
+
+Important implementation caveats:
+
+- GSM8K extraction can differ between strict `####` parsing and flexible number extraction.
+- Multiple-choice log-likelihood tasks can be distorted by chat templates.
+- `acc_norm` is preferred when answer-option length bias can change the ranking.
+- Metric extraction scripts must reject `stderr` and `alias` fields when looking for the actual score.
+- Runtime versions should be recorded with benchmark outputs because harness behavior can change across releases.
+
+## Earlier Development Signals
+
+Before the final Qwen3 8B -> 1.7B run, earlier experiments showed that sparse offline top-k KD could not consistently outperform strong baselines. Those runs were useful because they identified the bottleneck: sparse cached teacher logits were not dense enough to transfer deeper reasoning pathways.
+
+The final move to online full-vocabulary KD is the key methodological change behind the stronger final results.