Quintus/docs/experiment_timeline.md

# Experiment Timeline

This timeline explains why the final Quintus design looks the way it does. It focuses on the technical evolution from sparse offline distillation to the final online full-vocabulary pipeline.

## 1. Offline Top-K KD Prototype

The earliest design precomputed teacher logits to disk and trained the student from cached top-k supports.

Why it was attractive:

- Avoided loading teacher and student together.
- Reduced KD memory from full vocabulary to top-k support.
- Made cloud interruptions easier to survive because teacher logits were already saved.

Main lessons:

- Serialization contracts matter as much as loss math.
- Top-k token IDs need safe dtypes.
- Teacher-logit shards must preserve original row order.
- Missing or stale shards should fail loudly.

## 2. Static Audit And Fail-Fast Hardening

The project then moved through a static-audit phase focused on silent failure modes.

Major hardening themes:

- Dataset zero-retention checks.
- Missing-shard hard failures.
- Stale artifact cleanup.
- DeepSpeed accumulation correctness.
- Rank-safe writes.
- Explicit model revision and remote-code policy.
- Stronger provenance metadata.

This phase turned the code from a script bundle into a more reliable training pipeline.

## 3. Assistant-Only Supervision

The tokenization path originally risked supervising the whole conversation. That can over-train prompts, headers, and formatting tokens.

The corrected path derives `loss_mask` and trains only on assistant response tokens.

This changed the training contract:

- Prompt tokens provide context.
- Assistant tokens receive CE and KD loss.
- Rows without assistant targets are rejected.
- Checkpoints and datasets must agree on the mask schema.

## 4. Top-K Plus Residual Bucket

A later offline-KD pass improved the sparse support by adding an "other" bucket for teacher probability mass outside top-k.

This fixed a mathematical weakness: the student should be normalized against the full vocabulary before comparison, not only inside top-k. The residual bucket made offline KD less wrong, but it still compressed most of the teacher distribution into one scalar.

That design was useful, but not enough for flagship results.

## 5. Dataset And Objective Mismatch

Smoke runs showed a pattern that became important later: held-out KD validation loss can improve while benchmark quality worsens.

Key diagnosis:

- Matching teacher token distributions on a training corpus is not identical to improving GSM8K, ARC, coding, or open-ended assistant quality.
- Dataset order and first-N streaming can bias sample selection.
- Long reasoning traces can overweight style and process tokens relative to final answers.
- Small students can forget useful baseline behavior when full-parameter training is too aggressive.

This motivated stricter downstream evaluation gates.

## 6. Base Student Pivot

Several runs tested whether distilling into an already-instruct-tuned student caused destructive interference. The base-student hypothesis was sound: a raw base model has more plasticity and fewer alignment paths to overwrite.

The result was only a marginal improvement under offline top-k KD. That was the decisive clue.

Conclusion:

The student choice was not the main bottleneck. Offline top-k sparsity was the main bottleneck.

## 7. Offline Top-K Ceiling

With $k = 8$, the student saw only a tiny fraction of the teacher vocabulary distribution per target token:

$$
\frac{k}{|V|}
= \frac{8}{151{,}665}
\approx 5.3 \times 10^{-5}
= 0.0053\%
$$

Different $\alpha$ values, epochs, and student initializations did not remove this limit.

Offline top-k KD could perturb the student and sometimes improve narrow metrics, but it could not reliably transfer the teacher's broader reasoning distribution.

The project stopped treating offline top-k KD as the path to a flagship model.

## 8. Online Full-Vocabulary KD

![Offline vs Online KD](../assets/offline_vs_online_kd.png)

Online KD became the final architecture.

Instead of reading cached teacher shards, the training loop loads a frozen teacher and runs live teacher forward passes beside the student. The KD loss uses the teacher's full-vocabulary distribution.

Benefits:

- No top-k sparsity ceiling.
- No shard-order mismatch risk.
- No stale teacher-logit cache.
- Stronger transfer signal for reasoning.

Cost:

- Higher VRAM footprint.
- Teacher and student must fit together.
- KL computation needs chunking.
- Throughput depends heavily on packing and kernels.

## 9. Sequence Packing And B200 Tuning

Sequence packing converted padding waste into useful tokens.

The packing implementation:

- Packs only training data.
- Keeps validation easier to interpret.
- Uses fixed 4096-token bins.
- Inserts masked EOS separators.
- Stores packing metadata in checkpoints.
- Rejects packed/unpacked resume mismatches.

Development probes showed the expected utilization improvement and made online KD fast enough for serious single-GPU runs.

## 10. English-Only Final Data

The release run focuses on English samples.

Reasons:

- Reduce language drift in open-ended outputs.
- Keep the model's assistant behavior aligned with the intended release language.
- Make qualitative evaluation cleaner.
- Avoid CJK continuation artifacts after missed EOS.

The tradeoff is real: removing multilingual data can reduce access to some reasoning traces. For a public English assistant, language stability is worth that tradeoff.

## 11. Targeted SFT After KD

Online KD transferred capability, but raw KD is not a full assistant-alignment process.

Targeted SFT was added after KD to improve:

- identity grounding,
- chat format stability,
- practical assistant style,
- repetition control,
- response presentation.

This created the final two-stage public model:

```text
Qwen3-1.7B-Base
  -> online full-vocab KD from Qwen3-8B
  -> targeted SFT
  -> Quintus-1.7B
```

## 12. Release Verification

The final release surface combines:

- benchmark scoreboard,
- architecture documentation,
- evaluation methodology notes,
- pipeline hardening notes,
- weight audit,
- model-card draft.

The public docs focus on reusable methods, release results, and reproducible checks.