iamrahulreddy/Quintus
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: iamrahulreddy/Quintus
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-28 21:11:02 +08:00
commit 930b4e9f2c
53 changed files with 9557 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

88
docs/architecture.md Normal file
View File

@@ -0,0 +1,88 @@
# Architecture
Quintus is built as a two-stage model development pipeline:
1. Online full-vocabulary knowledge distillation from a larger Qwen3 teacher into a Qwen3-1.7B base student.
2. Targeted SFT to improve instruction-following behavior, persona consistency, and generation stability.
![Quintus Architecture](../assets/quintus_architecture.png)
## Core Training Path
The main training entry point is `src/train.py`. It supports three phases:
- `sft`: Cross-entropy training on assistant response tokens.
- `kd`: Offline top-k teacher-logit distillation, retained for compatibility and provenance checks.
- `online_kd`: The final preferred path. Teacher logits are produced live during the student forward pass.
The final KD objective is implemented in `src/losses.py`:
$$
\mathcal{L}_{\text{total}}
= \alpha \mathcal{L}_{\text{CE}}
+ (1 - \alpha)\mathcal{L}_{\text{KD}}
$$
For the final run, $\alpha = 0.3$ and $T = 2.0$. In this codebase, $\alpha$ is the cross-entropy weight. The complementary weight is assigned to the KD term.
## Data Flow
`src/download.py` prepares the training data. It handles both pre-tokenized rows and raw instruction data. For raw rows, it normalizes common conversation schemas, applies the tokenizer chat template, and builds an assistant-only `loss_mask`.
Important details:
- Prompt and formatting tokens are masked out.
- Assistant response tokens receive loss.
- Samples longer than `max_seq_len` are rejected rather than silently truncated.
- The tokenizer contract is later validated to avoid teacher/student vocabulary mismatches.
## Sequence Packing
`src/sequence_packing.py` implements deterministic first-fit decreasing packing. It places multiple shorter samples into fixed-length bins, separated by EOS tokens.
Packing properties:
- Training split is packed; validation can remain unpacked for interpretability.
- Bins are fixed at `pack_length = 4096` in the final profile.
- EOS separators have `loss_mask = 0`.
- The first token after a separator is optionally masked to avoid cross-sample target leakage.
- Attention masks are built from the true packed length, not by comparing token IDs against `pad_token_id`.
The attention-mask detail is important because Qwen tokenizers can reuse EOS-like IDs in ways that make token-identity-derived padding masks unsafe.
## Online KD Memory Strategy
Full-vocabulary KD is expensive because both student and teacher produce logits shaped as:
$$
\text{student\_logits},\ \text{teacher\_logits}
\in \mathbb{R}^{B \times S \times |V|}
$$
The implementation keeps this feasible by chunking along the token dimension with:
$$
C_{\text{KD}} = 2048
$$
Each chunk computes the teacher softmax, student log-softmax, and masked KL contribution, then accumulates the result. This preserves the dense teacher distribution while avoiding a single large KL workspace.
## Validation, Provenance, And Safety Checks
Several modules exist to prevent silent training corruption:
- `src/provenance.py`: Validates tokenizer contracts, vocab sizes, revisions, and teacher-logit metadata.
- `src/kd_contracts.py`: Builds deterministic tokenizer fingerprints.
- `src/training_schedule.py`: Aligns train/validation splits with batch and gradient-accumulation constraints.
- `src/checkpoints.py`: Saves model, tokenizer, scheduler, trainer state, and packing metadata; validates resume compatibility.
- `src/transformers_compat.py`: Resolves attention backend and formats model-loading errors.
## SFT Layer
The `sft/` directory contains the post-KD alignment layer:
- `sft/train_sft.py`: SFT training with optional sequence packing, LoRA/QLoRA paths, and built-in spot evaluations.
- `sft/evaluate.py`: EvalPlus and lm-evaluation-harness orchestration.
- `sft/chat.py`: Local interactive chat wrapper using the tokenizer chat template.
This stage is intentionally separate from KD. KD transfers the teacher's probability structure; SFT teaches the model how to expose that capability in the intended assistant format.

88
docs/benchmarks.md Normal file
View File

@@ -0,0 +1,88 @@
# Benchmarks
The release scoreboard compares Qwen3-1.7B-Base, Qwen3-1.7B-Instruct, and Quintus-1.7B. Evaluations use a mixture of EvalPlus and lm-evaluation-harness style benchmarks, with greedy or deterministic settings where applicable.
For the detailed benchmark-control rules, see [Evaluation Methodology](evaluation_methodology.md).
## Final Scoreboard
| Benchmark | Qwen3-1.7B-Base | Qwen3-1.7B-Instruct | Quintus-1.7B |
| :--- | :---: | :---: | :---: |
| HumanEval pass@1 | 67.1% | 70.7% | 67.7% |
| MBPP pass@1 | 67.2% | 58.2% | 64.8% |
| GSM8K, 10-shot flexible | 69.98% | 69.75% | 74.30% |
| ARC-Challenge acc_norm | 55.72% | 52.99% | 58.36% |
| WinoGrande, 5-shot | 65.67% | 61.01% | 66.38% |
| PIQA acc_norm | 75.63% | 72.09% | 75.57% |
## Full Checkpoint Matrix
The compact scoreboard above is the headline comparison. The full matrix below
records the broader evaluation suite across four checkpoints:
- `Base`: `Qwen/Qwen3-1.7B-Base`
- `Instruct`: `Qwen/Qwen3-1.7B-Instruct`
- `Pre-SFT`: online KD checkpoint before targeted SFT
- `Quintus SFT`: final public Quintus checkpoint
$\Delta$ vs Instruct is computed as Quintus SFT minus
`Qwen/Qwen3-1.7B-Instruct`, in percentage points. GSM8K strict and flexible
scores are listed separately because parser behavior and EOS handling can
change the measured result.
| Area | Benchmark | Base | Instruct | Pre-SFT | Quintus SFT | $\Delta$ vs Instruct |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: |
| Coding | HumanEval pass@1 | 67.1% | 70.7% | 68.3% | 67.7% | -3.0 pp |
| Coding | HumanEval+ | 60.4% | 64.0% | 62.8% | 60.4% | -3.6 pp |
| Coding | MBPP pass@1 | 67.2% | 58.2% | 63.0% | 64.8% | +6.6 pp |
| Coding | MBPP+ | 58.2% | 50.0% | 54.5% | 56.3% | +6.3 pp |
| Math | GSM8K flexible | 70.0% | 69.8% | 74.4% | 74.3% | +4.5 pp |
| Math | GSM8K strict | 69.6% | 69.8% | 74.1% | 60.9% | -8.9 pp |
| Reasoning/commonsense | WinoGrande, 5-shot | 65.7% | 61.0% | 66.0% | 66.4% | +5.4 pp |
| Reasoning/commonsense | ARC-Challenge acc | 51.5% | 49.5% | 51.9% | 54.8% | +5.3 pp |
| Reasoning/commonsense | ARC-Challenge acc_norm | 55.7% | 53.0% | 55.6% | 58.4% | +5.4 pp |
| Reasoning/commonsense | BoolQ | 79.0% | 77.5% | 77.3% | 71.6% | -5.9 pp |
| Reasoning/commonsense | PIQA acc | 75.6% | 72.9% | 75.8% | 75.2% | +2.3 pp |
| Reasoning/commonsense | PIQA acc_norm | 75.6% | 72.1% | 75.7% | 75.6% | +3.5 pp |
## Interpretation
The strongest result is the reasoning crossover: Quintus beats both the base and the official 1.7B instruct model on GSM8K, ARC-Challenge, and WinoGrande, despite remaining at the same parameter scale.
The coding picture is mixed but useful:
- HumanEval remains slightly below Qwen3-1.7B-Instruct.
- MBPP is substantially above Qwen3-1.7B-Instruct, though still below the base model.
This suggests the model gained useful instruction-following and reasoning behavior without fully matching larger or more heavily aligned code-specialized models.
## What The Benchmarks Support
These results support four claims:
1. Online KD transferred reasoning capability into a compact student.
2. The final model did not merely memorize assistant formatting; it improved several reasoning and commonsense metrics.
3. SFT helped expose the distilled capability in an assistant setting.
4. The model still has capacity limits typical of the 1.7B scale, especially on code execution reliability and long multi-step algorithm generation.
## Evaluation Caveats
Benchmark comparisons are sensitive to prompt format. Raw completion, chat-template generation, and log-likelihood multiple-choice scoring can produce different rankings. For fair interpretation:
- Compare raw models against raw models when measuring base reasoning.
- Compare chat-wrapped models against chat-wrapped models when measuring format alignment.
- Treat open-ended qualitative prompts as alignment tests, not as a replacement for standardized benchmarks.
Important implementation caveats:
- GSM8K extraction can differ between strict `####` parsing and flexible number extraction.
- Multiple-choice log-likelihood tasks can be distorted by chat templates.
- `acc_norm` is preferred when answer-option length bias can change the ranking.
- Metric extraction scripts must reject `stderr` and `alias` fields when looking for the actual score.
- Runtime versions should be recorded with benchmark outputs because harness behavior can change across releases.
## Earlier Development Signals
Before the final Qwen3 8B -> 1.7B run, earlier experiments showed that sparse offline top-k KD could not consistently outperform strong baselines. Those runs were useful because they identified the bottleneck: sparse cached teacher logits were not dense enough to transfer deeper reasoning pathways.
The final move to online full-vocabulary KD is the key methodological change behind the stronger final results.

152
docs/engineering_insights.md Normal file
View File

@@ -0,0 +1,152 @@
# Engineering Insights
This project evolved through several failed and successful training designs. The useful lessons are summarized here as public engineering notes.
For expanded operational detail, see [Training Playbook](training_playbook.md), [Pipeline Hardening](pipeline_hardening.md), and [Evaluation Methodology](evaluation_methodology.md).
## 1. Sparse Offline KD Hit A Ceiling
The earliest distillation path cached only a small top-k slice of teacher logits. That made training cheaper, but it discarded most of the teacher distribution. With a vocabulary of roughly 151K tokens and $k = 8$, the visible support was:
$$
\frac{k}{|V|}
= \frac{8}{151{,}665}
\approx 5.3 \times 10^{-5}
= 0.0053\%
$$
The result was clear: top-k KD could perturb the student, but it did not transfer enough "dark knowledge" to reliably improve reasoning. Different alphas, epochs, and student initializations could not escape this sparse-signal ceiling.
The final fix was to use online KD: load teacher and student together, run both forward passes, and compute KL against the teacher's full vocabulary distribution.
## 2. Base Student Was Better Than Fighting An Aligned Space
Distilling into an already instruction-tuned student can cause destructive interference. The student's weights already encode one aligned behavior manifold, while the teacher's soft logits pull toward another. Training can look numerically stable while reasoning metrics regress.
The final path uses `Qwen/Qwen3-1.7B-Base` as the student. The base model has more plasticity, while the CE term and later SFT stage teach assistant formatting.
## 3. KD And Alignment Are Different Problems
Standardized benchmarks showed that KD can improve reasoning and calibration, but open-ended chat quality still needs alignment data.
The important diagnosis:
- A distillation failure means the student did not absorb the teacher's useful probability structure.
- An alignment gap means the student has capability, but the generation path is not yet trained to behave like a polished assistant.
The project therefore separates the pipeline into KD first, then SFT.
## 4. Assistant-Only Loss Masking Matters
A key bug class was assigning loss to chat formatting tokens instead of only assistant response content. If the model is trained to optimize structural tokens too heavily, it can learn formatting before substance.
The current tokenization path derives an assistant-only `loss_mask`, so:
- User prompts are context, not targets.
- Chat headers and separators are masked.
- Assistant response tokens are the only supervised targets.
This keeps training focused on semantic outputs rather than wrapper reproduction.
## 5. Sequence Packing Was The Main Throughput Win
The dataset contains many sequences shorter than the maximum context length. Dynamic padding wastes a large fraction of compute. First-fit decreasing sequence packing converted that waste into useful tokens.
Observed engineering outcome:
- Unpacked B200 online KD ran around the low-20K tokens/sec range in earlier probes.
- Packed B200 online KD reached roughly the mid-40K tokens/sec range after warmup.
- Packed utilization was close to full 4096-token bins.
The final code keeps packing deterministic and stores packing metadata in checkpoints so packed/unpacked resume mismatches fail loudly.
## 6. Full-Vocab KD Needed Token Chunking
Online KD preserves the full teacher distribution, but a full KL workspace at Qwen vocabulary scale is too large to materialize casually:
$$
\text{KL workspace} \sim \mathbb{R}^{B \times S \times |V|}
$$
The solution is token-dimension chunking. The current implementation uses:
$$
C_{\text{KD}} = 2048
$$
Larger chunks reduce loop overhead, but increase temporary memory pressure. The selected value is a practical B200-oriented balance for the 8B -> 1.7B workload.
## 7. Shape Churn And Synchronization Can Quietly Drain Throughput
Several performance bugs were not correctness bugs:
- Dynamic sequence lengths caused allocator churn.
- Repeated `.item()` calls forced CPU-GPU synchronization.
- Single-GPU DeepSpeed could add overhead when the model already fit comfortably.
- `torch.compile` added memory overhead, dynamic-shape graph breaks, recompile overhead, and checkpoint portability risk.
The final training loop favors stable shapes, fewer scalar syncs, fused AdamW when available, FlashAttention when available, and Liger kernels where they do not conflict with KD logits.
## 8. Evaluation Requires Controlled Comparisons
Raw completion and chat-template evaluation activate different behavior. A base model can perform well in raw mode and poorly under chat markup. A chat-aligned model can underperform on raw continuation-style tasks if the benchmark asks for direct option likelihoods.
The project uses both controls:
- Raw-to-raw comparisons isolate distilled base capability.
- Chat-to-chat comparisons estimate template robustness and assistant-format alignment.
This distinction avoids blaming KD for failures that belong to alignment or benchmark formatting.
## 9. Post-KD SFT Is Not Optional For Assistant Quality
KD transfers probability structure; it does not guarantee careful behavior, refusal policy, calibrated uncertainty, or code reliability. Targeted SFT was added to address:
- Confident hallucination in open-ended answers.
- Persona and identity consistency.
- Repetition loops.
- Chat-format stability.
- Practical assistant presentation.
Preference training or DPO would be the natural next layer if the project continues beyond the current release.
## 10. Training Loss Is Not The Release Gate
Several development runs looked numerically healthy while downstream benchmarks moved in the wrong direction. That pattern is expected when the training objective is only a proxy for the release objective.
Useful release gates:
- Standardized benchmarks.
- Raw and chat controls.
- Mismatch inspection.
- Qualitative prompts after benchmark checks.
- Weight and checkpoint structure audits.
Held-out KD validation loss is important, but it cannot prove that the model improved on math, code, multiple-choice reasoning, or assistant behavior.
## 11. Fail-Fast Beats Silent Recovery
The pipeline hardened around a simple rule: corrupt artifacts should stop the run.
Examples:
- Missing teacher-logit shards fail instead of becoming zero tensors.
- Tokenization with zero usable rows fails immediately.
- Shard schema mismatches are rejected.
- Packed/unpacked checkpoint resume mismatches are rejected.
- Stale evaluation outputs are cleaned before new scores are written.
This makes errors louder, but it keeps published numbers trustworthy.
## 12. Public Docs Should Preserve Decisions
A release-quality project should expose durable engineering conclusions:
- why online KD replaced offline top-k KD,
- why assistant-only masking matters,
- why raw/chat evaluation controls are required,
- why sequence packing changed throughput,
- why SFT remains necessary after KD,
- why checkpoint and provenance checks exist.
That level of detail is enough for technical readers without turning the documentation into a chronological run journal.

234
docs/evaluation_methodology.md Normal file
View File

@@ -0,0 +1,234 @@
# Evaluation Methodology
Evaluation was one of the hardest parts of Quintus. Several early scores were misleading until prompt format, metric extraction, parser behavior, and runtime artifacts were audited carefully.
## Evaluation Principle
A model comparison is only meaningful when the prompt format and metric path match the question being asked.
Two distinct questions matter:
- Base capability: Does the distilled model improve raw reasoning and likelihood behavior?
- Assistant behavior: Does the distilled model handle chat formatting and produce usable responses?
Those questions need separate controls.
## Run Identity And Determinism
A benchmark record should identify the checkpoint role (`best` or `last`), exact model directory or revision, prompt mode, seeds, decoding mode, and runtime versions. Greedy decoding with fixed seeds makes repeated runs easier to compare, but hardware and kernel drift can still change edge cases.
Treat determinism as an artifact contract, not a vague claim.
## Raw-To-Raw And Chat-To-Chat
Raw completion and chat-template prompting activate different model behavior. A base model can be strong in raw mode and weak under chat markup. An instruct model can be strong in chat, but weak on raw continuation-style likelihood tasks.
Recommended controls:
- Raw-to-raw: compare base-style prompts against base-style prompts.
- Chat-to-chat: compare chat-wrapped prompts against chat-wrapped prompts.
- Raw-vs-chat within the same model: measure format tax.
Avoid comparing a chat-wrapped distilled model directly against a raw base baseline and treating the delta as pure capability transfer.
## Log-Likelihood Tasks Should Usually Stay Raw
Multiple-choice tasks such as ARC-Challenge, HellaSwag, and PIQA often score options by likelihood:
$$
P(\text{option}\mid\text{prompt})
$$
Wrapping the prompt in chat markup changes the next-token distribution. An aligned model may not want to begin a response with a bare option string after `<|im_start|>assistant`, so option likelihoods can fall for formatting reasons rather than reasoning reasons.
For log-likelihood tasks:
- Use raw completion format unless the benchmark was designed for chat.
- Prefer `acc_norm` where length bias matters.
- Record whether chat templates were applied.
## GSM8K Parser Traps
GSM8K evaluation can be distorted by parser behavior.
Two common filters behave differently:
- `strict-match`: looks for an answer after the `####` delimiter.
- `flexible-extract`: searches for numbers and may choose the last matched number.
A chat model can solve the problem, emit the correct `####` answer, miss EOS, and continue into a hallucinated next dialogue turn containing another number. In that case:
- `strict-match` may score the response correct.
- `flexible-extract` may grab the later hallucinated number and score it wrong.
This is not just a parser detail. It reveals an EOS and prompt-format interaction.
Mitigations:
- Register all relevant EOS tokens, including `<|im_end|>` and `<|endoftext|>`.
- Use deterministic generation for benchmark runs.
- Avoid excessive `fewshot_as_multiturn` wrapping unless the model was trained for that shape.
- Inspect mismatches, not just aggregate scores.
## Reasoning Models Need Enough Generation Budget
Instruction-tuned reasoning models may spend hundreds of tokens inside a reasoning trace before reaching the final answer. If `max_new_tokens` is too small, the model can be cut off before emitting the final answer marker.
That can make a capable model appear weak under exact-match metrics.
For fair GSM8K-style generation:
- Set a sufficient generation limit.
- Track truncation rate.
- Compare extracted answers against raw responses during audits.
## Batched Generation Details
Decoder-only batched generation should use left-padding. Right-padding can put the next-token position on padding for shorter prompts and make batched outputs differ from single-sample outputs.
Generation parsers should:
- Set `tokenizer.padding_side = "left"` for batched generation.
- Slice decoded continuations by each prompt's true input length.
- Stop at the first registered EOS token.
- Record truncation and empty-generation counts.
## English-Only Evaluation Controls
For English-only release checks, filtering the dataset is necessary but not sufficient. Evaluation should also use an English-only system instruction when chat prompts are enabled, register all relevant EOS IDs, and clean generated artifacts that continue into another language after the intended answer.
This cleanup is an evaluation-artifact guard. It is not a substitute for training data quality, SFT, preference tuning, or behavioral calibration.
## Metric Extraction Must Be Strict
Post-processing scripts should never fall back loosely to any metric key that starts with the right prefix. A loose fallback can accidentally read:
- `*_stderr`
- `alias`
- a different filter result
Robust extraction should:
- Match the exact metric and filter name.
- Ignore stderr and alias fields when extracting scores.
- Fail loudly if the expected key is absent.
## Boolean CLI Flags
Some harness flags use `action="store_true"`. Passing `"False"` after such a flag does not disable it; the presence of the flag enables it.
Correct pattern:
- Include the flag only when true.
- Omit the flag when false.
This matters for options such as multiturn few-shot formatting.
## Sample Log Format
`lm-evaluation-harness` may log different filters for the same document as separate JSONL objects with the same `doc_id`. A parser that assumes one object contains all filters can crash or silently compare the wrong fields.
Correct approach:
- Group sample records by `doc_id`.
- Index filter-specific records inside each group.
- Compare strict and flexible outputs from the same document.
## JSONL Parsing With Unicode Line Separators
Model outputs can contain Unicode line separator characters such as `\u2028` or `\u2029`. Calling `str.splitlines()` on a whole JSONL file can split a valid JSON string into invalid fragments.
Robust JSONL parsing:
```python
with open(path, "r", encoding="utf-8") as f:
for line in f:
if line.strip():
record = json.loads(line)
```
Iterating the file handle respects actual line endings and does not split on Unicode separators inside JSON strings.
## Hub Loading And Snapshot Hygiene
If weights or datasets are stored on the Hub, the client should be told the correct repository type. Download or snapshot the artifact first, verify that expected files exist, then pass the local directory to Transformers, vLLM, or the evaluation harness.
This separates transfer failures from engine construction and avoids repeated downloads during long benchmark runs.
Optional high-throughput Hub transfer backends such as `hf_transfer` can reduce setup time, but the correctness contract is still local snapshot validation.
## Path-Length And Output Artifacts
Evaluation tools can derive output paths from model paths. Deep Hugging Face cache paths can become extremely long after sanitization, especially on Windows.
Public guidance:
- Copy or symlink model weights to a short local directory before evaluation.
- Pass short relative paths to the evaluator.
- Keep result directories shallow.
- Fail if expected sample files are missing.
This prevents silent write failures and missing-output confusion.
## vLLM Evaluation Settings
For large benchmark runs, vLLM can greatly reduce runtime through continuous batching and KV-cache management.
Useful settings in development:
- `batch_size = auto`
- prefix caching enabled
- PagedAttention-backed KV-cache management when available
- bounded GPU memory utilization
- explicit `max_model_len` where context bounds matter
- explicit attention backend where the runtime supports it
- local pre-caching of model snapshots before engine construction
- explicit engine teardown between model runs
The benchmark artifact should record runtime versions for:
- `lm-eval`
- `vllm`
- `transformers`
- `torch`
- `datasets`
- `accelerate`
Version drift can change metric keys, generation behavior, attention backends, and output formats.
## Qualitative Evaluation
Open-ended prompt suites are useful, but they are not replacements for standardized benchmarks.
A good qualitative suite should:
- Compare raw and chat modes separately.
- Use fixed prompts and deterministic ordering.
- Include benchmark-template leakage probes.
- Include factual, math, code, system design, and LLM-internals prompts.
- Record complete outputs.
- Inspect inherited base-model errors separately from new chat-mode errors.
Qualitative failures should be classified:
- Distillation failure: the student did not absorb useful teacher probability structure.
- Alignment gap: capability exists, but the generation path lacks SFT, preference tuning, or calibration.
- Data contamination: the model repeats benchmark or pretraining artifacts.
- Code reliability gap: prose is correct, but generated code violates stated constraints.
This distinction prevents the wrong fix. Distillation failures need KD changes. Alignment gaps need SFT, DPO, RLHF, or curated behavior data.
## Release Gate
The final checkpoint should pass all of these before public claims are made:
- Benchmark tasks use the intended prompt format.
- Metric keys are exact.
- Sample counts match the full benchmark set.
- Raw and chat comparisons are not mixed.
- Generation limits are sufficient for the model style.
- Checkpoint identity is explicit.
- Missing requested checkpoints fail instead of falling back to older local weights.
- Runtime versions are recorded.
- Mismatch samples are inspected for parser artifacts.
- No stale result directory or old JSON file is reused.

181
docs/experiment_timeline.md Normal file
View File

@@ -0,0 +1,181 @@
# Experiment Timeline
This timeline explains why the final Quintus design looks the way it does. It focuses on the technical evolution from sparse offline distillation to the final online full-vocabulary pipeline.
## 1. Offline Top-K KD Prototype
The earliest design precomputed teacher logits to disk and trained the student from cached top-k supports.
Why it was attractive:
- Avoided loading teacher and student together.
- Reduced KD memory from full vocabulary to top-k support.
- Made cloud interruptions easier to survive because teacher logits were already saved.
Main lessons:
- Serialization contracts matter as much as loss math.
- Top-k token IDs need safe dtypes.
- Teacher-logit shards must preserve original row order.
- Missing or stale shards should fail loudly.
## 2. Static Audit And Fail-Fast Hardening
The project then moved through a static-audit phase focused on silent failure modes.
Major hardening themes:
- Dataset zero-retention checks.
- Missing-shard hard failures.
- Stale artifact cleanup.
- DeepSpeed accumulation correctness.
- Rank-safe writes.
- Explicit model revision and remote-code policy.
- Stronger provenance metadata.
This phase turned the code from a script bundle into a more reliable training pipeline.
## 3. Assistant-Only Supervision
The tokenization path originally risked supervising the whole conversation. That can over-train prompts, headers, and formatting tokens.
The corrected path derives `loss_mask` and trains only on assistant response tokens.
This changed the training contract:
- Prompt tokens provide context.
- Assistant tokens receive CE and KD loss.
- Rows without assistant targets are rejected.
- Checkpoints and datasets must agree on the mask schema.
## 4. Top-K Plus Residual Bucket
A later offline-KD pass improved the sparse support by adding an "other" bucket for teacher probability mass outside top-k.
This fixed a mathematical weakness: the student should be normalized against the full vocabulary before comparison, not only inside top-k. The residual bucket made offline KD less wrong, but it still compressed most of the teacher distribution into one scalar.
That design was useful, but not enough for flagship results.
## 5. Dataset And Objective Mismatch
Smoke runs showed a pattern that became important later: held-out KD validation loss can improve while benchmark quality worsens.
Key diagnosis:
- Matching teacher token distributions on a training corpus is not identical to improving GSM8K, ARC, coding, or open-ended assistant quality.
- Dataset order and first-N streaming can bias sample selection.
- Long reasoning traces can overweight style and process tokens relative to final answers.
- Small students can forget useful baseline behavior when full-parameter training is too aggressive.
This motivated stricter downstream evaluation gates.
## 6. Base Student Pivot
Several runs tested whether distilling into an already-instruct-tuned student caused destructive interference. The base-student hypothesis was sound: a raw base model has more plasticity and fewer alignment paths to overwrite.
The result was only a marginal improvement under offline top-k KD. That was the decisive clue.
Conclusion:
The student choice was not the main bottleneck. Offline top-k sparsity was the main bottleneck.
## 7. Offline Top-K Ceiling
With $k = 8$, the student saw only a tiny fraction of the teacher vocabulary distribution per target token:
$$
\frac{k}{|V|}
= \frac{8}{151{,}665}
\approx 5.3 \times 10^{-5}
= 0.0053\%
$$
Different $\alpha$ values, epochs, and student initializations did not remove this limit.
Offline top-k KD could perturb the student and sometimes improve narrow metrics, but it could not reliably transfer the teacher's broader reasoning distribution.
The project stopped treating offline top-k KD as the path to a flagship model.
## 8. Online Full-Vocabulary KD
![Offline vs Online KD](../assets/offline_vs_online_kd.png)
Online KD became the final architecture.
Instead of reading cached teacher shards, the training loop loads a frozen teacher and runs live teacher forward passes beside the student. The KD loss uses the teacher's full-vocabulary distribution.
Benefits:
- No top-k sparsity ceiling.
- No shard-order mismatch risk.
- No stale teacher-logit cache.
- Stronger transfer signal for reasoning.
Cost:
- Higher VRAM footprint.
- Teacher and student must fit together.
- KL computation needs chunking.
- Throughput depends heavily on packing and kernels.
## 9. Sequence Packing And B200 Tuning
Sequence packing converted padding waste into useful tokens.
The packing implementation:
- Packs only training data.
- Keeps validation easier to interpret.
- Uses fixed 4096-token bins.
- Inserts masked EOS separators.
- Stores packing metadata in checkpoints.
- Rejects packed/unpacked resume mismatches.
Development probes showed the expected utilization improvement and made online KD fast enough for serious single-GPU runs.
## 10. English-Only Final Data
The release run focuses on English samples.
Reasons:
- Reduce language drift in open-ended outputs.
- Keep the model's assistant behavior aligned with the intended release language.
- Make qualitative evaluation cleaner.
- Avoid CJK continuation artifacts after missed EOS.
The tradeoff is real: removing multilingual data can reduce access to some reasoning traces. For a public English assistant, language stability is worth that tradeoff.
## 11. Targeted SFT After KD
Online KD transferred capability, but raw KD is not a full assistant-alignment process.
Targeted SFT was added after KD to improve:
- identity grounding,
- chat format stability,
- practical assistant style,
- repetition control,
- response presentation.
This created the final two-stage public model:
```text
Qwen3-1.7B-Base
-> online full-vocab KD from Qwen3-8B
-> targeted SFT
-> Quintus-1.7B
```
## 12. Release Verification
The final release surface combines:
- benchmark scoreboard,
- architecture documentation,
- evaluation methodology notes,
- pipeline hardening notes,
- weight audit,
- model-card draft.
The public docs focus on reusable methods, release results, and reproducible checks.

178
docs/huggingface_model_card.md Normal file
View File

@@ -0,0 +1,178 @@
# Quintus-1.7B
Quintus-1.7B is a compact instruction-following assistant derived from `Qwen/Qwen3-1.7B-Base`. It was trained with online full-vocabulary knowledge distillation from a larger Qwen3-8B teacher, followed by targeted SFT for assistant behavior and generation stability.
## Model Details
- Base architecture: Qwen3-1.7B
- Base checkpoint: `Qwen/Qwen3-1.7B-Base`
- Distillation teacher: Qwen3-8B class teacher
- Training method: Online full-vocabulary KD + targeted SFT
- Context length used in training: 4096 tokens
- Primary language focus: English
- Release repository: `iamrahulreddy/Quintus`
- Attention path: FlashAttention-2 when available
- Training kernels: Liger kernels for compatible Qwen-family operators
- Optimizer: fused AdamW
## Intended Use
Quintus is intended for:
- General assistant use.
- Reasoning and math prompts.
- Lightweight coding assistance.
- Local experimentation with compact LLMs.
- Research into online KD and small-model alignment.
It is not intended as a safety-critical decision system. Like other compact language models, it can hallucinate and should be verified on high-stakes tasks.
## Training Summary
The training pipeline has two main stages:
1. Online KD: The student learns from the teacher's dense full-vocabulary probability distribution. This avoids the sparse top-k ceiling encountered in earlier offline KD experiments.
2. SFT: The distilled checkpoint is tuned on curated instruction/persona data to improve assistant-style behavior and reduce repetition or formatting drift.
The KD loss combines assistant-token cross entropy and teacher-student KL divergence:
$$
\mathcal{L}_{\text{total}}
= \alpha \mathcal{L}_{\text{CE}}
+ (1 - \alpha)\mathcal{L}_{\text{KD}}
$$
For the release run, $\alpha = 0.3$ and $T = 2.0$.
`torch.compile` was kept disabled for the final KD path because this workload showed high Inductor memory overhead, dynamic-shape graph breaks, recompile overhead, and checkpoint portability risk from `_orig_mod.` state-dict prefixes when compiled modules are not unwrapped before saving.
## Evaluation
| Benchmark | Qwen3-1.7B-Base | Qwen3-1.7B-Instruct | Quintus-1.7B |
| :--- | :---: | :---: | :---: |
| HumanEval pass@1 | 67.1% | 70.7% | 67.7% |
| MBPP pass@1 | 67.2% | 58.2% | 64.8% |
| GSM8K, 10-shot flexible | 69.98% | 69.75% | 74.30% |
| ARC-Challenge acc_norm | 55.72% | 52.99% | 58.36% |
| WinoGrande, 5-shot | 65.67% | 61.01% | 66.38% |
| PIQA acc_norm | 75.63% | 72.09% | 75.57% |
## Strengths
- Strong math and reasoning transfer for the 1.7B parameter scale.
- Good commonsense and ARC-style benchmark performance.
- Compact enough for lower-resource deployment compared with larger teachers.
- Public weight audit indicates healthy structural divergence from the base checkpoint without collapse.
## Limitations
- The model can still produce confident factual errors.
- Code generation can contradict stated complexity constraints.
- It is smaller than the teacher and inherits capacity limits of the 1.7B scale.
- Evaluation results depend on prompt format; raw and chat-template modes are not interchangeable.
- Additional preference tuning would likely improve calibration and refusal behavior.
## Example Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
PUBLIC_REPO_ID = "iamrahulreddy/Quintus"
print(f"Loading Quintus from {PUBLIC_REPO_ID}...")
tokenizer = AutoTokenizer.from_pretrained(PUBLIC_REPO_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
PUBLIC_REPO_ID,
device_map="auto",
dtype=torch.float16,
trust_remote_code=True,
)
stop_tokens = ["<|endoftext|>", "<|im_end|>"]
eos_token_ids = [tokenizer.eos_token_id] if tokenizer.eos_token_id is not None else []
for token in stop_tokens:
token_id = tokenizer.convert_tokens_to_ids(token)
if token_id is not None and token_id not in eos_token_ids:
eos_token_ids.append(token_id)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
conversation_history = [
{
"role": "system",
"content": (
"You are Quintus, a highly capable AI assistant created by "
"Muskula Rahul. You are helpful, precise, and logically sound."
),
}
]
print()
print("Quintus Chat (type 'quit' to exit)")
print()
while True:
try:
user_input = input("You: ").strip()
if user_input.lower() in ["quit", "exit"]:
print("\nGoodbye!")
break
if not user_input:
continue
conversation_history.append({"role": "user", "content": user_input})
prompt = tokenizer.apply_chat_template(
conversation_history,
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
print("Quintus: ", end="", flush=True)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
temperature=0.7,
top_p=0.9,
do_sample=True,
streamer=streamer,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=eos_token_ids,
)
generated_ids = outputs[0][inputs.input_ids.shape[-1]:]
assistant_response = tokenizer.decode(
generated_ids,
skip_special_tokens=True,
).strip()
conversation_history.append({"role": "assistant", "content": assistant_response})
print()
except KeyboardInterrupt:
print("\n\nGoodbye!")
break
```
## Credits
- [Qwen Team](https://qwenlm.github.io/) and the [Qwen Hugging Face organization](https://huggingface.co/Qwen) for the Qwen3 model family.
- [`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B), used as the distillation teacher.
- [`Qwen/Qwen3-1.7B-Base`](https://huggingface.co/Qwen/Qwen3-1.7B-Base), used as the base student checkpoint.
- [`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B), used for the tokenizer and chat-template contract.
- [Alibaba PAI](https://huggingface.co/alibaba-pai) for [`DistilQwen_100k`](https://huggingface.co/datasets/alibaba-pai/DistilQwen_100k), the primary instruction source after filtering.
- [Hugging Face Transformers](https://github.com/huggingface/transformers), [vLLM](https://github.com/vllm-project/vllm), [EvalPlus](https://github.com/evalplus/evalplus), [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), [FlashAttention](https://github.com/Dao-AILab/flash-attention), and [Liger Kernel](https://github.com/linkedin/Liger-Kernel) for training and evaluation infrastructure.
## License And Author
This software is distributed under the MIT License. Refer to the repository [LICENSE](../LICENSE) file for full text.
Author: Muskula Rahul - [@iamrahulreddy](https://github.com/iamrahulreddy)
## Citation
If you use this model or code, cite the repository and the upstream Qwen3 models.

42
docs/index.md Normal file
View File

@@ -0,0 +1,42 @@
# Quintus Documentation
Quintus-1.7B is a compact assistant built from the Qwen3-1.7B-Base architecture. The project uses online full-vocabulary knowledge distillation from a Qwen3-8B teacher, followed by targeted SFT for instruction style, identity grounding, and generation stability.
This documentation summarizes the public architecture, training decisions, evaluation controls, and release artifacts for the showcase branch.
## Reading Order
- [Architecture](architecture.md): End-to-end pipeline, modules, data flow, and training phases.
- [Experiment Timeline](experiment_timeline.md): How the project moved from offline top-k KD to final online full-vocabulary KD.
- [Training Playbook](training_playbook.md): Practical training choices, memory rules, packing, kernels, and checkpointing.
- [Pipeline Hardening](pipeline_hardening.md): Silent-failure classes and the safeguards added around artifacts, provenance, and runtime.
- [Evaluation Methodology](evaluation_methodology.md): Benchmark controls, parser traps, raw/chat comparisons, and qualitative evaluation rules.
- [Engineering Insights](engineering_insights.md): Condensed technical lessons and design decisions.
- [Benchmarks](benchmarks.md): Verified evaluation results and interpretation.
- [Weight Audit](weight_audit.md): Structural checkpoint verification and what the audit means.
- [Hugging Face Model Card](huggingface_model_card.md): Release-page text for the public model card.
## Project Summary
The core thesis is simple: a small base model can absorb useful reasoning behavior from a larger instruction model if the distillation signal is dense enough and the evaluation controls are fair.
The project initially explored sparse offline top-k distillation, but that approach hit a ceiling because the student only saw a tiny fraction of the teacher vocabulary distribution. The final pipeline pivots to online KD, where teacher and student are run together and the student receives the teacher's full-vocabulary probability distribution during training.
After KD, a small SFT stage teaches the model how to expose that knowledge in a conversational interface. This separation matters: KD transfers capability; SFT and later preference training improve behavior, style, and confidence calibration.
## Repository Map
```text
configs/ Training configuration and DeepSpeed template.
src/ Online KD, data loading, losses, checkpointing, and packing.
sft/ Post-KD supervised fine-tuning, chat, and consolidated evaluation runner.
weight_audit/ Checkpoint structure and weight-divergence audit.
docs/ Public architecture, training, evaluation, and release notes.
```
## Main Public Artifact
The final model weights are available at: [Quintus](https://huggingface.co/iamrahulreddy/Quintus)
The Colab quickstart is available at: [Colab Quick Chat](https://colab.research.google.com/drive/1TdMSN5HzD1mToCFVf_qQoj10NGZLy2V0?usp=sharing)

208
docs/pipeline_hardening.md Normal file
View File

@@ -0,0 +1,208 @@
# Pipeline Hardening
![Pipeline Hardening Flow](../assets/pipeline_hardening_flow.png)
This page summarizes the correctness and reliability lessons that shaped the Quintus codebase. Most of these are silent-failure classes: the pipeline can appear to run while producing invalid or misleading artifacts.
## Silent Serialization Bugs
Teacher token IDs must be stored in a dtype that can represent the tokenizer vocabulary.
An early offline-KD path stored top-k token IDs too narrowly. Qwen token IDs exceed signed 16-bit range, so IDs could wrap negative and later be clamped into valid-looking but wrong positions. Training could continue, but the KL support was corrupted.
Hardening rule:
- Store token IDs as `int32` or wider.
- Validate IDs on load.
- Reject negative IDs.
- Reject IDs outside the student vocabulary.
- Treat dtype as part of the shard contract.
## Row-Order Preservation
Teacher-logit extraction often sorts samples by length for throughput. Training usually expects logits to match the original tokenized row order.
If sorted extraction writes shards in sorted order without restoring original indices, the student receives teacher logits for the wrong sample. This is a model-poisoning bug, not a performance issue.
Hardening rule:
- Batch by sorted length if useful.
- Preserve `original_idx`.
- Write final shards in original dataset order.
- Verify teacher-logit length against the tokenized row length at training time.
## Dataset Schema And Decoding
Public instruction datasets do not share a single row schema. Some rows arrive as `messages`; others use Alpaca-style `instruction`, `input`, and `output` fields. Some content fields contain nested dict/list payloads that need structured coercion before templating.
Dataset streaming can also fail late when a compression codec or file decoder is missing. That failure should remain visible instead of being replaced by a generic "zero samples" result.
Hardening rule:
- Detect Alpaca-style instruction/output rows before chat-message conversion.
- Coerce nested dict/list content through structured serialization, then normalize to text.
- Normalize common role aliases before applying a chat template.
- Preserve the first real dataset exception when streaming fails.
- Validate dataset decoding and schema mapping before large model downloads.
## Zero-Data And Data-Erasure Guards
Data preparation should fail when no usable rows are produced. It should also distinguish "download only" from "tokenize and overwrite output".
Hardening rule:
- Abort if filtering retains zero samples.
- Abort if tokenization writes zero rows.
- Do not open tokenized output in write mode for asset-only setup.
- Use explicit flags for model-only or data-only phases.
## Missing Shards Must Fail
Replacing missing teacher-logit shards with zero tensors makes the training loop look healthy while removing the KD signal.
Hardening rule:
- Missing shard means hard failure.
- Stale shard directories are cleaned before extraction.
- `_provenance.json` is required for KD.
- Shard count, sample count, max sequence length, temperature, top-k, and schema version are checked before training.
## Provenance Contracts
Path equality is weak provenance because paths change across machines. Data identity should come from content and model contracts.
Useful provenance fields:
- schema version
- dataset fingerprint or SHA-256
- sample count
- shard count
- max sequence length
- top-k or full-vocab mode
- temperature
- teacher model ID and revision
- student model ID and revision
- tokenizer sizes
- tokenizer fingerprints
- shard dtypes
Tokenizer fingerprints can drift across library versions. Vocab size and schema compatibility should remain hard gates; fingerprint drift can be a warning when stronger invariants still match.
## Assistant-Only Loss Masks
Supervising prompt and chat-template tokens can teach formatting before substance. It can also make chat-mode behavior fragile.
Hardening rule:
- Tokenized rows must include `loss_mask`.
- Loss mask must be binary.
- Rows with zero assistant targets are rejected.
- User prompts, system prompts, separators, and padding are not targets.
- Assistant response tokens are the supervised region.
Prefix-stable mask derivation is useful when tokenizer-provided assistant masks are unavailable.
## Gradient Accumulation Semantics
DeepSpeed and non-DeepSpeed paths need different step-accounting logic.
DeepSpeed accumulation is global across the full run, not local to each epoch. Epoch-end remainder branches should not create phantom optimizer steps.
Non-DeepSpeed accumulation needs an explicit final flush when a leftover accumulation window exists. That flush must rescale gradients so the update represents the mean over the remainder, not a shrunken `remainder / grad_accum` update.
Hardening rule:
- Advance `global_step` only after a real optimizer update.
- Align scheduler steps with real updates.
- Log flush steps.
- Include flush steps in training-loss CSVs.
- Prefer validation split sizes that align with effective batch size.
## Checkpoint Semantics
`init_from_checkpoint` and `resume_from_checkpoint` are different operations.
- Initialization starts a new phase from an existing model.
- Resume continues an interrupted phase from training state.
Mixing the two can skip training, restart from the wrong model, or reuse stale state.
Hardening rule:
- Forbid simultaneous init and resume.
- Save trainer state and scheduler state.
- Search both `step_*` and `epoch_*` checkpoints for resume.
- Store batch offset for mid-epoch resume.
- Keep final model-loading checkpoints portable.
## Compiler Portability
Compiled PyTorch modules can save weights with `_orig_mod.` prefixes if not unwrapped. Standard Transformers and vLLM loaders do not expect those keys.
Hardening rule:
- Keep `torch.compile` opt-in.
- Treat dynamic-shape recompile overhead as a throughput risk, not just a startup cost.
- Unwrap compiled modules before saving.
- Strip `_orig_mod.` only as a repair path, not as the normal release path.
- Verify saved checkpoints load through standard APIs.
## Artifact Hygiene
Stale outputs are a real ML correctness problem. Old result JSONs, old plots, or old sample logs can make a failed run look successful.
Hardening rule:
- Clean evaluation output directories before a new run.
- Clean stale plots before rendering.
- Select result files by clear recency rules.
- Fail if expected task outputs are incomplete.
- Fail if a requested checkpoint is missing; do not fall back to older local weights.
- Include runtime versions in result summaries.
## Environment Contracts
Notebook and cloud images often contain mixed binary packages. Import success for `torch` alone does not prove the stack is healthy.
Hardening rule:
- Treat `torch`, `torchvision`, and `torchaudio` as one binary compatibility family.
- Use staged dependency manifests instead of ad hoc installs.
- Keep vLLM dependencies separate from HF-only evaluation dependencies.
- Prefer clear preflight errors over late framework crashes.
- Print exception chains, not only the outer error.
## Remote Code And Revisions
Model loading should be reproducible and explicit.
Hardening rule:
- Pin teacher, student, and tokenizer revisions when possible.
- Default remote-code trust to false.
- Provide an explicit override for models that need custom code.
- Explain remote-code failures clearly.
## Safe Logging
Training logs should be rich enough for issue diagnosis without dumping config internals.
Hardening rule:
- Avoid logging authentication values or full config payloads.
- Disable traceback local-variable dumps in rich tracebacks.
- Strip ANSI sequences from file logs while keeping colored notebook output if desired.
- Use UTF-8 file logs and replacement-safe console output for generated model text.
- Log checkpoint save/upload intent, output size, duration, and destination path without sensitive values.
## Public Release Rule
A project can be release-ready without every possible production safeguard. The line is crossed when:
- known silent corruption paths are removed,
- remaining tradeoffs are documented,
- artifacts are reproducible enough to audit,
- public docs focus on decisions, methods, and release artifacts,
- evaluation claims are tied to clear methodology.
For Quintus, the release surface should describe the engineering decisions and results.

199
docs/training_playbook.md Normal file
View File

@@ -0,0 +1,199 @@
# Training Playbook
This page captures the practical training lessons behind Quintus. It focuses on the engineering decisions that made the final online-KD run stable, reproducible, and fast enough to complete on large single-GPU hardware.
## Core Objective
The training objective combines assistant-token cross entropy with teacher-student KL divergence:
$$
\mathcal{L}_{\text{total}}
= \alpha \mathcal{L}_{\text{CE}}
+ (1 - \alpha)\mathcal{L}_{\text{KD}}
$$
For the final Qwen3 run:
$$
\alpha = 0.3,\quad
T = 2.0,\quad
C_{\text{KD}} = 2048,\quad
S_{\max} = 4096
$$
In this codebase, $\alpha$ is the cross-entropy weight. Lower $\alpha$ gives the teacher distribution more influence. Higher $\alpha$ gives hard assistant targets more influence.
## Why Online KD Replaced Offline Top-K KD
The early pipeline precomputed only a small top-k slice of the teacher distribution. That made storage and training cheaper, but it created a hard information ceiling.
With a Qwen vocabulary around 151K tokens:
$$
\frac{k}{|V|}
= \frac{8}{151{,}665}
\approx 5.3 \times 10^{-5}
= 0.0053\%
$$
That sparse signal was enough to disturb student weights, but not enough to reliably transfer deeper reasoning behavior. Several development probes changed alpha, epochs, and student initialization; the same ceiling remained.
The final online path removes that bottleneck. Teacher and student run together, and the KL term is computed from the live full-vocabulary teacher distribution.
## Memory Shape To Respect
Full-vocabulary KD is dominated by logits:
$$
\text{student\_logits},\ \text{teacher\_logits}
\in \mathbb{R}^{B \times S \times |V|}
$$
At Qwen vocabulary scale, increasing micro-batch size by one can add many GiB of temporary memory pressure. Effective batch size is not the same as memory cost. Peak memory is mostly driven by micro-batch size, sequence length, vocabulary width, activation storage, and the backward pass.
Useful rule:
$$
B_{\text{eff}} = B_{\mu} \times A
$$
Keeping $B_{\mu}$ lower and $A$ higher is often safer than a large micro-batch with the same effective batch size.
## Token Chunking
A naive full-vocabulary KL implementation materializes too much temporary state. Quintus computes KD over token chunks:
$$
C_{\text{KD}} = 2048
$$
Larger chunks reduce loop overhead but increase temporary memory. Smaller chunks save memory but can add kernel-launch and Python overhead. The final value is a B200-oriented balance for the 8B -> 1.7B workload.
## Sequence Packing
Sequence packing was the largest throughput win in development probes.
The packing strategy:
- Sort samples by length descending.
- Pack samples with deterministic first-fit decreasing binning.
- Insert EOS separators between samples.
- Set separator `loss_mask = 0`.
- Optionally mask the first token after each separator.
- Build `attention_mask` from true packed length, not from token identity.
The attention-mask detail matters because Qwen tokenizers can share EOS-like IDs with padding behavior. Deriving attention from `input_ids != pad_token_id` can accidentally mask real EOS separators inside packed rows.
Packing probes showed an unpacked B200 online-KD baseline around the low-20K tokens/sec range. Packed training reached roughly the mid-40K tokens/sec range after warmup. The final Qwen3 profile uses the same design principle with a conservative 8B -> 1.7B batch shape.
## B200-Oriented Final Shape
The Qwen3 config is intentionally conservative:
$$
B_{\mu}=4,\quad
A=2,\quad
B_{\text{eff}}=8,\quad
L_{\text{pack}}=4096
$$
Runtime choices:
- `gradient_checkpointing = false`
- `compile_model = false`
- `fused_adamw = true`
- `sequence_packing.enabled = true`
- FlashAttention-2 when available
- Liger kernels for compatible Qwen-family operators
The main reason is the 8B teacher plus 1.7B student online-KD footprint. A smaller teacher/student pair can use larger micro-batches, but the release workload reserves more headroom.
## Kernel Choices
FlashAttention-2 is the preferred stable attention path when available.
Liger kernels are useful for Qwen-family training, but KD places an important constraint on fusion:
- Safe to fuse: RMSNorm, RoPE, SwiGLU.
- Avoid for KD: fused linear cross entropy that hides raw student logits.
The KD loss needs raw student logits to compute teacher-student KL. Any optimization that bypasses logits entirely can break the objective.
## Why `torch.compile` Stayed Off
`torch.compile` can be useful for some SFT paths, but it was not the production choice for final KD.
Observed risks:
- Large Inductor memory overhead.
- Warmup cost on short-lived cloud instances.
- Dynamic-shape graph breaks from variable sequence lengths.
- Recompile overhead that reduced cumulative throughput in probes.
- `_orig_mod.` prefixes in saved checkpoints if compiled modules are not unwrapped before saving.
- Limited benefit after FlashAttention and Liger already fuse the major kernels.
For this workload, stable eager execution with targeted kernels was more predictable than compiler-driven fusion.
## DataLoader And Cloud Stability
Large worker counts can improve throughput on local systems, but notebook and cloud environments can deadlock through multiprocessing queues, IPC limits, or shared-memory pressure.
Practical policy:
- Start with conservative worker and prefetch settings.
- Treat a silent training hang as a DataLoader candidate, even when GPU utilization remains high.
- For some cloud notebook runs, `dataloader_workers = 0` was the most stable choice.
- For the release config, `dataloader_workers = 8` and `prefetch_factor = 2` are a controlled default, not a universal rule.
## Checkpointing And Resume
Cloud GPUs are preemptible and notebook sessions disappear. The training loop therefore treats checkpointing as a core training feature, not an afterthought.
Important design points:
- `best` is selected from validation loss where available.
- `last` is saved for final-state inspection.
- Step checkpoints can resume mid-epoch.
- Scheduler state is saved.
- Optimizer state may be intentionally omitted for very large runs to avoid massive checkpoint overhead.
- Resume semantics distinguish initialization from a completed checkpoint and continuation from an interrupted checkpoint.
This avoids the common trap where `resume_from_checkpoint` silently starts from the wrong phase or stale state.
## Provenance Rules
The pipeline is strict about artifact compatibility:
- Tokenizer vocabulary sizes must match the model contract.
- Teacher-logit metadata must match expected temperature, sample count, max sequence length, and tokenizer/model identity.
- Dataset fingerprints are preferred over path equality because paths are machine-local.
- Tokenizer fingerprints can drift across library versions, so hard checks should focus on vocab-size and schema invariants.
The principle is simple: train only when artifacts prove they belong together.
## Dataset Sampling
Taking the first N valid streamed examples can bias a run if the upstream dataset is ordered by source, task, difficulty, or language. Later configs added stream shuffling before selection.
The config uses a non-default seed:
```text
stream_shuffle_seed = 25
split_seed = 25
```
The number is intentionally explicit. Reproducibility needs stable seeds; it does not require the overused value `42`.
## Practical Watchpoints
During a run, these signals matter more than a single loss number:
- Loss stays finite from the first logging window.
- CE and KD move in plausible ranges.
- Rolling throughput remains stable after warmup.
- GPU memory is high but not near an unpredictable OOM edge.
- Validation loss is computed on the intended holdout.
- Saved checkpoints load in standard Transformers and vLLM paths.
- Downstream benchmark results agree with the training story.
Held-out KD loss is useful, but it is not the release gate. Standardized benchmarks and qualitative checks must decide whether the checkpoint improved the target behavior.

66
docs/weight_audit.md Normal file
View File

@@ -0,0 +1,66 @@
# Weight Audit
The `weight_audit/` directory contains a structural audit script and a generated report comparing the final distilled checkpoint against `Qwen/Qwen3-1.7B-Base`.
The audit is not a behavioral benchmark. It answers a narrower question: is the checkpoint structurally intact, same-architecture, and plausibly modified by training without signs of collapse?
## What Was Checked
The audit verifies:
- Base and distilled checkpoint commits.
- Architecture and config compatibility.
- Parameter counts and tensor keys.
- Weight tying between embeddings and LM head.
- Per-tensor statistics.
- Layer-type aggregate statistics.
- Isotropy of 2D weight matrices.
- Base-vs-distilled divergence for all shared tensors.
- Sparsity, dead rows, low cosine similarity, and low SNR warnings.
## Headline Result
The final report shows:
```text
shared tensors : 311
tensors changed vs base : 277 / 311
cosine similarity : mean = 0.999991 | median = 0.999992
relative error : mean = 0.001093 | median = 0.001293
SNR dB : mean = 81.86 | median = 47.79
high-sparsity layers (>10%) : 0
heavy-tail layers (|kurt_d|>5.0) : 0
dead-row layers : 0
low-cos layers (<0.95) : 0
low-SNR layers (<20 dB) : 0
```
## Interpretation
This is a healthy pattern for light-touch distillation:
- The architecture is unchanged.
- Most tensors changed.
- The changes are small relative to the original base weights.
- Projection matrices, embeddings, and MLP/attention layers moved.
- Some normalization tensors remained unchanged or changed only slightly.
- No layer shows obvious structural collapse.
The unchanged tensors are primarily normalization-related weights. That is not concerning by itself. It suggests the main semantic projection weights absorbed the training signal while basic scaling structure stayed stable.
## Why Isotropy Matters
The report's global isotropy score is close to zero. Near-zero average pairwise row cosine means the weight rows are not collapsing into one shared direction.
This is useful as a sanity check after KD. A collapsed model can sometimes load and produce text, but its internal geometry becomes degenerate. The audit does not show that pattern.
## What The Audit Does Not Prove
The weight audit does not prove that answers are correct, safe, or well calibrated. It should be read alongside:
- Standard benchmarks.
- Open-ended qualitative evaluations.
- SFT evaluation outputs.
- Manual regression prompts.
The audit says the checkpoint is structurally ready for downstream evaluation and release packaging.