67 lines
2.6 KiB
Markdown
67 lines
2.6 KiB
Markdown
|
# Weight Audit
|
||
|
|
|
||
|
|
The `weight_audit/` directory contains a structural audit script and a generated report comparing the final distilled checkpoint against `Qwen/Qwen3-1.7B-Base`.
|
||
|
|
|
||
|
|
The audit is not a behavioral benchmark. It answers a narrower question: is the checkpoint structurally intact, same-architecture, and plausibly modified by training without signs of collapse?
|
||
|
|
|
||
|
|
## What Was Checked
|
||
|
|
|
||
|
|
The audit verifies:
|
||
|
|
|
||
|
|
- Base and distilled checkpoint commits.
|
||
|
|
- Architecture and config compatibility.
|
||
|
|
- Parameter counts and tensor keys.
|
||
|
|
- Weight tying between embeddings and LM head.
|
||
|
|
- Per-tensor statistics.
|
||
|
|
- Layer-type aggregate statistics.
|
||
|
|
- Isotropy of 2D weight matrices.
|
||
|
|
- Base-vs-distilled divergence for all shared tensors.
|
||
|
|
- Sparsity, dead rows, low cosine similarity, and low SNR warnings.
|
||
|
|
|
||
|
|
## Headline Result
|
||
|
|
|
||
|
|
The final report shows:
|
||
|
|
|
||
|
|
```text
|
||
|
|
shared tensors : 311
|
||
|
|
tensors changed vs base : 277 / 311
|
||
|
|
cosine similarity : mean = 0.999991 | median = 0.999992
|
||
|
|
relative error : mean = 0.001093 | median = 0.001293
|
||
|
|
SNR dB : mean = 81.86 | median = 47.79
|
||
|
|
high-sparsity layers (>10%) : 0
|
||
|
|
heavy-tail layers (|kurt_d|>5.0) : 0
|
||
|
|
dead-row layers : 0
|
||
|
|
low-cos layers (<0.95) : 0
|
||
|
|
low-SNR layers (<20 dB) : 0
|
||
|
|
```
|
||
|
|
|
||
|
|
## Interpretation
|
||
|
|
|
||
|
|
This is a healthy pattern for light-touch distillation:
|
||
|
|
|
||
|
|
- The architecture is unchanged.
|
||
|
|
- Most tensors changed.
|
||
|
|
- The changes are small relative to the original base weights.
|
||
|
|
- Projection matrices, embeddings, and MLP/attention layers moved.
|
||
|
|
- Some normalization tensors remained unchanged or changed only slightly.
|
||
|
|
- No layer shows obvious structural collapse.
|
||
|
|
|
||
|
|
The unchanged tensors are primarily normalization-related weights. That is not concerning by itself. It suggests the main semantic projection weights absorbed the training signal while basic scaling structure stayed stable.
|
||
|
|
|
||
|
|
## Why Isotropy Matters
|
||
|
|
|
||
|
|
The report's global isotropy score is close to zero. Near-zero average pairwise row cosine means the weight rows are not collapsing into one shared direction.
|
||
|
|
|
||
|
|
This is useful as a sanity check after KD. A collapsed model can sometimes load and produce text, but its internal geometry becomes degenerate. The audit does not show that pattern.
|
||
|
|
|
||
|
|
## What The Audit Does Not Prove
|
||
|
|
|
||
|
|
The weight audit does not prove that answers are correct, safe, or well calibrated. It should be read alongside:
|
||
|
|
|
||
|
|
- Standard benchmarks.
|
||
|
|
- Open-ended qualitative evaluations.
|
||
|
|
- SFT evaluation outputs.
|
||
|
|
- Manual regression prompts.
|
||
|
|
|
||
|
|
The audit says the checkpoint is structurally ready for downstream evaluation and release packaging.
|