gpt2-chatml-fp32/walkthrough.md

# Walkthrough: GPT-2 Fine-Tuning on no_robots Dataset

We have completed the full supervised fine-tuning of GPT-2 on the human-curated **`HuggingFaceH4/no_robots`** SFT dataset. This walkthrough documents the training outcomes, GGUF conversion, importance matrix calibration, quantization file sizes, and final coherence outputs.

---

## 1. Fine-Tuning Metrics

- **Dataset**: `HuggingFaceH4/no_robots` (Full `train` split, ~9,500 conversations).
- **Training Setup**: 2 epochs, batch size of 16 (micro batch size 2, gradient accumulation 8).
- **Disk Cache Management**: Safely purged ~26 GB of stale model checkpoints from the Hugging Face cache to ensure 46 GB of free storage headroom.
- **Evaluation Outcomes**:
  - **Final Eval Loss**: `2.671` (Improved from `2.795` on the test split).
  - **Eval Perplexity**: `14.46` (Improved from `16.36` on the test split).

---

## 2. GGUF Conversion & Importance Matrix Calibration

1. **GGUF Conversion**:
   - Converted the final HF model weights using `convert_hf_to_gguf.py` under the `ggify` conda environment.
   - Result: `gpt2-f32.gguf` (252.5 MB). (Note: Newer llama.cpp scripts automatically convert token embeddings and specific weights to F16, resulting in a much more compact baseline).
2. **Importance Matrix**:
   - Generated using `llama-imatrix` with ChatML tags parsed.
   - **Calibration Perplexity**: `22.1067 +/- 0.17597`
   - Result: `imatrix.gguf` (260 KB).

---

## 3. Final GGUF Model Sizes

All GGUF files are located in `/home/maxgn/outputs/gpt2-chatml-no-robots/`:

| Model Filename | Quantization Type | Size | Description |
| :--- | :--- | :--- | :--- |
| [gpt2-f32.gguf](file:///home/maxgn/outputs/gpt2-chatml-no-robots/gpt2-f32.gguf) | F16-Embeddings baseline | 252.5 MB | Full precision baseline model |
| [ggml-model-Q8_0.gguf](file:///home/maxgn/outputs/gpt2-chatml-no-robots/ggml-model-Q8_0.gguf) | Q8_0 | 136.7 MB | High-fidelity 8-bit quantization |
| [ggml-model-IQ4_NL.gguf](file:///home/maxgn/outputs/gpt2-chatml-no-robots/ggml-model-IQ4_NL.gguf) | IQ4_NL (imatrix) | 84.8 MB | Highly-optimized 4-bit non-linear quantization |
| [ggml-model-IQ3_XXS.gguf](file:///home/maxgn/outputs/gpt2-chatml-no-robots/ggml-model-IQ3_XXS.gguf) | IQ3_XXS (imatrix) | 64.8 MB | Imatrix 3-bit quantization (Chromebook-compatible) |

---

## 4. Coherence Testing Results

Testing was performed using `llama-cli -st` on the final GGUF models.

### Prompt A: "What is the capital of France?"
* **FP32 (`gpt2-f32.gguf`)**: 
  > `The capital of France is Paris.` (Coherent & Correct, exited cleanly).
* **Q8_0 (`ggml-model-Q8_0.gguf`)**: 
  > `The capital of France is Paris.` (Coherent & Correct, exited cleanly).
* **IQ4_NL (`ggml-model-IQ4_NL.gguf`)**: 
  > `The capital of France is Paris.` (Coherent & Correct, exited cleanly).
* **IQ3_XXS (`ggml-model-IQ3_XXS.gguf`)**: 
  > `The capital of France is Paris, France.` (Coherent & Correct, exited cleanly).

### Prompt B: "Why is the sky blue? Explain in one short sentence."
* **FP32 (`gpt2-f32.gguf`)** (temp 0.1, repeat-penalty 1.15):
  > `The sun's rays have been shining brightly for millions of years, and it has always had its own unique color scheme that reflects sunlight well into space!` (Coherent & Confidently Incorrect).
* **IQ3_XXS (`ggml-model-IQ3_XXS.gguf`)**:
  > `The Sky Blue, or "Blue Moon" as it's commonly known from its origins to be called by some people an ancient symbol of wisdom and beauty, has been a popular choice for centuries...` (Coherent & Confidently Incorrect).

### Prompt C: "What is gravity?"
* **FP32 (`gpt2-f32.gguf`)** (temp 0.1, repeat-penalty 1.15):
  > `Gravity is the force that pulls us around on our own accord. It's not just an ordinary object like air or water; it can be anything from rocks to atoms in a vacuum, and even molecules.` (Coherent & Confidently Incorrect).
* **IQ3_XXS (`ggml-model-IQ3_XXS.gguf`)**:
  > `Gravity is the force that drives us through our bodies!` (Coherent & Confidently Incorrect/Hilarious).

---

## 5. Conclusion & Recommendations

1. **Stop-Token Obedience**: The full training run successfully taught the model to obey the `<|im_end|>` token, terminating generation cleanly instead of looping endlessly (which occurred during the test split).
2. **Quantization Integrity**: With the full 9,500 sample SFT, even the aggressive **IQ3_XXS** (64.8 MB) quantization remained stable and generated coherent sentences without getting stuck in infinite token loops.
3. **Chromebook Deployment**: The **IQ3_XXS** is highly recommended for ultra-low storage limits (such as Chromebook non-system partitions), while **IQ4_NL** (84.8 MB) offers the highest intelligence-to-size ratio.