gpt2-chatml-fp32/walkthrough.md

Walkthrough: GPT-2 Fine-Tuning on no_robots Dataset

We have completed the full supervised fine-tuning of GPT-2 on the human-curated HuggingFaceH4/no_robots SFT dataset. This walkthrough documents the training outcomes, GGUF conversion, importance matrix calibration, quantization file sizes, and final coherence outputs.

---

1. Fine-Tuning Metrics

• Dataset: HuggingFaceH4/no_robots (Full train split, ~9,500 conversations).
• Training Setup: 2 epochs, batch size of 16 (micro batch size 2, gradient accumulation 8).
• Disk Cache Management: Safely purged ~26 GB of stale model checkpoints from the Hugging Face cache to ensure 46 GB of free storage headroom.
• Evaluation Outcomes:
  • Final Eval Loss: 2.671 (Improved from 2.795 on the test split).
  • Eval Perplexity: 14.46 (Improved from 16.36 on the test split).

---

2. GGUF Conversion & Importance Matrix Calibration

1. GGUF Conversion:
   • Converted the final HF model weights using convert_hf_to_gguf.py under the ggify conda environment.
   • Result: gpt2-f32.gguf (252.5 MB). (Note: Newer llama.cpp scripts automatically convert token embeddings and specific weights to F16, resulting in a much more compact baseline).
2. Importance Matrix:
   • Generated using llama-imatrix with ChatML tags parsed.
   • Calibration Perplexity: 22.1067 +/- 0.17597
   • Result: imatrix.gguf (260 KB).

---

3. Final GGUF Model Sizes

All GGUF files are located in /home/maxgn/outputs/gpt2-chatml-no-robots/:

Model Filename	Quantization Type	Size	Description
gpt2-f32.gguf	F16-Embeddings baseline	252.5 MB	Full precision baseline model
ggml-model-Q8_0.gguf	Q8_0	136.7 MB	High-fidelity 8-bit quantization
ggml-model-IQ4_NL.gguf	IQ4_NL (imatrix)	84.8 MB	Highly-optimized 4-bit non-linear quantization
ggml-model-IQ3_XXS.gguf	IQ3_XXS (imatrix)	64.8 MB	Imatrix 3-bit quantization (Chromebook-compatible)

---

4. Coherence Testing Results

Testing was performed using llama-cli -st on the final GGUF models.

Prompt A: "What is the capital of France?"

• FP32 (gpt2-f32.gguf):

  The capital of France is Paris. (Coherent & Correct, exited cleanly).

• Q8_0 (ggml-model-Q8_0.gguf):

  The capital of France is Paris. (Coherent & Correct, exited cleanly).

• IQ4_NL (ggml-model-IQ4_NL.gguf):

  The capital of France is Paris. (Coherent & Correct, exited cleanly).

• IQ3_XXS (ggml-model-IQ3_XXS.gguf):

  The capital of France is Paris, France. (Coherent & Correct, exited cleanly).

Prompt B: "Why is the sky blue? Explain in one short sentence."

• FP32 (gpt2-f32.gguf) (temp 0.1, repeat-penalty 1.15):

  The sun's rays have been shining brightly for millions of years, and it has always had its own unique color scheme that reflects sunlight well into space! (Coherent & Confidently Incorrect).

• IQ3_XXS (ggml-model-IQ3_XXS.gguf):

  The Sky Blue, or "Blue Moon" as it's commonly known from its origins to be called by some people an ancient symbol of wisdom and beauty, has been a popular choice for centuries... (Coherent & Confidently Incorrect).

Prompt C: "What is gravity?"

• FP32 (gpt2-f32.gguf) (temp 0.1, repeat-penalty 1.15):

  Gravity is the force that pulls us around on our own accord. It's not just an ordinary object like air or water; it can be anything from rocks to atoms in a vacuum, and even molecules. (Coherent & Confidently Incorrect).

• IQ3_XXS (ggml-model-IQ3_XXS.gguf):

  Gravity is the force that drives us through our bodies! (Coherent & Confidently Incorrect/Hilarious).

---

5. Conclusion & Recommendations

1. Stop-Token Obedience: The full training run successfully taught the model to obey the <|im_end|> token, terminating generation cleanly instead of looping endlessly (which occurred during the test split).
2. Quantization Integrity: With the full 9,500 sample SFT, even the aggressive IQ3_XXS (64.8 MB) quantization remained stable and generated coherent sentences without getting stuck in infinite token loops.
3. Chromebook Deployment: The IQ3_XXS is highly recommended for ultra-low storage limits (such as Chromebook non-system partitions), while IQ4_NL (84.8 MB) offers the highest intelligence-to-size ratio.