gpt2-chatml-fp32/implementation_plan.md

Implementation Plan: Fine-Tuning GPT-2 on High-Quality SFT Dataset (Revised)

We will perform a new fine-tuning run using the high-quality HuggingFaceH4/no_robots SFT dataset. We will first train on a subset to verify training dynamics and output coherence, and then execute training on the full dataset.

---

User Review Required

Important

Please review the following clarifications and confirm approval to begin the autonomous run:

1. Continual Pre-training (FineWeb) vs. Direct SFT (no_robots)

You asked: Is it a good idea to SFT on a high quality document dataset, like fineweb-2, then IF on no_robots?

• Our Recommendation: We recommend direct SFT on no_robots from the base GPT-2 model.
• Why: Continual pre-training (on raw document datasets like FineWeb) is highly compute-intensive and requires billions of tokens to avoid "catastrophic forgetting" of the model's base language modeling capability. For a 124M model like GPT-2 under a 12-hour limit, training on FineWeb documents first would consume significant time and risk degrading the model's coherence or causing it to forget basic grammar before SFT even starts. Instruction-following (IF) alignment directly on no_robots is the most reliable way to teach it conversational structure within our timeframe.

2. Disk Space Cleanup Strategy

• Baseline: Currently, we have 21 GB of free disk space. However, /home/maxgn/.cache/huggingface/ contains 35 GB of cached data (29 GB in hub, 6.2 GB in datasets).
• Action: Before and during training, we will monitor disk space. If free space drops below 12 GB, we will clean up old caches (specifically unused hub models and datasets) to keep the host healthy.

3. Updated Test Split & Verification Formats

• Test Split: Adjusted to train[:2500] per your feedback.
• Coherence Verification Formats: Added IQ3_XXS to our test formats alongside IQ4_NL and Q8_0.

---

Proposed Changes

Configuration Changes

[MODIFY] axolotl-gpt2-chatml-fp32.yml

Update the dataset configuration to point to HuggingFaceH4/no_robots, map its standard fields, and direct outputs to a new dedicated folder:

datasets:
  - path: HuggingFaceH4/no_robots
    split: train[:2500] # Set to train[:2500] for test split, train for full run
    type: chat_template
    field_messages: messages
    message_property_mappings:
      role: role
      content: content
    roles_to_train:
      - assistant
    train_on_eos: turn
    default_system_message: "You are a helpful, concise assistant. Keep answers short and clear."

output_dir: /home/maxgn/outputs/gpt2-chatml-no-robots

---

Execution Plan (Autonomous Execution)

Once approved, we will execute the following steps:

Phase 1: Cache Setup & Pre-Flight Check

1. Verify free disk space. If needed, clear out old Hugging Face hub checkpoints to free up extra headroom.
2. Update axolotl-gpt2-chatml-fp32.yml to target the train[:2500] split.

Phase 2: Test Split Training

1. Launch training using the background command:

   export AXOLOTL_DO_NOT_TRACK=1 && source /home/maxgn/axolotl/.venv/bin/activate && accelerate launch --module --dynamo_backend no axolotl.cli.train /home/maxgn/gpt2-chatml-web-chat/axolotl-gpt2-chatml-fp32.yml --debug=False --debug-text-only=False --debug-num-examples=0 --shard=False
   

2. Monitor validation loss in debug.log.

Phase 3: Test Split GGUF Quantization & Verification

1. Convert the test checkpoint to GGUF format: gpt2-f32.gguf.
2. Generate an imatrix file on the calibration-data.txt file.
3. Quantize the model into Q8_0, IQ4_NL, and IQ3_XXS.
4. Run inference tests on:
   • Prompt A: "What is the capital of France?"
   • Prompt B: "Why is the sky blue? Explain in one short sentence."
5. Evaluation:
   • If output quality is equal/better than our previous run, proceed to Phase 5.
   • If output quality is worse (e.g. repetitive loops or structure loss), proceed to Phase 4 (Troubleshooting).

Phase 4: Troubleshooting (If needed)

1. Adjust hyperparameters (e.g., reduce learning rate to 2e-5, increase epochs, or tune repetition penalty).
2. Re-run Phase 2 and 3 until verified.

Phase 5: Full Dataset Training & Final Quantization

1. Update axolotl-gpt2-chatml-fp32.yml to split: train.
2. Run full training (~1.5 hours).
3. Convert final checkpoint to GGUF and quantize into Q8_0, IQ4_NL, and IQ3_XXS using imatrix.
4. Perform final coherence verification.

---

Verification Plan

Automated Tests

• Run llama-cli -st on the quantized models and log responses to confirm coherence.
• Output files and sizes will be verified and logged in walkthrough.md.