# How to resume training from step 5000

This guide assumes you stopped training at step 5000 (or any saved checkpoint) and now want to continue without restarting from scratch.

## What you need

1. **A GPU** with at least 80 GB VRAM (B200 / H200 / 2×H100 80GB).
2. **The full checkpoint folder** — published on this repo on the branch `resumable-step-5000`. Contains:
   - `adapter_model.safetensors` (LoRA weights, ~155 MB)
   - `adapter_config.json`
   - `optimizer.pt` (8-bit AdamW state, ~80 MB)
   - `scheduler.pt` (cosine LR schedule state)
   - `rng_state.pth` (random state — needed for shuffle resume)
   - `trainer_state.json` (step counter, loss history)
   - `training_args.bin` (config snapshot)
   - tokenizer files
3. **The training data** — 7.9 GB JSONL of `(bible→chapter)` examples, NOT published here (copyright). The owner has it staged at:
   - workstation: `/home/fordoilcorp/booktrain/training/train_quality.jsonl`
   - or wherever the owner has cold-storage backups
4. **The exact pinned Python stack** — see `requirements.txt` (also on this branch).
5. **The training script** — `train_qlora_full.py` (also on this branch).

## Step-by-step resume

```bash
# 1. Set up the host (assumes Ubuntu 22.04 + CUDA 12.8 base image)
apt-get update && apt-get install -y python3-pip rsync git curl
pip install --upgrade pip

# 2. Install the pinned stack
pip install hf_transfer huggingface-hub==0.36.2
pip install torch==2.12.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt

# 3. CRITICAL env var for bitsandbytes to find libnvJitLink.so.13
export LD_LIBRARY_PATH=$(python3 -c "import torch; import os; print(os.path.dirname(torch.__file__))")/../nvidia/cu13/lib:$LD_LIBRARY_PATH
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export HF_HUB_ENABLE_HF_TRANSFER=1
export HF_TOKEN=<your hf token>

# 4. Pull the resumable checkpoint
mkdir -p /workspace/checkpoints
huggingface-cli download Fordentinc/book-builder-bookwriter-v1 \
  --revision resumable-step-5000 \
  --local-dir /workspace/checkpoints/checkpoint-5000

# 5. Pull the training script
huggingface-cli download Fordentinc/book-builder-bookwriter-v1 \
  --revision resumable-step-5000 \
  --include "train_qlora_full.py" \
  --local-dir /workspace/training

# 6. Get the training data onto the host
#    Option A: rsync from your workstation:
#      rsync -av /home/fordoilcorp/booktrain/training/train_quality.jsonl <pod>:/workspace/data/
#      rsync -av /home/fordoilcorp/booktrain/training/eval_quality.jsonl <pod>:/workspace/data/
#    Option B: pull from a private HF dataset if one is set up later.

# 7. Resume training
cd /workspace/training
python3 train_qlora_full.py \
  --base_model Qwen/Qwen2.5-7B \
  --train_jsonl /workspace/data/train_quality.jsonl \
  --eval_jsonl /workspace/data/eval_quality.jsonl \
  --output_dir /workspace/checkpoints \
  --max_seq 2048 \
  --per_device_train_batch_size 4 \
  --grad_accum 8 \
  --lr 2e-4 \
  --num_epochs 1.0 \
  --warmup_ratio 0.03 \
  --logging_steps 10 \
  --save_steps 500 \
  --eval_steps 500 \
  --save_total_limit 3 \
  --hub_repo_id Fordentinc/book-builder-bookwriter-v1 \
  --hub_token $HF_TOKEN \
  --resume_from_checkpoint /workspace/checkpoints/checkpoint-5000
```

The trainer will pick up at step 5001, with the optimizer momentum, the LR schedule position, and the RNG state restored exactly. Expected continuation: step 5001 → 9697 = 4696 steps × ~4.78s = **~6h 14m** of training, ~$26 on a $4.17/hr B200.

## What was tested

- Mechanism validated on this repo using `resumable-test-4500` branch (proved a full checkpoint folder uploads and re-downloads cleanly with all files intact).

## Things that are NOT in this branch (and where they live)

- **Base model `Qwen/Qwen2.5-7B`** — pulled from `huggingface.co/Qwen/Qwen2.5-7B` (Apache 2.0, public, will remain available).
- **Training corpus** — see point 3 above. NOT on HF.
- **Tokenize cache** — regenerated automatically on first run (~7 min on B200). Not worth shipping.

## Verifying the checkpoint before launching real training

After step 5 above, sanity check by loading the checkpoint:

```python
from transformers import AutoModelForCausalLM
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", torch_dtype=torch.bfloat16, device_map="cuda:0")
model = PeftModel.from_pretrained(base, "/workspace/checkpoints/checkpoint-5000")
print("adapter loaded OK, trainable params:", sum(p.numel() for p in model.parameters() if p.requires_grad)/1e6, "M")
```

Expect: `~40.4 M`. If you see that, you can resume safely.

---

Last updated: 2026-05-28