book-builder-bookwriter-v1/resume/RESUME.md

How to resume training from step 5000

This guide assumes you stopped training at step 5000 (or any saved checkpoint) and now want to continue without restarting from scratch.

What you need

1. A GPU with at least 80 GB VRAM (B200 / H200 / 2×H100 80GB).
2. The full checkpoint folder — published on this repo on the branch resumable-step-5000. Contains:
   • adapter_model.safetensors (LoRA weights, ~155 MB)
   • adapter_config.json
   • optimizer.pt (8-bit AdamW state, ~80 MB)
   • scheduler.pt (cosine LR schedule state)
   • rng_state.pth (random state — needed for shuffle resume)
   • trainer_state.json (step counter, loss history)
   • training_args.bin (config snapshot)
   • tokenizer files
3. The training data — 7.9 GB JSONL of (bible→chapter) examples, NOT published here (copyright). The owner has it staged at:
   • workstation: /home/fordoilcorp/booktrain/training/train_quality.jsonl
   • or wherever the owner has cold-storage backups
4. The exact pinned Python stack — see requirements.txt (also on this branch).
5. The training script — train_qlora_full.py (also on this branch).

Step-by-step resume

# 1. Set up the host (assumes Ubuntu 22.04 + CUDA 12.8 base image)
apt-get update && apt-get install -y python3-pip rsync git curl
pip install --upgrade pip

# 2. Install the pinned stack
pip install hf_transfer huggingface-hub==0.36.2
pip install torch==2.12.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt

# 3. CRITICAL env var for bitsandbytes to find libnvJitLink.so.13
export LD_LIBRARY_PATH=$(python3 -c "import torch; import os; print(os.path.dirname(torch.__file__))")/../nvidia/cu13/lib:$LD_LIBRARY_PATH
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export HF_HUB_ENABLE_HF_TRANSFER=1
export HF_TOKEN=<your hf token>

# 4. Pull the resumable checkpoint
mkdir -p /workspace/checkpoints
huggingface-cli download Fordentinc/book-builder-bookwriter-v1 \
  --revision resumable-step-5000 \
  --local-dir /workspace/checkpoints/checkpoint-5000

# 5. Pull the training script
huggingface-cli download Fordentinc/book-builder-bookwriter-v1 \
  --revision resumable-step-5000 \
  --include "train_qlora_full.py" \
  --local-dir /workspace/training

# 6. Get the training data onto the host
#    Option A: rsync from your workstation:
#      rsync -av /home/fordoilcorp/booktrain/training/train_quality.jsonl <pod>:/workspace/data/
#      rsync -av /home/fordoilcorp/booktrain/training/eval_quality.jsonl <pod>:/workspace/data/
#    Option B: pull from a private HF dataset if one is set up later.

# 7. Resume training
cd /workspace/training
python3 train_qlora_full.py \
  --base_model Qwen/Qwen2.5-7B \
  --train_jsonl /workspace/data/train_quality.jsonl \
  --eval_jsonl /workspace/data/eval_quality.jsonl \
  --output_dir /workspace/checkpoints \
  --max_seq 2048 \
  --per_device_train_batch_size 4 \
  --grad_accum 8 \
  --lr 2e-4 \
  --num_epochs 1.0 \
  --warmup_ratio 0.03 \
  --logging_steps 10 \
  --save_steps 500 \
  --eval_steps 500 \
  --save_total_limit 3 \
  --hub_repo_id Fordentinc/book-builder-bookwriter-v1 \
  --hub_token $HF_TOKEN \
  --resume_from_checkpoint /workspace/checkpoints/checkpoint-5000

The trainer will pick up at step 5001, with the optimizer momentum, the LR schedule position, and the RNG state restored exactly. Expected continuation: step 5001 → 9697 = 4696 steps × ~4.78s = ~6h 14m of training, ~$26 on a $4.17/hr B200.

What was tested

• Mechanism validated on this repo using resumable-test-4500 branch (proved a full checkpoint folder uploads and re-downloads cleanly with all files intact).

Things that are NOT in this branch (and where they live)

• Base model Qwen/Qwen2.5-7B — pulled from huggingface.co/Qwen/Qwen2.5-7B (Apache 2.0, public, will remain available).
• Training corpus — see point 3 above. NOT on HF.
• Tokenize cache — regenerated automatically on first run (~7 min on B200). Not worth shipping.

Verifying the checkpoint before launching real training

After step 5 above, sanity check by loading the checkpoint:

from transformers import AutoModelForCausalLM
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", torch_dtype=torch.bfloat16, device_map="cuda:0")
model = PeftModel.from_pretrained(base, "/workspace/checkpoints/checkpoint-5000")
print("adapter loaded OK, trainable params:", sum(p.numel() for p in model.parameters() if p.requires_grad)/1e6, "M")

Expect: ~40.4 M. If you see that, you can resume safely.

---

Last updated: 2026-05-28