112 lines
4.6 KiB
Markdown
112 lines
4.6 KiB
Markdown
# How to resume training from step 5000
|
||
|
||
This guide assumes you stopped training at step 5000 (or any saved checkpoint) and now want to continue without restarting from scratch.
|
||
|
||
## What you need
|
||
|
||
1. **A GPU** with at least 80 GB VRAM (B200 / H200 / 2×H100 80GB).
|
||
2. **The full checkpoint folder** — published on this repo on the branch `resumable-step-5000`. Contains:
|
||
- `adapter_model.safetensors` (LoRA weights, ~155 MB)
|
||
- `adapter_config.json`
|
||
- `optimizer.pt` (8-bit AdamW state, ~80 MB)
|
||
- `scheduler.pt` (cosine LR schedule state)
|
||
- `rng_state.pth` (random state — needed for shuffle resume)
|
||
- `trainer_state.json` (step counter, loss history)
|
||
- `training_args.bin` (config snapshot)
|
||
- tokenizer files
|
||
3. **The training data** — 7.9 GB JSONL of `(bible→chapter)` examples, NOT published here (copyright). The owner has it staged at:
|
||
- workstation: `/home/fordoilcorp/booktrain/training/train_quality.jsonl`
|
||
- or wherever the owner has cold-storage backups
|
||
4. **The exact pinned Python stack** — see `requirements.txt` (also on this branch).
|
||
5. **The training script** — `train_qlora_full.py` (also on this branch).
|
||
|
||
## Step-by-step resume
|
||
|
||
```bash
|
||
# 1. Set up the host (assumes Ubuntu 22.04 + CUDA 12.8 base image)
|
||
apt-get update && apt-get install -y python3-pip rsync git curl
|
||
pip install --upgrade pip
|
||
|
||
# 2. Install the pinned stack
|
||
pip install hf_transfer huggingface-hub==0.36.2
|
||
pip install torch==2.12.0 --index-url https://download.pytorch.org/whl/cu128
|
||
pip install -r requirements.txt
|
||
|
||
# 3. CRITICAL env var for bitsandbytes to find libnvJitLink.so.13
|
||
export LD_LIBRARY_PATH=$(python3 -c "import torch; import os; print(os.path.dirname(torch.__file__))")/../nvidia/cu13/lib:$LD_LIBRARY_PATH
|
||
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
|
||
export HF_HUB_ENABLE_HF_TRANSFER=1
|
||
export HF_TOKEN=<your hf token>
|
||
|
||
# 4. Pull the resumable checkpoint
|
||
mkdir -p /workspace/checkpoints
|
||
huggingface-cli download Fordentinc/book-builder-bookwriter-v1 \
|
||
--revision resumable-step-5000 \
|
||
--local-dir /workspace/checkpoints/checkpoint-5000
|
||
|
||
# 5. Pull the training script
|
||
huggingface-cli download Fordentinc/book-builder-bookwriter-v1 \
|
||
--revision resumable-step-5000 \
|
||
--include "train_qlora_full.py" \
|
||
--local-dir /workspace/training
|
||
|
||
# 6. Get the training data onto the host
|
||
# Option A: rsync from your workstation:
|
||
# rsync -av /home/fordoilcorp/booktrain/training/train_quality.jsonl <pod>:/workspace/data/
|
||
# rsync -av /home/fordoilcorp/booktrain/training/eval_quality.jsonl <pod>:/workspace/data/
|
||
# Option B: pull from a private HF dataset if one is set up later.
|
||
|
||
# 7. Resume training
|
||
cd /workspace/training
|
||
python3 train_qlora_full.py \
|
||
--base_model Qwen/Qwen2.5-7B \
|
||
--train_jsonl /workspace/data/train_quality.jsonl \
|
||
--eval_jsonl /workspace/data/eval_quality.jsonl \
|
||
--output_dir /workspace/checkpoints \
|
||
--max_seq 2048 \
|
||
--per_device_train_batch_size 4 \
|
||
--grad_accum 8 \
|
||
--lr 2e-4 \
|
||
--num_epochs 1.0 \
|
||
--warmup_ratio 0.03 \
|
||
--logging_steps 10 \
|
||
--save_steps 500 \
|
||
--eval_steps 500 \
|
||
--save_total_limit 3 \
|
||
--hub_repo_id Fordentinc/book-builder-bookwriter-v1 \
|
||
--hub_token $HF_TOKEN \
|
||
--resume_from_checkpoint /workspace/checkpoints/checkpoint-5000
|
||
```
|
||
|
||
The trainer will pick up at step 5001, with the optimizer momentum, the LR schedule position, and the RNG state restored exactly. Expected continuation: step 5001 → 9697 = 4696 steps × ~4.78s = **~6h 14m** of training, ~$26 on a $4.17/hr B200.
|
||
|
||
## What was tested
|
||
|
||
- Mechanism validated on this repo using `resumable-test-4500` branch (proved a full checkpoint folder uploads and re-downloads cleanly with all files intact).
|
||
|
||
## Things that are NOT in this branch (and where they live)
|
||
|
||
- **Base model `Qwen/Qwen2.5-7B`** — pulled from `huggingface.co/Qwen/Qwen2.5-7B` (Apache 2.0, public, will remain available).
|
||
- **Training corpus** — see point 3 above. NOT on HF.
|
||
- **Tokenize cache** — regenerated automatically on first run (~7 min on B200). Not worth shipping.
|
||
|
||
## Verifying the checkpoint before launching real training
|
||
|
||
After step 5 above, sanity check by loading the checkpoint:
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM
|
||
from peft import PeftModel
|
||
import torch
|
||
|
||
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", torch_dtype=torch.bfloat16, device_map="cuda:0")
|
||
model = PeftModel.from_pretrained(base, "/workspace/checkpoints/checkpoint-5000")
|
||
print("adapter loaded OK, trainable params:", sum(p.numel() for p in model.parameters() if p.requires_grad)/1e6, "M")
|
||
```
|
||
|
||
Expect: `~40.4 M`. If you see that, you can resume safely.
|
||
|
||
---
|
||
|
||
Last updated: 2026-05-28
|