初始化项目，由ModelHub XC社区提供模型

Model: Fordentinc/book-builder-bookwriter-v1 Source: Original Platform
2026-08-05 12:49:18 +08:00
commit 078bb5f6b6
35 changed files with 152939 additions and 0 deletions
--- a/resume/RESUME.md
+++ b/resume/RESUME.md
@@ -0,0 +1,111 @@
+# How to resume training from step 5000
+
+This guide assumes you stopped training at step 5000 (or any saved checkpoint) and now want to continue without restarting from scratch.
+
+## What you need
+
+1. **A GPU** with at least 80 GB VRAM (B200 / H200 / 2×H100 80GB).
+2. **The full checkpoint folder** — published on this repo on the branch `resumable-step-5000`. Contains:
+   - `adapter_model.safetensors` (LoRA weights, ~155 MB)
+   - `adapter_config.json`
+   - `optimizer.pt` (8-bit AdamW state, ~80 MB)
+   - `scheduler.pt` (cosine LR schedule state)
+   - `rng_state.pth` (random state — needed for shuffle resume)
+   - `trainer_state.json` (step counter, loss history)
+   - `training_args.bin` (config snapshot)
+   - tokenizer files
+3. **The training data** — 7.9 GB JSONL of `(bible→chapter)` examples, NOT published here (copyright). The owner has it staged at:
+   - workstation: `/home/fordoilcorp/booktrain/training/train_quality.jsonl`
+   - or wherever the owner has cold-storage backups
+4. **The exact pinned Python stack** — see `requirements.txt` (also on this branch).
+5. **The training script** — `train_qlora_full.py` (also on this branch).
+
+## Step-by-step resume
+
+```bash
+# 1. Set up the host (assumes Ubuntu 22.04 + CUDA 12.8 base image)
+apt-get update && apt-get install -y python3-pip rsync git curl
+pip install --upgrade pip
+
+# 2. Install the pinned stack
+pip install hf_transfer huggingface-hub==0.36.2
+pip install torch==2.12.0 --index-url https://download.pytorch.org/whl/cu128
+pip install -r requirements.txt
+
+# 3. CRITICAL env var for bitsandbytes to find libnvJitLink.so.13
+export LD_LIBRARY_PATH=$(python3 -c "import torch; import os; print(os.path.dirname(torch.__file__))")/../nvidia/cu13/lib:$LD_LIBRARY_PATH
+export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
+export HF_HUB_ENABLE_HF_TRANSFER=1
+export HF_TOKEN=<your hf token>
+
+# 4. Pull the resumable checkpoint
+mkdir -p /workspace/checkpoints
+huggingface-cli download Fordentinc/book-builder-bookwriter-v1 \
+  --revision resumable-step-5000 \
+  --local-dir /workspace/checkpoints/checkpoint-5000
+
+# 5. Pull the training script
+huggingface-cli download Fordentinc/book-builder-bookwriter-v1 \
+  --revision resumable-step-5000 \
+  --include "train_qlora_full.py" \
+  --local-dir /workspace/training
+
+# 6. Get the training data onto the host
+#    Option A: rsync from your workstation:
+#      rsync -av /home/fordoilcorp/booktrain/training/train_quality.jsonl <pod>:/workspace/data/
+#      rsync -av /home/fordoilcorp/booktrain/training/eval_quality.jsonl <pod>:/workspace/data/
+#    Option B: pull from a private HF dataset if one is set up later.
+
+# 7. Resume training
+cd /workspace/training
+python3 train_qlora_full.py \
+  --base_model Qwen/Qwen2.5-7B \
+  --train_jsonl /workspace/data/train_quality.jsonl \
+  --eval_jsonl /workspace/data/eval_quality.jsonl \
+  --output_dir /workspace/checkpoints \
+  --max_seq 2048 \
+  --per_device_train_batch_size 4 \
+  --grad_accum 8 \
+  --lr 2e-4 \
+  --num_epochs 1.0 \
+  --warmup_ratio 0.03 \
+  --logging_steps 10 \
+  --save_steps 500 \
+  --eval_steps 500 \
+  --save_total_limit 3 \
+  --hub_repo_id Fordentinc/book-builder-bookwriter-v1 \
+  --hub_token $HF_TOKEN \
+  --resume_from_checkpoint /workspace/checkpoints/checkpoint-5000
+```
+
+The trainer will pick up at step 5001, with the optimizer momentum, the LR schedule position, and the RNG state restored exactly. Expected continuation: step 5001 → 9697 = 4696 steps × ~4.78s = **~6h 14m** of training, ~$26 on a $4.17/hr B200.
+
+## What was tested
+
+- Mechanism validated on this repo using `resumable-test-4500` branch (proved a full checkpoint folder uploads and re-downloads cleanly with all files intact).
+
+## Things that are NOT in this branch (and where they live)
+
+- **Base model `Qwen/Qwen2.5-7B`** — pulled from `huggingface.co/Qwen/Qwen2.5-7B` (Apache 2.0, public, will remain available).
+- **Training corpus** — see point 3 above. NOT on HF.
+- **Tokenize cache** — regenerated automatically on first run (~7 min on B200). Not worth shipping.
+
+## Verifying the checkpoint before launching real training
+
+After step 5 above, sanity check by loading the checkpoint:
+
+```python
+from transformers import AutoModelForCausalLM
+from peft import PeftModel
+import torch
+
+base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", torch_dtype=torch.bfloat16, device_map="cuda:0")
+model = PeftModel.from_pretrained(base, "/workspace/checkpoints/checkpoint-5000")
+print("adapter loaded OK, trainable params:", sum(p.numel() for p in model.parameters() if p.requires_grad)/1e6, "M")
+```
+
+Expect: `~40.4 M`. If you see that, you can resume safely.
+
+---
+
+Last updated: 2026-05-28
--- a/resume/requirements.txt
+++ b/resume/requirements.txt
@@ -0,0 +1,24 @@
+# Exact pinned versions for resuming book-builder-bookwriter-v1 training.
+# These versions were verified working end-to-end on B200 (sm_100) and 4090 (sm_89).
+# Install with:
+#   pip install -r requirements.txt --index-url https://download.pytorch.org/whl/cu128
+# (only torch needs cu128 wheel; rest install from PyPI normally — see install commands below)
+
+# === Core (install via cu128 PyTorch index) ===
+torch==2.12.0
+triton==3.7.0
+
+# === HF stack ===
+transformers==4.46.3
+accelerate==1.1.1
+datasets==3.1.0
+peft==0.13.2
+bitsandbytes==0.49.2
+huggingface-hub==0.36.2
+hf-transfer==0.1.9
+safetensors==0.7.0
+sentencepiece==0.2.1
+protobuf==7.35.0
+tokenizers==0.20.3
+tensorboard==2.20.0
+numpy==2.4.6
--- a/resume/train_qlora_full.py
+++ b/resume/train_qlora_full.py
@@ -0,0 +1,208 @@
+#!/usr/bin/env python3
+"""
+Production QLoRA training for BookBuilder bookwriter v1.
+Same recipe as smoke_qlora.py, scaled up via HF Trainer.
+Default target: Qwen 2.5 7B, 1.82B-token corpus, 1 epoch, ctx 2048.
+"""
+import os, sys, json, argparse, time
+from pathlib import Path
+
+import torch
+from datasets import load_dataset
+from transformers import (
+    AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig,
+    Trainer, TrainingArguments, DataCollatorForLanguageModeling,
+)
+from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
+
+
+def parse_args():
+    p = argparse.ArgumentParser()
+    p.add_argument("--base_model", default="Qwen/Qwen2.5-7B")
+    p.add_argument("--train_jsonl", required=True)
+    p.add_argument("--eval_jsonl", default=None)
+    p.add_argument("--output_dir", required=True)
+    p.add_argument("--max_seq", type=int, default=2048)
+    p.add_argument("--per_device_train_batch_size", type=int, default=8)
+    p.add_argument("--per_device_eval_batch_size", type=int, default=8)
+    p.add_argument("--grad_accum", type=int, default=4)
+    p.add_argument("--lr", type=float, default=2e-4)
+    p.add_argument("--num_epochs", type=float, default=1.0)
+    p.add_argument("--warmup_ratio", type=float, default=0.03)
+    p.add_argument("--lora_r", type=int, default=16)
+    p.add_argument("--lora_alpha", type=int, default=32)
+    p.add_argument("--lora_dropout", type=float, default=0.05)
+    p.add_argument("--save_steps", type=int, default=500)
+    p.add_argument("--eval_steps", type=int, default=500)
+    p.add_argument("--logging_steps", type=int, default=20)
+    p.add_argument("--save_total_limit", type=int, default=3)
+    p.add_argument("--hub_repo_id", default=None,
+                   help="If set, push checkpoints to this HF repo each save.")
+    p.add_argument("--hub_token", default=None)
+    p.add_argument("--max_train_samples", type=int, default=None,
+                   help="Cap for sanity runs.")
+    p.add_argument("--seed", type=int, default=42)
+    p.add_argument("--gradient_checkpointing", action="store_true", default=False)
+    p.add_argument("--no_eval", action="store_true")
+    return p.parse_args()
+
+
+def main():
+    args = parse_args()
+    print(f"=== BookBuilder QLoRA train ===", flush=True)
+    print(f"base: {args.base_model}")
+    print(f"train: {args.train_jsonl}")
+    print(f"eval : {args.eval_jsonl}")
+    print(f"out  : {args.output_dir}")
+    print(f"ctx  : {args.max_seq}  bs/dev: {args.per_device_train_batch_size}  ga: {args.grad_accum}  lr: {args.lr}", flush=True)
+
+    # ---------- tokenizer ----------
+    tok = AutoTokenizer.from_pretrained(args.base_model, use_fast=True)
+    if tok.pad_token is None:
+        tok.pad_token = tok.eos_token
+    tok.padding_side = "right"
+
+    # ---------- model: 4-bit NF4 ----------
+    bnb = BitsAndBytesConfig(
+        load_in_4bit=True,
+        bnb_4bit_quant_type="nf4",
+        bnb_4bit_compute_dtype=torch.bfloat16,
+        bnb_4bit_use_double_quant=True,
+    )
+    print("Loading 4-bit base...", flush=True)
+    t0 = time.time()
+    model = AutoModelForCausalLM.from_pretrained(
+        args.base_model,
+        quantization_config=bnb,
+        torch_dtype=torch.bfloat16,
+        attn_implementation="sdpa",
+        device_map="cuda:0",
+    )
+    model.config._attn_implementation = "sdpa"
+    model.config.use_cache = False
+    print(f"  base loaded in {time.time()-t0:.1f}s | VRAM {torch.cuda.memory_allocated()/1e9:.2f} GB", flush=True)
+
+    model = prepare_model_for_kbit_training(
+        model, use_gradient_checkpointing=args.gradient_checkpointing,
+    )
+    lora = LoraConfig(
+        r=args.lora_r,
+        lora_alpha=args.lora_alpha,
+        lora_dropout=args.lora_dropout,
+        bias="none",
+        task_type="CAUSAL_LM",
+        target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
+    )
+    model = get_peft_model(model, lora)
+    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
+    total     = sum(p.numel() for p in model.parameters())
+    print(f"  trainable: {trainable/1e6:.2f}M / {total/1e6:.1f}M ({100*trainable/total:.3f}%)", flush=True)
+
+    # ---------- dataset ----------
+    data_files = {"train": args.train_jsonl}
+    if args.eval_jsonl and not args.no_eval:
+        data_files["eval"] = args.eval_jsonl
+    ds = load_dataset("json", data_files=data_files)
+    if args.max_train_samples:
+        ds["train"] = ds["train"].shuffle(seed=args.seed).select(range(args.max_train_samples))
+    print(f"  train rows: {len(ds['train']):,}", flush=True)
+    if "eval" in ds:
+        print(f"  eval rows : {len(ds['eval']):,}", flush=True)
+
+    def tokenize_fn(batch):
+        out = tok(
+            batch["text"],
+            truncation=True,
+            max_length=args.max_seq,
+            padding=False,
+        )
+        return out
+
+    # Explicit cache paths — HF datasets' default fingerprint isn't stable across
+    # Python processes (tokenizer object hash differs), so a plain .map() re-tokenizes
+    # every fresh launch. Pin the cache to a deterministic path so subsequent runs hit it.
+    cache_root = os.path.dirname(os.path.abspath(args.train_jsonl))
+    tag = f"qwen-ctx{args.max_seq}"
+    cache_files = {
+        split: os.path.join(cache_root, f"tok_cache_{split}_{tag}.arrow")
+        for split in ds.keys()
+    }
+    for split, path in cache_files.items():
+        print(f"  tokenize cache[{split}] -> {path}", flush=True)
+    tokenized = ds.map(
+        tokenize_fn,
+        batched=True,
+        remove_columns=ds["train"].column_names,
+        num_proc=8,
+        desc="tokenize",
+        cache_file_names=cache_files,
+        load_from_cache_file=True,
+    )
+
+    collator = DataCollatorForLanguageModeling(tokenizer=tok, mlm=False)
+
+    # ---------- trainer ----------
+    push_to_hub = bool(args.hub_repo_id)
+    targs = TrainingArguments(
+        output_dir=args.output_dir,
+        overwrite_output_dir=False,
+        seed=args.seed,
+        num_train_epochs=args.num_epochs,
+        per_device_train_batch_size=args.per_device_train_batch_size,
+        per_device_eval_batch_size=args.per_device_eval_batch_size,
+        gradient_accumulation_steps=args.grad_accum,
+        learning_rate=args.lr,
+        warmup_ratio=args.warmup_ratio,
+        lr_scheduler_type="cosine",
+        bf16=True,
+        fp16=False,
+        tf32=True,
+        gradient_checkpointing=args.gradient_checkpointing,
+        logging_steps=args.logging_steps,
+        save_steps=args.save_steps,
+        eval_strategy="steps" if "eval" in tokenized else "no",
+        eval_steps=args.eval_steps if "eval" in tokenized else None,
+        save_total_limit=args.save_total_limit,
+        report_to="tensorboard",
+        push_to_hub=push_to_hub,
+        hub_model_id=args.hub_repo_id if push_to_hub else None,
+        hub_token=args.hub_token,
+        hub_strategy="every_save" if push_to_hub else "end",
+        hub_private_repo=False,
+        dataloader_num_workers=4,
+        dataloader_pin_memory=True,
+        optim="paged_adamw_8bit",
+        remove_unused_columns=False,
+    )
+
+    trainer_kwargs = dict(
+        model=model,
+        args=targs,
+        train_dataset=tokenized["train"],
+        eval_dataset=tokenized.get("eval"),
+        data_collator=collator,
+    )
+    # transformers 4.46 uses `tokenizer=`; 5.x uses `processing_class=`
+    import inspect
+    if "processing_class" in inspect.signature(Trainer.__init__).parameters:
+        trainer_kwargs["processing_class"] = tok
+    else:
+        trainer_kwargs["tokenizer"] = tok
+    trainer = Trainer(**trainer_kwargs)
+
+    print(f"=== begin training ===", flush=True)
+    trainer.train()
+
+    print(f"=== saving final adapter ===", flush=True)
+    trainer.save_model(args.output_dir)
+    tok.save_pretrained(args.output_dir)
+    if push_to_hub:
+        trainer.push_to_hub(commit_message="final adapter")
+
+    # Sentinel
+    Path(args.output_dir).joinpath(".train_complete").touch()
+    print("DONE.", flush=True)
+
+
+if __name__ == "__main__":
+    main()