--- license: llama3.2 base_model: meta-llama/Llama-3.2-1B-Instruct datasets: - Anthropic/hh-rlhf language: - en library_name: transformers pipeline_tag: text-generation tags: - dpo - preference-optimization - rlhf - llama-3.2 - from-scratch --- # Llama-3.2-1B-Instruct-DPO-HH `meta-llama/Llama-3.2-1B-Instruct` aligned with **Direct Preference Optimization** on Anthropic HH-RLHF, using a from-scratch DPO implementation — no TRL, no `DPOTrainer`, no Axolotl, no Lightning. The loss, sequence scoring, completion masking, reference handling and training loop are all explicit PyTorch. This is research and learning code. Read the limitations before using it. ## Training | | | |---|---| | Base model | `meta-llama/Llama-3.2-1B-Instruct` | | Objective | DPO (Rafailov et al., 2023), summed completion log-probabilities | | Reference | Frozen copy of the base model, live (not cached) | | Dataset | `Anthropic/hh-rlhf`, `train` split | | Pairs after parsing | 159,384 | | Pairs seen | **79,360** (50% of one epoch, 1,240 steps) | | Hardware | 1× NVIDIA H200 | ### Hyperparameters | | | |---|---| | β | 0.1 | | Learning rate | 5e-07 | | Schedule | cosine to 0, 10% warmup | | Optimizer | AdamW, β₁ 0.9, β₂ 0.95, wd 0.0 | | Effective batch | 64 pairs (8 per device × 8 accumulation) | | Max grad norm | 1.0 | | Parameter dtype | **float32** (BF16 autocast for compute) | | Max length | 1024 (prompt 640, completion 384) | | Seed | 42 | **On precision.** Parameters are stored in FP32 and BF16 is used only for forward/backward compute. Storing *trainable* parameters in BF16 silently breaks preference tuning: at a weight of 0.01 the gap between representable BF16 values is 6.1e-5, while an AdamW update at these learning rates is ~1e-6, so updates round to a no-op while the loss curve still looks plausible. This pipeline rejects BF16 parameter storage for training outright. ## Results Training metrics only — **no held-out evaluation was run.** These are in-training statistics on the optimized data, not a measure of generalization. | steps | loss | reward margin | reward accuracy | |---|---|---|---| | 0–248 | 0.6809 | +0.0420 | 0.548 | | 248–496 | 0.6513 | +0.1620 | 0.596 | | 496–744 | 0.6453 | +0.1894 | 0.631 | | 744–992 | 0.6437 | +0.2152 | 0.618 | | 992–1240 | 0.6435 | +0.2027 | 0.627 | | **final** | **0.6455** | **+0.2002** | **0.615** | DPO's loss is exactly `log 2 = 0.693147` when policy and reference are identical. This run began at `0.690420` with implicit rewards of `-0.0112` / `-0.0176`, confirming the frozen reference and the completion masking were correct at step 0. **Where the margin comes from.** `chosen_reward` moved +0.1242 → +0.0333 and `rejected_reward` +0.0595 → -0.1797. The separation is produced mainly by pushing the *rejected* responses below the reference, not by making the chosen ones more likely — the likelihood-displacement behaviour DPO is known for. Read the margin as "less likely to produce the dispreferred response", which is not the same claim as "more likely to produce the preferred one". ## How large is the effect, really? Stated plainly, because a reward margin alone does not tell you this: - Relative weight change from the base model: **0.000354** - Greedy generations that are byte-identical to the base model: **0 of 4** spot-check prompts This is a *modest* perturbation of the base model, which is what half an epoch at lr 5e-7 with β=0.1 should produce — β exists precisely to keep the policy near its reference. On general prompts the outputs are recognisably the base model's, with differences in phrasing and formatting. Do not expect a dramatically different assistant. ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_id = "jackf857/Llama-3.2-1B-Instruct-DPO-HH" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto") messages = [{"role": "user", "content": "Explain why the sky is blue, briefly."}] ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt") out = model.generate(ids.to(model.device), max_new_tokens=128) print(tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True)) ``` The Llama 3 chat template injects the current date unless `date_string` is pinned. Training used `date_string="26 Jul 2024"`. ## Data processing HH transcripts were parsed into canonical `(messages, chosen, rejected)` triples with no chat markup, then rendered through the official Llama 3 chat template. Only the final assistant response is scored; the completion mask is exactly `[0]*prompt_len + [1]*completion_len`, enforced by the prompt-prefix invariant. Verified against the real Llama 3 tokenizer: exactly one BOS per sequence, 0% BPE merges across the prompt/completion boundary, every completion ending in `<|eot_id|>`. ## Limitations - **No held-out evaluation.** Every number above is a training metric. There is no evidence here that this model is better than its base — only that DPO optimized what it was asked to optimize. - **50% of one epoch on a 1B model.** A short run on a small model. - **HH-RLHF is noisy.** Preference labels are known to be inconsistent, and many pairs have no clear quality difference. Some labels prefer epistemic humility ("I don't know") over confident answers, so the model may become more hedging. - **Safety is not established.** No safety evaluation was performed. Do not deploy where harmful output matters. Inherits all limitations of the base model. - English only; 1B models hallucinate readily. ## License Governed by the [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE), inherited from the base model. That license requires derivative model names to begin with "Llama". Anthropic HH-RLHF is MIT licensed. ## Citation ```bibtex @inproceedings{rafailov2023direct, title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model}, author = {Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Ermon, Stefano and Manning, Christopher D. and Finn, Chelsea}, booktitle = {Advances in Neural Information Processing Systems}, year = {2023} } ```