157 lines
6.2 KiB
Markdown
157 lines
6.2 KiB
Markdown
---
|
||
license: llama3.2
|
||
base_model: meta-llama/Llama-3.2-1B-Instruct
|
||
datasets:
|
||
- Anthropic/hh-rlhf
|
||
language:
|
||
- en
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- dpo
|
||
- preference-optimization
|
||
- rlhf
|
||
- llama-3.2
|
||
- from-scratch
|
||
---
|
||
|
||
# Llama-3.2-1B-Instruct-DPO-HH
|
||
|
||
`meta-llama/Llama-3.2-1B-Instruct` aligned with **Direct Preference Optimization** on Anthropic HH-RLHF,
|
||
using a from-scratch DPO implementation — no TRL, no `DPOTrainer`, no Axolotl, no
|
||
Lightning. The loss, sequence scoring, completion masking, reference handling and
|
||
training loop are all explicit PyTorch.
|
||
|
||
This is research and learning code. Read the limitations before using it.
|
||
|
||
## Training
|
||
|
||
| | |
|
||
|---|---|
|
||
| Base model | `meta-llama/Llama-3.2-1B-Instruct` |
|
||
| Objective | DPO (Rafailov et al., 2023), summed completion log-probabilities |
|
||
| Reference | Frozen copy of the base model, live (not cached) |
|
||
| Dataset | `Anthropic/hh-rlhf`, `train` split |
|
||
| Pairs after parsing | 159,384 |
|
||
| Pairs seen | **79,360** (50% of one epoch, 1,240 steps) |
|
||
| Hardware | 1× NVIDIA H200 |
|
||
|
||
### Hyperparameters
|
||
|
||
| | |
|
||
|---|---|
|
||
| β | 0.1 |
|
||
| Learning rate | 5e-07 |
|
||
| Schedule | cosine to 0, 10% warmup |
|
||
| Optimizer | AdamW, β₁ 0.9, β₂ 0.95, wd 0.0 |
|
||
| Effective batch | 64 pairs (8 per device × 8 accumulation) |
|
||
| Max grad norm | 1.0 |
|
||
| Parameter dtype | **float32** (BF16 autocast for compute) |
|
||
| Max length | 1024 (prompt 640, completion 384) |
|
||
| Seed | 42 |
|
||
|
||
**On precision.** Parameters are stored in FP32 and BF16 is used only for
|
||
forward/backward compute. Storing *trainable* parameters in BF16 silently breaks
|
||
preference tuning: at a weight of 0.01 the gap between representable BF16 values is
|
||
6.1e-5, while an AdamW update at these learning rates is ~1e-6, so updates round to a
|
||
no-op while the loss curve still looks plausible. This pipeline rejects BF16 parameter
|
||
storage for training outright.
|
||
|
||
## Results
|
||
|
||
Training metrics only — **no held-out evaluation was run.** These are in-training
|
||
statistics on the optimized data, not a measure of generalization.
|
||
|
||
| steps | loss | reward margin | reward accuracy |
|
||
|---|---|---|---|
|
||
| 0–248 | 0.6809 | +0.0420 | 0.548 |
|
||
| 248–496 | 0.6513 | +0.1620 | 0.596 |
|
||
| 496–744 | 0.6453 | +0.1894 | 0.631 |
|
||
| 744–992 | 0.6437 | +0.2152 | 0.618 |
|
||
| 992–1240 | 0.6435 | +0.2027 | 0.627 |
|
||
| **final** | **0.6455** | **+0.2002** | **0.615** |
|
||
|
||
DPO's loss is exactly `log 2 = 0.693147` when policy and reference are identical. This
|
||
run began at `0.690420` with implicit rewards of `-0.0112` /
|
||
`-0.0176`, confirming the frozen reference and the completion masking were
|
||
correct at step 0.
|
||
|
||
**Where the margin comes from.** `chosen_reward` moved +0.1242 → +0.0333
|
||
and `rejected_reward` +0.0595 → -0.1797. The separation is
|
||
produced mainly by pushing the *rejected* responses below the reference, not by making the
|
||
chosen ones more likely — the likelihood-displacement behaviour DPO is known for. Read the
|
||
margin as "less likely to produce the dispreferred response", which is not the same claim
|
||
as "more likely to produce the preferred one".
|
||
|
||
## How large is the effect, really?
|
||
|
||
Stated plainly, because a reward margin alone does not tell you this:
|
||
|
||
- Relative weight change from the base model: **0.000354**
|
||
- Greedy generations that are byte-identical to the base model: **0 of 4** spot-check prompts
|
||
|
||
This is a *modest* perturbation of the base model, which is what half an epoch at lr 5e-7 with β=0.1 should produce — β exists precisely to keep the policy near its reference. On general prompts the outputs are recognisably the base model's, with differences in phrasing and formatting. Do not expect a dramatically different assistant.
|
||
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
model_id = "jackf857/Llama-3.2-1B-Instruct-DPO-HH"
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
|
||
|
||
messages = [{"role": "user", "content": "Explain why the sky is blue, briefly."}]
|
||
ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
|
||
out = model.generate(ids.to(model.device), max_new_tokens=128)
|
||
print(tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
The Llama 3 chat template injects the current date unless `date_string` is pinned.
|
||
Training used `date_string="26 Jul 2024"`.
|
||
|
||
## Data processing
|
||
|
||
HH transcripts were parsed into canonical `(messages, chosen, rejected)` triples with no
|
||
chat markup, then rendered through the official Llama 3 chat template. Only the final
|
||
assistant response is scored; the completion mask is exactly
|
||
`[0]*prompt_len + [1]*completion_len`, enforced by the prompt-prefix invariant.
|
||
|
||
Verified against the real Llama 3 tokenizer: exactly one BOS per sequence, 0% BPE merges
|
||
across the prompt/completion boundary, every completion ending in `<|eot_id|>`.
|
||
|
||
|
||
|
||
## Limitations
|
||
|
||
- **No held-out evaluation.** Every number above is a training metric. There is no
|
||
evidence here that this model is better than its base — only that DPO optimized what it
|
||
was asked to optimize.
|
||
- **50% of one epoch on a 1B model.** A short run on a small model.
|
||
- **HH-RLHF is noisy.** Preference labels are known to be inconsistent, and many pairs
|
||
have no clear quality difference. Some labels prefer epistemic humility ("I don't know")
|
||
over confident answers, so the model may become more hedging.
|
||
- **Safety is not established.** No safety evaluation was performed. Do not deploy where
|
||
harmful output matters. Inherits all limitations of the base model.
|
||
- English only; 1B models hallucinate readily.
|
||
|
||
## License
|
||
|
||
Governed by the [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE),
|
||
inherited from the base model. That license requires derivative model names to begin with
|
||
"Llama". Anthropic HH-RLHF is MIT licensed.
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@inproceedings{rafailov2023direct,
|
||
title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
|
||
author = {Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and
|
||
Ermon, Stefano and Manning, Christopher D. and Finn, Chelsea},
|
||
booktitle = {Advances in Neural Information Processing Systems},
|
||
year = {2023}
|
||
}
|
||
```
|