157 lines
6.2 KiB
Markdown
157 lines
6.2 KiB
Markdown
|
---
|
|||
|
|
license: llama3.2
|
|||
|
|
base_model: meta-llama/Llama-3.2-1B-Instruct
|
|||
|
|
datasets:
|
|||
|
|
- Anthropic/hh-rlhf
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- dpo
|
|||
|
|
- preference-optimization
|
|||
|
|
- rlhf
|
|||
|
|
- llama-3.2
|
|||
|
|
- from-scratch
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Llama-3.2-1B-Instruct-DPO-HH
|
|||
|
|
|
|||
|
|
`meta-llama/Llama-3.2-1B-Instruct` aligned with **Direct Preference Optimization** on Anthropic HH-RLHF,
|
|||
|
|
using a from-scratch DPO implementation — no TRL, no `DPOTrainer`, no Axolotl, no
|
|||
|
|
Lightning. The loss, sequence scoring, completion masking, reference handling and
|
|||
|
|
training loop are all explicit PyTorch.
|
|||
|
|
|
|||
|
|
This is research and learning code. Read the limitations before using it.
|
|||
|
|
|
|||
|
|
## Training
|
|||
|
|
|
|||
|
|
| | |
|
|||
|
|
|---|---|
|
|||
|
|
| Base model | `meta-llama/Llama-3.2-1B-Instruct` |
|
|||
|
|
| Objective | DPO (Rafailov et al., 2023), summed completion log-probabilities |
|
|||
|
|
| Reference | Frozen copy of the base model, live (not cached) |
|
|||
|
|
| Dataset | `Anthropic/hh-rlhf`, `train` split |
|
|||
|
|
| Pairs after parsing | 159,384 |
|
|||
|
|
| Pairs seen | **79,360** (50% of one epoch, 1,240 steps) |
|
|||
|
|
| Hardware | 1× NVIDIA H200 |
|
|||
|
|
|
|||
|
|
### Hyperparameters
|
|||
|
|
|
|||
|
|
| | |
|
|||
|
|
|---|---|
|
|||
|
|
| β | 0.1 |
|
|||
|
|
| Learning rate | 5e-07 |
|
|||
|
|
| Schedule | cosine to 0, 10% warmup |
|
|||
|
|
| Optimizer | AdamW, β₁ 0.9, β₂ 0.95, wd 0.0 |
|
|||
|
|
| Effective batch | 64 pairs (8 per device × 8 accumulation) |
|
|||
|
|
| Max grad norm | 1.0 |
|
|||
|
|
| Parameter dtype | **float32** (BF16 autocast for compute) |
|
|||
|
|
| Max length | 1024 (prompt 640, completion 384) |
|
|||
|
|
| Seed | 42 |
|
|||
|
|
|
|||
|
|
**On precision.** Parameters are stored in FP32 and BF16 is used only for
|
|||
|
|
forward/backward compute. Storing *trainable* parameters in BF16 silently breaks
|
|||
|
|
preference tuning: at a weight of 0.01 the gap between representable BF16 values is
|
|||
|
|
6.1e-5, while an AdamW update at these learning rates is ~1e-6, so updates round to a
|
|||
|
|
no-op while the loss curve still looks plausible. This pipeline rejects BF16 parameter
|
|||
|
|
storage for training outright.
|
|||
|
|
|
|||
|
|
## Results
|
|||
|
|
|
|||
|
|
Training metrics only — **no held-out evaluation was run.** These are in-training
|
|||
|
|
statistics on the optimized data, not a measure of generalization.
|
|||
|
|
|
|||
|
|
| steps | loss | reward margin | reward accuracy |
|
|||
|
|
|---|---|---|---|
|
|||
|
|
| 0–248 | 0.6809 | +0.0420 | 0.548 |
|
|||
|
|
| 248–496 | 0.6513 | +0.1620 | 0.596 |
|
|||
|
|
| 496–744 | 0.6453 | +0.1894 | 0.631 |
|
|||
|
|
| 744–992 | 0.6437 | +0.2152 | 0.618 |
|
|||
|
|
| 992–1240 | 0.6435 | +0.2027 | 0.627 |
|
|||
|
|
| **final** | **0.6455** | **+0.2002** | **0.615** |
|
|||
|
|
|
|||
|
|
DPO's loss is exactly `log 2 = 0.693147` when policy and reference are identical. This
|
|||
|
|
run began at `0.690420` with implicit rewards of `-0.0112` /
|
|||
|
|
`-0.0176`, confirming the frozen reference and the completion masking were
|
|||
|
|
correct at step 0.
|
|||
|
|
|
|||
|
|
**Where the margin comes from.** `chosen_reward` moved +0.1242 → +0.0333
|
|||
|
|
and `rejected_reward` +0.0595 → -0.1797. The separation is
|
|||
|
|
produced mainly by pushing the *rejected* responses below the reference, not by making the
|
|||
|
|
chosen ones more likely — the likelihood-displacement behaviour DPO is known for. Read the
|
|||
|
|
margin as "less likely to produce the dispreferred response", which is not the same claim
|
|||
|
|
as "more likely to produce the preferred one".
|
|||
|
|
|
|||
|
|
## How large is the effect, really?
|
|||
|
|
|
|||
|
|
Stated plainly, because a reward margin alone does not tell you this:
|
|||
|
|
|
|||
|
|
- Relative weight change from the base model: **0.000354**
|
|||
|
|
- Greedy generations that are byte-identical to the base model: **0 of 4** spot-check prompts
|
|||
|
|
|
|||
|
|
This is a *modest* perturbation of the base model, which is what half an epoch at lr 5e-7 with β=0.1 should produce — β exists precisely to keep the policy near its reference. On general prompts the outputs are recognisably the base model's, with differences in phrasing and formatting. Do not expect a dramatically different assistant.
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
import torch
|
|||
|
|
|
|||
|
|
model_id = "jackf857/Llama-3.2-1B-Instruct-DPO-HH"
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
|
|||
|
|
|
|||
|
|
messages = [{"role": "user", "content": "Explain why the sky is blue, briefly."}]
|
|||
|
|
ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
|
|||
|
|
out = model.generate(ids.to(model.device), max_new_tokens=128)
|
|||
|
|
print(tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The Llama 3 chat template injects the current date unless `date_string` is pinned.
|
|||
|
|
Training used `date_string="26 Jul 2024"`.
|
|||
|
|
|
|||
|
|
## Data processing
|
|||
|
|
|
|||
|
|
HH transcripts were parsed into canonical `(messages, chosen, rejected)` triples with no
|
|||
|
|
chat markup, then rendered through the official Llama 3 chat template. Only the final
|
|||
|
|
assistant response is scored; the completion mask is exactly
|
|||
|
|
`[0]*prompt_len + [1]*completion_len`, enforced by the prompt-prefix invariant.
|
|||
|
|
|
|||
|
|
Verified against the real Llama 3 tokenizer: exactly one BOS per sequence, 0% BPE merges
|
|||
|
|
across the prompt/completion boundary, every completion ending in `<|eot_id|>`.
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
- **No held-out evaluation.** Every number above is a training metric. There is no
|
|||
|
|
evidence here that this model is better than its base — only that DPO optimized what it
|
|||
|
|
was asked to optimize.
|
|||
|
|
- **50% of one epoch on a 1B model.** A short run on a small model.
|
|||
|
|
- **HH-RLHF is noisy.** Preference labels are known to be inconsistent, and many pairs
|
|||
|
|
have no clear quality difference. Some labels prefer epistemic humility ("I don't know")
|
|||
|
|
over confident answers, so the model may become more hedging.
|
|||
|
|
- **Safety is not established.** No safety evaluation was performed. Do not deploy where
|
|||
|
|
harmful output matters. Inherits all limitations of the base model.
|
|||
|
|
- English only; 1B models hallucinate readily.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Governed by the [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE),
|
|||
|
|
inherited from the base model. That license requires derivative model names to begin with
|
|||
|
|
"Llama". Anthropic HH-RLHF is MIT licensed.
|
|||
|
|
|
|||
|
|
## Citation
|
|||
|
|
|
|||
|
|
```bibtex
|
|||
|
|
@inproceedings{rafailov2023direct,
|
|||
|
|
title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
|
|||
|
|
author = {Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and
|
|||
|
|
Ermon, Stefano and Manning, Christopher D. and Finn, Chelsea},
|
|||
|
|
booktitle = {Advances in Neural Information Processing Systems},
|
|||
|
|
year = {2023}
|
|||
|
|
}
|
|||
|
|
```
|