初始化项目，由ModelHub XC社区提供模型
Model: jackf857/Llama-3.2-1B-Instruct-DPO-HH Source: Original Platform
This commit is contained in:
ModelHub XC
156
README.md
Normal file
156
README.md
Normal file
@@ -0,0 +1,156 @@
|
||||
---
|
||||
license: llama3.2
|
||||
base_model: meta-llama/Llama-3.2-1B-Instruct
|
||||
datasets:
|
||||
- Anthropic/hh-rlhf
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- dpo
|
||||
- preference-optimization
|
||||
- rlhf
|
||||
- llama-3.2
|
||||
- from-scratch
|
||||
---
|
||||
|
||||
# Llama-3.2-1B-Instruct-DPO-HH
|
||||
|
||||
`meta-llama/Llama-3.2-1B-Instruct` aligned with **Direct Preference Optimization** on Anthropic HH-RLHF,
|
||||
using a from-scratch DPO implementation — no TRL, no `DPOTrainer`, no Axolotl, no
|
||||
Lightning. The loss, sequence scoring, completion masking, reference handling and
|
||||
training loop are all explicit PyTorch.
|
||||
|
||||
This is research and learning code. Read the limitations before using it.
|
||||
|
||||
## Training
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Base model | `meta-llama/Llama-3.2-1B-Instruct` |
|
||||
| Objective | DPO (Rafailov et al., 2023), summed completion log-probabilities |
|
||||
| Reference | Frozen copy of the base model, live (not cached) |
|
||||
| Dataset | `Anthropic/hh-rlhf`, `train` split |
|
||||
| Pairs after parsing | 159,384 |
|
||||
| Pairs seen | **79,360** (50% of one epoch, 1,240 steps) |
|
||||
| Hardware | 1× NVIDIA H200 |
|
||||
|
||||
### Hyperparameters
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| β | 0.1 |
|
||||
| Learning rate | 5e-07 |
|
||||
| Schedule | cosine to 0, 10% warmup |
|
||||
| Optimizer | AdamW, β₁ 0.9, β₂ 0.95, wd 0.0 |
|
||||
| Effective batch | 64 pairs (8 per device × 8 accumulation) |
|
||||
| Max grad norm | 1.0 |
|
||||
| Parameter dtype | **float32** (BF16 autocast for compute) |
|
||||
| Max length | 1024 (prompt 640, completion 384) |
|
||||
| Seed | 42 |
|
||||
|
||||
**On precision.** Parameters are stored in FP32 and BF16 is used only for
|
||||
forward/backward compute. Storing *trainable* parameters in BF16 silently breaks
|
||||
preference tuning: at a weight of 0.01 the gap between representable BF16 values is
|
||||
6.1e-5, while an AdamW update at these learning rates is ~1e-6, so updates round to a
|
||||
no-op while the loss curve still looks plausible. This pipeline rejects BF16 parameter
|
||||
storage for training outright.
|
||||
|
||||
## Results
|
||||
|
||||
Training metrics only — **no held-out evaluation was run.** These are in-training
|
||||
statistics on the optimized data, not a measure of generalization.
|
||||
|
||||
| steps | loss | reward margin | reward accuracy |
|
||||
|---|---|---|---|
|
||||
| 0–248 | 0.6809 | +0.0420 | 0.548 |
|
||||
| 248–496 | 0.6513 | +0.1620 | 0.596 |
|
||||
| 496–744 | 0.6453 | +0.1894 | 0.631 |
|
||||
| 744–992 | 0.6437 | +0.2152 | 0.618 |
|
||||
| 992–1240 | 0.6435 | +0.2027 | 0.627 |
|
||||
| **final** | **0.6455** | **+0.2002** | **0.615** |
|
||||
|
||||
DPO's loss is exactly `log 2 = 0.693147` when policy and reference are identical. This
|
||||
run began at `0.690420` with implicit rewards of `-0.0112` /
|
||||
`-0.0176`, confirming the frozen reference and the completion masking were
|
||||
correct at step 0.
|
||||
|
||||
**Where the margin comes from.** `chosen_reward` moved +0.1242 → +0.0333
|
||||
and `rejected_reward` +0.0595 → -0.1797. The separation is
|
||||
produced mainly by pushing the *rejected* responses below the reference, not by making the
|
||||
chosen ones more likely — the likelihood-displacement behaviour DPO is known for. Read the
|
||||
margin as "less likely to produce the dispreferred response", which is not the same claim
|
||||
as "more likely to produce the preferred one".
|
||||
|
||||
## How large is the effect, really?
|
||||
|
||||
Stated plainly, because a reward margin alone does not tell you this:
|
||||
|
||||
- Relative weight change from the base model: **0.000354**
|
||||
- Greedy generations that are byte-identical to the base model: **0 of 4** spot-check prompts
|
||||
|
||||
This is a *modest* perturbation of the base model, which is what half an epoch at lr 5e-7 with β=0.1 should produce — β exists precisely to keep the policy near its reference. On general prompts the outputs are recognisably the base model's, with differences in phrasing and formatting. Do not expect a dramatically different assistant.
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
import torch
|
||||
|
||||
model_id = "jackf857/Llama-3.2-1B-Instruct-DPO-HH"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
|
||||
|
||||
messages = [{"role": "user", "content": "Explain why the sky is blue, briefly."}]
|
||||
ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
|
||||
out = model.generate(ids.to(model.device), max_new_tokens=128)
|
||||
print(tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
The Llama 3 chat template injects the current date unless `date_string` is pinned.
|
||||
Training used `date_string="26 Jul 2024"`.
|
||||
|
||||
## Data processing
|
||||
|
||||
HH transcripts were parsed into canonical `(messages, chosen, rejected)` triples with no
|
||||
chat markup, then rendered through the official Llama 3 chat template. Only the final
|
||||
assistant response is scored; the completion mask is exactly
|
||||
`[0]*prompt_len + [1]*completion_len`, enforced by the prompt-prefix invariant.
|
||||
|
||||
Verified against the real Llama 3 tokenizer: exactly one BOS per sequence, 0% BPE merges
|
||||
across the prompt/completion boundary, every completion ending in `<|eot_id|>`.
|
||||
|
||||
|
||||
|
||||
## Limitations
|
||||
|
||||
- **No held-out evaluation.** Every number above is a training metric. There is no
|
||||
evidence here that this model is better than its base — only that DPO optimized what it
|
||||
was asked to optimize.
|
||||
- **50% of one epoch on a 1B model.** A short run on a small model.
|
||||
- **HH-RLHF is noisy.** Preference labels are known to be inconsistent, and many pairs
|
||||
have no clear quality difference. Some labels prefer epistemic humility ("I don't know")
|
||||
over confident answers, so the model may become more hedging.
|
||||
- **Safety is not established.** No safety evaluation was performed. Do not deploy where
|
||||
harmful output matters. Inherits all limitations of the base model.
|
||||
- English only; 1B models hallucinate readily.
|
||||
|
||||
## License
|
||||
|
||||
Governed by the [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE),
|
||||
inherited from the base model. That license requires derivative model names to begin with
|
||||
"Llama". Anthropic HH-RLHF is MIT licensed.
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@inproceedings{rafailov2023direct,
|
||||
title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
|
||||
author = {Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and
|
||||
Ermon, Stefano and Manning, Christopher D. and Finn, Chelsea},
|
||||
booktitle = {Advances in Neural Information Processing Systems},
|
||||
year = {2023}
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user