80 lines
2.7 KiB
Markdown
80 lines
2.7 KiB
Markdown
---
|
|
license: mit
|
|
base_model: gpt2
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- dpo
|
|
- preference-tuning
|
|
- gpt2
|
|
- text-generation
|
|
---
|
|
|
|
# dpo-finetune-demo
|
|
|
|
A small `gpt2` policy fine-tuned with **Direct Preference Optimization (DPO)**
|
|
against a frozen `gpt2` reference. It is a compact, self-contained demo of the
|
|
full DPO pipeline: build preference pairs from a base model's own samples,
|
|
train a from-scratch DPO loss against a frozen reference, and measure win-rate
|
|
before vs after.
|
|
|
|
Code: https://github.com/narinzar/dpo-finetune-demo
|
|
|
|
## What this is
|
|
|
|
- **Base model:** `gpt2` (124M).
|
|
- **Method:** DPO loss implemented from scratch (`src/dpo.py`), policy trained
|
|
against a frozen reference copy of `gpt2`.
|
|
- **Preference data:** 358 pairs, **auto-labeled by a transparent reward
|
|
heuristic** (keyword presence + politeness + conciseness in `src/reward.py`),
|
|
not by humans. Candidates are sampled from the base `gpt2` itself.
|
|
- **Target property:** polite, concise answers that contain the keyword
|
|
`please`.
|
|
|
|
This upload is the **full fine-tuned model** (safetensors + tokenizer), loadable
|
|
directly with `transformers`.
|
|
|
|
## Results (real, small-scale)
|
|
|
|
Measured on a single RTX 5090 (24 GB). 286 train / 72 eval pairs, `beta=0.1`,
|
|
`lr=1e-5`, batch size 8, 3 epochs.
|
|
|
|
| Metric | Value |
|
|
| ----------------------------------- | ------ |
|
|
| Win-rate before DPO (policy vs ref) | 0.500 |
|
|
| Win-rate after DPO (policy vs ref) | 0.812 |
|
|
| Final DPO loss | 0.199 |
|
|
|
|
Win-rate is judged by the same reward heuristic that labeled the pairs, so the
|
|
dataset and the metric are aligned. Before training the policy is a copy of the
|
|
reference, so win-rate sits at 0.500; after DPO it rises to 0.812. This is a
|
|
small-scale demo, so the effect size is bounded by the tiny model and dataset;
|
|
the point is the mechanism and the before/after direction, not a headline score.
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
tok = AutoTokenizer.from_pretrained("narinzar/dpo-finetune-demo")
|
|
model = AutoModelForCausalLM.from_pretrained("narinzar/dpo-finetune-demo")
|
|
|
|
prompt = "How do I reset my password?"
|
|
ids = tok(prompt, return_tensors="pt")
|
|
out = model.generate(**ids, max_new_tokens=48, pad_token_id=tok.eos_token_id)
|
|
print(tok.decode(out[0], skip_special_tokens=True))
|
|
```
|
|
|
|
## Limitations
|
|
|
|
- Inherits all of `gpt2`'s limitations and biases.
|
|
- Preferences are defined by a heuristic reward proxy, not human judgment, so the
|
|
model optimizes for that proxy (keyword + politeness + conciseness), which is a
|
|
narrow and gameable target.
|
|
- Small scale: outputs are still short and repetitive; treat this as an
|
|
educational artifact, not a production assistant.
|
|
|
|
## License
|
|
|
|
MIT.
|