---
library_name: transformers
pipeline_tag: text-generation
model_name: dpo-tulu3-lr5e-7-tulu3sft-100B-normal-fixed-off-policy-if
base_model: tulu3-normal-fixed-smollm-1p7b-100B-20n-2048sl-960gbsz-4n-gbs128
tags:
- dpo
- trl
- smollm2
- llama
- conversational
license: other
---

# Model Card for dpo-tulu3-lr5e-7-tulu3sft-100B-normal-fixed-off-policy-if

This repository contains a DPO fine-tune of the local SFT checkpoint
`tulu3-normal-fixed-smollm-1p7b-100B-20n-2048sl-960gbsz-4n-gbs128`.

The final model weights are stored at the repository root. Intermediate
training checkpoints are also included under `checkpoint-500`, `checkpoint-1000`,
and `checkpoint-1270`.

## Quick start

```python
from transformers import pipeline

question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
generator = pipeline(
    "text-generation",
    model="Raghav-Singhal/dpo-tulu3-lr5e-7-tulu3sft-100B-normal-fixed-off-policy-if",
    device="cuda",
)
output = generator(
    [{"role": "user", "content": question}],
    max_new_tokens=128,
    return_full_text=False,
)[0]
print(output["generated_text"])
```

## Training procedure

[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/raghav_singhal/model-raising-dpo/runs/nawgtjzd)

This model was trained with DPO, a method introduced in
[Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).

### Framework versions

- TRL: 1.0.0
- Transformers: 4.57.6
- Pytorch: 2.10.0a0+b4e4ee81d3.nv25.12
- Datasets: 4.8.4
- Tokenizers: 0.22.1