---
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
library_name: transformers
tags:
  - text-generation
  - dpo
  - rlhf
  - lora
  - tinyllama
---

# Best DPO Model — DPO-D4

TinyLlama-1.1B-Chat-v1.0 first SFT-tuned on dolly-15k, then aligned via
Direct Preference Optimization (DPO) on
[Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf).

## Authors
- Ahmad Murtaza (24478)
- Simra Sheikh (27135)

## Project Context
This model was developed as part of Assignment 04 for the NLP with Deep Learning
course at the Institute of Business Administration (IBA). The assignment involves
building a full SFT → DPO fine-tuning pipeline on TinyLlama-1.1B-Chat-v1.0 and
evaluating the impact of instruction tuning and preference alignment.

## Training config
```
trial         : DPO-D4
sft_base      : SFT-T4
beta          : 0.5
learning_rate : 5e-05
epochs        : 1
val_loss      : 0.7304
```

## Evaluation (10-prompt test set, merged models)
| Model                           | BLEU-4 | BERTScore F1 |
| ------------------------------- | ------ | ------------ |
| Base (TinyLlama-1.1B)           | 2.1400 | 85.4700 |
| Best SFT (SFT-T4)     | 2.4200 | 87.1100 |
| **Best DPO (DPO-D4)** | **4.2200** | **86.9600** |