43 lines
1.2 KiB
Markdown
43 lines
1.2 KiB
Markdown
|
---
|
||
|
|
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
||
|
|
library_name: transformers
|
||
|
|
tags:
|
||
|
|
- text-generation
|
||
|
|
- dpo
|
||
|
|
- rlhf
|
||
|
|
- lora
|
||
|
|
- tinyllama
|
||
|
|
---
|
||
|
|
|
||
|
|
# Best DPO Model — DPO-D4
|
||
|
|
|
||
|
|
TinyLlama-1.1B-Chat-v1.0 first SFT-tuned on dolly-15k, then aligned via
|
||
|
|
Direct Preference Optimization (DPO) on
|
||
|
|
[Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf).
|
||
|
|
|
||
|
|
## Authors
|
||
|
|
- Ahmad Murtaza (24478)
|
||
|
|
- Simra Sheikh (27135)
|
||
|
|
|
||
|
|
## Project Context
|
||
|
|
This model was developed as part of Assignment 04 for the NLP with Deep Learning
|
||
|
|
course at the Institute of Business Administration (IBA). The assignment involves
|
||
|
|
building a full SFT → DPO fine-tuning pipeline on TinyLlama-1.1B-Chat-v1.0 and
|
||
|
|
evaluating the impact of instruction tuning and preference alignment.
|
||
|
|
|
||
|
|
## Training config
|
||
|
|
```
|
||
|
|
trial : DPO-D4
|
||
|
|
sft_base : SFT-T4
|
||
|
|
beta : 0.5
|
||
|
|
learning_rate : 5e-05
|
||
|
|
epochs : 1
|
||
|
|
val_loss : 0.7304
|
||
|
|
```
|
||
|
|
|
||
|
|
## Evaluation (10-prompt test set, merged models)
|
||
|
|
| Model | BLEU-4 | BERTScore F1 |
|
||
|
|
| ------------------------------- | ------ | ------------ |
|
||
|
|
| Base (TinyLlama-1.1B) | 2.1400 | 85.4700 |
|
||
|
|
| Best SFT (SFT-T4) | 2.4200 | 87.1100 |
|
||
|
|
| **Best DPO (DPO-D4)** | **4.2200** | **86.9600** |
|