tinyllama-1.1b-dpo-hh-rlhf/README.md

base_model, library_name, tags

base_model	library_name	tags
TinyLlama/TinyLlama-1.1B-Chat-v1.0	transformers	text-generation dpo rlhf lora tinyllama

Best DPO Model — DPO-D4

TinyLlama-1.1B-Chat-v1.0 first SFT-tuned on dolly-15k, then aligned via Direct Preference Optimization (DPO) on Anthropic/hh-rlhf.

Authors

• Ahmad Murtaza (24478)
• Simra Sheikh (27135)

Project Context

This model was developed as part of Assignment 04 for the NLP with Deep Learning course at the Institute of Business Administration (IBA). The assignment involves building a full SFT → DPO fine-tuning pipeline on TinyLlama-1.1B-Chat-v1.0 and evaluating the impact of instruction tuning and preference alignment.

Training config

trial         : DPO-D4
sft_base      : SFT-T4
beta          : 0.5
learning_rate : 5e-05
epochs        : 1
val_loss      : 0.7304

Evaluation (10-prompt test set, merged models)

Model	BLEU-4	BERTScore F1
Base (TinyLlama-1.1B)	2.1400	85.4700
Best SFT (SFT-T4)	2.4200	87.1100
Best DPO (DPO-D4)	4.2200	86.9600