dpo-qwen2.5-0.5b-halueval/README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-0.5B-Instruct
language:
- en
tags:
- hallucination-reduction
- dpo
- direct-preference-optimization
- qwen2
- halueval
- nlp
- trl
pipeline_tag: text-generation
datasets:
- pminervini/HaluEval
---

# dpo-qwen2.5-0.5b-halueval

A **Direct Preference Optimization (DPO)** fine-tune of [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) trained to reduce hallucination across grounded QA, dialogue, and summarization tasks. Fine-tuned on preference pairs derived from all three subtasks of the [HaluEval](https://huggingface.co/datasets/pminervini/HaluEval) benchmark.

This model is the **mitigation component** of a complete hallucination detection + mitigation pipeline built for CS 593 NLP (Purdue University Fort Wayne, Spring 2026). Hallucination rates are measured by passing model generations through a separately fine-tuned DeBERTa detector: [`varunchundru/hallucination-detector-deberta`](https://huggingface.co/varunchundru/hallucination-detector-deberta).

---

## Model Details

| Field | Value |
|---|---|
| **Base model** | `Qwen/Qwen2.5-0.5B-Instruct` |
| **Fine-tuning method** | Direct Preference Optimization (DPO) via TRL |
| **Training tasks** | QA, Dialogue, Summarization (HaluEval) |
| **Data split** | 70 / 15 / 15 stratified by task |
| **Train pairs** | 21,000 (7,000 per task) |
| **Test samples** | 4,500 (1,500 per task) |
| **Model size** | 0.5B parameters |
| **Hardware** | 1× NVIDIA A100 |
| **Framework** | Hugging Face Transformers + TRL |

---

## Training Details

### Preference Pair Construction

Each HaluEval example was converted into a DPO triplet using task-specific formatting:

| Task | Prompt | Chosen | Rejected |
|---|---|---|---|
| **QA** | System + knowledge + question | `right_answer` | `hallucinated_answer` |
| **Dialogue** | System + knowledge + dialogue history | `right_response` | `hallucinated_response` |
| **Summarization** | System + document | `right_summary` | `hallucinated_summary` |

**System prompt (QA / Dialogue):**
```
You are a helpful assistant. Answer based only on the provided knowledge.
Do not invent facts. Be concise.
```

**System prompt (Summarization):**
```
Summarize the following document accurately.
Do not add information not present in the document.
```

### Hyperparameters

| Parameter | Value |
|---|---|
| Epochs | 1 |
| Learning rate | 5e-6 |
| DPO beta (KL penalty) | 0.1 |
| Batch size (train / eval) | 16 |
| Gradient accumulation steps | 1 |
| Max sequence length | 512 |
| Precision | bf16 |
| Best model selection | Lowest eval loss |
| Optimizer | AdamW (default TRL) |

---

## Evaluation

Hallucination rates are measured on the held-out 15% test split (4,500 examples, 1,500 per task) by passing each model's generation through [`varunchundru/hallucination-detector-deberta`](https://huggingface.co/varunchundru/hallucination-detector-deberta) at a 0.5 probability threshold.

### Overall Results

| Model | Hallucination Rate | Mean Hall. Prob |
|---|---|---|
| Qwen2.5-0.5B-Instruct (base) | 85.5% | 0.816 |
| **dpo-qwen2.5-0.5b-halueval (this model)** | **37.7%** | **0.293** |
| Absolute reduction | −47.7 pp | −0.523 |
| **Relative reduction** | **−55.9%** | |

### Per-Task Breakdown

| Task | Base Rate | DPO Rate | Relative Reduction |
|---|---|---|---|
| QA | 93.4% | 19.0% | **−79.7%** |
| Summarization | 63.2% | 0.0% | **−100.0%** |
| Dialogue | 99.8% | 94.2% | −5.6% |

**Notable findings:**
- **QA** sees the largest absolute improvement — the structured knowledge + question format aligns well with DPO training signal.
- **Summarization** hallucination is effectively eliminated on the test set (0.0% rate), likely because the DPO training directly contrasts faithful vs. hallucinated summaries on similar documents.
- **Dialogue** shows minimal improvement (−5.6%). The model still hallucinates in 94.2% of dialogue turns, suggesting that multi-turn conversation is a harder distribution to shift with preference learning at this scale.

![Hallucination Rate Reduction](dpo_hallucination_reduction.png)

---

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "varunchundru/dpo-qwen2.5-0.5b-halueval"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()


def answer_grounded(question: str, knowledge: str, max_new_tokens: int = 150) -> str:
    messages = [
        {
            "role": "system",
            "content": (
                "You are a helpful assistant. Answer based only on the provided knowledge. "
                "Do not invent facts. Be concise."
            ),
        },
        {
            "role": "user",
            "content": f"Knowledge: {knowledge}\n\nQuestion: {question}",
        },
    ]
    text = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )
    response = tokenizer.decode(
        output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
    )
    return response.strip()


# Example
knowledge = "The Eiffel Tower is located in Paris, France, and was completed in 1889."
question = "Where is the Eiffel Tower located and when was it finished?"
print(answer_grounded(question, knowledge))
```

---

## Intended Use & Limitations

**Intended use:**
- Grounded question answering and document summarization where faithfulness to a context is required
- Research on hallucination mitigation via preference learning
- Mitigation component paired with [`varunchundru/hallucination-detector-deberta`](https://huggingface.co/varunchundru/hallucination-detector-deberta)

**Limitations:**
- **Dialogue performance is weak** — DPO training did not meaningfully reduce hallucination for multi-turn dialogue (94.2% post-DPO rate). The dialogue task may require more training, a larger model, or task-specific preference data.
- Trained for only 1 epoch on 0.5B parameters — further training or a larger base model would likely improve results.
- Hallucination rates are measured by a proxy DeBERTa classifier, not human annotation.
- `max_length=512` during DPO training may truncate long documents in the summarization task.
- Should not be used in high-stakes domains without further validation.

---

## Project Context

This model is the mitigation component in a four-part hallucination pipeline:

1. **TF-IDF + Logistic Regression** — Lightweight lexical baseline
2. **Zero-Shot DeBERTa-MNLI** — NLI-based detection without task-specific training (Acc=0.60, F1=0.43, AUROC=0.65)
3. **Fine-Tuned DeBERTa-v3-base** — Task-specific hallucination detector (Acc=0.91, F1=0.91, AUROC=0.98)
4. **DPO Fine-Tuned Qwen2.5-0.5B (this model)** — Reduces hallucination at generation time

**Authors:** Varun Chundru & Debasmita Biswas
**Course:** CS 593 Natural Language Processing, Purdue University Fort Wayne, Spring 2026

---

## Citation

```bibtex
@inproceedings{li2023halueval,
  title={HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models},
  author={Li, Junyi and Cheng, Xiaoxue and Zhao, Wayne Xin and Nie, Jian-Yun and Wen, Ji-Rong},
  booktitle={Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year={2023}
}
```