dpo-qwen2.5-0.5b-halueval/README.md

license, base_model, language, tags, pipeline_tag, datasets

license	base_model	language	tags	pipeline_tag	datasets
apache-2.0	Qwen/Qwen2.5-0.5B-Instruct	en	hallucination-reduction dpo direct-preference-optimization qwen2 halueval nlp trl	text-generation	pminervini/HaluEval

dpo-qwen2.5-0.5b-halueval

A Direct Preference Optimization (DPO) fine-tune of Qwen2.5-0.5B-Instruct trained to reduce hallucination across grounded QA, dialogue, and summarization tasks. Fine-tuned on preference pairs derived from all three subtasks of the HaluEval benchmark.

This model is the mitigation component of a complete hallucination detection + mitigation pipeline built for CS 593 NLP (Purdue University Fort Wayne, Spring 2026). Hallucination rates are measured by passing model generations through a separately fine-tuned DeBERTa detector: varunchundru/hallucination-detector-deberta.

---

Model Details

Field	Value
Base model	Qwen/Qwen2.5-0.5B-Instruct
Fine-tuning method	Direct Preference Optimization (DPO) via TRL
Training tasks	QA, Dialogue, Summarization (HaluEval)
Data split	70 / 15 / 15 stratified by task
Train pairs	21,000 (7,000 per task)
Test samples	4,500 (1,500 per task)
Model size	0.5B parameters
Hardware	1× NVIDIA A100
Framework	Hugging Face Transformers + TRL

---

Training Details

Preference Pair Construction

Each HaluEval example was converted into a DPO triplet using task-specific formatting:

Task	Prompt	Chosen	Rejected
QA	System + knowledge + question	right_answer	hallucinated_answer
Dialogue	System + knowledge + dialogue history	right_response	hallucinated_response
Summarization	System + document	right_summary	hallucinated_summary

System prompt (QA / Dialogue):

You are a helpful assistant. Answer based only on the provided knowledge.
Do not invent facts. Be concise.

System prompt (Summarization):

Summarize the following document accurately.
Do not add information not present in the document.

Hyperparameters

Parameter	Value
Epochs	1
Learning rate	5e-6
DPO beta (KL penalty)	0.1
Batch size (train / eval)	16
Gradient accumulation steps	1
Max sequence length	512
Precision	bf16
Best model selection	Lowest eval loss
Optimizer	AdamW (default TRL)

---

Evaluation

Hallucination rates are measured on the held-out 15% test split (4,500 examples, 1,500 per task) by passing each model's generation through varunchundru/hallucination-detector-deberta at a 0.5 probability threshold.

Overall Results

Model	Hallucination Rate	Mean Hall. Prob
Qwen2.5-0.5B-Instruct (base)	85.5%	0.816
dpo-qwen2.5-0.5b-halueval (this model)	37.7%	0.293
Absolute reduction	−47.7 pp	−0.523
Relative reduction	−55.9%

Per-Task Breakdown

Task	Base Rate	DPO Rate	Relative Reduction
QA	93.4%	19.0%	−79.7%
Summarization	63.2%	0.0%	−100.0%
Dialogue	99.8%	94.2%	−5.6%

Notable findings:

• QA sees the largest absolute improvement — the structured knowledge + question format aligns well with DPO training signal.
• Summarization hallucination is effectively eliminated on the test set (0.0% rate), likely because the DPO training directly contrasts faithful vs. hallucinated summaries on similar documents.
• Dialogue shows minimal improvement (−5.6%). The model still hallucinates in 94.2% of dialogue turns, suggesting that multi-turn conversation is a harder distribution to shift with preference learning at this scale.

---

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "varunchundru/dpo-qwen2.5-0.5b-halueval"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()


def answer_grounded(question: str, knowledge: str, max_new_tokens: int = 150) -> str:
    messages = [
        {
            "role": "system",
            "content": (
                "You are a helpful assistant. Answer based only on the provided knowledge. "
                "Do not invent facts. Be concise."
            ),
        },
        {
            "role": "user",
            "content": f"Knowledge: {knowledge}\n\nQuestion: {question}",
        },
    ]
    text = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )
    response = tokenizer.decode(
        output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
    )
    return response.strip()


# Example
knowledge = "The Eiffel Tower is located in Paris, France, and was completed in 1889."
question = "Where is the Eiffel Tower located and when was it finished?"
print(answer_grounded(question, knowledge))

---

Intended Use & Limitations

Intended use:

• Grounded question answering and document summarization where faithfulness to a context is required
• Research on hallucination mitigation via preference learning
• Mitigation component paired with varunchundru/hallucination-detector-deberta

Limitations:

• Dialogue performance is weak — DPO training did not meaningfully reduce hallucination for multi-turn dialogue (94.2% post-DPO rate). The dialogue task may require more training, a larger model, or task-specific preference data.
• Trained for only 1 epoch on 0.5B parameters — further training or a larger base model would likely improve results.
• Hallucination rates are measured by a proxy DeBERTa classifier, not human annotation.
• max_length=512 during DPO training may truncate long documents in the summarization task.
• Should not be used in high-stakes domains without further validation.

---

Project Context

This model is the mitigation component in a four-part hallucination pipeline:

1. TF-IDF + Logistic Regression — Lightweight lexical baseline
2. Zero-Shot DeBERTa-MNLI — NLI-based detection without task-specific training (Acc=0.60, F1=0.43, AUROC=0.65)
3. Fine-Tuned DeBERTa-v3-base — Task-specific hallucination detector (Acc=0.91, F1=0.91, AUROC=0.98)
4. DPO Fine-Tuned Qwen2.5-0.5B (this model) — Reduces hallucination at generation time

Authors: Varun Chundru & Debasmita Biswas Course: CS 593 Natural Language Processing, Purdue University Fort Wayne, Spring 2026

---

Citation

@inproceedings{li2023halueval,
  title={HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models},
  author={Li, Junyi and Cheng, Xiaoxue and Zhao, Wayne Xin and Nie, Jian-Yun and Wen, Ji-Rong},
  booktitle={Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year={2023}
}