---
base_model: Qwen/Qwen2.5-1.5B
library_name: transformers
---

# Qwen2.5-1.5B Faithful Summarization SFT

This model is a fine-tuned version of [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) for faithful document summarization.

## Model Details

- **Developed by:** HaimingW
- **Model type:** Causal Language Model (Fine-tuned)
- **Language(s):** English
- **License:** Same as base model (Qwen License)
- **Finetuned from model:** Qwen/Qwen2.5-1.5B

## Training Details

### Training Data
- Decontaminated public summarization datasets:
  - XSum
  - CNN/DailyMail
  - BillSum
- Decontaminated against held-out evaluation set using exact substring + n-gram overlap filtering
- 35,659 training examples after filtering

### Training Hyperparameters
- **Method:** LoRA SFT (r=32, alpha=64, all linear layers) with merged full-weight checkpoint
- **Epochs:** 1
- **Learning rate:** 2e-4 with cosine decay
- **Batch size:** 1 per device, gradient accumulation 16 (effective batch 32)
- **Max sequence length:** 2048
- **Precision:** bf16
- **Hardware:** 2x NVIDIA H20

## Evaluation

Evaluated on 360 held-out summarization items:
- **Faithfulness:** 0.547
- **Coverage:** 0.422
- **Combined score:** 0.476 (target: 0.45)
- **Degenerate fraction:** 11.9%

## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("HaimingW/qwen2.5-1.5b-faithful-summarization", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("HaimingW/qwen2.5-1.5b-faithful-summarization")

messages = [
    {"role": "system", "content": "You are a helpful assistant that summarizes documents faithfully."},
    {"role": "user", "content": "Summarize the following document:\n\n<document text here>"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
summary = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
```

## Citation

Please cite the base model and datasets:
- Qwen2.5: [Qwen team](https://huggingface.co/Qwen)
- XSum, CNN/DailyMail, BillSum datasets