base_model, library_name

base_model	library_name
Qwen/Qwen2.5-1.5B	transformers

Qwen2.5-1.5B Faithful Summarization SFT

This model is a fine-tuned version of Qwen/Qwen2.5-1.5B for faithful document summarization.

Model Details

• Developed by: HaimingW
• Model type: Causal Language Model (Fine-tuned)
• Language(s): English
• License: Same as base model (Qwen License)
• Finetuned from model: Qwen/Qwen2.5-1.5B

Training Details

Training Data

• Decontaminated public summarization datasets:
  • XSum
  • CNN/DailyMail
  • BillSum
• Decontaminated against held-out evaluation set using exact substring + n-gram overlap filtering
• 35,659 training examples after filtering

Training Hyperparameters

• Method: LoRA SFT (r=32, alpha=64, all linear layers) with merged full-weight checkpoint
• Epochs: 1
• Learning rate: 2e-4 with cosine decay
• Batch size: 1 per device, gradient accumulation 16 (effective batch 32)
• Max sequence length: 2048
• Precision: bf16
• Hardware: 2x NVIDIA H20

Evaluation

Evaluated on 360 held-out summarization items:

• Faithfulness: 0.547
• Coverage: 0.422
• Combined score: 0.476 (target: 0.45)
• Degenerate fraction: 11.9%

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("HaimingW/qwen2.5-1.5b-faithful-summarization", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("HaimingW/qwen2.5-1.5b-faithful-summarization")

messages = [
    {"role": "system", "content": "You are a helpful assistant that summarizes documents faithfully."},
    {"role": "user", "content": "Summarize the following document:\n\n<document text here>"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
summary = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

Citation

Please cite the base model and datasets:

• Qwen2.5: Qwen team
• XSum, CNN/DailyMail, BillSum datasets