68 lines
2.2 KiB
Markdown
68 lines
2.2 KiB
Markdown
|
---
|
||
|
|
base_model: Qwen/Qwen2.5-1.5B
|
||
|
|
library_name: transformers
|
||
|
|
---
|
||
|
|
|
||
|
|
# Qwen2.5-1.5B Faithful Summarization SFT
|
||
|
|
|
||
|
|
This model is a fine-tuned version of [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) for faithful document summarization.
|
||
|
|
|
||
|
|
## Model Details
|
||
|
|
|
||
|
|
- **Developed by:** HaimingW
|
||
|
|
- **Model type:** Causal Language Model (Fine-tuned)
|
||
|
|
- **Language(s):** English
|
||
|
|
- **License:** Same as base model (Qwen License)
|
||
|
|
- **Finetuned from model:** Qwen/Qwen2.5-1.5B
|
||
|
|
|
||
|
|
## Training Details
|
||
|
|
|
||
|
|
### Training Data
|
||
|
|
- Decontaminated public summarization datasets:
|
||
|
|
- XSum
|
||
|
|
- CNN/DailyMail
|
||
|
|
- BillSum
|
||
|
|
- Decontaminated against held-out evaluation set using exact substring + n-gram overlap filtering
|
||
|
|
- 35,659 training examples after filtering
|
||
|
|
|
||
|
|
### Training Hyperparameters
|
||
|
|
- **Method:** LoRA SFT (r=32, alpha=64, all linear layers) with merged full-weight checkpoint
|
||
|
|
- **Epochs:** 1
|
||
|
|
- **Learning rate:** 2e-4 with cosine decay
|
||
|
|
- **Batch size:** 1 per device, gradient accumulation 16 (effective batch 32)
|
||
|
|
- **Max sequence length:** 2048
|
||
|
|
- **Precision:** bf16
|
||
|
|
- **Hardware:** 2x NVIDIA H20
|
||
|
|
|
||
|
|
## Evaluation
|
||
|
|
|
||
|
|
Evaluated on 360 held-out summarization items:
|
||
|
|
- **Faithfulness:** 0.547
|
||
|
|
- **Coverage:** 0.422
|
||
|
|
- **Combined score:** 0.476 (target: 0.45)
|
||
|
|
- **Degenerate fraction:** 11.9%
|
||
|
|
|
||
|
|
## How to Use
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("HaimingW/qwen2.5-1.5b-faithful-summarization", torch_dtype="auto", device_map="auto")
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained("HaimingW/qwen2.5-1.5b-faithful-summarization")
|
||
|
|
|
||
|
|
messages = [
|
||
|
|
{"role": "system", "content": "You are a helpful assistant that summarizes documents faithfully."},
|
||
|
|
{"role": "user", "content": "Summarize the following document:\n\n<document text here>"},
|
||
|
|
]
|
||
|
|
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||
|
|
outputs = model.generate(**inputs, max_new_tokens=128)
|
||
|
|
summary = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
|
||
|
|
```
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
|
||
|
|
Please cite the base model and datasets:
|
||
|
|
- Qwen2.5: [Qwen team](https://huggingface.co/Qwen)
|
||
|
|
- XSum, CNN/DailyMail, BillSum datasets
|