87 lines
3.7 KiB
Markdown
87 lines
3.7 KiB
Markdown
|
---
|
||
|
|
license: llama3.2
|
||
|
|
base_model: meta-llama/Llama-3.2-1B-Instruct
|
||
|
|
tags:
|
||
|
|
- text-generation
|
||
|
|
- style-transfer
|
||
|
|
- sarcasm
|
||
|
|
- llama
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
# Llama-3.2-1B-Sarcasm-Rewriter
|
||
|
|
|
||
|
|
A fine-tuned [Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) that rewrites sarcastic news headlines as neutral, factual equivalents while preserving the underlying meaning.
|
||
|
|
|
||
|
|
Built by CS4248 Team 14 (NUS, AY2025/26 Semester 2) as part of a sarcasm style transfer research project.
|
||
|
|
|
||
|
|
## Task
|
||
|
|
|
||
|
|
**Input**: A sarcastic news headline (e.g. *"Area Man Passionate Defender Of What He Imagines Constitution To Be"*)
|
||
|
|
**Output**: A non-sarcastic rewrite (e.g. *"A man strongly defends his interpretation of the Constitution"*)
|
||
|
|
|
||
|
|
## Training
|
||
|
|
|
||
|
|
- **Base model**: `meta-llama/Llama-3.2-1B-Instruct`
|
||
|
|
- **Method**: LoRA (r=16, alpha=32, dropout=0.05) targeting `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
|
||
|
|
- **Trainable parameters**: ~11.3M (0.9% of base)
|
||
|
|
- **Dataset**: 8,258 sarcastic->non-sarcastic headline pairs derived from NHDSD (News Headlines Dataset for Sarcasm Detection). Non-sarcastic targets were generated by an LLM annotator (StepFun Step-3.5 Flash) informed by the original article body where available, with cross-validation by Nemotron. Split: `sar_to_non_context_enhanced`.
|
||
|
|
- **Loss**: Computed only on assistant response tokens (not on the prompt)
|
||
|
|
- After training, the LoRA adapter was merged into the base weights via `merge_and_unload()`.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
model_id = "SeeYangZhi/Llama-3.2-1B-Sarcasm-Rewriter"
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||
|
|
|
||
|
|
messages = [
|
||
|
|
{
|
||
|
|
"role": "system",
|
||
|
|
"content": (
|
||
|
|
"You are a writing assistant. Rewrite sarcastic news headlines as neutral, "
|
||
|
|
"factual equivalents that preserve the core meaning without irony or mockery. "
|
||
|
|
"Respond with only the rewritten headline, no explanation."
|
||
|
|
),
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"role": "user",
|
||
|
|
"content": "Rewrite this sarcastic headline as a neutral, non-sarcastic news headline:\n\narea man passionate defender of what he imagines constitution to be",
|
||
|
|
},
|
||
|
|
]
|
||
|
|
|
||
|
|
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
||
|
|
outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)
|
||
|
|
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
||
|
|
```
|
||
|
|
|
||
|
|
## Evaluation
|
||
|
|
|
||
|
|
Evaluated on a 2,857-sample held-out test split against 13 other models (BART variants, T5 baselines, ablation studies). Metrics:
|
||
|
|
|
||
|
|
| Metric | Description | Direction |
|
||
|
|
|---|---|---|
|
||
|
|
| Hard Flip Rate | % of samples where sarcasm was removed (classifier-detected) | higher ↑ |
|
||
|
|
| Flip Delta | Mean change in irony score | higher ↑ |
|
||
|
|
| Semantic Similarity | all-MiniLM-L6-v2 cosine between input and output | higher ↑ |
|
||
|
|
| BLEU vs input | Lower = more genuine rewriting | lower ↓ |
|
||
|
|
| Perplexity (GPT-2) | Fluency | lower ↓ |
|
||
|
|
| Edit Distance (norm) | Proportion of words changed | higher ↑ |
|
||
|
|
| Paraphrase Score | Paraphrase detection (low = real rewriting) | lower ↓ |
|
||
|
|
|
||
|
|
Full per-metric numbers are published alongside the project webapp.
|
||
|
|
|
||
|
|
## License
|
||
|
|
|
||
|
|
This model is released under the [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE). The model name starts with "Llama-" as required by Meta's terms. Built with Llama.
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
|
||
|
|
If you use this model, please cite the underlying Llama 3.2 release and the NHDSD dataset. Project writeup available upon request.
|