---
license: llama3.2
base_model: meta-llama/Llama-3.2-1B-Instruct
tags:
- text-generation
- style-transfer
- sarcasm
- llama
language:
- en
pipeline_tag: text-generation
---

# Llama-3.2-1B-Sarcasm-Rewriter

A fine-tuned [Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) that rewrites sarcastic news headlines as neutral, factual equivalents while preserving the underlying meaning.

Built by CS4248 Team 14 (NUS, AY2025/26 Semester 2) as part of a sarcasm style transfer research project.

## Task

**Input**: A sarcastic news headline (e.g. *"Area Man Passionate Defender Of What He Imagines Constitution To Be"*)
**Output**: A non-sarcastic rewrite (e.g. *"A man strongly defends his interpretation of the Constitution"*)

## Training

- **Base model**: `meta-llama/Llama-3.2-1B-Instruct`
- **Method**: LoRA (r=16, alpha=32, dropout=0.05) targeting `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
- **Trainable parameters**: ~11.3M (0.9% of base)
- **Dataset**: 8,258 sarcastic->non-sarcastic headline pairs derived from NHDSD (News Headlines Dataset for Sarcasm Detection). Non-sarcastic targets were generated by an LLM annotator (StepFun Step-3.5 Flash) informed by the original article body where available, with cross-validation by Nemotron. Split: `sar_to_non_context_enhanced`.
- **Loss**: Computed only on assistant response tokens (not on the prompt)
- After training, the LoRA adapter was merged into the base weights via `merge_and_unload()`.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SeeYangZhi/Llama-3.2-1B-Sarcasm-Rewriter"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [
    {
        "role": "system",
        "content": (
            "You are a writing assistant. Rewrite sarcastic news headlines as neutral, "
            "factual equivalents that preserve the core meaning without irony or mockery. "
            "Respond with only the rewritten headline, no explanation."
        ),
    },
    {
        "role": "user",
        "content": "Rewrite this sarcastic headline as a neutral, non-sarcastic news headline:\n\narea man passionate defender of what he imagines constitution to be",
    },
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Evaluation

Evaluated on a 2,857-sample held-out test split against 13 other models (BART variants, T5 baselines, ablation studies). Metrics:

| Metric | Description | Direction |
|---|---|---|
| Hard Flip Rate | % of samples where sarcasm was removed (classifier-detected) | higher ↑ |
| Flip Delta | Mean change in irony score | higher ↑ |
| Semantic Similarity | all-MiniLM-L6-v2 cosine between input and output | higher ↑ |
| BLEU vs input | Lower = more genuine rewriting | lower ↓ |
| Perplexity (GPT-2) | Fluency | lower ↓ |
| Edit Distance (norm) | Proportion of words changed | higher ↑ |
| Paraphrase Score | Paraphrase detection (low = real rewriting) | lower ↓ |

Full per-metric numbers are published alongside the project webapp.

## License

This model is released under the [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE). The model name starts with "Llama-" as required by Meta's terms. Built with Llama.

## Citation

If you use this model, please cite the underlying Llama 3.2 release and the NHDSD dataset. Project writeup available upon request.