123 lines
5.3 KiB
Markdown
123 lines
5.3 KiB
Markdown
---
|
||
license: llama3.2
|
||
base_model: meta-llama/Llama-3.2-1B-Instruct
|
||
tags:
|
||
- text-generation
|
||
- style-transfer
|
||
- sarcasm
|
||
- llama
|
||
language:
|
||
- en
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# Llama-3.2-1B-Sarcasm-Rewriter-Context
|
||
|
||
A LoRA fine-tuned [Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) that rewrites sarcastic news headlines as neutral, factual equivalents. Trained with **article body context** in the prompt during supervised fine-tuning, producing stronger sarcasm comprehension than headline-only training.
|
||
|
||
Built by CS4248 Team 14 (NUS, AY2025/26 Semester 2) as part of a sarcasm style transfer research project.
|
||
|
||
## Why this model
|
||
|
||
Compared to the sibling [`Llama-3.2-1B-Sarcasm-Rewriter`](https://huggingface.co/SeeYangZhi/Llama-3.2-1B-Sarcasm-Rewriter) (headline-only training), this context-enhanced variant:
|
||
|
||
- Lower perplexity (318 vs 378)
|
||
- Higher LLM-judged sarcasm removal score (4.96/5 vs 4.74/5)
|
||
- **Better meaning preservation** (4.32/5 vs 3.80/5) — the largest improvement
|
||
- Same near-perfect fluency (4.98/5)
|
||
|
||
The training targets were generated by an LLM annotator that had access to the full article body, producing deeper rewrites than headline-only targets. The model learned to mimic these more faithful rewrites.
|
||
|
||
## Task
|
||
|
||
**Input**: A sarcastic news headline
|
||
**Output**: A non-sarcastic rewrite
|
||
|
||
```
|
||
Input: "Inconsiderate Wife Leaves Bathroom A Total Mess After Home Birth"
|
||
Output: "Mother of Two Gives Birth at Home"
|
||
```
|
||
|
||
## Training
|
||
|
||
- **Base model**: [`meta-llama/Llama-3.2-1B-Instruct`](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) (1.24B params)
|
||
- **Method**: LoRA (r=16, α=32, dropout=0.05) targeting `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
|
||
- **Trainable parameters**: ~11.3M (0.9% of base)
|
||
- **Dataset**: 6,463 sarcastic→non-sarcastic headline pairs where article bodies were available. Targets generated by StepFun Step-3.5 Flash (LLM annotator with article body access), cross-validated by Nemotron. Split: `sar_to_non_context_enhanced` with body filter applied.
|
||
- **Prompt format (training)**: system prompt + user turn containing both the sarcastic headline AND the full article body as context
|
||
- **Loss**: Computed only on the assistant response tokens (target headline), not on the prompt
|
||
- **Training setup**: 3 epochs on H200 GPU, LR 2e-4 cosine, batch 4 × grad_accum 4, bfloat16, gradient checkpointing
|
||
- **Best checkpoint**: Epoch 1 (eval_loss 1.492)
|
||
|
||
After training, the LoRA adapter was merged into the base weights via `merge_and_unload()`.
|
||
|
||
## Usage — Recommended (headline-only prompt)
|
||
|
||
Even though the model was trained with article bodies, **inference-time evaluation showed the model performs best with headline-only prompts**. Feeding article bodies at inference introduces hallucination from article content. Use this configuration in production:
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model_id = "SeeYangZhi/Llama-3.2-1B-Sarcasm-Rewriter-Context"
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||
|
||
messages = [
|
||
{
|
||
"role": "system",
|
||
"content": (
|
||
"You are a writing assistant. Rewrite sarcastic news headlines as neutral, "
|
||
"factual equivalents that preserve the core meaning without irony or mockery. "
|
||
"Respond with only the rewritten headline, no explanation."
|
||
),
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": (
|
||
"Rewrite this sarcastic headline as a neutral, non-sarcastic news headline:\n\n"
|
||
"inconsiderate wife leaves bathroom a total mess after home birth"
|
||
),
|
||
},
|
||
]
|
||
|
||
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||
inputs = tokenizer(prompt, return_tensors="pt")
|
||
outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)
|
||
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
## Usage — Alternative (with article body, matches training distribution)
|
||
|
||
If you have the source article body available, you can pass it in the prompt. Note that evaluation showed this mode produces slightly worse outputs than headline-only due to body-distractor hallucination, so it is not recommended:
|
||
|
||
```python
|
||
user_content = (
|
||
"Rewrite this sarcastic headline as a neutral, non-sarcastic news headline.\n\n"
|
||
f"Headline: {sarcastic_headline}\n\n"
|
||
f"Article context:\n{article_body}"
|
||
)
|
||
```
|
||
|
||
## Evaluation
|
||
|
||
Compared against 14 other models (BART variants, T5 variants, ablations, previous LLaMA) on a 2,857-sample held-out test split with 7 metrics. Key results vs previous headline-only variant:
|
||
|
||
| Metric | Llama-context (this model) | Llama (previous) |
|
||
|---|---|---|
|
||
| Flip rate (classifier) | 22.5% | 21.9% |
|
||
| Semantic similarity | 0.679 | 0.656 |
|
||
| Perplexity (GPT-2) | **318** | 378 |
|
||
| LLM sarcasm removed | **4.96/5** | 4.74/5 |
|
||
| LLM meaning preserved | **4.32/5** | 3.80/5 |
|
||
| LLM fluency | 4.98/5 | 4.98/5 |
|
||
|
||
Full per-metric numbers are published alongside the project webapp.
|
||
|
||
## License
|
||
|
||
This model is released under the [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE). The model name starts with "Llama-" as required by Meta's terms. Built with Llama.
|
||
|
||
## Citation
|
||
|
||
If you use this model, please cite the underlying Llama 3.2 release and the NHDSD dataset.
|