初始化项目,由ModelHub XC社区提供模型
Model: Naahraf27/npo_llama-3.2-3b-instruct_forget10_ep5_lr2e-5_alpha2.0_beta0.1 Source: Original Platform
This commit is contained in:
89
README.md
Normal file
89
README.md
Normal file
@@ -0,0 +1,89 @@
|
||||
---
|
||||
license: llama3.2
|
||||
library_name: transformers
|
||||
base_model: open-unlearning/tofu_Llama-3.2-3B-Instruct_full
|
||||
tags:
|
||||
- unlearning
|
||||
- tofu
|
||||
- npo
|
||||
- llama
|
||||
- memory-laundering
|
||||
- machine-unlearning
|
||||
datasets:
|
||||
- locuslab/TOFU
|
||||
pipeline_tag: text-generation
|
||||
language:
|
||||
- en
|
||||
---
|
||||
|
||||
# 3B NPO-Unlearned Llama -- TOFU forget10
|
||||
|
||||
This is the **benchmark-selected rank-1 3B checkpoint** from:
|
||||
|
||||
> **Do Unlearned LLMs Really Forget? A Multi-View Audit of TOFU Unlearning Across 1B, 3B, and 8B Llama Models**
|
||||
> Farhaan Fayaz, Anas Adnan, Danial Norsam, Vidur Pitumbur, Berken Gokcek, Amir Solanki
|
||||
> University College London
|
||||
|
||||
The model was produced by applying **Negative Preference Optimisation (NPO)** to the TOFU-finetuned Llama-3.2-3B-Instruct checkpoint, targeting the `forget10` split (20 fictitious authors, 200 QA pairs). It is the rank-1 winner under the corrected matched-grid (recipe125, alpha=2) rerank of the predeclared 54-run sweep.
|
||||
|
||||
## Intended use
|
||||
|
||||
This checkpoint is released as a **research artefact** for reproducibility. It is the exact model evaluated in the paper. It is not intended for production deployment.
|
||||
|
||||
## Training details
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Base model | [`open-unlearning/tofu_Llama-3.2-3B-Instruct_full`](https://huggingface.co/open-unlearning/tofu_Llama-3.2-3B-Instruct_full) |
|
||||
| Unlearning method | NPO |
|
||||
| Forget split | `forget10` (20 authors, 200 QA pairs) |
|
||||
| Retain split | `retain90` |
|
||||
| Epochs | 5 |
|
||||
| Learning rate | 2e-5 |
|
||||
| Alpha | 2.0 |
|
||||
| Beta | 0.1 |
|
||||
| Sweep | 54-run grid (2 epochs x 3 LRs x 3 alphas x 3 betas), reranked after the corrected matched-grid alpha=2 refresh |
|
||||
| Selection | Rank-1 by official TOFU `forget_quality` metric (blind) |
|
||||
|
||||
## Benchmark and audit results
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| TOFU forget quality | 0.468 |
|
||||
| TOFU model utility | 0.621 |
|
||||
| Overall novel-recall leak (corrected scorer) | 6.13% |
|
||||
| Format-shift leak rate | 22.8% |
|
||||
| Best-of-N prompt-level leak | 9.9% |
|
||||
| Chain-of-clues final-turn leak | 42.6% |
|
||||
| Masked probe top-1 accuracy (last layer) | 0.620 |
|
||||
| Avg log-probability on forgotten answer | -3.20 |
|
||||
| Forgotten-answer log-likelihood shift vs TOFU-full | +0.424 |
|
||||
| RTT recovery delta | +0.51 pp |
|
||||
| Quantization delta (INT8 vs FP16) | -0.03 pp |
|
||||
|
||||
Under the corrected novel-recall scorer (which excludes prompt-echoed content), base models leak near zero (0.3%), confirming that detected leakage is genuine TOFU-specific knowledge. The TOFU-full model leaks 7.85% at this scale. This unlearned checkpoint leaks 6.13% -- a reduction of 1.73 pp. However, chain-of-clues multi-turn prompting still extracts 42.6% of the forgotten facts, and the forgotten-answer log-likelihood actually *increases* by +0.424 relative to TOFU-full, indicating that the internal target signal is not consistently suppressed even though behavioural suppression partially succeeds.
|
||||
|
||||
Full audit details, including per-family breakdowns, are reported in the paper and the [GitHub repository](https://github.com/Naahraf27/memory-laundering).
|
||||
|
||||
## How to load
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "Naahraf27/npo_llama-3.2-3b-instruct_forget10_ep5_lr2e-5_alpha2.0_beta0.1"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")
|
||||
```
|
||||
|
||||
## Citation
|
||||
|
||||
If you use this checkpoint, please cite:
|
||||
|
||||
```bibtex
|
||||
@article{fayaz2026memory,
|
||||
title={Do Unlearned LLMs Really Forget? A Multi-View Audit of TOFU Unlearning Across 1B, 3B, and 8B Llama Models},
|
||||
author={Fayaz, Farhaan and Adnan, Anas and Norsam, Danial and Pitumbur, Vidur and Gokcek, Berken and Solanki, Amir},
|
||||
year={2026},
|
||||
institution={University College London}
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user