[2025-05-02 21:35:12,013][model][INFO] - Setting pad_token as eos token: <|eot_id|>
[2025-05-02 21:35:12,016][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_DPO/evals
[2025-05-02 21:35:12,017][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2025-05-02 21:35:12,017][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_DPO/evals/TOFU_EVAL.json
[2025-05-02 21:35:12,017][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_DPO/evals/TOFU_SUMMARY.json
[2025-05-02 21:35:15,625][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
[2025-05-02 21:35:15,636][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2025-05-02 21:35:24,926][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
[2025-05-02 21:35:24,943][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2025-05-02 21:35:56,842][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
[2025-05-02 21:35:56,853][metrics][INFO] - Evaluating forget_truth_ratio
[2025-05-02 21:35:56,855][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
[2025-05-02 21:35:56,863][metrics][INFO] - Evaluating forget_quality
[2025-05-02 21:35:56,864][evaluator][INFO] - Result for metric forget_quality:	5.805666105234786e-14
[2025-05-02 21:35:59,226][metrics][INFO] - Evaluating forget_Q_A_Prob
[2025-05-02 21:36:05,740][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.5762331750616432
[2025-05-02 21:36:07,530][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2025-05-02 21:36:20,707][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.04170014203058363
[2025-05-02 21:36:22,500][metrics][INFO] - Evaluating retain_Q_A_Prob
[2025-05-02 21:36:29,800][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2025-05-02 21:36:49,706][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2025-05-02 21:36:57,683][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2025-05-02 21:37:27,050][metrics][INFO] - Evaluating retain_Truth_Ratio
[2025-05-02 21:37:29,127][metrics][INFO] - Evaluating ra_Q_A_Prob
[2025-05-02 21:37:31,606][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2025-05-02 21:37:34,641][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2025-05-02 21:37:36,149][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2025-05-02 21:37:38,730][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2025-05-02 21:37:38,730][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2025-05-02 21:37:38,731][metrics][INFO] - Evaluating ra_Truth_Ratio
[2025-05-02 21:37:41,024][metrics][INFO] - Evaluating wf_Q_A_Prob
[2025-05-02 21:37:43,272][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2025-05-02 21:37:46,377][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2025-05-02 21:37:48,087][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2025-05-02 21:37:53,552][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2025-05-02 21:37:53,552][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2025-05-02 21:37:53,552][metrics][INFO] - Evaluating wf_Truth_Ratio
[2025-05-02 21:37:53,553][metrics][INFO] - Evaluating model_utility
[2025-05-02 21:37:53,553][evaluator][INFO] - Result for metric model_utility:	0.3277496519583408
[2025-05-02 21:37:57,014][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
[2025-05-02 21:37:57,032][metrics][INFO] - Evaluating mia_min_k
[2025-05-02 21:38:05,882][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
[2025-05-02 21:38:05,891][metrics][INFO] - Evaluating privleak
[2025-05-02 21:38:05,891][evaluator][INFO] - Result for metric privleak:	-96.0889072091442
[2025-05-02 21:38:07,692][metrics][INFO] - Evaluating extraction_strength
[2025-05-02 21:38:12,357][evaluator][INFO] - Result for metric extraction_strength:	0.261045404211332
[2025-05-09 20:28:33,178][model][INFO] - Setting pad_token as eos token: <|eot_id|>
[2025-05-09 20:28:33,182][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_DPO/evals
[2025-05-09 20:28:33,183][evaluator][INFO] - Loading existing evaluations from saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_DPO/evals/TOFU_EVAL.json
[2025-05-09 20:28:33,196][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2025-05-09 20:28:33,197][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_DPO/evals/TOFU_EVAL.json
[2025-05-09 20:28:33,197][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_DPO/evals/TOFU_SUMMARY.json
[2025-05-09 20:28:33,197][evaluator][INFO] - Skipping forget_quality, already evaluated.
[2025-05-09 20:28:33,197][evaluator][INFO] - Result for metric forget_quality:	5.805666105234786e-14
[2025-05-09 20:28:33,199][evaluator][INFO] - Skipping forget_Q_A_Prob, already evaluated.
[2025-05-09 20:28:33,199][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.5762331750616432
[2025-05-09 20:28:33,218][evaluator][INFO] - Skipping forget_Q_A_ROUGE, already evaluated.
[2025-05-09 20:28:33,218][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.04170014203058363
[2025-05-09 20:28:33,241][evaluator][INFO] - Skipping model_utility, already evaluated.
[2025-05-09 20:28:33,241][evaluator][INFO] - Result for metric model_utility:	0.3277496519583408
[2025-05-09 20:28:33,247][evaluator][INFO] - Skipping privleak, already evaluated.
[2025-05-09 20:28:33,247][evaluator][INFO] - Result for metric privleak:	-96.0889072091442
[2025-05-09 20:28:33,256][evaluator][INFO] - Skipping extraction_strength, already evaluated.
[2025-05-09 20:28:33,256][evaluator][INFO] - Result for metric extraction_strength:	0.261045404211332
[2025-05-13 09:25:54,344][model][INFO] - Setting pad_token as eos token: <|eot_id|>
[2025-05-13 09:25:54,347][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_DPO/evals
[2025-05-13 09:25:54,348][evaluator][INFO] - Loading existing evaluations from saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_DPO/evals/TOFU_EVAL.json
[2025-05-13 09:25:54,361][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2025-05-13 09:25:54,361][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_DPO/evals/TOFU_EVAL.json
[2025-05-13 09:25:54,361][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_DPO/evals/TOFU_SUMMARY.json
[2025-05-13 09:25:54,361][evaluator][INFO] - Skipping forget_quality, already evaluated.
[2025-05-13 09:25:54,361][evaluator][INFO] - Result for metric forget_quality:	5.805666105234786e-14
[2025-05-13 09:25:54,370][evaluator][INFO] - Skipping forget_Q_A_Prob, already evaluated.
[2025-05-13 09:25:54,370][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.5762331750616432
[2025-05-13 09:25:54,372][evaluator][INFO] - Skipping forget_Q_A_ROUGE, already evaluated.
[2025-05-13 09:25:54,372][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.04170014203058363
[2025-05-13 09:25:54,373][evaluator][INFO] - Skipping model_utility, already evaluated.
[2025-05-13 09:25:54,374][evaluator][INFO] - Result for metric model_utility:	0.3277496519583408
[2025-05-13 09:25:54,375][evaluator][INFO] - Skipping privleak, already evaluated.
[2025-05-13 09:25:54,375][evaluator][INFO] - Result for metric privleak:	-96.0889072091442
[2025-05-13 09:25:54,376][evaluator][INFO] - Skipping extraction_strength, already evaluated.
[2025-05-13 09:25:54,376][evaluator][INFO] - Result for metric extraction_strength:	0.261045404211332