[2026-05-21 13:41:49,828][model][INFO] - Setting pad_token as eos token: <|eot_id|>
[2026-05-21 13:41:53,824][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4
[2026-05-21 13:41:55,071][trainer][INFO] - SimNPO Trainer loaded, output_dir: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4
[2026-05-21 13:41:55,642][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 13:41:55,642][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-0/evals/TOFU_EVAL.json
[2026-05-21 13:41:55,642][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-0/evals/TOFU_SUMMARY.json
[2026-05-21 13:41:57,435][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 13:42:14,138][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 13:42:49,130][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 13:42:49,131][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6448834990772232
[2026-05-21 13:42:49,137][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 13:42:49,138][metrics][INFO] - Evaluating forget_quality
[2026-05-21 13:42:49,138][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 13:42:49,138][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 13:42:51,141][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 13:42:55,954][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.10887884757248685
[2026-05-21 13:42:57,618][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 13:46:07,243][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.27890823225307393
[2026-05-21 13:46:08,886][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 13:46:16,399][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 13:47:36,147][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 13:47:43,718][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 13:48:10,569][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 13:48:12,248][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 13:48:15,546][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 13:48:19,756][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 13:48:21,599][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 13:48:58,918][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 13:48:58,918][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:48:58,918][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 13:49:00,580][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 13:49:02,793][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 13:49:06,973][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 13:49:08,385][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 13:50:05,059][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 13:50:05,060][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:50:05,060][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 13:50:05,060][metrics][INFO] - Evaluating model_utility
[2026-05-21 13:50:05,061][evaluator][INFO] - Result for metric model_utility:	0.2185219733034594
[2026-05-21 13:50:08,176][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 13:50:14,964][metrics][INFO] - Evaluating privleak
[2026-05-21 13:50:14,965][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 13:50:14,965][evaluator][INFO] - Result for metric privleak:	-47.96999999040602
[2026-05-21 13:50:16,641][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 13:50:31,558][evaluator][INFO] - Result for metric extraction_strength:	0.05119585323242784
[2026-05-21 13:52:32,141][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 13:52:32,141][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-25/evals/TOFU_EVAL.json
[2026-05-21 13:52:32,141][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-25/evals/TOFU_SUMMARY.json
[2026-05-21 13:52:34,515][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 13:52:42,120][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 13:53:11,351][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 13:53:11,352][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6414198592494345
[2026-05-21 13:53:11,358][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 13:53:11,358][metrics][INFO] - Evaluating forget_quality
[2026-05-21 13:53:11,359][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 13:53:11,359][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 13:53:13,191][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 13:53:19,003][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.20349505057558417
[2026-05-21 13:53:20,753][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 13:54:33,238][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.3363523902144183
[2026-05-21 13:54:35,217][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 13:54:42,714][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 13:55:40,635][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 13:55:45,320][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 13:56:01,893][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 13:56:03,568][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 13:56:05,655][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 13:56:08,019][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 13:56:09,276][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 13:56:57,945][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 13:56:57,946][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:56:57,946][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 13:56:59,881][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 13:57:01,754][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 13:57:03,621][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 13:57:04,874][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 13:57:10,920][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 13:57:10,921][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:57:10,921][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 13:57:10,921][metrics][INFO] - Evaluating model_utility
[2026-05-21 13:57:10,922][evaluator][INFO] - Result for metric model_utility:	0.27790674597563475
[2026-05-21 13:57:14,071][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 13:57:15,644][metrics][INFO] - Evaluating privleak
[2026-05-21 13:57:15,644][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 13:57:15,644][evaluator][INFO] - Result for metric privleak:	-66.64374998667122
[2026-05-21 13:57:16,981][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 13:57:18,384][evaluator][INFO] - Result for metric extraction_strength:	0.062394619827353906
[2026-05-21 13:58:05,566][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 13:58:05,566][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-50/evals/TOFU_EVAL.json
[2026-05-21 13:58:05,566][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-50/evals/TOFU_SUMMARY.json
[2026-05-21 13:58:07,331][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 13:58:15,414][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 13:58:43,721][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 13:58:43,722][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6230931944181684
[2026-05-21 13:58:43,729][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 13:58:43,729][metrics][INFO] - Evaluating forget_quality
[2026-05-21 13:58:43,730][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 13:58:43,730][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 13:58:45,430][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 13:58:51,436][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.3349009375087917
[2026-05-21 13:58:53,116][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 14:00:00,893][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.38400245016174755
[2026-05-21 14:00:02,716][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 14:00:10,290][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 14:00:55,664][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 14:01:03,361][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 14:01:32,607][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 14:01:34,300][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 14:01:36,977][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 14:01:41,003][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 14:01:42,266][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 14:01:51,299][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 14:01:51,299][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:01:51,299][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 14:01:52,974][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 14:01:55,310][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 14:01:58,871][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 14:02:00,270][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 14:02:09,462][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 14:02:09,463][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:02:09,463][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 14:02:09,463][metrics][INFO] - Evaluating model_utility
[2026-05-21 14:02:09,464][evaluator][INFO] - Result for metric model_utility:	0.3793508809768731
[2026-05-21 14:02:12,459][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 14:02:18,002][metrics][INFO] - Evaluating privleak
[2026-05-21 14:02:18,002][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 14:02:18,002][evaluator][INFO] - Result for metric privleak:	-82.57624998348474
[2026-05-21 14:02:19,846][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 14:02:32,706][evaluator][INFO] - Result for metric extraction_strength:	0.08291476394201429
[2026-05-21 14:03:43,858][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 14:03:43,858][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-75/evals/TOFU_EVAL.json
[2026-05-21 14:03:43,858][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-75/evals/TOFU_SUMMARY.json
[2026-05-21 14:03:45,532][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 14:03:50,772][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 14:04:07,357][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 14:04:07,358][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6189158593014839
[2026-05-21 14:04:07,365][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 14:04:07,365][metrics][INFO] - Evaluating forget_quality
[2026-05-21 14:04:07,365][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 14:04:07,365][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 14:04:09,123][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 14:04:12,470][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.34760695200413466
[2026-05-21 14:04:14,478][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 14:04:53,084][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.38394973457576287
[2026-05-21 14:04:54,774][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 14:05:02,239][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 14:05:30,694][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 14:05:36,590][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 14:05:59,909][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 14:06:01,526][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 14:06:03,492][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 14:06:07,836][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 14:06:09,116][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 14:06:18,131][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 14:06:18,131][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:06:18,131][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 14:06:19,811][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 14:06:22,281][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 14:06:25,957][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 14:06:27,284][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 14:06:34,497][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 14:06:34,498][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:06:34,498][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 14:06:34,498][metrics][INFO] - Evaluating model_utility
[2026-05-21 14:06:34,499][evaluator][INFO] - Result for metric model_utility:	0.38425409126783
[2026-05-21 14:06:38,348][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 14:06:43,812][metrics][INFO] - Evaluating privleak
[2026-05-21 14:06:43,813][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 14:06:43,813][evaluator][INFO] - Result for metric privleak:	-83.71749998325649
[2026-05-21 14:06:45,548][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 14:07:08,688][evaluator][INFO] - Result for metric extraction_strength:	0.08463917597529186
[2026-05-21 14:08:21,983][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 14:08:21,984][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-100/evals/TOFU_EVAL.json
[2026-05-21 14:08:21,984][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-100/evals/TOFU_SUMMARY.json
[2026-05-21 14:08:23,767][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 14:08:31,957][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 14:09:01,576][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 14:09:01,577][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6181410546581807
[2026-05-21 14:09:01,583][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 14:09:01,584][metrics][INFO] - Evaluating forget_quality
[2026-05-21 14:09:01,584][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 14:09:01,584][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 14:09:03,337][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 14:09:09,301][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.35111760402098297
[2026-05-21 14:09:11,122][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 14:10:10,724][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.3848115314453001
[2026-05-21 14:10:12,398][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 14:10:19,849][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 14:10:59,083][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 14:11:05,060][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 14:11:33,858][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 14:11:35,531][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 14:11:37,848][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 14:11:40,658][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 14:11:41,930][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 14:11:49,146][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 14:11:49,146][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:11:49,146][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 14:11:50,817][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 14:11:53,075][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 14:11:56,630][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 14:11:57,891][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 14:12:13,895][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 14:12:13,896][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:12:13,896][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 14:12:13,896][metrics][INFO] - Evaluating model_utility
[2026-05-21 14:12:13,897][evaluator][INFO] - Result for metric model_utility:	0.38945632253152707
[2026-05-21 14:12:17,302][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 14:12:18,792][metrics][INFO] - Evaluating privleak
[2026-05-21 14:12:18,792][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 14:12:18,792][evaluator][INFO] - Result for metric privleak:	-83.93999998321199
[2026-05-21 14:12:20,070][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 14:12:21,679][evaluator][INFO] - Result for metric extraction_strength:	0.08717998187117214
[2026-05-21 14:13:07,731][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 14:13:07,731][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-125/evals/TOFU_EVAL.json
[2026-05-21 14:13:07,731][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-125/evals/TOFU_SUMMARY.json
[2026-05-21 14:13:09,455][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 14:13:15,520][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 14:13:44,789][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 14:13:44,790][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6182047623953388
[2026-05-21 14:13:44,796][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 14:13:44,797][metrics][INFO] - Evaluating forget_quality
[2026-05-21 14:13:44,797][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 14:13:44,797][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 14:13:46,438][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 14:13:51,259][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.3523250863514841
[2026-05-21 14:13:53,033][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 14:14:57,204][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.3850742811515734
[2026-05-21 14:14:58,901][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 14:15:06,651][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 14:15:48,169][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 14:15:54,181][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 14:16:22,227][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 14:16:23,925][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 14:16:26,570][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 14:16:30,718][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 14:16:32,038][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 14:16:41,046][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 14:16:41,046][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:16:41,046][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 14:16:42,723][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 14:16:45,377][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 14:16:48,836][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 14:16:50,303][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 14:17:06,455][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 14:17:06,455][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:17:06,456][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 14:17:06,456][metrics][INFO] - Evaluating model_utility
[2026-05-21 14:17:06,456][evaluator][INFO] - Result for metric model_utility:	0.3867871595048101
[2026-05-21 14:17:09,811][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 14:17:15,247][metrics][INFO] - Evaluating privleak
[2026-05-21 14:17:15,248][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 14:17:15,248][evaluator][INFO] - Result for metric privleak:	-84.0524999831895
[2026-05-21 14:17:17,040][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 14:17:32,756][evaluator][INFO] - Result for metric extraction_strength:	0.08658014531492801
[2026-05-21 14:18:42,024][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 14:18:42,024][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-150/evals/TOFU_EVAL.json
[2026-05-21 14:18:42,024][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-150/evals/TOFU_SUMMARY.json
[2026-05-21 14:18:43,899][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 14:18:51,499][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 14:19:17,007][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 14:19:17,008][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6177007290305743
[2026-05-21 14:19:17,014][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 14:19:17,015][metrics][INFO] - Evaluating forget_quality
[2026-05-21 14:19:17,015][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 14:19:17,015][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 14:19:18,823][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 14:19:24,824][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.35286230804398655
[2026-05-21 14:19:26,482][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 14:20:05,951][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.38548027127365925
[2026-05-21 14:20:07,665][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 14:20:14,399][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 14:20:46,571][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 14:20:54,476][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 14:21:21,302][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 14:21:23,506][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 14:21:26,158][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 14:21:30,200][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 14:21:31,450][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 14:21:39,581][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 14:21:39,582][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:21:39,582][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 14:21:41,267][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 14:21:43,599][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 14:21:47,062][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 14:21:48,317][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 14:22:04,276][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 14:22:04,276][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:22:04,276][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 14:22:04,276][metrics][INFO] - Evaluating model_utility
[2026-05-21 14:22:04,277][evaluator][INFO] - Result for metric model_utility:	0.389570021337874
[2026-05-21 14:22:07,214][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 14:22:12,747][metrics][INFO] - Evaluating privleak
[2026-05-21 14:22:12,747][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 14:22:12,747][evaluator][INFO] - Result for metric privleak:	-84.09624998318074
[2026-05-21 14:22:14,458][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 14:22:37,635][evaluator][INFO] - Result for metric extraction_strength:	0.08737657392397029
[2026-05-21 14:23:50,677][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 14:23:50,678][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-175/evals/TOFU_EVAL.json
[2026-05-21 14:23:50,678][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-175/evals/TOFU_SUMMARY.json
[2026-05-21 14:23:52,366][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 14:24:00,068][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 14:24:29,732][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 14:24:29,733][evaluator][INFO] - Result for metric forget_truth_ratio:	0.61761660709215
[2026-05-21 14:24:29,740][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 14:24:29,740][metrics][INFO] - Evaluating forget_quality
[2026-05-21 14:24:29,741][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 14:24:29,741][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 14:24:31,434][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 14:24:37,439][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.3529359891824424
[2026-05-21 14:24:39,138][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 14:25:27,518][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.3858334433873904
[2026-05-21 14:25:29,212][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 14:25:36,119][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 14:26:13,244][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 14:26:19,837][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 14:26:48,531][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 14:26:50,239][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 14:26:52,370][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 14:26:54,727][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 14:26:55,999][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 14:27:02,297][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 14:27:02,297][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:27:02,297][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 14:27:04,329][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 14:27:06,676][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 14:27:09,610][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 14:27:10,887][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 14:27:23,803][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 14:27:23,803][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:27:23,803][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 14:27:23,804][metrics][INFO] - Evaluating model_utility
[2026-05-21 14:27:23,804][evaluator][INFO] - Result for metric model_utility:	0.38596120600351796
[2026-05-21 14:27:26,809][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 14:27:32,599][metrics][INFO] - Evaluating privleak
[2026-05-21 14:27:32,600][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 14:27:32,600][evaluator][INFO] - Result for metric privleak:	-84.109999983178
[2026-05-21 14:27:34,343][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 14:27:39,904][evaluator][INFO] - Result for metric extraction_strength:	0.08715686518263624
[2026-05-21 14:28:56,633][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 14:28:56,633][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-200/evals/TOFU_EVAL.json
[2026-05-21 14:28:56,633][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-200/evals/TOFU_SUMMARY.json
[2026-05-21 14:28:58,297][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 14:29:05,868][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 14:29:35,483][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 14:29:35,484][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6176789569373814
[2026-05-21 14:29:35,490][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 14:29:35,491][metrics][INFO] - Evaluating forget_quality
[2026-05-21 14:29:35,491][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 14:29:35,491][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 14:29:37,291][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 14:29:43,198][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.3530563661269844
[2026-05-21 14:29:44,887][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 14:30:35,000][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.3858238347046325
[2026-05-21 14:30:36,945][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 14:30:44,482][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 14:31:29,528][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 14:31:37,866][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 14:32:04,499][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 14:32:06,162][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 14:32:08,832][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 14:32:13,184][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 14:32:14,438][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 14:32:21,953][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 14:32:21,953][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:32:21,953][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 14:32:23,774][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 14:32:25,562][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 14:32:28,749][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 14:32:30,166][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 14:32:46,413][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 14:32:46,414][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:32:46,414][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 14:32:46,414][metrics][INFO] - Evaluating model_utility
[2026-05-21 14:32:46,415][evaluator][INFO] - Result for metric model_utility:	0.38865403947163873
[2026-05-21 14:32:49,531][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 14:32:55,132][metrics][INFO] - Evaluating privleak
[2026-05-21 14:32:55,132][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 14:32:55,132][evaluator][INFO] - Result for metric privleak:	-84.05874998318824
[2026-05-21 14:32:56,939][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 14:32:59,389][evaluator][INFO] - Result for metric extraction_strength:	0.0845172504995537
[2026-05-21 14:33:52,576][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 14:33:52,576][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-225/evals/TOFU_EVAL.json
[2026-05-21 14:33:52,576][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-225/evals/TOFU_SUMMARY.json
[2026-05-21 14:33:54,211][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 14:33:59,807][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 14:34:23,369][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 14:34:23,370][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6176135041820996
[2026-05-21 14:34:23,376][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 14:34:23,377][metrics][INFO] - Evaluating forget_quality
[2026-05-21 14:34:23,377][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 14:34:23,377][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 14:34:25,048][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 14:34:31,293][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.3531488827429712
[2026-05-21 14:34:38,330][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 14:35:32,093][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.38595172349337675
[2026-05-21 14:35:33,813][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 14:35:41,578][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 14:36:23,376][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 14:36:31,267][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 14:36:58,530][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 14:37:00,381][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 14:37:03,478][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 14:37:07,621][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 14:37:09,032][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 14:37:17,101][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 14:37:17,101][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:37:17,101][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 14:37:19,088][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 14:37:22,208][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 14:37:25,753][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 14:37:27,011][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 14:37:43,790][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 14:37:43,791][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:37:43,791][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 14:37:43,791][metrics][INFO] - Evaluating model_utility
[2026-05-21 14:37:43,792][evaluator][INFO] - Result for metric model_utility:	0.38656524448187823
[2026-05-21 14:37:46,811][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 14:37:52,207][metrics][INFO] - Evaluating privleak
[2026-05-21 14:37:52,207][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 14:37:52,207][evaluator][INFO] - Result for metric privleak:	-84.14999998316999
[2026-05-21 14:37:54,588][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 14:38:16,613][evaluator][INFO] - Result for metric extraction_strength:	0.08690805565882671
[2026-05-21 14:39:24,024][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 14:39:24,024][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-250/evals/TOFU_EVAL.json
[2026-05-21 14:39:24,024][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4/checkpoint-250/evals/TOFU_SUMMARY.json
[2026-05-21 14:39:25,713][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 14:39:33,358][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 14:39:58,593][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 14:39:58,594][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6175223364978973
[2026-05-21 14:39:58,600][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 14:39:58,601][metrics][INFO] - Evaluating forget_quality
[2026-05-21 14:39:58,601][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 14:39:58,601][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 14:40:00,392][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 14:40:06,399][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.3530087990872562
[2026-05-21 14:40:08,232][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 14:40:45,290][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.3857650261516718
[2026-05-21 14:40:47,150][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 14:40:52,766][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 14:41:25,175][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 14:41:33,115][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 14:42:00,329][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 14:42:01,993][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 14:42:04,966][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 14:42:09,061][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 14:42:10,361][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 14:42:19,191][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 14:42:19,192][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:42:19,192][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 14:42:20,838][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 14:42:23,848][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 14:42:27,388][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 14:42:28,653][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 14:42:44,760][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 14:42:44,761][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 14:42:44,761][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 14:42:44,761][metrics][INFO] - Evaluating model_utility
[2026-05-21 14:42:44,762][evaluator][INFO] - Result for metric model_utility:	0.3884310780989943
[2026-05-21 14:42:48,251][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 14:42:53,673][metrics][INFO] - Evaluating privleak
[2026-05-21 14:42:53,674][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 14:42:53,674][evaluator][INFO] - Result for metric privleak:	-84.10499998317901
[2026-05-21 14:42:55,653][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 14:43:17,396][evaluator][INFO] - Result for metric extraction_strength:	0.08670724689551795