tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/RMU.log

[2026-05-21 11:57:33,698][model][INFO] - Setting pad_token as eos token: <|eot_id|>
[2026-05-21 11:57:37,813][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4
[2026-05-21 11:57:39,435][trainer][INFO] - RMU Trainer loaded, output_dir: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4
[2026-05-21 11:57:40,023][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 11:57:40,023][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-0/evals/TOFU_EVAL.json
[2026-05-21 11:57:40,023][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-0/evals/TOFU_SUMMARY.json
[2026-05-21 11:57:41,506][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 11:57:58,264][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 11:58:38,873][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 11:58:38,874][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6448834990772232
[2026-05-21 11:58:38,880][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 11:58:38,881][metrics][INFO] - Evaluating forget_quality
[2026-05-21 11:58:38,881][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 11:58:38,881][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 11:58:40,563][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 11:58:46,744][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.10887884757248685
[2026-05-21 11:58:48,505][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 12:02:17,635][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.27890823225307393
[2026-05-21 12:02:19,330][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 12:02:27,753][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 12:04:33,587][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 12:04:41,251][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 12:05:12,905][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 12:05:14,652][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 12:05:18,181][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 12:05:21,757][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 12:05:23,000][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 12:06:07,138][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 12:06:07,139][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:06:07,139][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 12:06:08,822][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 12:06:11,152][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 12:06:15,810][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 12:06:17,088][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 12:07:14,539][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 12:07:14,539][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:07:14,539][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 12:07:14,540][metrics][INFO] - Evaluating model_utility
[2026-05-21 12:07:14,544][evaluator][INFO] - Result for metric model_utility:	0.2185219733034594
[2026-05-21 12:07:17,574][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 12:07:21,214][metrics][INFO] - Evaluating privleak
[2026-05-21 12:07:21,214][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 12:07:21,214][evaluator][INFO] - Result for metric privleak:	-47.96999999040602
[2026-05-21 12:07:22,668][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 12:07:23,965][evaluator][INFO] - Result for metric extraction_strength:	0.05119585323242784
[2026-05-21 12:08:53,751][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 12:08:53,751][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-25/evals/TOFU_EVAL.json
[2026-05-21 12:08:53,751][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-25/evals/TOFU_SUMMARY.json
[2026-05-21 12:08:55,728][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 12:09:02,060][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 12:09:26,356][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 12:09:26,357][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6614321776157142
[2026-05-21 12:09:26,364][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 12:09:26,364][metrics][INFO] - Evaluating forget_quality
[2026-05-21 12:09:26,364][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 12:09:26,364][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 12:09:28,002][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 12:09:33,162][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.044040033797500655
[2026-05-21 12:09:34,852][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 12:11:29,404][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.19825707891118127
[2026-05-21 12:11:31,096][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 12:11:39,019][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 12:13:45,155][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 12:13:52,720][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 12:14:21,986][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 12:14:23,693][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 12:14:26,349][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 12:14:30,389][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 12:14:31,749][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 12:15:44,420][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 12:15:44,421][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:15:44,421][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 12:15:46,154][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 12:15:48,537][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 12:15:51,980][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 12:15:53,337][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 12:16:22,460][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 12:16:22,460][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:16:22,460][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 12:16:22,461][metrics][INFO] - Evaluating model_utility
[2026-05-21 12:16:22,461][evaluator][INFO] - Result for metric model_utility:	0.08605862105946929
[2026-05-21 12:16:25,761][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 12:16:31,351][metrics][INFO] - Evaluating privleak
[2026-05-21 12:16:31,351][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 12:16:31,351][evaluator][INFO] - Result for metric privleak:	-22.12749999557451
[2026-05-21 12:16:33,075][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 12:16:42,339][evaluator][INFO] - Result for metric extraction_strength:	0.04014118794649353
[2026-05-21 12:17:06,543][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 12:17:06,543][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-50/evals/TOFU_EVAL.json
[2026-05-21 12:17:06,544][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-50/evals/TOFU_SUMMARY.json
[2026-05-21 12:17:08,209][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 12:17:16,386][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 12:17:44,453][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 12:17:44,454][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6646114327873448
[2026-05-21 12:17:44,460][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 12:17:44,461][metrics][INFO] - Evaluating forget_quality
[2026-05-21 12:17:44,461][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 12:17:44,461][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 12:17:46,123][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 12:17:51,886][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.028171376134705498
[2026-05-21 12:17:53,595][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 12:19:56,169][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.18136216120754584
[2026-05-21 12:19:58,075][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 12:20:05,461][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 12:22:11,777][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 12:22:19,434][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 12:22:48,615][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 12:22:50,385][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 12:22:53,191][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 12:22:57,341][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 12:22:58,995][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 12:23:38,005][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 12:23:38,005][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:23:38,005][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 12:23:39,646][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 12:23:42,040][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 12:23:45,003][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 12:23:46,257][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 12:24:23,368][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 12:24:23,369][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:24:23,369][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 12:24:23,369][metrics][INFO] - Evaluating model_utility
[2026-05-21 12:24:23,370][evaluator][INFO] - Result for metric model_utility:	0.07766678967490327
[2026-05-21 12:24:26,397][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 12:24:31,922][metrics][INFO] - Evaluating privleak
[2026-05-21 12:24:31,923][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 12:24:31,923][evaluator][INFO] - Result for metric privleak:	-16.837499996632495
[2026-05-21 12:24:33,630][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 12:24:42,580][evaluator][INFO] - Result for metric extraction_strength:	0.03555271291935598
[2026-05-21 12:25:09,643][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 12:25:09,643][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-75/evals/TOFU_EVAL.json
[2026-05-21 12:25:09,643][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-75/evals/TOFU_SUMMARY.json
[2026-05-21 12:25:11,296][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 12:25:19,068][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 12:25:36,970][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 12:25:36,971][evaluator][INFO] - Result for metric forget_truth_ratio:	0.666956633608444
[2026-05-21 12:25:36,977][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 12:25:36,978][metrics][INFO] - Evaluating forget_quality
[2026-05-21 12:25:36,978][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 12:25:36,978][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 12:25:38,613][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 12:25:44,827][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.025599835189350415
[2026-05-21 12:25:46,467][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 12:27:48,567][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.17665323136017774
[2026-05-21 12:27:50,319][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 12:27:58,030][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 12:30:05,340][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 12:30:12,963][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 12:30:39,143][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 12:30:40,835][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 12:30:43,488][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 12:30:47,643][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 12:30:48,950][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 12:31:28,106][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 12:31:28,107][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:31:28,107][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 12:31:29,859][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 12:31:32,220][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 12:31:35,651][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 12:31:36,902][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 12:32:16,255][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 12:32:16,256][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:32:16,256][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 12:32:16,256][metrics][INFO] - Evaluating model_utility
[2026-05-21 12:32:16,257][evaluator][INFO] - Result for metric model_utility:	0.025169849004668194
[2026-05-21 12:32:19,201][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 12:32:24,841][metrics][INFO] - Evaluating privleak
[2026-05-21 12:32:24,841][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 12:32:24,841][evaluator][INFO] - Result for metric privleak:	-15.60999999687799
[2026-05-21 12:32:26,598][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 12:32:30,837][evaluator][INFO] - Result for metric extraction_strength:	0.03542379317218397
[2026-05-21 12:32:55,301][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 12:32:55,301][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-100/evals/TOFU_EVAL.json
[2026-05-21 12:32:55,301][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-100/evals/TOFU_SUMMARY.json
[2026-05-21 12:32:56,927][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 12:33:02,940][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 12:33:32,151][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 12:33:32,152][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6650891487506957
[2026-05-21 12:33:32,158][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 12:33:32,159][metrics][INFO] - Evaluating forget_quality
[2026-05-21 12:33:32,159][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 12:33:32,159][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 12:33:33,843][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 12:33:38,385][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.024368888099124887
[2026-05-21 12:33:39,662][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 12:35:08,076][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.17340090189465512
[2026-05-21 12:35:09,710][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 12:35:14,668][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 12:37:19,876][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 12:37:27,741][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 12:37:55,967][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 12:37:57,746][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 12:38:00,481][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 12:38:04,623][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 12:38:06,064][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 12:38:40,084][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 12:38:40,084][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:38:40,084][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 12:38:41,711][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 12:38:44,062][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 12:38:47,631][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 12:38:48,878][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 12:39:28,314][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 12:39:28,314][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:39:28,314][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 12:39:28,315][metrics][INFO] - Evaluating model_utility
[2026-05-21 12:39:28,315][evaluator][INFO] - Result for metric model_utility:	0.07572111776600261
[2026-05-21 12:39:31,238][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 12:39:36,722][metrics][INFO] - Evaluating privleak
[2026-05-21 12:39:36,722][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 12:39:36,722][evaluator][INFO] - Result for metric privleak:	-15.054999996988997
[2026-05-21 12:39:38,473][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 12:39:45,286][evaluator][INFO] - Result for metric extraction_strength:	0.034881082155764
[2026-05-21 12:40:09,613][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 12:40:09,613][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-125/evals/TOFU_EVAL.json
[2026-05-21 12:40:09,613][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-125/evals/TOFU_SUMMARY.json
[2026-05-21 12:40:11,550][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 12:40:19,424][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 12:40:49,178][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 12:40:49,179][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6653264816436281
[2026-05-21 12:40:49,185][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 12:40:49,186][metrics][INFO] - Evaluating forget_quality
[2026-05-21 12:40:49,186][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 12:40:49,186][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 12:40:50,857][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 12:40:56,868][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.023090821445148322
[2026-05-21 12:40:58,563][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 12:42:59,770][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.17290647721627245
[2026-05-21 12:43:01,413][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 12:43:08,531][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 12:44:48,920][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 12:44:56,881][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 12:45:26,437][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 12:45:28,234][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 12:45:31,003][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 12:45:35,107][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 12:45:36,368][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 12:46:14,962][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 12:46:14,962][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:46:14,962][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 12:46:16,815][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 12:46:19,204][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 12:46:21,563][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 12:46:22,945][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 12:46:59,910][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 12:46:59,911][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:46:59,911][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 12:46:59,911][metrics][INFO] - Evaluating model_utility
[2026-05-21 12:46:59,912][evaluator][INFO] - Result for metric model_utility:	0.061282254233379956
[2026-05-21 12:47:02,933][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 12:47:08,271][metrics][INFO] - Evaluating privleak
[2026-05-21 12:47:08,271][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 12:47:08,271][evaluator][INFO] - Result for metric privleak:	-14.518749997096252
[2026-05-21 12:47:10,069][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 12:47:32,061][evaluator][INFO] - Result for metric extraction_strength:	0.03492969775585019
[2026-05-21 12:47:52,836][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 12:47:52,836][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-150/evals/TOFU_EVAL.json
[2026-05-21 12:47:52,836][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-150/evals/TOFU_SUMMARY.json
[2026-05-21 12:47:54,493][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 12:48:02,146][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 12:48:31,769][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 12:48:31,770][evaluator][INFO] - Result for metric forget_truth_ratio:	0.665408632731608
[2026-05-21 12:48:31,776][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 12:48:31,777][metrics][INFO] - Evaluating forget_quality
[2026-05-21 12:48:31,777][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 12:48:31,777][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 12:48:33,422][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 12:48:37,862][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.022637412079275235
[2026-05-21 12:48:39,116][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 12:50:46,079][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.16969368902404242
[2026-05-21 12:50:48,146][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 12:50:54,741][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 12:52:56,349][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 12:53:02,778][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 12:53:20,514][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 12:53:22,173][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 12:53:25,211][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 12:53:29,602][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 12:53:31,058][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 12:54:09,744][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 12:54:09,745][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:54:09,745][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 12:54:11,393][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 12:54:13,787][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 12:54:17,260][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 12:54:18,762][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 12:54:58,987][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 12:54:58,988][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 12:54:58,988][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 12:54:58,988][metrics][INFO] - Evaluating model_utility
[2026-05-21 12:54:58,989][evaluator][INFO] - Result for metric model_utility:	0.07293884758887383
[2026-05-21 12:55:01,964][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 12:55:04,458][metrics][INFO] - Evaluating privleak
[2026-05-21 12:55:04,458][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 12:55:04,458][evaluator][INFO] - Result for metric privleak:	-14.298749997140254
[2026-05-21 12:55:06,059][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 12:55:09,527][evaluator][INFO] - Result for metric extraction_strength:	0.034710650136802565
[2026-05-21 12:55:31,076][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 12:55:31,076][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-175/evals/TOFU_EVAL.json
[2026-05-21 12:55:31,076][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-175/evals/TOFU_SUMMARY.json
[2026-05-21 12:55:32,704][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 12:55:38,755][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 12:56:06,854][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 12:56:06,855][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6661571899518759
[2026-05-21 12:56:06,861][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 12:56:06,862][metrics][INFO] - Evaluating forget_quality
[2026-05-21 12:56:06,862][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 12:56:06,862][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 12:56:08,508][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 12:56:14,367][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.022689154595791478
[2026-05-21 12:56:16,040][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 12:58:20,768][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.17071292253126477
[2026-05-21 12:58:22,468][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 12:58:29,886][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 13:00:37,081][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 13:00:43,094][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 13:01:10,854][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 13:01:12,568][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 13:01:15,228][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 13:01:19,302][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 13:01:20,611][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 13:01:54,745][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 13:01:54,745][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:01:54,745][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 13:01:56,526][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 13:01:58,924][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 13:02:02,375][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 13:02:03,624][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 13:02:27,784][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 13:02:27,784][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:02:27,784][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 13:02:27,784][metrics][INFO] - Evaluating model_utility
[2026-05-21 13:02:27,785][evaluator][INFO] - Result for metric model_utility:	0.024938334488339972
[2026-05-21 13:02:30,834][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 13:02:35,356][metrics][INFO] - Evaluating privleak
[2026-05-21 13:02:35,356][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 13:02:35,356][evaluator][INFO] - Result for metric privleak:	-14.50124999709975
[2026-05-21 13:02:36,768][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 13:02:52,494][evaluator][INFO] - Result for metric extraction_strength:	0.03454297434412678
[2026-05-21 13:03:16,128][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 13:03:16,129][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-200/evals/TOFU_EVAL.json
[2026-05-21 13:03:16,129][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-200/evals/TOFU_SUMMARY.json
[2026-05-21 13:03:17,801][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 13:03:25,631][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 13:03:56,057][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 13:03:56,058][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6657672475603854
[2026-05-21 13:03:56,064][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 13:03:56,065][metrics][INFO] - Evaluating forget_quality
[2026-05-21 13:03:56,065][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 13:03:56,065][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 13:03:57,750][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 13:04:02,208][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.022478839480972967
[2026-05-21 13:04:03,476][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 13:06:08,123][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.17469059451335844
[2026-05-21 13:06:09,777][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 13:06:17,186][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 13:08:23,471][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 13:08:31,153][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 13:09:00,265][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 13:09:01,944][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 13:09:04,573][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 13:09:07,999][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 13:09:09,674][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 13:09:45,248][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 13:09:45,249][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:09:45,249][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 13:09:46,946][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 13:09:49,056][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 13:09:52,602][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 13:09:54,048][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 13:10:30,231][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 13:10:30,231][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:10:30,231][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 13:10:30,232][metrics][INFO] - Evaluating model_utility
[2026-05-21 13:10:30,232][evaluator][INFO] - Result for metric model_utility:	0.02489956345780598
[2026-05-21 13:10:33,282][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 13:10:36,848][metrics][INFO] - Evaluating privleak
[2026-05-21 13:10:36,849][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 13:10:36,849][evaluator][INFO] - Result for metric privleak:	-13.848749997230259
[2026-05-21 13:10:38,234][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 13:10:56,184][evaluator][INFO] - Result for metric extraction_strength:	0.03481065013680257
[2026-05-21 13:11:08,088][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 13:11:08,088][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-225/evals/TOFU_EVAL.json
[2026-05-21 13:11:08,088][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-225/evals/TOFU_SUMMARY.json
[2026-05-21 13:11:09,829][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 13:11:17,751][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 13:11:46,542][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 13:11:46,543][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6661828793220076
[2026-05-21 13:11:46,550][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 13:11:46,550][metrics][INFO] - Evaluating forget_quality
[2026-05-21 13:11:46,550][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 13:11:46,550][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 13:11:48,331][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 13:11:54,570][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.02249195015472651
[2026-05-21 13:11:56,226][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 13:13:55,029][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.16997095669432055
[2026-05-21 13:13:56,746][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 13:14:04,157][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 13:16:10,552][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 13:16:18,183][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 13:16:47,385][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 13:16:49,063][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 13:16:51,612][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 13:16:55,772][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 13:16:57,083][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 13:17:36,254][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 13:17:36,254][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:17:36,254][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 13:17:37,889][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 13:17:40,234][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 13:17:43,663][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 13:17:45,073][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 13:18:21,334][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 13:18:21,334][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:18:21,334][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 13:18:21,335][metrics][INFO] - Evaluating model_utility
[2026-05-21 13:18:21,335][evaluator][INFO] - Result for metric model_utility:	0.034445471917355024
[2026-05-21 13:18:24,581][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 13:18:30,118][metrics][INFO] - Evaluating privleak
[2026-05-21 13:18:30,118][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 13:18:30,118][evaluator][INFO] - Result for metric privleak:	-13.99624999720075
[2026-05-21 13:18:31,791][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 13:18:33,441][evaluator][INFO] - Result for metric extraction_strength:	0.03500545533160776
[2026-05-21 13:18:48,997][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-21 13:18:48,998][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-250/evals/TOFU_EVAL.json
[2026-05-21 13:18:48,998][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_RMU_qat-int4/checkpoint-250/evals/TOFU_SUMMARY.json
[2026-05-21 13:18:50,692][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-21 13:18:58,986][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-21 13:19:25,257][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-21 13:19:25,258][evaluator][INFO] - Result for metric forget_truth_ratio:	0.6682139209275343
[2026-05-21 13:19:25,264][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-21 13:19:25,265][metrics][INFO] - Evaluating forget_quality
[2026-05-21 13:19:25,265][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-21 13:19:25,265][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-21 13:19:26,973][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-21 13:19:32,982][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.022312334740927326
[2026-05-21 13:19:34,673][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-21 13:21:16,645][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.16821955009936282
[2026-05-21 13:21:18,356][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-21 13:21:25,924][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-21 13:23:30,694][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-21 13:23:38,159][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-21 13:24:07,357][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-21 13:24:09,157][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-21 13:24:11,661][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-21 13:24:15,807][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-21 13:24:17,063][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-21 13:24:56,249][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-21 13:24:56,250][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:24:56,250][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-21 13:24:57,929][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-21 13:25:00,268][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-21 13:25:03,792][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-21 13:25:05,243][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-21 13:25:41,021][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-21 13:25:41,021][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-21 13:25:41,021][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-21 13:25:41,021][metrics][INFO] - Evaluating model_utility
[2026-05-21 13:25:41,022][evaluator][INFO] - Result for metric model_utility:	0.024945798930326533
[2026-05-21 13:25:44,127][metrics][INFO] - Evaluating mia_min_k
[2026-05-21 13:25:49,445][metrics][INFO] - Evaluating privleak
[2026-05-21 13:25:49,445][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-21 13:25:49,445][evaluator][INFO] - Result for metric privleak:	-14.171249997165742
[2026-05-21 13:25:52,550][metrics][INFO] - Evaluating extraction_strength
[2026-05-21 13:26:14,618][evaluator][INFO] - Result for metric extraction_strength:	0.034667360093512525