467 lines
41 KiB
Plaintext
467 lines
41 KiB
Plaintext
[2026-05-18 13:54:31,762][model][WARNING] - Model open-unlearning/tofu_Llama-3.2-1B-Instruct_full requested with {'attn_implementation': 'flash_attention_2'}
|
|
[2026-05-18 13:54:57,079][model][INFO] - Setting pad_token as eos token: <|eot_id|>
|
|
[2026-05-18 13:55:00,796][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off
|
|
[2026-05-18 13:55:01,818][trainer][INFO] - NPO Trainer loaded, output_dir: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off
|
|
[2026-05-18 13:55:02,370][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-05-18 13:55:02,370][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-0/evals/TOFU_EVAL.json
|
|
[2026-05-18 13:55:02,370][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-0/evals/TOFU_SUMMARY.json
|
|
[2026-05-18 13:55:03,973][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-05-18 13:55:20,129][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-05-18 13:55:53,342][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-05-18 13:55:53,343][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4751472517287268
|
|
[2026-05-18 13:55:53,349][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
|
|
[2026-05-18 13:55:53,350][metrics][INFO] - Evaluating forget_quality
|
|
[2026-05-18 13:55:53,350][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
|
|
[2026-05-18 13:55:53,350][evaluator][INFO] - Result for metric forget_quality: None
|
|
[2026-05-18 13:55:55,130][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2026-05-18 13:55:59,596][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.8804745058715343
|
|
[2026-05-18 13:56:00,882][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-05-18 13:56:47,342][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.8224148587081459
|
|
[2026-05-18 13:56:48,995][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-05-18 13:56:55,825][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-05-18 13:57:06,460][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-05-18 13:57:12,461][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-05-18 13:57:32,576][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-05-18 13:57:34,236][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-05-18 13:57:36,995][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-05-18 13:57:39,580][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-05-18 13:57:40,827][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-05-18 13:57:47,676][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-05-18 13:57:47,676][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 13:57:47,677][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-05-18 13:57:49,344][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-05-18 13:57:51,569][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-05-18 13:57:55,229][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-05-18 13:57:56,601][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-05-18 13:58:07,839][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-05-18 13:58:07,839][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 13:58:07,839][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-05-18 13:58:07,840][metrics][INFO] - Evaluating model_utility
|
|
[2026-05-18 13:58:07,840][evaluator][INFO] - Result for metric model_utility: 0.5986573345007462
|
|
[2026-05-18 13:58:10,704][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-05-18 13:58:13,403][metrics][INFO] - Evaluating privleak
|
|
[2026-05-18 13:58:13,404][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
|
|
[2026-05-18 13:58:13,404][evaluator][INFO] - Result for metric privleak: -99.33374998013325
|
|
[2026-05-18 13:58:14,694][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-05-18 13:58:15,308][evaluator][INFO] - Result for metric extraction_strength: 0.7035078413166893
|
|
[2026-05-18 14:00:28,716][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-05-18 14:00:28,716][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-25/evals/TOFU_EVAL.json
|
|
[2026-05-18 14:00:28,716][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-25/evals/TOFU_SUMMARY.json
|
|
[2026-05-18 14:00:30,797][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-05-18 14:00:34,916][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-05-18 14:00:48,840][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-05-18 14:00:48,841][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5215462041054256
|
|
[2026-05-18 14:00:48,847][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
|
|
[2026-05-18 14:00:48,847][metrics][INFO] - Evaluating forget_quality
|
|
[2026-05-18 14:00:48,848][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
|
|
[2026-05-18 14:00:48,848][evaluator][INFO] - Result for metric forget_quality: None
|
|
[2026-05-18 14:00:50,789][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2026-05-18 14:00:53,593][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.7682410891354085
|
|
[2026-05-18 14:00:54,905][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-05-18 14:01:00,411][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.6679393669272393
|
|
[2026-05-18 14:01:02,103][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-05-18 14:01:06,134][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-05-18 14:01:11,627][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-05-18 14:01:16,346][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-05-18 14:01:30,082][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-05-18 14:01:31,788][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-05-18 14:01:33,753][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-05-18 14:01:35,615][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-05-18 14:01:36,873][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-05-18 14:01:37,587][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:01:37,587][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:01:37,587][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-05-18 14:01:38,856][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-05-18 14:01:40,600][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-05-18 14:01:42,178][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-05-18 14:01:43,527][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-05-18 14:01:44,439][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:01:44,439][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:01:44,439][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-05-18 14:01:44,440][metrics][INFO] - Evaluating model_utility
|
|
[2026-05-18 14:01:44,440][evaluator][INFO] - Result for metric model_utility: 0.5802270741620733
|
|
[2026-05-18 14:01:47,079][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-05-18 14:01:48,071][metrics][INFO] - Evaluating privleak
|
|
[2026-05-18 14:01:48,071][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
|
|
[2026-05-18 14:01:48,071][evaluator][INFO] - Result for metric privleak: -98.86124998022777
|
|
[2026-05-18 14:01:49,359][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-05-18 14:01:50,046][evaluator][INFO] - Result for metric extraction_strength: 0.4727573242328045
|
|
[2026-05-18 14:02:43,850][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-05-18 14:02:43,850][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-50/evals/TOFU_EVAL.json
|
|
[2026-05-18 14:02:43,850][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-50/evals/TOFU_SUMMARY.json
|
|
[2026-05-18 14:02:45,429][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-05-18 14:02:49,505][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-05-18 14:03:03,449][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-05-18 14:03:03,450][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6758352799741583
|
|
[2026-05-18 14:03:03,456][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
|
|
[2026-05-18 14:03:03,456][metrics][INFO] - Evaluating forget_quality
|
|
[2026-05-18 14:03:03,456][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
|
|
[2026-05-18 14:03:03,456][evaluator][INFO] - Result for metric forget_quality: None
|
|
[2026-05-18 14:03:05,160][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2026-05-18 14:03:07,961][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.23828835071995855
|
|
[2026-05-18 14:03:09,224][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-05-18 14:03:35,722][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.266978506986982
|
|
[2026-05-18 14:03:37,475][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-05-18 14:03:41,707][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-05-18 14:03:45,668][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-05-18 14:03:49,814][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-05-18 14:04:03,639][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-05-18 14:04:05,383][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-05-18 14:04:07,352][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-05-18 14:04:09,228][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-05-18 14:04:10,572][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-05-18 14:04:11,267][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:04:11,267][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:04:11,267][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-05-18 14:04:13,555][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-05-18 14:04:15,327][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-05-18 14:04:16,908][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-05-18 14:04:18,170][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-05-18 14:04:18,873][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:04:18,874][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:04:18,874][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-05-18 14:04:18,874][metrics][INFO] - Evaluating model_utility
|
|
[2026-05-18 14:04:18,875][evaluator][INFO] - Result for metric model_utility: 0.365025395428697
|
|
[2026-05-18 14:04:21,447][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-05-18 14:04:22,456][metrics][INFO] - Evaluating privleak
|
|
[2026-05-18 14:04:22,457][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
|
|
[2026-05-18 14:04:22,457][evaluator][INFO] - Result for metric privleak: -75.04874998499025
|
|
[2026-05-18 14:04:23,844][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-05-18 14:04:24,658][evaluator][INFO] - Result for metric extraction_strength: 0.08614304082608436
|
|
[2026-05-18 14:05:19,095][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-05-18 14:05:19,095][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-75/evals/TOFU_EVAL.json
|
|
[2026-05-18 14:05:19,095][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-75/evals/TOFU_SUMMARY.json
|
|
[2026-05-18 14:05:20,882][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-05-18 14:05:25,058][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-05-18 14:05:39,034][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-05-18 14:05:39,035][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6668836791633079
|
|
[2026-05-18 14:05:39,041][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
|
|
[2026-05-18 14:05:39,042][metrics][INFO] - Evaluating forget_quality
|
|
[2026-05-18 14:05:39,042][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
|
|
[2026-05-18 14:05:39,042][evaluator][INFO] - Result for metric forget_quality: None
|
|
[2026-05-18 14:05:40,713][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2026-05-18 14:05:43,524][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.22939431543461977
|
|
[2026-05-18 14:05:44,787][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-05-18 14:05:48,594][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.19464342599083026
|
|
[2026-05-18 14:05:49,944][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-05-18 14:05:53,959][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-05-18 14:05:59,042][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-05-18 14:06:03,088][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-05-18 14:06:16,837][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-05-18 14:06:18,531][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-05-18 14:06:20,445][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-05-18 14:06:22,314][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-05-18 14:06:23,557][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-05-18 14:06:24,902][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:06:24,903][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:06:24,903][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-05-18 14:06:26,318][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-05-18 14:06:28,106][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-05-18 14:06:29,685][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-05-18 14:06:31,014][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-05-18 14:06:31,984][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:06:31,984][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:06:31,984][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-05-18 14:06:31,984][metrics][INFO] - Evaluating model_utility
|
|
[2026-05-18 14:06:31,985][evaluator][INFO] - Result for metric model_utility: 0.34932386649395714
|
|
[2026-05-18 14:06:34,498][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-05-18 14:06:35,497][metrics][INFO] - Evaluating privleak
|
|
[2026-05-18 14:06:35,498][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
|
|
[2026-05-18 14:06:35,498][evaluator][INFO] - Result for metric privleak: -65.46124998690773
|
|
[2026-05-18 14:06:36,810][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-05-18 14:06:37,620][evaluator][INFO] - Result for metric extraction_strength: 0.09391103318955707
|
|
[2026-05-18 14:07:30,784][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-05-18 14:07:30,785][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-100/evals/TOFU_EVAL.json
|
|
[2026-05-18 14:07:30,785][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-100/evals/TOFU_SUMMARY.json
|
|
[2026-05-18 14:07:32,467][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-05-18 14:07:36,654][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-05-18 14:07:50,605][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-05-18 14:07:50,606][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6531148071926944
|
|
[2026-05-18 14:07:50,612][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
|
|
[2026-05-18 14:07:50,612][metrics][INFO] - Evaluating forget_quality
|
|
[2026-05-18 14:07:50,613][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
|
|
[2026-05-18 14:07:50,613][evaluator][INFO] - Result for metric forget_quality: None
|
|
[2026-05-18 14:07:52,345][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2026-05-18 14:07:55,159][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.1875428356032353
|
|
[2026-05-18 14:07:56,408][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-05-18 14:08:01,500][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.22033509820856984
|
|
[2026-05-18 14:08:03,199][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-05-18 14:08:07,214][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-05-18 14:08:13,022][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-05-18 14:08:17,053][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-05-18 14:08:30,813][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-05-18 14:08:32,534][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-05-18 14:08:34,570][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-05-18 14:08:36,441][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-05-18 14:08:37,679][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-05-18 14:08:39,458][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:08:39,458][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:08:39,458][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-05-18 14:08:40,743][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-05-18 14:08:42,502][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-05-18 14:08:44,083][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-05-18 14:08:45,415][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-05-18 14:08:46,470][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:08:46,470][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:08:46,470][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-05-18 14:08:46,471][metrics][INFO] - Evaluating model_utility
|
|
[2026-05-18 14:08:46,471][evaluator][INFO] - Result for metric model_utility: 0.43717298699454965
|
|
[2026-05-18 14:08:48,984][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-05-18 14:08:49,984][metrics][INFO] - Evaluating privleak
|
|
[2026-05-18 14:08:49,984][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
|
|
[2026-05-18 14:08:49,985][evaluator][INFO] - Result for metric privleak: -38.19124999236173
|
|
[2026-05-18 14:08:51,268][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-05-18 14:08:52,071][evaluator][INFO] - Result for metric extraction_strength: 0.09873176679531875
|
|
[2026-05-18 14:09:41,834][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-05-18 14:09:41,834][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-125/evals/TOFU_EVAL.json
|
|
[2026-05-18 14:09:41,834][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-125/evals/TOFU_SUMMARY.json
|
|
[2026-05-18 14:09:43,508][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-05-18 14:09:47,641][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-05-18 14:10:01,592][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-05-18 14:10:01,592][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6237154687561808
|
|
[2026-05-18 14:10:01,598][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
|
|
[2026-05-18 14:10:01,599][metrics][INFO] - Evaluating forget_quality
|
|
[2026-05-18 14:10:01,599][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
|
|
[2026-05-18 14:10:01,599][evaluator][INFO] - Result for metric forget_quality: None
|
|
[2026-05-18 14:10:03,260][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2026-05-18 14:10:06,063][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.15545873703609686
|
|
[2026-05-18 14:10:07,313][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-05-18 14:10:12,073][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2513310748335476
|
|
[2026-05-18 14:10:13,325][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-05-18 14:10:17,327][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-05-18 14:10:23,249][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-05-18 14:10:27,286][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-05-18 14:10:41,035][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-05-18 14:10:42,745][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-05-18 14:10:44,708][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-05-18 14:10:46,577][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-05-18 14:10:47,822][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-05-18 14:10:48,838][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:10:48,838][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:10:48,838][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-05-18 14:10:50,197][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-05-18 14:10:52,000][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-05-18 14:10:53,577][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-05-18 14:10:54,817][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-05-18 14:10:56,881][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:10:56,882][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:10:56,882][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-05-18 14:10:56,882][metrics][INFO] - Evaluating model_utility
|
|
[2026-05-18 14:10:56,883][evaluator][INFO] - Result for metric model_utility: 0.49362893415678766
|
|
[2026-05-18 14:10:59,475][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-05-18 14:11:00,462][metrics][INFO] - Evaluating privleak
|
|
[2026-05-18 14:11:00,462][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
|
|
[2026-05-18 14:11:00,462][evaluator][INFO] - Result for metric privleak: -13.094999997381013
|
|
[2026-05-18 14:11:01,746][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-05-18 14:11:02,542][evaluator][INFO] - Result for metric extraction_strength: 0.09858533284264409
|
|
[2026-05-18 14:11:53,342][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-05-18 14:11:53,343][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-150/evals/TOFU_EVAL.json
|
|
[2026-05-18 14:11:53,343][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-150/evals/TOFU_SUMMARY.json
|
|
[2026-05-18 14:11:55,079][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-05-18 14:11:59,252][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-05-18 14:12:13,207][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-05-18 14:12:13,207][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5806421898784745
|
|
[2026-05-18 14:12:13,213][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
|
|
[2026-05-18 14:12:13,214][metrics][INFO] - Evaluating forget_quality
|
|
[2026-05-18 14:12:13,214][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
|
|
[2026-05-18 14:12:13,214][evaluator][INFO] - Result for metric forget_quality: None
|
|
[2026-05-18 14:12:14,796][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2026-05-18 14:12:17,599][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.15392853119468783
|
|
[2026-05-18 14:12:18,838][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-05-18 14:12:23,915][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.27986895187799854
|
|
[2026-05-18 14:12:25,549][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-05-18 14:12:29,551][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-05-18 14:12:35,561][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-05-18 14:12:39,971][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-05-18 14:12:53,737][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-05-18 14:12:55,355][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-05-18 14:12:57,221][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-05-18 14:12:59,097][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-05-18 14:13:00,360][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-05-18 14:13:01,499][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:13:01,499][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:13:01,499][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-05-18 14:13:02,741][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-05-18 14:13:04,731][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-05-18 14:13:06,308][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-05-18 14:13:07,535][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-05-18 14:13:08,742][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:13:08,742][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:13:08,742][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-05-18 14:13:08,743][metrics][INFO] - Evaluating model_utility
|
|
[2026-05-18 14:13:08,743][evaluator][INFO] - Result for metric model_utility: 0.5353537708875081
|
|
[2026-05-18 14:13:11,314][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-05-18 14:13:12,318][metrics][INFO] - Evaluating privleak
|
|
[2026-05-18 14:13:12,318][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
|
|
[2026-05-18 14:13:12,318][evaluator][INFO] - Result for metric privleak: -21.444999995711004
|
|
[2026-05-18 14:13:13,720][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-05-18 14:13:14,523][evaluator][INFO] - Result for metric extraction_strength: 0.10802013307923694
|
|
[2026-05-18 14:14:07,060][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-05-18 14:14:07,060][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-175/evals/TOFU_EVAL.json
|
|
[2026-05-18 14:14:07,060][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-175/evals/TOFU_SUMMARY.json
|
|
[2026-05-18 14:14:08,719][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-05-18 14:14:12,789][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-05-18 14:14:26,764][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-05-18 14:14:26,764][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5689891586427206
|
|
[2026-05-18 14:14:26,770][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
|
|
[2026-05-18 14:14:26,771][metrics][INFO] - Evaluating forget_quality
|
|
[2026-05-18 14:14:26,771][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
|
|
[2026-05-18 14:14:26,771][evaluator][INFO] - Result for metric forget_quality: None
|
|
[2026-05-18 14:14:28,494][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2026-05-18 14:14:31,298][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.14874292317486834
|
|
[2026-05-18 14:14:32,623][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-05-18 14:14:38,203][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2956713274905
|
|
[2026-05-18 14:14:39,847][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-05-18 14:14:43,836][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-05-18 14:14:50,807][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-05-18 14:14:54,853][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-05-18 14:15:08,629][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-05-18 14:15:10,462][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-05-18 14:15:12,391][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-05-18 14:15:14,266][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-05-18 14:15:15,515][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-05-18 14:15:16,487][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:15:16,487][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:15:16,488][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-05-18 14:15:17,760][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-05-18 14:15:19,489][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-05-18 14:15:21,071][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-05-18 14:15:22,347][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-05-18 14:15:24,289][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:15:24,289][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:15:24,289][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-05-18 14:15:24,290][metrics][INFO] - Evaluating model_utility
|
|
[2026-05-18 14:15:24,290][evaluator][INFO] - Result for metric model_utility: 0.5515531045256538
|
|
[2026-05-18 14:15:26,783][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-05-18 14:15:27,785][metrics][INFO] - Evaluating privleak
|
|
[2026-05-18 14:15:27,785][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
|
|
[2026-05-18 14:15:27,785][evaluator][INFO] - Result for metric privleak: -15.800624996839888
|
|
[2026-05-18 14:15:29,043][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-05-18 14:15:29,839][evaluator][INFO] - Result for metric extraction_strength: 0.11600868539246466
|
|
[2026-05-18 14:16:24,603][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-05-18 14:16:24,603][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-200/evals/TOFU_EVAL.json
|
|
[2026-05-18 14:16:24,603][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-200/evals/TOFU_SUMMARY.json
|
|
[2026-05-18 14:16:26,252][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-05-18 14:16:30,454][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-05-18 14:16:44,412][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-05-18 14:16:44,413][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5614269492250329
|
|
[2026-05-18 14:16:44,419][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
|
|
[2026-05-18 14:16:44,419][metrics][INFO] - Evaluating forget_quality
|
|
[2026-05-18 14:16:44,419][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
|
|
[2026-05-18 14:16:44,419][evaluator][INFO] - Result for metric forget_quality: None
|
|
[2026-05-18 14:16:46,111][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2026-05-18 14:16:48,925][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.14210461506576394
|
|
[2026-05-18 14:16:50,235][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-05-18 14:16:55,885][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2970728548646609
|
|
[2026-05-18 14:16:57,523][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-05-18 14:17:01,535][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-05-18 14:17:08,651][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-05-18 14:17:12,659][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-05-18 14:17:26,399][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-05-18 14:17:28,037][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-05-18 14:17:29,907][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-05-18 14:17:31,779][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-05-18 14:17:33,006][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-05-18 14:17:33,823][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:17:33,823][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:17:33,823][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-05-18 14:17:35,264][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-05-18 14:17:36,972][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-05-18 14:17:38,549][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-05-18 14:17:39,786][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-05-18 14:17:43,814][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:17:43,815][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:17:43,815][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-05-18 14:17:43,815][metrics][INFO] - Evaluating model_utility
|
|
[2026-05-18 14:17:43,816][evaluator][INFO] - Result for metric model_utility: 0.5591748848281174
|
|
[2026-05-18 14:17:46,742][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-05-18 14:17:47,745][metrics][INFO] - Evaluating privleak
|
|
[2026-05-18 14:17:47,745][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
|
|
[2026-05-18 14:17:47,745][evaluator][INFO] - Result for metric privleak: -17.166249996566755
|
|
[2026-05-18 14:17:49,081][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-05-18 14:17:49,881][evaluator][INFO] - Result for metric extraction_strength: 0.11815252416178619
|
|
[2026-05-18 14:18:39,514][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-05-18 14:18:39,514][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-225/evals/TOFU_EVAL.json
|
|
[2026-05-18 14:18:39,514][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-225/evals/TOFU_SUMMARY.json
|
|
[2026-05-18 14:18:41,139][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-05-18 14:18:45,205][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-05-18 14:18:59,191][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-05-18 14:18:59,193][evaluator][INFO] - Result for metric forget_truth_ratio: 0.559432289746286
|
|
[2026-05-18 14:18:59,199][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
|
|
[2026-05-18 14:18:59,199][metrics][INFO] - Evaluating forget_quality
|
|
[2026-05-18 14:18:59,199][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
|
|
[2026-05-18 14:18:59,199][evaluator][INFO] - Result for metric forget_quality: None
|
|
[2026-05-18 14:19:00,827][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2026-05-18 14:19:03,629][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.13278577822376975
|
|
[2026-05-18 14:19:04,861][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-05-18 14:19:10,156][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.30219775268292687
|
|
[2026-05-18 14:19:11,820][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-05-18 14:19:15,860][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-05-18 14:19:22,552][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-05-18 14:19:26,622][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-05-18 14:19:40,363][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-05-18 14:19:42,025][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-05-18 14:19:44,233][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-05-18 14:19:46,108][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-05-18 14:19:47,431][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-05-18 14:19:48,347][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:19:48,348][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:19:48,348][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-05-18 14:19:50,409][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-05-18 14:19:52,353][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-05-18 14:19:53,929][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-05-18 14:19:55,162][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-05-18 14:19:56,423][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:19:56,423][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:19:56,423][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-05-18 14:19:56,423][metrics][INFO] - Evaluating model_utility
|
|
[2026-05-18 14:19:56,424][evaluator][INFO] - Result for metric model_utility: 0.5593146400206448
|
|
[2026-05-18 14:19:59,178][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-05-18 14:20:00,182][metrics][INFO] - Evaluating privleak
|
|
[2026-05-18 14:20:00,183][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
|
|
[2026-05-18 14:20:00,183][evaluator][INFO] - Result for metric privleak: -14.034999997192998
|
|
[2026-05-18 14:20:01,562][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-05-18 14:20:02,368][evaluator][INFO] - Result for metric extraction_strength: 0.11882007020428687
|
|
[2026-05-18 14:20:54,400][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-05-18 14:20:54,400][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-250/evals/TOFU_EVAL.json
|
|
[2026-05-18 14:20:54,400][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_NPO_qat-off/checkpoint-250/evals/TOFU_SUMMARY.json
|
|
[2026-05-18 14:20:56,086][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-05-18 14:21:00,205][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-05-18 14:21:14,170][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-05-18 14:21:14,170][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5612359669808998
|
|
[2026-05-18 14:21:14,176][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
|
|
[2026-05-18 14:21:14,177][metrics][INFO] - Evaluating forget_quality
|
|
[2026-05-18 14:21:14,177][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
|
|
[2026-05-18 14:21:14,177][evaluator][INFO] - Result for metric forget_quality: None
|
|
[2026-05-18 14:21:15,906][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2026-05-18 14:21:18,726][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.13566936177085154
|
|
[2026-05-18 14:21:20,061][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-05-18 14:21:25,547][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.3076188844200944
|
|
[2026-05-18 14:21:27,366][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-05-18 14:21:31,504][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-05-18 14:21:38,204][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-05-18 14:21:42,261][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-05-18 14:21:56,026][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-05-18 14:21:57,712][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-05-18 14:21:59,628][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-05-18 14:22:01,503][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-05-18 14:22:02,770][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-05-18 14:22:03,597][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:22:03,597][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:22:03,597][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-05-18 14:22:04,849][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-05-18 14:22:06,654][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-05-18 14:22:08,232][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-05-18 14:22:09,479][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-05-18 14:22:10,670][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-05-18 14:22:10,671][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-05-18 14:22:10,671][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-05-18 14:22:10,671][metrics][INFO] - Evaluating model_utility
|
|
[2026-05-18 14:22:10,671][evaluator][INFO] - Result for metric model_utility: 0.5599261205924377
|
|
[2026-05-18 14:22:13,279][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-05-18 14:22:14,284][metrics][INFO] - Evaluating privleak
|
|
[2026-05-18 14:22:14,284][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
|
|
[2026-05-18 14:22:14,284][evaluator][INFO] - Result for metric privleak: -13.192499997361507
|
|
[2026-05-18 14:22:15,573][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-05-18 14:22:16,382][evaluator][INFO] - Result for metric extraction_strength: 0.11907237943929123
|