60 lines
5.3 KiB
Plaintext
60 lines
5.3 KiB
Plaintext
[2026-08-24 16:02:28,521][model][INFO] - Setting pad_token as eos token: <|eot_id|>
|
|
[2026-08-24 16:02:28,525][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/baselines/tofu/NPO/Llama-3.1-8B-Instruct/forget05/evals
|
|
[2026-08-24 16:02:28,527][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2026-08-24 16:02:28,527][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/baselines/tofu/NPO/Llama-3.1-8B-Instruct/forget05/evals/TOFU_EVAL.json
|
|
[2026-08-24 16:02:28,527][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/baselines/tofu/NPO/Llama-3.1-8B-Instruct/forget05/evals/TOFU_SUMMARY.json
|
|
[2026-08-24 16:02:30,471][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.1-8B-Instruct_retain95/TOFU_EVAL.json
|
|
[2026-08-24 16:02:30,481][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2026-08-24 16:02:37,358][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.1-8B-Instruct_retain95/TOFU_EVAL.json
|
|
[2026-08-24 16:02:37,368][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2026-08-24 16:03:02,886][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.1-8B-Instruct_retain95/TOFU_EVAL.json
|
|
[2026-08-24 16:03:02,896][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2026-08-24 16:03:02,897][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.1-8B-Instruct_retain95/TOFU_EVAL.json
|
|
[2026-08-24 16:03:02,905][metrics][INFO] - Evaluating forget_quality
|
|
[2026-08-24 16:03:02,906][evaluator][INFO] - Result for metric forget_quality: 0.011843449760085422
|
|
[2026-08-24 16:03:04,138][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2026-08-24 16:03:14,659][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2026-08-24 16:04:00,406][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2026-08-24 16:04:11,558][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2026-08-24 16:05:00,809][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2026-08-24 16:05:02,042][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2026-08-24 16:05:04,603][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2026-08-24 16:05:09,445][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2026-08-24 16:05:10,390][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2026-08-24 16:05:18,593][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2026-08-24 16:05:18,593][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2026-08-24 16:05:18,593][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2026-08-24 16:05:19,837][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2026-08-24 16:05:22,437][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2026-08-24 16:05:27,584][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2026-08-24 16:05:28,434][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2026-08-24 16:05:40,357][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2026-08-24 16:05:40,357][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2026-08-24 16:05:40,357][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2026-08-24 16:05:40,357][metrics][INFO] - Evaluating model_utility
|
|
[2026-08-24 16:05:40,358][evaluator][INFO] - Result for metric model_utility: 0.5678715338202777
|
|
[2026-08-24 16:05:41,597][metrics][INFO] - Evaluating exact_memorization
|
|
[2026-08-24 16:05:45,731][evaluator][INFO] - Result for metric exact_memorization: 0.5072669323906303
|
|
[2026-08-24 16:05:46,739][metrics][INFO] - Evaluating extraction_strength
|
|
[2026-08-24 16:05:51,044][evaluator][INFO] - Result for metric extraction_strength: 0.05579637056005085
|
|
[2026-08-24 16:05:51,073][evaluator][INFO] - Skipping forget_Q_A_PARA_Prob, already evaluated.
|
|
[2026-08-24 16:05:51,074][evaluator][INFO] - Result for metric forget_Q_A_PARA_Prob: 0.023710637748226873
|
|
[2026-08-24 16:05:51,074][evaluator][INFO] - Skipping forget_truth_ratio, already evaluated.
|
|
[2026-08-24 16:05:51,074][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6632387214939777
|
|
[2026-08-24 16:05:51,977][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2026-08-24 16:06:21,886][metrics][INFO] - Evaluating forget_Q_A_gibberish
|
|
[2026-08-24 16:06:22,437][evaluator][INFO] - Result for metric forget_Q_A_gibberish: 0.8972614184241684
|
|
[2026-08-24 16:06:24,641][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.1-8B-Instruct_retain95/TOFU_EVAL.json
|
|
[2026-08-24 16:06:24,651][metrics][INFO] - Evaluating mia_min_k
|
|
[2026-08-24 16:06:33,158][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.1-8B-Instruct_retain95/TOFU_EVAL.json
|
|
[2026-08-24 16:06:33,167][metrics][INFO] - Evaluating privleak
|
|
[2026-08-24 16:06:33,167][evaluator][INFO] - Result for metric privleak: 44.8683597320801
|
|
[2026-08-24 16:06:35,337][metrics][INFO] - Evaluating mia_min_k_plus_plus
|
|
[2026-08-24 16:06:52,206][evaluator][INFO] - Result for metric mia_min_k_plus_plus: 0.120575
|
|
[2026-08-24 16:06:52,242][evaluator][INFO] - Skipping mia_min_k, already evaluated.
|
|
[2026-08-24 16:06:52,242][evaluator][INFO] - Result for metric mia_min_k: 0.06733750000000001
|
|
[2026-08-24 16:06:54,534][metrics][INFO] - Evaluating mia_loss
|
|
[2026-08-24 16:07:04,898][evaluator][INFO] - Result for metric mia_loss: 0.06501250000000001
|
|
[2026-08-24 16:07:07,081][metrics][INFO] - Evaluating mia_zlib
|
|
[2026-08-24 16:07:17,516][evaluator][INFO] - Result for metric mia_zlib: 0.10685
|