83 lines
7.9 KiB
Plaintext
83 lines
7.9 KiB
Plaintext
[2025-05-02 20:53:06,934][model][INFO] - Setting pad_token as eos token: <|eot_id|>
|
|
[2025-05-02 20:53:06,936][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_NPO/evals
|
|
[2025-05-02 20:53:06,938][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2025-05-02 20:53:06,938][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_NPO/evals/TOFU_EVAL.json
|
|
[2025-05-02 20:53:06,938][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_NPO/evals/TOFU_SUMMARY.json
|
|
[2025-05-02 20:53:10,115][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 20:53:10,127][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2025-05-02 20:53:18,794][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 20:53:18,805][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2025-05-02 20:53:50,629][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 20:53:50,640][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2025-05-02 20:53:50,641][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 20:53:50,649][metrics][INFO] - Evaluating forget_quality
|
|
[2025-05-02 20:53:50,651][evaluator][INFO] - Result for metric forget_quality: 0.02985302150862578
|
|
[2025-05-02 20:53:53,280][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2025-05-02 20:53:59,822][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.1261680691310903
|
|
[2025-05-02 20:54:01,537][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2025-05-02 20:55:02,203][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.25210551424004063
|
|
[2025-05-02 20:55:03,996][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2025-05-02 20:55:11,697][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2025-05-02 20:56:06,684][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2025-05-02 20:56:14,365][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2025-05-02 20:56:43,732][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2025-05-02 20:56:45,414][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2025-05-02 20:56:47,734][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2025-05-02 20:56:50,768][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2025-05-02 20:56:52,037][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2025-05-02 20:57:05,532][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2025-05-02 20:57:05,532][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2025-05-02 20:57:05,532][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2025-05-02 20:57:07,690][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2025-05-02 20:57:09,966][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2025-05-02 20:57:13,072][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2025-05-02 20:57:14,642][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2025-05-02 20:57:35,200][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2025-05-02 20:57:35,200][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2025-05-02 20:57:35,200][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2025-05-02 20:57:35,200][metrics][INFO] - Evaluating model_utility
|
|
[2025-05-02 20:57:35,201][evaluator][INFO] - Result for metric model_utility: 0.5338083414962111
|
|
[2025-05-02 20:57:38,914][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 20:57:38,926][metrics][INFO] - Evaluating mia_min_k
|
|
[2025-05-02 20:57:47,740][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 20:57:47,750][metrics][INFO] - Evaluating privleak
|
|
[2025-05-02 20:57:47,750][evaluator][INFO] - Result for metric privleak: 18.97581351860541
|
|
[2025-05-02 20:57:50,357][metrics][INFO] - Evaluating extraction_strength
|
|
[2025-05-02 20:57:55,078][evaluator][INFO] - Result for metric extraction_strength: 0.059796485372586014
|
|
[2025-05-09 20:17:24,911][model][INFO] - Setting pad_token as eos token: <|eot_id|>
|
|
[2025-05-09 20:17:24,914][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_NPO/evals
|
|
[2025-05-09 20:17:24,915][evaluator][INFO] - Loading existing evaluations from saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_NPO/evals/TOFU_EVAL.json
|
|
[2025-05-09 20:17:25,004][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2025-05-09 20:17:25,004][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_NPO/evals/TOFU_EVAL.json
|
|
[2025-05-09 20:17:25,004][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_NPO/evals/TOFU_SUMMARY.json
|
|
[2025-05-09 20:17:25,004][evaluator][INFO] - Skipping forget_quality, already evaluated.
|
|
[2025-05-09 20:17:25,004][evaluator][INFO] - Result for metric forget_quality: 0.02985302150862578
|
|
[2025-05-09 20:17:25,008][evaluator][INFO] - Skipping forget_Q_A_Prob, already evaluated.
|
|
[2025-05-09 20:17:25,008][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.1261680691310903
|
|
[2025-05-09 20:17:25,011][evaluator][INFO] - Skipping forget_Q_A_ROUGE, already evaluated.
|
|
[2025-05-09 20:17:25,011][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.25210551424004063
|
|
[2025-05-09 20:17:25,013][evaluator][INFO] - Skipping model_utility, already evaluated.
|
|
[2025-05-09 20:17:25,013][evaluator][INFO] - Result for metric model_utility: 0.5338083414962111
|
|
[2025-05-09 20:17:25,015][evaluator][INFO] - Skipping privleak, already evaluated.
|
|
[2025-05-09 20:17:25,015][evaluator][INFO] - Result for metric privleak: 18.97581351860541
|
|
[2025-05-09 20:17:25,018][evaluator][INFO] - Skipping extraction_strength, already evaluated.
|
|
[2025-05-09 20:17:25,018][evaluator][INFO] - Result for metric extraction_strength: 0.059796485372586014
|
|
[2025-05-13 08:20:30,111][model][INFO] - Setting pad_token as eos token: <|eot_id|>
|
|
[2025-05-13 08:20:30,113][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_NPO/evals
|
|
[2025-05-13 08:20:30,115][evaluator][INFO] - Loading existing evaluations from saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_NPO/evals/TOFU_EVAL.json
|
|
[2025-05-13 08:20:30,165][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2025-05-13 08:20:30,165][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_NPO/evals/TOFU_EVAL.json
|
|
[2025-05-13 08:20:30,165][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_NPO/evals/TOFU_SUMMARY.json
|
|
[2025-05-13 08:20:30,165][evaluator][INFO] - Skipping forget_quality, already evaluated.
|
|
[2025-05-13 08:20:30,165][evaluator][INFO] - Result for metric forget_quality: 0.02985302150862578
|
|
[2025-05-13 08:20:30,187][evaluator][INFO] - Skipping forget_Q_A_Prob, already evaluated.
|
|
[2025-05-13 08:20:30,187][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.1261680691310903
|
|
[2025-05-13 08:20:30,190][evaluator][INFO] - Skipping forget_Q_A_ROUGE, already evaluated.
|
|
[2025-05-13 08:20:30,190][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.25210551424004063
|
|
[2025-05-13 08:20:30,204][evaluator][INFO] - Skipping model_utility, already evaluated.
|
|
[2025-05-13 08:20:30,204][evaluator][INFO] - Result for metric model_utility: 0.5338083414962111
|
|
[2025-05-13 08:20:30,206][evaluator][INFO] - Skipping privleak, already evaluated.
|
|
[2025-05-13 08:20:30,206][evaluator][INFO] - Result for metric privleak: 18.97581351860541
|
|
[2025-05-13 08:20:30,215][evaluator][INFO] - Skipping extraction_strength, already evaluated.
|
|
[2025-05-13 08:20:30,215][evaluator][INFO] - Result for metric extraction_strength: 0.059796485372586014
|