101 lines
9.9 KiB
Plaintext
101 lines
9.9 KiB
Plaintext
[2025-05-02 21:51:40,732][model][INFO] - Setting pad_token as eos token: <|eot_id|>
|
|
[2025-05-02 21:51:40,735][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals
|
|
[2025-05-02 21:51:40,736][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2025-05-02 21:51:40,736][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals/TOFU_EVAL.json
|
|
[2025-05-02 21:51:40,736][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals/TOFU_SUMMARY.json
|
|
[2025-05-02 21:51:43,929][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 21:51:43,940][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
|
|
[2025-05-02 21:51:52,652][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 21:51:52,664][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
|
|
[2025-05-02 21:52:24,471][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 21:52:24,484][metrics][INFO] - Evaluating forget_truth_ratio
|
|
[2025-05-02 21:52:24,485][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 21:52:24,494][metrics][INFO] - Evaluating forget_quality
|
|
[2025-05-02 21:52:24,495][evaluator][INFO] - Result for metric forget_quality: 4.353260441808186e-19
|
|
[2025-05-02 21:52:26,684][metrics][INFO] - Evaluating forget_Q_A_Prob
|
|
[2025-05-02 21:52:33,182][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.00043882395156970234
|
|
[2025-05-02 21:52:35,322][metrics][INFO] - Evaluating forget_Q_A_ROUGE
|
|
[2025-05-02 21:53:43,279][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.042342875545698926
|
|
[2025-05-02 21:53:45,045][metrics][INFO] - Evaluating retain_Q_A_Prob
|
|
[2025-05-02 21:53:52,390][metrics][INFO] - Evaluating retain_Q_A_ROUGE
|
|
[2025-05-02 21:54:18,330][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
|
|
[2025-05-02 21:54:26,148][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
|
|
[2025-05-02 21:54:55,552][metrics][INFO] - Evaluating retain_Truth_Ratio
|
|
[2025-05-02 21:54:57,295][metrics][INFO] - Evaluating ra_Q_A_Prob
|
|
[2025-05-02 21:54:59,941][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
|
|
[2025-05-02 21:55:02,973][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
|
|
[2025-05-02 21:55:04,235][metrics][INFO] - Evaluating ra_Q_A_ROUGE
|
|
[2025-05-02 21:55:07,779][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
|
|
[2025-05-02 21:55:07,780][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
|
|
[2025-05-02 21:55:07,780][metrics][INFO] - Evaluating ra_Truth_Ratio
|
|
[2025-05-02 21:55:09,044][metrics][INFO] - Evaluating wf_Q_A_Prob
|
|
[2025-05-02 21:55:11,287][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
|
|
[2025-05-02 21:55:14,390][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
|
|
[2025-05-02 21:55:15,655][metrics][INFO] - Evaluating wf_Q_A_ROUGE
|
|
[2025-05-02 21:55:20,650][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
|
|
[2025-05-02 21:55:20,650][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
|
|
[2025-05-02 21:55:20,651][metrics][INFO] - Evaluating wf_Truth_Ratio
|
|
[2025-05-02 21:55:20,651][metrics][INFO] - Evaluating model_utility
|
|
[2025-05-02 21:55:20,651][evaluator][INFO] - Result for metric model_utility: 0.6712234811279283
|
|
[2025-05-02 21:55:23,773][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 21:55:23,785][metrics][INFO] - Evaluating mia_min_k
|
|
[2025-05-02 21:55:32,575][metrics][INFO] - Loading evaluations from saves/eval/tofu_Llama-3.2-3B-Instruct_retain90/TOFU_EVAL.json
|
|
[2025-05-02 21:55:32,583][metrics][INFO] - Evaluating privleak
|
|
[2025-05-02 21:55:32,583][evaluator][INFO] - Result for metric privleak: 62.431024691878754
|
|
[2025-05-02 21:55:34,474][metrics][INFO] - Evaluating extraction_strength
|
|
[2025-05-02 21:55:39,203][evaluator][INFO] - Result for metric extraction_strength: 0.03262797759418284
|
|
[2025-05-09 18:48:58,401][model][INFO] - Setting pad_token as eos token: <|eot_id|>
|
|
[2025-05-09 18:48:58,403][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals
|
|
[2025-05-09 18:48:58,404][evaluator][INFO] - Loading existing evaluations from saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals/TOFU_EVAL.json
|
|
[2025-05-09 18:48:58,457][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2025-05-09 18:48:58,457][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals/TOFU_EVAL.json
|
|
[2025-05-09 18:48:58,457][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals/TOFU_SUMMARY.json
|
|
[2025-05-09 18:48:58,457][evaluator][INFO] - Skipping forget_quality, already evaluated.
|
|
[2025-05-09 18:48:58,457][evaluator][INFO] - Result for metric forget_quality: 4.353260441808186e-19
|
|
[2025-05-09 18:48:58,462][evaluator][INFO] - Skipping forget_Q_A_Prob, already evaluated.
|
|
[2025-05-09 18:48:58,462][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.00043882395156970234
|
|
[2025-05-09 18:48:58,465][evaluator][INFO] - Skipping forget_Q_A_ROUGE, already evaluated.
|
|
[2025-05-09 18:48:58,465][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.042342875545698926
|
|
[2025-05-09 18:48:58,468][evaluator][INFO] - Skipping model_utility, already evaluated.
|
|
[2025-05-09 18:48:58,468][evaluator][INFO] - Result for metric model_utility: 0.6712234811279283
|
|
[2025-05-09 18:48:58,471][evaluator][INFO] - Skipping privleak, already evaluated.
|
|
[2025-05-09 18:48:58,472][evaluator][INFO] - Result for metric privleak: 62.431024691878754
|
|
[2025-05-09 18:48:58,475][evaluator][INFO] - Skipping extraction_strength, already evaluated.
|
|
[2025-05-09 18:48:58,475][evaluator][INFO] - Result for metric extraction_strength: 0.03262797759418284
|
|
[2025-05-09 19:03:19,094][model][INFO] - Setting pad_token as eos token: <|eot_id|>
|
|
[2025-05-09 19:03:19,096][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals
|
|
[2025-05-09 19:03:19,098][evaluator][INFO] - Loading existing evaluations from saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals/TOFU_EVAL.json
|
|
[2025-05-09 19:03:19,138][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2025-05-09 19:03:19,138][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals/TOFU_EVAL.json
|
|
[2025-05-09 19:03:19,138][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals/TOFU_SUMMARY.json
|
|
[2025-05-09 19:03:19,138][evaluator][INFO] - Skipping forget_quality, already evaluated.
|
|
[2025-05-09 19:03:19,138][evaluator][INFO] - Result for metric forget_quality: 4.353260441808186e-19
|
|
[2025-05-09 19:03:19,147][evaluator][INFO] - Skipping forget_Q_A_Prob, already evaluated.
|
|
[2025-05-09 19:03:19,147][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.00043882395156970234
|
|
[2025-05-09 19:03:19,148][evaluator][INFO] - Skipping forget_Q_A_ROUGE, already evaluated.
|
|
[2025-05-09 19:03:19,148][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.042342875545698926
|
|
[2025-05-09 19:03:19,154][evaluator][INFO] - Skipping model_utility, already evaluated.
|
|
[2025-05-09 19:03:19,154][evaluator][INFO] - Result for metric model_utility: 0.6712234811279283
|
|
[2025-05-09 19:03:19,156][evaluator][INFO] - Skipping privleak, already evaluated.
|
|
[2025-05-09 19:03:19,156][evaluator][INFO] - Result for metric privleak: 62.431024691878754
|
|
[2025-05-09 19:03:19,157][evaluator][INFO] - Skipping extraction_strength, already evaluated.
|
|
[2025-05-09 19:03:19,158][evaluator][INFO] - Result for metric extraction_strength: 0.03262797759418284
|
|
[2025-05-13 07:04:33,050][model][INFO] - Setting pad_token as eos token: <|eot_id|>
|
|
[2025-05-13 07:04:33,052][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals
|
|
[2025-05-13 07:04:33,054][evaluator][INFO] - Loading existing evaluations from saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals/TOFU_EVAL.json
|
|
[2025-05-13 07:04:33,089][evaluator][INFO] - ***** Running TOFU evaluation suite *****
|
|
[2025-05-13 07:04:33,089][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals/TOFU_EVAL.json
|
|
[2025-05-13 07:04:33,089][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/tofu_Llama-3.2-3B-Instruct_forget10_RMU/evals/TOFU_SUMMARY.json
|
|
[2025-05-13 07:04:33,089][evaluator][INFO] - Skipping forget_quality, already evaluated.
|
|
[2025-05-13 07:04:33,089][evaluator][INFO] - Result for metric forget_quality: 4.353260441808186e-19
|
|
[2025-05-13 07:04:33,115][evaluator][INFO] - Skipping forget_Q_A_Prob, already evaluated.
|
|
[2025-05-13 07:04:33,115][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.00043882395156970234
|
|
[2025-05-13 07:04:33,119][evaluator][INFO] - Skipping forget_Q_A_ROUGE, already evaluated.
|
|
[2025-05-13 07:04:33,119][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.042342875545698926
|
|
[2025-05-13 07:04:33,121][evaluator][INFO] - Skipping model_utility, already evaluated.
|
|
[2025-05-13 07:04:33,121][evaluator][INFO] - Result for metric model_utility: 0.6712234811279283
|
|
[2025-05-13 07:04:33,124][evaluator][INFO] - Skipping privleak, already evaluated.
|
|
[2025-05-13 07:04:33,124][evaluator][INFO] - Result for metric privleak: 62.431024691878754
|
|
[2025-05-13 07:04:33,127][evaluator][INFO] - Skipping extraction_strength, already evaluated.
|
|
[2025-05-13 07:04:33,127][evaluator][INFO] - Result for metric extraction_strength: 0.03262797759418284
|