2026-03-05 03:23:38 [INFO] ======================================================================== 2026-03-05 03:23:38 [INFO] FRANKENSTALLM 3B — Full Evaluation Pipeline 2026-03-05 03:23:38 [INFO] ======================================================================== 2026-03-05 03:23:38 [INFO] Checkpoint : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/checkpoints/korean_3b_fp8_run1/checkpoint-0057000 2026-03-05 03:23:38 [INFO] Tokenizer : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/tokenizer/korean_sp/tokenizer.json 2026-03-05 03:23:38 [INFO] Data dir : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/data 2026-03-05 03:23:38 [INFO] Output dir : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0323 2026-03-05 03:23:38 [INFO] GPUs : [2, 3, 4, 5, 6, 7] 2026-03-05 03:23:38 [INFO] SEQ_LEN : 2048 STRIDE: 512 BATCH_SIZE: 32 2026-03-05 03:23:38 [INFO] Phases : phase0=skip phase1=run phase2=run 2026-03-05 03:23:38 [INFO] 2026-03-05 03:23:38 [INFO] ------------------------------------------------------------------------ 2026-03-05 03:23:38 [INFO] PHASE 0 — HF Checkpoint Conversion 2026-03-05 03:23:38 [INFO] ------------------------------------------------------------------------ 2026-03-05 03:23:38 [INFO] Skipping Phase 0 — reusing found: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000 2026-03-05 03:23:38 [INFO] 2026-03-05 03:23:38 [INFO] ------------------------------------------------------------------------ 2026-03-05 03:23:38 [INFO] PHASE 1 — Internal Evaluation — 6 GPU Parallel 2026-03-05 03:23:38 [INFO] ------------------------------------------------------------------------ 2026-03-05 03:23:38 [INFO] Submitted: GPU 5 — Calibration + Token NLL 2026-03-05 03:23:38 [INFO] Submitted: GPU 6 — Generation (15 prompts × 4 temps) 2026-03-05 03:23:38 [INFO] Submitted: GPU 7 — Repetition grid (12 × 5) 2026-03-05 03:23:38 [INFO] Submitted: GPU 2 — PPL: 3b_val.bin 2026-03-05 03:23:38 [INFO] Submitted: GPU 3 — PPL: korean_c4 + korean_val 2026-03-05 03:23:38 [INFO] Submitted: GPU 4 — PPL: hplt_ko + cc100_ko + PPL: 7 cosmo files + PPL: 7 remaining files /usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden. Overriding a previously registered kernel for the same operator and the same dispatch key operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 dispatch key: ADInplaceOrView previous kernel: no debug info new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.) self.m.impl( /usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden. Overriding a previously registered kernel for the same operator and the same dispatch key operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 dispatch key: ADInplaceOrView previous kernel: no debug info new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.) self.m.impl( /usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden. Overriding a previously registered kernel for the same operator and the same dispatch key operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 dispatch key: ADInplaceOrView previous kernel: no debug info new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.) self.m.impl( /usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden. Overriding a previously registered kernel for the same operator and the same dispatch key operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 dispatch key: ADInplaceOrView previous kernel: no debug info new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.) self.m.impl( /usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden. Overriding a previously registered kernel for the same operator and the same dispatch key operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 dispatch key: ADInplaceOrView previous kernel: no debug info new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.) self.m.impl( /usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden. Overriding a previously registered kernel for the same operator and the same dispatch key operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 dispatch key: ADInplaceOrView previous kernel: no debug info new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.) self.m.impl( [PPL_MULTI cuda:0] Loading model once for 2 files... [PPL cuda:0] korean_c4: 15,159,838 tokens, 30.3 MB [PPL cuda:0] Loading model for 3b... [PPL cuda:0] 3b: 75,681,623 tokens, 151.4 MB [CALIB cuda:0] Loading model... [CALIB cuda:0] Using 50,000 tokens from 3b_val.bin [PPL_MULTI cuda:0] Loading model once for 16 files... [PPL cuda:0] hplt_ko: 16,165,735 tokens, 32.3 MB [CALIB cuda:0] DONE top1=0.6875, top5=0.8164, top10=0.8593, entropy=1.5682, 3.8s [NLL cuda:0] Loading model... [NLL cuda:0] Using 50,000 tokens from 3b_val.bin 2026-03-05 03:24:05 [INFO] [DONE] GPU 5 — Calibration + Token NLL 2026-03-05 03:25:42 [ERROR] [FAILED] GPU 6 — Generation (15 prompts × 4 temps) Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. 2026-03-05 03:25:42 [ERROR] [FAILED] GPU 7 — Repetition grid (12 × 5) Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. 2026-03-05 03:25:42 [ERROR] [FAILED] GPU 2 — PPL: 3b_val.bin Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. 2026-03-05 03:25:42 [ERROR] [FAILED] GPU 3 — PPL: korean_c4 + korean_val Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. 2026-03-05 03:25:42 [ERROR] [FAILED] GPU 4 — PPL: hplt_ko + cc100_ko + PPL: 7 cosmo files + PPL: 7 remaining files Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 569, in run_phase1 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. 2026-03-05 03:25:45 [INFO] Phase 1 complete: 1 succeeded, 5 failed 2026-03-05 03:25:45 [INFO] Phase 1 results saved: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0323/phase1_results.json 2026-03-05 03:25:45 [INFO] Phase 1 complete in 2m 6s. 2026-03-05 03:25:45 [INFO] 2026-03-05 03:25:45 [INFO] ------------------------------------------------------------------------ 2026-03-05 03:25:45 [INFO] PHASE 2 — lm-eval Benchmarks — 6 GPU Parallel 2026-03-05 03:25:45 [INFO] ------------------------------------------------------------------------ 2026-03-05 03:25:45 [INFO] Running 0-shot benchmarks on 8 GPUs ... 2026-03-05 03:25:45 [INFO] Submitted: [0-shot] GPU 2 — KoBEST: boolq + copa 2026-03-05 03:25:45 [INFO] Submitted: [0-shot] GPU 3 — KoBEST: hellaswag + sentineg 2026-03-05 03:25:45 [INFO] Submitted: [0-shot] GPU 4 — KoBEST: wic 2026-03-05 03:25:45 [INFO] Submitted: [0-shot] GPU 5 — HAE-RAE (all subtasks) 2026-03-05 03:25:45 [INFO] Submitted: [0-shot] GPU 6 — MMLU-KO part 1/2 2026-03-05 03:25:45 [INFO] Submitted: [0-shot] GPU 7 — MMLU-KO part 2/2 [Phase 2 [0-shot] GPU 2 — KoBEST: boolq + copa] Starting on cuda:2 — tasks: ['kobest_boolq', 'kobest_copa'], 0-shot [Phase 2 [0-shot] GPU 3 — KoBEST: hellaswag + sentineg] Starting on cuda:3 — tasks: ['kobest_hellaswag', 'kobest_sentineg'], 0-shot [Phase 2 [0-shot] GPU 4 — KoBEST: wic] Starting on cuda:4 — tasks: ['kobest_wic'], 0-shot [Phase 2 [0-shot] GPU 5 — HAE-RAE (all subtasks)] Starting on cuda:5 — tasks: ['haerae'], 0-shot [Phase 2 [0-shot] GPU 7 — MMLU-KO part 2/2] Starting on cuda:7 — tasks: ['global_mmlu_ko_high_school_statistics', 'global_mmlu_ko_high_school_us_history', 'global_mmlu_ko_high_school_world_history', 'global_mmlu_ko_human_aging', 'global_mmlu_ko_human_sexuality', 'global_mmlu_ko_international_law', 'global_mmlu_ko_jurisprudence', 'global_mmlu_ko_logical_fallacies', 'global_mmlu_ko_machine_learning', 'global_mmlu_ko_management', 'global_mmlu_ko_marketing', 'global_mmlu_ko_medical_genetics', 'global_mmlu_ko_miscellaneous', 'global_mmlu_ko_moral_disputes', 'global_mmlu_ko_moral_scenarios', 'global_mmlu_ko_nutrition', 'global_mmlu_ko_philosophy', 'global_mmlu_ko_prehistory', 'global_mmlu_ko_professional_accounting', 'global_mmlu_ko_professional_law', 'global_mmlu_ko_professional_medicine', 'global_mmlu_ko_professional_psychology', 'global_mmlu_ko_public_relations', 'global_mmlu_ko_security_studies', 'global_mmlu_ko_sociology', 'global_mmlu_ko_us_foreign_policy', 'global_mmlu_ko_virology', 'global_mmlu_ko_world_religions'], 0-shot [Phase 2 [0-shot] GPU 6 — MMLU-KO part 1/2] Starting on cuda:6 — tasks: ['global_mmlu_ko_abstract_algebra', 'global_mmlu_ko_anatomy', 'global_mmlu_ko_astronomy', 'global_mmlu_ko_business_ethics', 'global_mmlu_ko_clinical_knowledge', 'global_mmlu_ko_college_biology', 'global_mmlu_ko_college_chemistry', 'global_mmlu_ko_college_computer_science', 'global_mmlu_ko_college_mathematics', 'global_mmlu_ko_college_medicine', 'global_mmlu_ko_college_physics', 'global_mmlu_ko_computer_security', 'global_mmlu_ko_conceptual_physics', 'global_mmlu_ko_econometrics', 'global_mmlu_ko_electrical_engineering', 'global_mmlu_ko_elementary_mathematics', 'global_mmlu_ko_formal_logic', 'global_mmlu_ko_global_facts', 'global_mmlu_ko_high_school_biology', 'global_mmlu_ko_high_school_chemistry', 'global_mmlu_ko_high_school_computer_science', 'global_mmlu_ko_high_school_european_history', 'global_mmlu_ko_high_school_geography', 'global_mmlu_ko_high_school_government_and_politics', 'global_mmlu_ko_high_school_macroeconomics', 'global_mmlu_ko_high_school_mathematics', 'global_mmlu_ko_high_school_microeconomics', 'global_mmlu_ko_high_school_physics', 'global_mmlu_ko_high_school_psychology'], 0-shot 2026-03-05 03:25:45 [INFO] TensorFlow version 2.20.0 available. 2026-03-05 03:25:45 [INFO] TensorFlow version 2.20.0 available. 2026-03-05 03:25:45 [INFO] TensorFlow version 2.20.0 available. 2026-03-05 03:25:45 [INFO] TensorFlow version 2.20.0 available. 2026-03-05 03:25:45 [INFO] TensorFlow version 2.20.0 available. 2026-03-05 03:25:45 [INFO] TensorFlow version 2.20.0 available. 2026-03-05 03:25:48 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2026-03-05 03:25:48 [INFO] Initializing hf model, with arguments: {'pretrained': '/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype': 'bfloat16', 'device': 'cuda:0'} 2026-03-05 03:25:48 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2026-03-05 03:25:48 [INFO] Initializing hf model, with arguments: {'pretrained': '/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype': 'bfloat16', 'device': 'cuda:0'} 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_statistics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_us_history' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_world_history' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_human_aging' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_human_sexuality' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_international_law' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_jurisprudence' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_logical_fallacies' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_machine_learning' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_management' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_marketing' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_medical_genetics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_miscellaneous' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_moral_disputes' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_moral_scenarios' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_nutrition' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_philosophy' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_prehistory' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_accounting' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_law' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_medicine' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_psychology' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_public_relations' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_security_studies' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_sociology' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_us_foreign_policy' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_virology' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_world_religions' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 [LM_EVAL] Starting on cuda:7 (CUDA_VISIBLE_DEVICES=7), tasks=['global_mmlu_ko_high_school_statistics', 'global_mmlu_ko_high_school_us_history', 'global_mmlu_ko_high_school_world_history', 'global_mmlu_ko_human_aging', 'global_mmlu_ko_human_sexuality', 'global_mmlu_ko_international_law', 'global_mmlu_ko_jurisprudence', 'global_mmlu_ko_logical_fallacies', 'global_mmlu_ko_machine_learning', 'global_mmlu_ko_management', 'global_mmlu_ko_marketing', 'global_mmlu_ko_medical_genetics', 'global_mmlu_ko_miscellaneous', 'global_mmlu_ko_moral_disputes', 'global_mmlu_ko_moral_scenarios', 'global_mmlu_ko_nutrition', 'global_mmlu_ko_philosophy', 'global_mmlu_ko_prehistory', 'global_mmlu_ko_professional_accounting', 'global_mmlu_ko_professional_law', 'global_mmlu_ko_professional_medicine', 'global_mmlu_ko_professional_psychology', 'global_mmlu_ko_public_relations', 'global_mmlu_ko_security_studies', 'global_mmlu_ko_sociology', 'global_mmlu_ko_us_foreign_policy', 'global_mmlu_ko_virology', 'global_mmlu_ko_world_religions'], num_fewshot=0 [LM_EVAL] No valid tasks to evaluate. [Phase 2 [0-shot] GPU 7 — MMLU-KO part 2/2] Done. 2026-03-05 03:25:48 [INFO] Initializing hf model, with arguments: {'pretrained': '/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype': 'bfloat16', 'device': 'cuda:0'} 2026-03-05 03:25:48 [INFO] [DONE] [0-shot] GPU 7 — MMLU-KO part 2/2 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_abstract_algebra' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_anatomy' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_astronomy' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_business_ethics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_clinical_knowledge' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_biology' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_chemistry' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_computer_science' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_mathematics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_medicine' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_physics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_computer_security' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_conceptual_physics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_econometrics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_electrical_engineering' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_elementary_mathematics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_formal_logic' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_global_facts' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_biology' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_chemistry' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_computer_science' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_european_history' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_geography' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_government_and_politics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_macroeconomics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_mathematics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_microeconomics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_physics' not found in lm_eval registry — skipping. 2026-03-05 03:25:48 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_psychology' not found in lm_eval registry — skipping. [LM_EVAL] Starting on cuda:6 (CUDA_VISIBLE_DEVICES=6), tasks=['global_mmlu_ko_abstract_algebra', 'global_mmlu_ko_anatomy', 'global_mmlu_ko_astronomy', 'global_mmlu_ko_business_ethics', 'global_mmlu_ko_clinical_knowledge', 'global_mmlu_ko_college_biology', 'global_mmlu_ko_college_chemistry', 'global_mmlu_ko_college_computer_science', 'global_mmlu_ko_college_mathematics', 'global_mmlu_ko_college_medicine', 'global_mmlu_ko_college_physics', 'global_mmlu_ko_computer_security', 'global_mmlu_ko_conceptual_physics', 'global_mmlu_ko_econometrics', 'global_mmlu_ko_electrical_engineering', 'global_mmlu_ko_elementary_mathematics', 'global_mmlu_ko_formal_logic', 'global_mmlu_ko_global_facts', 'global_mmlu_ko_high_school_biology', 'global_mmlu_ko_high_school_chemistry', 'global_mmlu_ko_high_school_computer_science', 'global_mmlu_ko_high_school_european_history', 'global_mmlu_ko_high_school_geography', 'global_mmlu_ko_high_school_government_and_politics', 'global_mmlu_ko_high_school_macroeconomics', 'global_mmlu_ko_high_school_mathematics', 'global_mmlu_ko_high_school_microeconomics', 'global_mmlu_ko_high_school_physics', 'global_mmlu_ko_high_school_psychology'], num_fewshot=0 [LM_EVAL] No valid tasks to evaluate. [Phase 2 [0-shot] GPU 6 — MMLU-KO part 1/2] Done. 2026-03-05 03:25:48 [INFO] [DONE] [0-shot] GPU 6 — MMLU-KO part 1/2 2026-03-05 03:25:48 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2026-03-05 03:25:48 [INFO] Initializing hf model, with arguments: {'pretrained': '/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype': 'bfloat16', 'device': 'cuda:0'} 2026-03-05 03:25:53 [INFO] Using device 'cuda:0' 2026-03-05 03:25:53 [INFO] Using device 'cuda:0' 2026-03-05 03:25:53 [INFO] Using device 'cuda:0' 2026-03-05 03:25:53 [INFO] Using device 'cuda:0' 2026-03-05 03:25:53 [INFO] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} 2026-03-05 03:25:53 [INFO] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} 2026-03-05 03:25:53 [INFO] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} 2026-03-05 03:25:53 [INFO] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} [LM_EVAL] Starting on cuda:2 (CUDA_VISIBLE_DEVICES=2), tasks=['kobest_boolq', 'kobest_copa'], num_fewshot=0 [LM_EVAL] Evaluating 2 task(s) together: ['kobest_boolq', 'kobest_copa'] Loading weights: 0%| | 0/171 [00:00: import of module 'utils' failed """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 762, in run_phase2 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception _pickle.PicklingError: Can't pickle : import of module 'utils' failed 56%|█████▌ | 786/1404 [00:00<00:00, 2026.51it/s] 78%|███████▊ | 1095/1404 [00:00<00:00, 2367.95it/s]2026-03-05 03:26:13 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/data/rare_words-00000-of-00001-4f34314c09209524.parquet "HTTP/1.1 307 Temporary Redirect" 2026-03-05 03:26:13 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/skt/kobest_v1/resolve/a5ea15e3ac77ed694b79f6204eb31889a2ba989f/sentineg/test_originated.jsonl "HTTP/1.1 307 Temporary Redirect" 100%|█████████▉| 1399/1404 [00:00<00:00, 2578.88it/s] 100%|██████████| 1404/1404 [00:00<00:00, 2109.35it/s] 2026-03-05 03:26:13 [INFO] Running loglikelihood requests 2026-03-05 03:26:13 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/datasets/skt/kobest_v1/a5ea15e3ac77ed694b79f6204eb31889a2ba989f/sentineg%2Ftest_originated.jsonl "HTTP/1.1 200 OK" 2026-03-05 03:26:13 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/datasets/skt/kobest_v1/a5ea15e3ac77ed694b79f6204eb31889a2ba989f/sentineg%2Ftest_originated.jsonl "HTTP/1.1 200 OK" 2026-03-05 03:26:13 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH_1.1/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/data/rare_words-00000-of-00001-4f34314c09209524.parquet "HTTP/1.1 302 Found" 2026-03-05 03:26:13 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/skt/kobest_v1/resolve/a5ea15e3ac77ed694b79f6204eb31889a2ba989f/sentineg/validation.jsonl "HTTP/1.1 307 Temporary Redirect" 2026-03-05 03:26:13 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/datasets/skt/kobest_v1/a5ea15e3ac77ed694b79f6204eb31889a2ba989f/sentineg%2Fvalidation.jsonl "HTTP/1.1 200 OK" 2026-03-05 03:26:14 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/datasets/skt/kobest_v1/a5ea15e3ac77ed694b79f6204eb31889a2ba989f/sentineg%2Fvalidation.jsonl "HTTP/1.1 200 OK" Generating train split: 0%| | 0/3649 [00:00: import of module 'utils' failed """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 762, in run_phase2 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception _pickle.PicklingError: Can't pickle : import of module 'utils' failed Running loglikelihood requests: 76%|███████▌ | 3650/4808 [00:05<00:00, 1756.64it/s] Running loglikelihood requests: 81%|████████ | 3906/4808 [00:05<00:00, 1956.62it/s] Running loglikelihood requests: 88%|████████▊ | 4226/4808 [00:05<00:00, 2181.32it/s]2026-03-05 03:26:19 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH_1.1/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/HAE_RAE_BENCH.py "HTTP/1.1 404 Not Found" Running loglikelihood requests: 94%|█████████▍| 4536/4808 [00:05<00:00, 2429.48it/s]2026-03-05 03:26:19 [INFO] HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/HAERAE-HUB/HAE_RAE_BENCH/HAERAE-HUB/HAE_RAE_BENCH.py "HTTP/1.1 404 Not Found" Running loglikelihood requests: 100%|█████████▉| 4802/4808 [00:05<00:00, 2237.16it/s] Running loglikelihood requests: 100%|██████████| 4808/4808 [00:05<00:00, 854.01it/s] 2026-03-05 03:26:20 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/.huggingface.yaml "HTTP/1.1 307 Temporary Redirect" 2026-03-05 03:26:20 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH_1.1/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/.huggingface.yaml "HTTP/1.1 404 Not Found" 2026-03-05 03:26:20 [INFO] HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=HAERAE-HUB/HAE_RAE_BENCH "HTTP/1.1 404 Not Found" 2026-03-05 03:26:20 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/dataset_infos.json "HTTP/1.1 307 Temporary Redirect" 2026-03-05 03:26:21 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH_1.1/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/dataset_infos.json "HTTP/1.1 404 Not Found" Passed argument batch_size = auto:1. Detecting largest batch size Determined largest batch size: 64 [LM_EVAL] Evaluation complete in 32.8s [LM_EVAL] kobest_boolq: acc=0.5014 [LM_EVAL] kobest_copa: acc=0.4940 [Phase 2 [0-shot] GPU 2 — KoBEST: boolq + copa] Done. 2026-03-05 03:26:21 [ERROR] [FAILED] [0-shot] GPU 2 — KoBEST: boolq + copa concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.12/concurrent/futures/process.py", line 219, in _sendback_result result_queue.put(_ResultItem(work_id, result=result, File "/usr/lib/python3.12/multiprocessing/queues.py", line 393, in put obj = _ForkingPickler.dumps(obj) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) _pickle.PicklingError: Can't pickle : import of module 'utils' failed """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 762, in run_phase2 result = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception _pickle.PicklingError: Can't pickle : import of module 'utils' failed 2026-03-05 03:26:21 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/data/history-00000-of-00001-9ad551d475d58413.parquet "HTTP/1.1 307 Temporary Redirect" 2026-03-05 03:26:21 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH_1.1/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/data/history-00000-of-00001-9ad551d475d58413.parquet "HTTP/1.1 302 Found" Generating test split: 0 examples [00:00, ? examples/s] Generating test split: 188 examples [00:00, 112102.52 examples/s] 2026-03-05 03:26:22 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH/resolve/main/README.md "HTTP/1.1 307 Temporary Redirect" 2026-03-05 03:26:22 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH_1.1/resolve/main/README.md "HTTP/1.1 307 Temporary Redirect" Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. 2026-03-05 03:26:22 [WARNING] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. 2026-03-05 03:26:22 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/datasets/HAERAE-HUB/HAE_RAE_BENCH_1.1/b480e81024913f27783d2b05d2f0b10089db19ad/README.md "HTTP/1.1 200 OK" 2026-03-05 03:26:23 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/HAE_RAE_BENCH.py "HTTP/1.1 307 Temporary Redirect" 2026-03-05 03:26:23 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH_1.1/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/HAE_RAE_BENCH.py "HTTP/1.1 404 Not Found" 2026-03-05 03:26:23 [INFO] HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/HAERAE-HUB/HAE_RAE_BENCH/HAERAE-HUB/HAE_RAE_BENCH.py "HTTP/1.1 404 Not Found" 2026-03-05 03:26:23 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/.huggingface.yaml "HTTP/1.1 307 Temporary Redirect" 2026-03-05 03:26:24 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH_1.1/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/.huggingface.yaml "HTTP/1.1 404 Not Found" 2026-03-05 03:26:24 [INFO] HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=HAERAE-HUB/HAE_RAE_BENCH "HTTP/1.1 404 Not Found" 2026-03-05 03:26:24 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/dataset_infos.json "HTTP/1.1 307 Temporary Redirect" 2026-03-05 03:26:24 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH_1.1/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/dataset_infos.json "HTTP/1.1 404 Not Found" 2026-03-05 03:26:25 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/data/general_knowledge-00000-of-00001-7418d1a0b8c4a615.parquet "HTTP/1.1 307 Temporary Redirect" 2026-03-05 03:26:25 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/HAERAE-HUB/HAE_RAE_BENCH_1.1/resolve/b480e81024913f27783d2b05d2f0b10089db19ad/data/general_knowledge-00000-of-00001-7418d1a0b8c4a615.parquet "HTTP/1.1 302 Found" Generating test split: 0 examples [00:00, ? examples/s] Generating test split: 176 examples [00:00, 106291.94 examples/s] 2026-03-05 03:26:25 [INFO] Selected tasks: 2026-03-05 03:26:25 [INFO] Group: haerae 2026-03-05 03:26:25 [INFO] ConfigurableGroup(group=haerae,group_alias=None): {'haerae_general_knowledge': ConfigurableTask(task_name=haerae_general_knowledge,output_type=multiple_choice,num_fewshot=None,num_samples=176), 'haerae_history': ConfigurableTask(task_name=haerae_history,output_type=multiple_choice,num_fewshot=None,num_samples=188), 'haerae_loan_word': ConfigurableTask(task_name=haerae_loan_word,output_type=multiple_choice,num_fewshot=None,num_samples=169), 'haerae_rare_word': ConfigurableTask(task_name=haerae_rare_word,output_type=multiple_choice,num_fewshot=None,num_samples=405), 'haerae_standard_nomenclature': ConfigurableTask(task_name=haerae_standard_nomenclature,output_type=multiple_choice,num_fewshot=None,num_samples=153)} 2026-03-05 03:26:25 [WARNING] Overwriting default num_fewshot of haerae_general_knowledge from None to 0 2026-03-05 03:26:25 [WARNING] Overwriting default num_fewshot of haerae_history from None to 0 2026-03-05 03:26:25 [WARNING] Overwriting default num_fewshot of haerae_loan_word from None to 0 2026-03-05 03:26:25 [WARNING] Overwriting default num_fewshot of haerae_rare_word from None to 0 2026-03-05 03:26:25 [WARNING] Overwriting default num_fewshot of haerae_standard_nomenclature from None to 0 2026-03-05 03:26:25 [INFO] Building contexts for haerae_general_knowledge on rank 0... 0%| | 0/176 [00:00: import of module 'utils' failed """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 796, in run_phase2 r = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception _pickle.PicklingError: Can't pickle : import of module 'utils' failed Running loglikelihood requests: 60%|█████▉ | 1665/2794 [00:08<00:03, 295.08it/s] Running loglikelihood requests: 62%|██████▏ | 1729/2794 [00:08<00:03, 297.70it/s] Running loglikelihood requests: 4%|▍ | 193/4808 [00:04<01:04, 71.54it/s] Running loglikelihood requests: 64%|██████▍ | 1793/2794 [00:08<00:03, 300.01it/s] Running loglikelihood requests: 66%|██████▋ | 1857/2794 [00:08<00:03, 301.92it/s] Running loglikelihood requests: 5%|▌ | 257/4808 [00:04<00:49, 91.81it/s] Running loglikelihood requests: 69%|██████▉ | 1921/2794 [00:09<00:02, 306.34it/s] Running loglikelihood requests: 71%|███████ | 1985/2794 [00:09<00:02, 312.16it/s] Running loglikelihood requests: 7%|▋ | 321/4808 [00:05<00:41, 109.12it/s] Running loglikelihood requests: 80%|███████▉ | 2225/2794 [00:09<00:00, 593.61it/s] Running loglikelihood requests: 89%|████████▉ | 2481/2794 [00:09<00:00, 839.87it/s] Running loglikelihood requests: 8%|▊ | 385/4808 [00:05<00:35, 123.21it/s] Running loglikelihood requests: 98%|█████████▊| 2737/2794 [00:09<00:00, 1077.41it/s] Running loglikelihood requests: 100%|██████████| 2794/2794 [00:09<00:00, 286.78it/s] Running loglikelihood requests: 9%|▉ | 449/4808 [00:05<00:32, 135.03it/s] Running loglikelihood requests: 11%|█ | 513/4808 [00:06<00:29, 144.58it/s] Running loglikelihood requests: 12%|█▏ | 577/4808 [00:06<00:27, 152.03it/s]Passed argument batch_size = auto:1. Detecting largest batch size Determined largest batch size: 64 [LM_EVAL] Evaluation complete in 33.9s [LM_EVAL] kobest_hellaswag: acc=0.2160, acc_norm=0.2180 [LM_EVAL] kobest_sentineg: acc=0.4987 [Phase 2 [5-shot] GPU 3 — KoBEST: hellaswag + sentineg] Done. 2026-03-05 03:27:07 [ERROR] [FAILED] [5-shot] GPU 3 — KoBEST: hellaswag + sentineg concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.12/concurrent/futures/process.py", line 219, in _sendback_result result_queue.put(_ResultItem(work_id, result=result, File "/usr/lib/python3.12/multiprocessing/queues.py", line 393, in put obj = _ForkingPickler.dumps(obj) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) _pickle.PicklingError: Can't pickle : import of module 'utils' failed """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 796, in run_phase2 r = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception _pickle.PicklingError: Can't pickle : import of module 'utils' failed Running loglikelihood requests: 13%|█▎ | 641/4808 [00:07<00:26, 157.67it/s] Running loglikelihood requests: 0%| | 1/5455 [00:03<5:47:47, 3.83s/it] Running loglikelihood requests: 15%|█▍ | 705/4808 [00:07<00:25, 162.50it/s] Running loglikelihood requests: 1%| | 65/5455 [00:04<04:21, 20.62it/s] Running loglikelihood requests: 16%|█▌ | 769/4808 [00:07<00:24, 166.40it/s] Running loglikelihood requests: 17%|█▋ | 833/4808 [00:08<00:23, 169.36it/s] Running loglikelihood requests: 2%|▏ | 129/5455 [00:04<02:08, 41.59it/s] Running loglikelihood requests: 19%|█▊ | 897/4808 [00:08<00:22, 171.19it/s] Running loglikelihood requests: 4%|▎ | 193/5455 [00:05<01:24, 62.33it/s] Running loglikelihood requests: 20%|█▉ | 961/4808 [00:08<00:22, 173.76it/s] Running loglikelihood requests: 5%|▍ | 257/5455 [00:05<01:03, 82.09it/s] Running loglikelihood requests: 21%|██▏ | 1025/4808 [00:09<00:21, 176.03it/s] Running loglikelihood requests: 6%|▌ | 321/5455 [00:05<00:51, 99.95it/s] Running loglikelihood requests: 23%|██▎ | 1089/4808 [00:09<00:20, 177.57it/s] Running loglikelihood requests: 7%|▋ | 385/5455 [00:06<00:43, 115.49it/s] Running loglikelihood requests: 24%|██▍ | 1153/4808 [00:09<00:20, 178.87it/s] Running loglikelihood requests: 8%|▊ | 449/5455 [00:06<00:38, 129.04it/s] Running loglikelihood requests: 25%|██▌ | 1217/4808 [00:10<00:19, 180.23it/s] Running loglikelihood requests: 9%|▉ | 513/5455 [00:07<00:35, 140.70it/s] Running loglikelihood requests: 27%|██▋ | 1281/4808 [00:10<00:19, 181.47it/s] Running loglikelihood requests: 11%|█ | 577/5455 [00:07<00:32, 149.98it/s] Running loglikelihood requests: 28%|██▊ | 1345/4808 [00:10<00:18, 182.59it/s] Running loglikelihood requests: 12%|█▏ | 641/5455 [00:07<00:30, 158.17it/s] Running loglikelihood requests: 29%|██▉ | 1409/4808 [00:11<00:18, 183.94it/s] Running loglikelihood requests: 13%|█▎ | 705/5455 [00:08<00:28, 164.94it/s] Running loglikelihood requests: 31%|███ | 1473/4808 [00:11<00:18, 185.09it/s] Running loglikelihood requests: 14%|█▍ | 769/5455 [00:08<00:27, 170.94it/s] Running loglikelihood requests: 32%|███▏ | 1537/4808 [00:11<00:17, 186.57it/s] Running loglikelihood requests: 15%|█▌ | 833/5455 [00:08<00:26, 176.12it/s] Running loglikelihood requests: 33%|███▎ | 1601/4808 [00:12<00:17, 187.86it/s] Running loglikelihood requests: 16%|█▋ | 897/5455 [00:09<00:25, 180.14it/s] Running loglikelihood requests: 35%|███▍ | 1665/4808 [00:12<00:16, 189.21it/s] Running loglikelihood requests: 18%|█▊ | 961/5455 [00:09<00:24, 183.08it/s] Running loglikelihood requests: 36%|███▌ | 1729/4808 [00:12<00:16, 190.75it/s] Running loglikelihood requests: 19%|█▉ | 1025/5455 [00:09<00:23, 185.40it/s] Running loglikelihood requests: 37%|███▋ | 1793/4808 [00:13<00:15, 192.00it/s] Running loglikelihood requests: 20%|█▉ | 1089/5455 [00:10<00:23, 186.79it/s] Running loglikelihood requests: 39%|███▊ | 1857/4808 [00:13<00:15, 193.08it/s] Running loglikelihood requests: 21%|██ | 1153/5455 [00:10<00:22, 188.65it/s] Running loglikelihood requests: 40%|███▉ | 1921/4808 [00:13<00:14, 194.15it/s] Running loglikelihood requests: 22%|██▏ | 1217/5455 [00:10<00:22, 189.46it/s] Running loglikelihood requests: 41%|████▏ | 1985/4808 [00:14<00:14, 195.46it/s] Running loglikelihood requests: 23%|██▎ | 1281/5455 [00:11<00:21, 190.60it/s] Running loglikelihood requests: 43%|████▎ | 2049/4808 [00:14<00:14, 195.08it/s] Running loglikelihood requests: 25%|██▍ | 1345/5455 [00:11<00:21, 191.96it/s] Running loglikelihood requests: 44%|████▍ | 2113/4808 [00:14<00:13, 195.55it/s] Running loglikelihood requests: 26%|██▌ | 1409/5455 [00:11<00:21, 192.29it/s] Running loglikelihood requests: 45%|████▌ | 2177/4808 [00:15<00:13, 197.36it/s] Running loglikelihood requests: 47%|████▋ | 2241/4808 [00:15<00:12, 198.18it/s] Running loglikelihood requests: 27%|██▋ | 1473/5455 [00:12<00:20, 192.66it/s] Running loglikelihood requests: 48%|████▊ | 2305/4808 [00:15<00:12, 199.11it/s] Running loglikelihood requests: 28%|██▊ | 1537/5455 [00:12<00:20, 193.05it/s] Running loglikelihood requests: 49%|████▉ | 2369/4808 [00:16<00:12, 200.30it/s] Running loglikelihood requests: 29%|██▉ | 1601/5455 [00:12<00:19, 193.81it/s] Running loglikelihood requests: 51%|█████ | 2433/4808 [00:16<00:11, 203.73it/s] Running loglikelihood requests: 31%|███ | 1665/5455 [00:13<00:19, 194.57it/s] Running loglikelihood requests: 52%|█████▏ | 2497/4808 [00:16<00:11, 206.49it/s] Running loglikelihood requests: 32%|███▏ | 1729/5455 [00:13<00:19, 195.05it/s] Running loglikelihood requests: 53%|█████▎ | 2561/4808 [00:17<00:10, 208.41it/s] Running loglikelihood requests: 33%|███▎ | 1793/5455 [00:13<00:18, 195.54it/s] Running loglikelihood requests: 55%|█████▍ | 2625/4808 [00:17<00:10, 211.64it/s] Running loglikelihood requests: 34%|███▍ | 1857/5455 [00:14<00:18, 196.11it/s] Running loglikelihood requests: 56%|█████▌ | 2689/4808 [00:17<00:09, 214.92it/s] Running loglikelihood requests: 35%|███▌ | 1921/5455 [00:14<00:18, 196.30it/s] Running loglikelihood requests: 57%|█████▋ | 2753/4808 [00:17<00:09, 220.11it/s] Running loglikelihood requests: 60%|█████▉ | 2881/4808 [00:18<00:05, 322.54it/s] Running loglikelihood requests: 36%|███▋ | 1985/5455 [00:14<00:17, 201.04it/s] Running loglikelihood requests: 63%|██████▎ | 3009/4808 [00:18<00:04, 414.30it/s] Running loglikelihood requests: 38%|███▊ | 2049/5455 [00:15<00:17, 200.04it/s] Running loglikelihood requests: 65%|██████▌ | 3137/4808 [00:18<00:03, 494.08it/s] Running loglikelihood requests: 68%|██████▊ | 3265/4808 [00:18<00:02, 559.09it/s] Running loglikelihood requests: 71%|███████ | 3393/4808 [00:18<00:02, 632.46it/s] Running loglikelihood requests: 39%|███▊ | 2113/5455 [00:15<00:16, 199.40it/s] Running loglikelihood requests: 73%|███████▎ | 3521/4808 [00:18<00:01, 696.55it/s] Running loglikelihood requests: 76%|███████▌ | 3649/4808 [00:19<00:01, 747.96it/s] Running loglikelihood requests: 40%|███▉ | 2177/5455 [00:15<00:16, 198.95it/s] Running loglikelihood requests: 79%|███████▊ | 3777/4808 [00:19<00:01, 760.20it/s] Running loglikelihood requests: 81%|████████ | 3905/4808 [00:19<00:01, 801.19it/s] Running loglikelihood requests: 41%|████ | 2241/5455 [00:16<00:16, 198.58it/s] Running loglikelihood requests: 84%|████████▍ | 4033/4808 [00:19<00:00, 832.37it/s] Running loglikelihood requests: 87%|████████▋ | 4161/4808 [00:19<00:00, 860.75it/s] Running loglikelihood requests: 42%|████▏ | 2305/5455 [00:16<00:15, 198.57it/s] Running loglikelihood requests: 89%|████████▉ | 4289/4808 [00:19<00:00, 881.61it/s] Running loglikelihood requests: 92%|█████████▏| 4417/4808 [00:19<00:00, 898.29it/s] Running loglikelihood requests: 95%|█████████▍| 4545/4808 [00:20<00:00, 874.75it/s] Running loglikelihood requests: 43%|████▎ | 2369/5455 [00:16<00:15, 197.58it/s] Running loglikelihood requests: 97%|█████████▋| 4673/4808 [00:20<00:00, 866.78it/s] Running loglikelihood requests: 100%|█████████▉| 4801/4808 [00:20<00:00, 924.35it/s] Running loglikelihood requests: 100%|██████████| 4808/4808 [00:20<00:00, 236.33it/s] Running loglikelihood requests: 45%|████▍ | 2433/5455 [00:16<00:15, 198.01it/s] Running loglikelihood requests: 46%|████▌ | 2497/5455 [00:17<00:14, 197.68it/s] Running loglikelihood requests: 47%|████▋ | 2561/5455 [00:17<00:14, 197.89it/s] Running loglikelihood requests: 48%|████▊ | 2625/5455 [00:17<00:14, 198.45it/s] Running loglikelihood requests: 49%|████▉ | 2689/5455 [00:18<00:13, 199.30it/s]Passed argument batch_size = auto:1. Detecting largest batch size Determined largest batch size: 64 [LM_EVAL] Evaluation complete in 48.9s [LM_EVAL] kobest_boolq: acc=0.5021 [LM_EVAL] kobest_copa: acc=0.4750 [Phase 2 [5-shot] GPU 2 — KoBEST: boolq + copa] Done. 2026-03-05 03:27:22 [ERROR] [FAILED] [5-shot] GPU 2 — KoBEST: boolq + copa concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.12/concurrent/futures/process.py", line 219, in _sendback_result result_queue.put(_ResultItem(work_id, result=result, File "/usr/lib/python3.12/multiprocessing/queues.py", line 393, in put obj = _ForkingPickler.dumps(obj) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) _pickle.PicklingError: Can't pickle : import of module 'utils' failed """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 796, in run_phase2 r = fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception _pickle.PicklingError: Can't pickle : import of module 'utils' failed Running loglikelihood requests: 50%|█████ | 2753/5455 [00:18<00:13, 199.41it/s] Running loglikelihood requests: 52%|█████▏ | 2817/5455 [00:18<00:13, 199.63it/s] Running loglikelihood requests: 53%|█████▎ | 2881/5455 [00:19<00:12, 199.80it/s] Running loglikelihood requests: 54%|█████▍ | 2945/5455 [00:19<00:12, 205.34it/s] Running loglikelihood requests: 55%|█████▌ | 3009/5455 [00:19<00:11, 204.43it/s] Running loglikelihood requests: 56%|█████▋ | 3073/5455 [00:20<00:11, 208.61it/s] Running loglikelihood requests: 58%|█████▊ | 3137/5455 [00:20<00:11, 206.50it/s] Running loglikelihood requests: 59%|█████▊ | 3201/5455 [00:20<00:10, 205.71it/s] Running loglikelihood requests: 60%|█████▉ | 3265/5455 [00:21<00:10, 204.99it/s] Running loglikelihood requests: 61%|██████ | 3329/5455 [00:21<00:10, 204.37it/s] Running loglikelihood requests: 62%|██████▏ | 3393/5455 [00:21<00:10, 204.01it/s] Running loglikelihood requests: 63%|██████▎ | 3457/5455 [00:22<00:09, 204.26it/s] Running loglikelihood requests: 65%|██████▍ | 3521/5455 [00:22<00:09, 204.58it/s] Running loglikelihood requests: 66%|██████▌ | 3585/5455 [00:22<00:09, 204.49it/s] Running loglikelihood requests: 67%|██████▋ | 3649/5455 [00:22<00:08, 205.01it/s] Running loglikelihood requests: 68%|██████▊ | 3713/5455 [00:23<00:08, 204.95it/s] Running loglikelihood requests: 69%|██████▉ | 3777/5455 [00:23<00:08, 204.99it/s] Running loglikelihood requests: 70%|███████ | 3841/5455 [00:23<00:07, 207.19it/s] Running loglikelihood requests: 72%|███████▏ | 3905/5455 [00:24<00:07, 209.70it/s] Running loglikelihood requests: 73%|███████▎ | 3969/5455 [00:24<00:07, 211.63it/s] Running loglikelihood requests: 74%|███████▍ | 4033/5455 [00:24<00:06, 212.83it/s] Running loglikelihood requests: 75%|███████▌ | 4097/5455 [00:25<00:06, 213.62it/s] Running loglikelihood requests: 76%|███████▋ | 4161/5455 [00:25<00:06, 214.37it/s] Running loglikelihood requests: 77%|███████▋ | 4225/5455 [00:25<00:05, 214.75it/s] Running loglikelihood requests: 79%|███████▊ | 4289/5455 [00:25<00:05, 220.21it/s] Running loglikelihood requests: 80%|███████▉ | 4353/5455 [00:26<00:05, 218.84it/s] Running loglikelihood requests: 81%|████████ | 4417/5455 [00:26<00:04, 218.59it/s] Running loglikelihood requests: 82%|████████▏ | 4481/5455 [00:26<00:04, 218.41it/s] Running loglikelihood requests: 83%|████████▎ | 4545/5455 [00:27<00:04, 219.14it/s] Running loglikelihood requests: 84%|████████▍ | 4609/5455 [00:27<00:03, 220.12it/s] Running loglikelihood requests: 86%|████████▌ | 4673/5455 [00:27<00:03, 220.65it/s] Running loglikelihood requests: 87%|████████▋ | 4737/5455 [00:27<00:03, 221.59it/s] Running loglikelihood requests: 88%|████████▊ | 4801/5455 [00:28<00:02, 222.32it/s] Running loglikelihood requests: 89%|████████▉ | 4865/5455 [00:28<00:02, 222.68it/s] Running loglikelihood requests: 90%|█████████ | 4929/5455 [00:28<00:02, 223.24it/s] Running loglikelihood requests: 92%|█████████▏| 4993/5455 [00:29<00:02, 224.16it/s] Running loglikelihood requests: 93%|█████████▎| 5057/5455 [00:29<00:01, 230.65it/s] Running loglikelihood requests: 94%|█████████▍| 5121/5455 [00:29<00:01, 229.04it/s] Running loglikelihood requests: 95%|█████████▌| 5185/5455 [00:29<00:01, 227.73it/s] Running loglikelihood requests: 96%|█████████▌| 5249/5455 [00:30<00:00, 227.40it/s] Running loglikelihood requests: 97%|█████████▋| 5313/5455 [00:30<00:00, 227.10it/s] Running loglikelihood requests: 99%|█████████▊| 5377/5455 [00:30<00:00, 226.40it/s] Running loglikelihood requests: 100%|██████████| 5455/5455 [00:30<00:00, 176.62it/s] Passed argument batch_size = auto:1. Detecting largest batch size Determined largest batch size: 64 [LM_EVAL] Evaluation complete in 62.6s [LM_EVAL] haerae: acc=0.2081, acc_norm=0.2081 [Phase 2 [5-shot] GPU 5 — HAE-RAE (all subtasks)] Done. 2026-03-05 03:27:37 [INFO] [DONE] [5-shot] GPU 5 — HAE-RAE (all subtasks) 2026-03-05 03:27:39 [INFO] Phase 2 (5-shot) complete: 3 succeeded, 3 failed 2026-03-05 03:27:39 [INFO] Phase 2 results saved: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0323/phase2_results.json 2026-03-05 03:27:39 [INFO] Phase 2 complete in 1m 53s. 2026-03-05 03:27:39 [INFO] 2026-03-05 03:27:39 [INFO] ------------------------------------------------------------------------ 2026-03-05 03:27:39 [INFO] PHASE 3 — Report Generation 2026-03-05 03:27:39 [INFO] ------------------------------------------------------------------------ 2026-03-05 03:27:39 [INFO] Report saved: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0323/full_eval_report.md 2026-03-05 03:27:39 [INFO] Phase 3 complete in 0s. 2026-03-05 03:27:39 [INFO] ======================================================================== 2026-03-05 03:27:39 [INFO] PIPELINE COMPLETE 2026-03-05 03:27:39 [INFO] ======================================================================== 2026-03-05 03:27:39 [INFO] Total time : 4m 0s 2026-03-05 03:27:39 [INFO] Output dir : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0323 2026-03-05 03:27:39 [INFO] Phase 1 results : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0323/phase1_results.json 2026-03-05 03:27:39 [INFO] Phase 2 results : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0323/phase2_results.json 2026-03-05 03:27:39 [INFO] Gen samples : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0323/generation_samples.json 2026-03-05 03:27:39 [INFO] Report : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0323/full_eval_report.md 2026-03-05 03:27:39 [INFO] Phase 1 tasks : 1 OK / 5 failed 2026-03-05 03:27:39 [INFO] Phase 2 tasks : 2 OK / 4 failed 2026-03-05 03:27:39 [INFO] ======================================================================== /usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 64 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '