657 lines
64 KiB
Plaintext
657 lines
64 KiB
Plaintext
2026-03-05 03:18:13 [INFO] ========================================================================
|
||
2026-03-05 03:18:13 [INFO] FRANKENSTALLM 3B — Full Evaluation Pipeline
|
||
2026-03-05 03:18:13 [INFO] ========================================================================
|
||
2026-03-05 03:18:13 [INFO] Checkpoint : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/checkpoints/korean_3b_fp8_run1/checkpoint-0057000
|
||
2026-03-05 03:18:13 [INFO] Tokenizer : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/tokenizer/korean_sp/tokenizer.json
|
||
2026-03-05 03:18:13 [INFO] Data dir : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/data
|
||
2026-03-05 03:18:13 [INFO] Output dir : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318
|
||
2026-03-05 03:18:13 [INFO] GPUs : [2, 3, 4, 5, 6, 7]
|
||
2026-03-05 03:18:13 [INFO] SEQ_LEN : 2048 STRIDE: 512 BATCH_SIZE: 32
|
||
2026-03-05 03:18:13 [INFO] Phases : phase0=run phase1=run phase2=run
|
||
2026-03-05 03:18:13 [INFO]
|
||
2026-03-05 03:18:13 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:18:13 [INFO] PHASE 0 — HF Checkpoint Conversion
|
||
2026-03-05 03:18:13 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:18:13 [INFO] Running: /usr/bin/python /PROJECT/0325120031_A/ghong/taketimes/llm-bang/scripts/convert_to_hf.py --checkpoint /PROJECT/0325120031_A/ghong/taketimes/llm-bang/checkpoints/korean_3b_fp8_run1/checkpoint-0057000 --output /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000 --tokenizer /PROJECT/0325120031_A/ghong/taketimes/llm-bang/tokenizer/korean_sp/tokenizer.json
|
||
/usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922
|
||
dispatch key: ADInplaceOrView
|
||
previous kernel: no debug info
|
||
new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
self.m.impl(
|
||
Checkpoint : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/checkpoints/korean_3b_fp8_run1/checkpoint-0057000
|
||
Output : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000
|
||
Model : d_model=3072, n_layers=28, vocab_size=64000, use_fp8=True
|
||
Loading model.pt ...
|
||
Source keys: 255
|
||
Remapping weight names ...
|
||
Destination keys: 171
|
||
Saving model.safetensors ...
|
||
Saved config.json
|
||
Copied tokenizer: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/tokenizer/korean_sp/tokenizer.json -> /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000/tokenizer.json
|
||
|
||
Done! HF model saved to: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000
|
||
Verify: ls -lh /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000
|
||
2026-03-05 03:18:30 [INFO] HF checkpoint saved to: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000
|
||
2026-03-05 03:18:30 [INFO] Phase 0 complete in 17s.
|
||
2026-03-05 03:18:30 [INFO]
|
||
2026-03-05 03:18:30 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:18:30 [INFO] PHASE 1 — Internal Evaluation — 6 GPU Parallel
|
||
2026-03-05 03:18:30 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:18:30 [INFO] Submitted: GPU 5 — Calibration + Token NLL
|
||
2026-03-05 03:18:30 [INFO] Submitted: GPU 6 — Generation (15 prompts × 4 temps)
|
||
2026-03-05 03:18:30 [INFO] Submitted: GPU 7 — Repetition grid (12 × 5)
|
||
2026-03-05 03:18:30 [INFO] Submitted: GPU 2 — PPL: 3b_val.bin
|
||
2026-03-05 03:18:30 [INFO] Submitted: GPU 3 — PPL: korean_c4 + korean_val
|
||
2026-03-05 03:18:30 [INFO] Submitted: GPU 4 — PPL: hplt_ko + cc100_ko + PPL: 7 cosmo files + PPL: 7 remaining files
|
||
/usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922
|
||
dispatch key: ADInplaceOrView
|
||
previous kernel: no debug info
|
||
new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
self.m.impl(
|
||
/usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922
|
||
dispatch key: ADInplaceOrView
|
||
previous kernel: no debug info
|
||
new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
self.m.impl(
|
||
/usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922
|
||
dispatch key: ADInplaceOrView
|
||
previous kernel: no debug info
|
||
new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
self.m.impl(
|
||
/usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922
|
||
dispatch key: ADInplaceOrView
|
||
previous kernel: no debug info
|
||
new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
self.m.impl(
|
||
/usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922
|
||
dispatch key: ADInplaceOrView
|
||
previous kernel: no debug info
|
||
new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
self.m.impl(
|
||
/usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922
|
||
dispatch key: ADInplaceOrView
|
||
previous kernel: no debug info
|
||
new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
self.m.impl(
|
||
[PPL cuda:2] Loading model for 3b...
|
||
[PPL cuda:2] 3b: 75,681,623 tokens, 151.4 MB
|
||
[PPL_MULTI cuda:3] Loading model once for 2 files...
|
||
[PPL cuda:3] korean_c4: 15,159,838 tokens, 30.3 MB
|
||
[CALIB cuda:5] Loading model...
|
||
[CALIB cuda:5] Using 50,000 tokens from 3b_val.bin
|
||
[PPL_MULTI cuda:4] Loading model once for 16 files...
|
||
[PPL cuda:4] hplt_ko: 16,165,735 tokens, 32.3 MB
|
||
[DCTN-0301014756:3083641:0:3084057] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
|
||
2026-03-05 03:18:48 [ERROR] [FAILED] GPU 5 — Calibration + Token NLL
|
||
Traceback (most recent call last):
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
|
||
|
||
2026-03-05 03:18:48 [ERROR] [FAILED] GPU 6 — Generation (15 prompts × 4 temps)
|
||
Traceback (most recent call last):
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
|
||
|
||
2026-03-05 03:18:48 [ERROR] [FAILED] GPU 7 — Repetition grid (12 × 5)
|
||
Traceback (most recent call last):
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
|
||
|
||
2026-03-05 03:18:48 [ERROR] [FAILED] GPU 2 — PPL: 3b_val.bin
|
||
Traceback (most recent call last):
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
|
||
|
||
2026-03-05 03:18:48 [ERROR] [FAILED] GPU 3 — PPL: korean_c4 + korean_val
|
||
Traceback (most recent call last):
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
|
||
|
||
2026-03-05 03:18:48 [ERROR] [FAILED] GPU 4 — PPL: hplt_ko + cc100_ko + PPL: 7 cosmo files + PPL: 7 remaining files
|
||
Traceback (most recent call last):
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py", line 515, in run_phase1
|
||
result = fut.result()
|
||
^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||
return self.__get_result()
|
||
^^^^^^^^^^^^^^^^^^^
|
||
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||
raise self._exception
|
||
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
|
||
|
||
2026-03-05 03:18:50 [INFO] Phase 1 complete: 0 succeeded, 6 failed
|
||
2026-03-05 03:18:50 [INFO] Phase 1 results saved: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/phase1_results.json
|
||
2026-03-05 03:18:50 [INFO] Phase 1 complete in 19s.
|
||
2026-03-05 03:18:50 [INFO]
|
||
2026-03-05 03:18:50 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:18:50 [INFO] PHASE 2 — lm-eval Benchmarks — 6 GPU Parallel
|
||
2026-03-05 03:18:50 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:18:50 [INFO] Running 0-shot benchmarks on 8 GPUs ...
|
||
2026-03-05 03:18:50 [INFO] Submitted: [0-shot] GPU 2 — KoBEST: boolq + copa
|
||
2026-03-05 03:18:50 [INFO] Submitted: [0-shot] GPU 3 — KoBEST: hellaswag + sentineg
|
||
2026-03-05 03:18:50 [INFO] Submitted: [0-shot] GPU 4 — KoBEST: wic
|
||
2026-03-05 03:18:50 [INFO] Submitted: [0-shot] GPU 5 — HAE-RAE (all subtasks)
|
||
2026-03-05 03:18:50 [INFO] Submitted: [0-shot] GPU 6 — MMLU-KO part 1/2
|
||
2026-03-05 03:18:50 [INFO] Submitted: [0-shot] GPU 7 — MMLU-KO part 2/2
|
||
[Phase 2 [0-shot] GPU 2 — KoBEST: boolq + copa] Starting on cuda:2 — tasks: ['kobest_boolq', 'kobest_copa'], 0-shot
|
||
[Phase 2 [0-shot] GPU 3 — KoBEST: hellaswag + sentineg] Starting on cuda:3 — tasks: ['kobest_hellaswag', 'kobest_sentineg'], 0-shot
|
||
[Phase 2 [0-shot] GPU 4 — KoBEST: wic] Starting on cuda:4 — tasks: ['kobest_wic'], 0-shot
|
||
[Phase 2 [0-shot] GPU 6 — MMLU-KO part 1/2] Starting on cuda:6 — tasks: ['global_mmlu_ko_abstract_algebra', 'global_mmlu_ko_anatomy', 'global_mmlu_ko_astronomy', 'global_mmlu_ko_business_ethics', 'global_mmlu_ko_clinical_knowledge', 'global_mmlu_ko_college_biology', 'global_mmlu_ko_college_chemistry', 'global_mmlu_ko_college_computer_science', 'global_mmlu_ko_college_mathematics', 'global_mmlu_ko_college_medicine', 'global_mmlu_ko_college_physics', 'global_mmlu_ko_computer_security', 'global_mmlu_ko_conceptual_physics', 'global_mmlu_ko_econometrics', 'global_mmlu_ko_electrical_engineering', 'global_mmlu_ko_elementary_mathematics', 'global_mmlu_ko_formal_logic', 'global_mmlu_ko_global_facts', 'global_mmlu_ko_high_school_biology', 'global_mmlu_ko_high_school_chemistry', 'global_mmlu_ko_high_school_computer_science', 'global_mmlu_ko_high_school_european_history', 'global_mmlu_ko_high_school_geography', 'global_mmlu_ko_high_school_government_and_politics', 'global_mmlu_ko_high_school_macroeconomics', 'global_mmlu_ko_high_school_mathematics', 'global_mmlu_ko_high_school_microeconomics', 'global_mmlu_ko_high_school_physics', 'global_mmlu_ko_high_school_psychology'], 0-shot
|
||
[Phase 2 [0-shot] GPU 5 — HAE-RAE (all subtasks)] Starting on cuda:5 — tasks: ['haerae'], 0-shot
|
||
[Phase 2 [0-shot] GPU 7 — MMLU-KO part 2/2] Starting on cuda:7 — tasks: ['global_mmlu_ko_high_school_statistics', 'global_mmlu_ko_high_school_us_history', 'global_mmlu_ko_high_school_world_history', 'global_mmlu_ko_human_aging', 'global_mmlu_ko_human_sexuality', 'global_mmlu_ko_international_law', 'global_mmlu_ko_jurisprudence', 'global_mmlu_ko_logical_fallacies', 'global_mmlu_ko_machine_learning', 'global_mmlu_ko_management', 'global_mmlu_ko_marketing', 'global_mmlu_ko_medical_genetics', 'global_mmlu_ko_miscellaneous', 'global_mmlu_ko_moral_disputes', 'global_mmlu_ko_moral_scenarios', 'global_mmlu_ko_nutrition', 'global_mmlu_ko_philosophy', 'global_mmlu_ko_prehistory', 'global_mmlu_ko_professional_accounting', 'global_mmlu_ko_professional_law', 'global_mmlu_ko_professional_medicine', 'global_mmlu_ko_professional_psychology', 'global_mmlu_ko_public_relations', 'global_mmlu_ko_security_studies', 'global_mmlu_ko_sociology', 'global_mmlu_ko_us_foreign_policy', 'global_mmlu_ko_virology', 'global_mmlu_ko_world_religions'], 0-shot
|
||
2026-03-05 03:18:50 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:50 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:50 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:50 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:50 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:50 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:53 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:53 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:53 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:53 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:53 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_statistics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_us_history' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_world_history' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_human_aging' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_human_sexuality' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_international_law' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_jurisprudence' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_logical_fallacies' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_machine_learning' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_management' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_marketing' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_medical_genetics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_miscellaneous' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_moral_disputes' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_moral_scenarios' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_nutrition' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_philosophy' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_prehistory' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_accounting' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_law' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_medicine' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_psychology' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_public_relations' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_security_studies' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_sociology' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_us_foreign_policy' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_virology' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_world_religions' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_abstract_algebra' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_anatomy' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_astronomy' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_business_ethics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_clinical_knowledge' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_biology' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_chemistry' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_computer_science' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_mathematics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_medicine' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_physics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_computer_security' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_conceptual_physics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_econometrics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_electrical_engineering' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_elementary_mathematics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_formal_logic' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_global_facts' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_biology' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_chemistry' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_computer_science' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_european_history' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_geography' not found in lm_eval registry — skipping.
|
||
[LM_EVAL] Starting on cuda:7 (CUDA_VISIBLE_DEVICES=7), tasks=['global_mmlu_ko_high_school_statistics', 'global_mmlu_ko_high_school_us_history', 'global_mmlu_ko_high_school_world_history', 'global_mmlu_ko_human_aging', 'global_mmlu_ko_human_sexuality', 'global_mmlu_ko_international_law', 'global_mmlu_ko_jurisprudence', 'global_mmlu_ko_logical_fallacies', 'global_mmlu_ko_machine_learning', 'global_mmlu_ko_management', 'global_mmlu_ko_marketing', 'global_mmlu_ko_medical_genetics', 'global_mmlu_ko_miscellaneous', 'global_mmlu_ko_moral_disputes', 'global_mmlu_ko_moral_scenarios', 'global_mmlu_ko_nutrition', 'global_mmlu_ko_philosophy', 'global_mmlu_ko_prehistory', 'global_mmlu_ko_professional_accounting', 'global_mmlu_ko_professional_law', 'global_mmlu_ko_professional_medicine', 'global_mmlu_ko_professional_psychology', 'global_mmlu_ko_public_relations', 'global_mmlu_ko_security_studies', 'global_mmlu_ko_sociology', 'global_mmlu_ko_us_foreign_policy', 'global_mmlu_ko_virology', 'global_mmlu_ko_world_religions'], num_fewshot=0
|
||
[LM_EVAL] No valid tasks to evaluate.
|
||
[Phase 2 [0-shot] GPU 7 — MMLU-KO part 2/2] Done.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_government_and_politics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_macroeconomics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_mathematics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_microeconomics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_physics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_psychology' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:53 [INFO] [DONE] [0-shot] GPU 7 — MMLU-KO part 2/2
|
||
[LM_EVAL] Starting on cuda:6 (CUDA_VISIBLE_DEVICES=6), tasks=['global_mmlu_ko_abstract_algebra', 'global_mmlu_ko_anatomy', 'global_mmlu_ko_astronomy', 'global_mmlu_ko_business_ethics', 'global_mmlu_ko_clinical_knowledge', 'global_mmlu_ko_college_biology', 'global_mmlu_ko_college_chemistry', 'global_mmlu_ko_college_computer_science', 'global_mmlu_ko_college_mathematics', 'global_mmlu_ko_college_medicine', 'global_mmlu_ko_college_physics', 'global_mmlu_ko_computer_security', 'global_mmlu_ko_conceptual_physics', 'global_mmlu_ko_econometrics', 'global_mmlu_ko_electrical_engineering', 'global_mmlu_ko_elementary_mathematics', 'global_mmlu_ko_formal_logic', 'global_mmlu_ko_global_facts', 'global_mmlu_ko_high_school_biology', 'global_mmlu_ko_high_school_chemistry', 'global_mmlu_ko_high_school_computer_science', 'global_mmlu_ko_high_school_european_history', 'global_mmlu_ko_high_school_geography', 'global_mmlu_ko_high_school_government_and_politics', 'global_mmlu_ko_high_school_macroeconomics', 'global_mmlu_ko_high_school_mathematics', 'global_mmlu_ko_high_school_microeconomics', 'global_mmlu_ko_high_school_physics', 'global_mmlu_ko_high_school_psychology'], num_fewshot=0
|
||
[LM_EVAL] No valid tasks to evaluate.
|
||
[Phase 2 [0-shot] GPU 6 — MMLU-KO part 1/2] Done.
|
||
2026-03-05 03:18:53 [INFO] [DONE] [0-shot] GPU 6 — MMLU-KO part 1/2
|
||
2026-03-05 03:18:53 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:53 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:54 [WARNING] [LM_EVAL] Batch evaluation failed (lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'). Falling back to per-task evaluation.
|
||
2026-03-05 03:18:54 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:54 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:54 [WARNING] [LM_EVAL] Task 'kobest_wic' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
[LM_EVAL] Starting on cuda:4 (CUDA_VISIBLE_DEVICES=4), tasks=['kobest_wic'], num_fewshot=0
|
||
[LM_EVAL] Evaluating 1 task(s) together: ['kobest_wic']
|
||
[LM_EVAL] Evaluating task 'kobest_wic' individually...
|
||
[LM_EVAL] Evaluation complete in 1.6s
|
||
[LM_EVAL] Skipped tasks: ['kobest_wic']
|
||
[Phase 2 [0-shot] GPU 4 — KoBEST: wic] Done.
|
||
2026-03-05 03:18:54 [INFO] [DONE] [0-shot] GPU 4 — KoBEST: wic
|
||
2026-03-05 03:18:55 [WARNING] [LM_EVAL] Batch evaluation failed (lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'). Falling back to per-task evaluation.
|
||
2026-03-05 03:18:55 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:55 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:55 [WARNING] [LM_EVAL] Task 'kobest_hellaswag' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
2026-03-05 03:18:55 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:55 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:55 [WARNING] [LM_EVAL] Task 'kobest_sentineg' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
[LM_EVAL] Starting on cuda:3 (CUDA_VISIBLE_DEVICES=3), tasks=['kobest_hellaswag', 'kobest_sentineg'], num_fewshot=0
|
||
[LM_EVAL] Evaluating 2 task(s) together: ['kobest_hellaswag', 'kobest_sentineg']
|
||
[LM_EVAL] Evaluating task 'kobest_hellaswag' individually...
|
||
[LM_EVAL] Evaluating task 'kobest_sentineg' individually...
|
||
[LM_EVAL] Evaluation complete in 1.6s
|
||
[LM_EVAL] Skipped tasks: ['kobest_hellaswag', 'kobest_sentineg']
|
||
[Phase 2 [0-shot] GPU 3 — KoBEST: hellaswag + sentineg] Done.
|
||
2026-03-05 03:18:55 [INFO] [DONE] [0-shot] GPU 3 — KoBEST: hellaswag + sentineg
|
||
2026-03-05 03:18:55 [WARNING] [LM_EVAL] Batch evaluation failed (lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'). Falling back to per-task evaluation.
|
||
2026-03-05 03:18:55 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:55 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:55 [WARNING] [LM_EVAL] Task 'kobest_boolq' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
2026-03-05 03:18:55 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:55 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:55 [WARNING] [LM_EVAL] Task 'kobest_copa' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
[LM_EVAL] Starting on cuda:2 (CUDA_VISIBLE_DEVICES=2), tasks=['kobest_boolq', 'kobest_copa'], num_fewshot=0
|
||
[LM_EVAL] Evaluating 2 task(s) together: ['kobest_boolq', 'kobest_copa']
|
||
[LM_EVAL] Evaluating task 'kobest_boolq' individually...
|
||
[LM_EVAL] Evaluating task 'kobest_copa' individually...
|
||
[LM_EVAL] Evaluation complete in 1.6s
|
||
[LM_EVAL] Skipped tasks: ['kobest_boolq', 'kobest_copa']
|
||
[Phase 2 [0-shot] GPU 2 — KoBEST: boolq + copa] Done.
|
||
2026-03-05 03:18:55 [INFO] [DONE] [0-shot] GPU 2 — KoBEST: boolq + copa
|
||
2026-03-05 03:18:55 [WARNING] [LM_EVAL] Batch evaluation failed (lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'). Falling back to per-task evaluation.
|
||
2026-03-05 03:18:55 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:55 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:55 [WARNING] [LM_EVAL] Task 'haerae' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
[LM_EVAL] Starting on cuda:5 (CUDA_VISIBLE_DEVICES=5), tasks=['haerae'], num_fewshot=0
|
||
[LM_EVAL] Evaluating 1 task(s) together: ['haerae']
|
||
[LM_EVAL] Evaluating task 'haerae' individually...
|
||
[LM_EVAL] Evaluation complete in 1.6s
|
||
[LM_EVAL] Skipped tasks: ['haerae']
|
||
[Phase 2 [0-shot] GPU 5 — HAE-RAE (all subtasks)] Done.
|
||
2026-03-05 03:18:55 [INFO] [DONE] [0-shot] GPU 5 — HAE-RAE (all subtasks)
|
||
2026-03-05 03:18:55 [INFO] Phase 2 (0-shot) complete: 6 succeeded, 0 failed
|
||
2026-03-05 03:18:55 [INFO] Attempting 5-shot benchmarks ...
|
||
2026-03-05 03:18:55 [INFO] Submitted: [5-shot] GPU 2 — KoBEST: boolq + copa
|
||
2026-03-05 03:18:55 [INFO] Submitted: [5-shot] GPU 3 — KoBEST: hellaswag + sentineg
|
||
2026-03-05 03:18:55 [INFO] Submitted: [5-shot] GPU 4 — KoBEST: wic
|
||
2026-03-05 03:18:55 [INFO] Submitted: [5-shot] GPU 5 — HAE-RAE (all subtasks)
|
||
2026-03-05 03:18:55 [INFO] Submitted: [5-shot] GPU 6 — MMLU-KO part 1/2
|
||
2026-03-05 03:18:55 [INFO] Submitted: [5-shot] GPU 7 — MMLU-KO part 2/2
|
||
[Phase 2 [5-shot] GPU 2 — KoBEST: boolq + copa] Starting on cuda:2 — tasks: ['kobest_boolq', 'kobest_copa'], 5-shot
|
||
[Phase 2 [5-shot] GPU 3 — KoBEST: hellaswag + sentineg] Starting on cuda:3 — tasks: ['kobest_hellaswag', 'kobest_sentineg'], 5-shot
|
||
[Phase 2 [5-shot] GPU 4 — KoBEST: wic] Starting on cuda:4 — tasks: ['kobest_wic'], 5-shot
|
||
[Phase 2 [5-shot] GPU 5 — HAE-RAE (all subtasks)] Starting on cuda:5 — tasks: ['haerae'], 5-shot
|
||
[Phase 2 [5-shot] GPU 6 — MMLU-KO part 1/2] Starting on cuda:6 — tasks: ['global_mmlu_ko_abstract_algebra', 'global_mmlu_ko_anatomy', 'global_mmlu_ko_astronomy', 'global_mmlu_ko_business_ethics', 'global_mmlu_ko_clinical_knowledge', 'global_mmlu_ko_college_biology', 'global_mmlu_ko_college_chemistry', 'global_mmlu_ko_college_computer_science', 'global_mmlu_ko_college_mathematics', 'global_mmlu_ko_college_medicine', 'global_mmlu_ko_college_physics', 'global_mmlu_ko_computer_security', 'global_mmlu_ko_conceptual_physics', 'global_mmlu_ko_econometrics', 'global_mmlu_ko_electrical_engineering', 'global_mmlu_ko_elementary_mathematics', 'global_mmlu_ko_formal_logic', 'global_mmlu_ko_global_facts', 'global_mmlu_ko_high_school_biology', 'global_mmlu_ko_high_school_chemistry', 'global_mmlu_ko_high_school_computer_science', 'global_mmlu_ko_high_school_european_history', 'global_mmlu_ko_high_school_geography', 'global_mmlu_ko_high_school_government_and_politics', 'global_mmlu_ko_high_school_macroeconomics', 'global_mmlu_ko_high_school_mathematics', 'global_mmlu_ko_high_school_microeconomics', 'global_mmlu_ko_high_school_physics', 'global_mmlu_ko_high_school_psychology'], 5-shot
|
||
[Phase 2 [5-shot] GPU 7 — MMLU-KO part 2/2] Starting on cuda:7 — tasks: ['global_mmlu_ko_high_school_statistics', 'global_mmlu_ko_high_school_us_history', 'global_mmlu_ko_high_school_world_history', 'global_mmlu_ko_human_aging', 'global_mmlu_ko_human_sexuality', 'global_mmlu_ko_international_law', 'global_mmlu_ko_jurisprudence', 'global_mmlu_ko_logical_fallacies', 'global_mmlu_ko_machine_learning', 'global_mmlu_ko_management', 'global_mmlu_ko_marketing', 'global_mmlu_ko_medical_genetics', 'global_mmlu_ko_miscellaneous', 'global_mmlu_ko_moral_disputes', 'global_mmlu_ko_moral_scenarios', 'global_mmlu_ko_nutrition', 'global_mmlu_ko_philosophy', 'global_mmlu_ko_prehistory', 'global_mmlu_ko_professional_accounting', 'global_mmlu_ko_professional_law', 'global_mmlu_ko_professional_medicine', 'global_mmlu_ko_professional_psychology', 'global_mmlu_ko_public_relations', 'global_mmlu_ko_security_studies', 'global_mmlu_ko_sociology', 'global_mmlu_ko_us_foreign_policy', 'global_mmlu_ko_virology', 'global_mmlu_ko_world_religions'], 5-shot
|
||
2026-03-05 03:18:56 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:56 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:56 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:56 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:56 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:56 [INFO] TensorFlow version 2.20.0 available.
|
||
2026-03-05 03:18:58 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:58 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:58 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:58 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:58 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:58 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:58 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:18:58 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_statistics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_us_history' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_world_history' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_human_aging' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_human_sexuality' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_international_law' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_jurisprudence' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_logical_fallacies' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_machine_learning' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_management' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_marketing' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_medical_genetics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_miscellaneous' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_moral_disputes' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_moral_scenarios' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_nutrition' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_philosophy' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_prehistory' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_accounting' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_law' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_medicine' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_professional_psychology' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_public_relations' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_security_studies' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_sociology' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_us_foreign_policy' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_virology' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_world_religions' not found in lm_eval registry — skipping.
|
||
[LM_EVAL] Starting on cuda:7 (CUDA_VISIBLE_DEVICES=7), tasks=['global_mmlu_ko_high_school_statistics', 'global_mmlu_ko_high_school_us_history', 'global_mmlu_ko_high_school_world_history', 'global_mmlu_ko_human_aging', 'global_mmlu_ko_human_sexuality', 'global_mmlu_ko_international_law', 'global_mmlu_ko_jurisprudence', 'global_mmlu_ko_logical_fallacies', 'global_mmlu_ko_machine_learning', 'global_mmlu_ko_management', 'global_mmlu_ko_marketing', 'global_mmlu_ko_medical_genetics', 'global_mmlu_ko_miscellaneous', 'global_mmlu_ko_moral_disputes', 'global_mmlu_ko_moral_scenarios', 'global_mmlu_ko_nutrition', 'global_mmlu_ko_philosophy', 'global_mmlu_ko_prehistory', 'global_mmlu_ko_professional_accounting', 'global_mmlu_ko_professional_law', 'global_mmlu_ko_professional_medicine', 'global_mmlu_ko_professional_psychology', 'global_mmlu_ko_public_relations', 'global_mmlu_ko_security_studies', 'global_mmlu_ko_sociology', 'global_mmlu_ko_us_foreign_policy', 'global_mmlu_ko_virology', 'global_mmlu_ko_world_religions'], num_fewshot=5
|
||
[LM_EVAL] No valid tasks to evaluate.
|
||
[Phase 2 [5-shot] GPU 7 — MMLU-KO part 2/2] Done.
|
||
2026-03-05 03:18:58 [INFO] [DONE] [5-shot] GPU 7 — MMLU-KO part 2/2
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_abstract_algebra' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_anatomy' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_astronomy' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_business_ethics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_clinical_knowledge' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_biology' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_chemistry' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_computer_science' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_mathematics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_medicine' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_college_physics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_computer_security' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_conceptual_physics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_econometrics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_electrical_engineering' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_elementary_mathematics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_formal_logic' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_global_facts' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_biology' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_chemistry' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_computer_science' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_european_history' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_geography' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_government_and_politics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_macroeconomics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_mathematics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_microeconomics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_physics' not found in lm_eval registry — skipping.
|
||
2026-03-05 03:18:58 [WARNING] [LM_EVAL] Task 'global_mmlu_ko_high_school_psychology' not found in lm_eval registry — skipping.
|
||
[LM_EVAL] Starting on cuda:6 (CUDA_VISIBLE_DEVICES=6), tasks=['global_mmlu_ko_abstract_algebra', 'global_mmlu_ko_anatomy', 'global_mmlu_ko_astronomy', 'global_mmlu_ko_business_ethics', 'global_mmlu_ko_clinical_knowledge', 'global_mmlu_ko_college_biology', 'global_mmlu_ko_college_chemistry', 'global_mmlu_ko_college_computer_science', 'global_mmlu_ko_college_mathematics', 'global_mmlu_ko_college_medicine', 'global_mmlu_ko_college_physics', 'global_mmlu_ko_computer_security', 'global_mmlu_ko_conceptual_physics', 'global_mmlu_ko_econometrics', 'global_mmlu_ko_electrical_engineering', 'global_mmlu_ko_elementary_mathematics', 'global_mmlu_ko_formal_logic', 'global_mmlu_ko_global_facts', 'global_mmlu_ko_high_school_biology', 'global_mmlu_ko_high_school_chemistry', 'global_mmlu_ko_high_school_computer_science', 'global_mmlu_ko_high_school_european_history', 'global_mmlu_ko_high_school_geography', 'global_mmlu_ko_high_school_government_and_politics', 'global_mmlu_ko_high_school_macroeconomics', 'global_mmlu_ko_high_school_mathematics', 'global_mmlu_ko_high_school_microeconomics', 'global_mmlu_ko_high_school_physics', 'global_mmlu_ko_high_school_psychology'], num_fewshot=5
|
||
[LM_EVAL] No valid tasks to evaluate.
|
||
[Phase 2 [5-shot] GPU 6 — MMLU-KO part 1/2] Done.
|
||
2026-03-05 03:18:58 [INFO] [DONE] [5-shot] GPU 6 — MMLU-KO part 1/2
|
||
2026-03-05 03:19:00 [WARNING] [LM_EVAL] Batch evaluation failed (lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'). Falling back to per-task evaluation.
|
||
2026-03-05 03:19:00 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:19:00 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:19:00 [WARNING] [LM_EVAL] Task 'kobest_hellaswag' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
2026-03-05 03:19:00 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:19:00 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:19:00 [WARNING] [LM_EVAL] Task 'kobest_sentineg' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
[LM_EVAL] Starting on cuda:3 (CUDA_VISIBLE_DEVICES=3), tasks=['kobest_hellaswag', 'kobest_sentineg'], num_fewshot=5
|
||
[LM_EVAL] Evaluating 2 task(s) together: ['kobest_hellaswag', 'kobest_sentineg']
|
||
[LM_EVAL] Evaluating task 'kobest_hellaswag' individually...
|
||
[LM_EVAL] Evaluating task 'kobest_sentineg' individually...
|
||
[LM_EVAL] Evaluation complete in 1.6s
|
||
[LM_EVAL] Skipped tasks: ['kobest_hellaswag', 'kobest_sentineg']
|
||
[Phase 2 [5-shot] GPU 3 — KoBEST: hellaswag + sentineg] Done.
|
||
2026-03-05 03:19:00 [INFO] [DONE] [5-shot] GPU 3 — KoBEST: hellaswag + sentineg
|
||
2026-03-05 03:19:00 [WARNING] [LM_EVAL] Batch evaluation failed (lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'). Falling back to per-task evaluation.
|
||
2026-03-05 03:19:00 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:19:00 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:19:00 [WARNING] [LM_EVAL] Task 'kobest_boolq' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
2026-03-05 03:19:00 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:19:00 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:19:00 [WARNING] [LM_EVAL] Task 'kobest_copa' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
[LM_EVAL] Starting on cuda:2 (CUDA_VISIBLE_DEVICES=2), tasks=['kobest_boolq', 'kobest_copa'], num_fewshot=5
|
||
[LM_EVAL] Evaluating 2 task(s) together: ['kobest_boolq', 'kobest_copa']
|
||
[LM_EVAL] Evaluating task 'kobest_boolq' individually...
|
||
[LM_EVAL] Evaluating task 'kobest_copa' individually...
|
||
[LM_EVAL] Evaluation complete in 1.6s
|
||
[LM_EVAL] Skipped tasks: ['kobest_boolq', 'kobest_copa']
|
||
[Phase 2 [5-shot] GPU 2 — KoBEST: boolq + copa] Done.
|
||
2026-03-05 03:19:00 [INFO] [DONE] [5-shot] GPU 2 — KoBEST: boolq + copa
|
||
2026-03-05 03:19:00 [WARNING] [LM_EVAL] Batch evaluation failed (lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'). Falling back to per-task evaluation.
|
||
2026-03-05 03:19:00 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:19:00 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:19:00 [WARNING] [LM_EVAL] Task 'haerae' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
[LM_EVAL] Starting on cuda:5 (CUDA_VISIBLE_DEVICES=5), tasks=['haerae'], num_fewshot=5
|
||
[LM_EVAL] Evaluating 1 task(s) together: ['haerae']
|
||
[LM_EVAL] Evaluating task 'haerae' individually...
|
||
[LM_EVAL] Evaluation complete in 1.7s
|
||
[LM_EVAL] Skipped tasks: ['haerae']
|
||
[Phase 2 [5-shot] GPU 5 — HAE-RAE (all subtasks)] Done.
|
||
2026-03-05 03:19:00 [WARNING] [LM_EVAL] Batch evaluation failed (lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'). Falling back to per-task evaluation.
|
||
2026-03-05 03:19:00 [INFO] [DONE] [5-shot] GPU 5 — HAE-RAE (all subtasks)
|
||
2026-03-05 03:19:00 [INFO] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
2026-03-05 03:19:00 [INFO] Initializing hf model, with arguments: {'pretrained':
|
||
'/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000', 'dtype':
|
||
'bfloat16', 'device': 'cuda:0'}
|
||
2026-03-05 03:19:00 [WARNING] [LM_EVAL] Task 'kobest_wic' failed: lm_eval.models.huggingface.HFLM() got multiple values for keyword argument 'device'
|
||
[LM_EVAL] Starting on cuda:4 (CUDA_VISIBLE_DEVICES=4), tasks=['kobest_wic'], num_fewshot=5
|
||
[LM_EVAL] Evaluating 1 task(s) together: ['kobest_wic']
|
||
[LM_EVAL] Evaluating task 'kobest_wic' individually...
|
||
[LM_EVAL] Evaluation complete in 1.7s
|
||
[LM_EVAL] Skipped tasks: ['kobest_wic']
|
||
[Phase 2 [5-shot] GPU 4 — KoBEST: wic] Done.
|
||
2026-03-05 03:19:00 [INFO] [DONE] [5-shot] GPU 4 — KoBEST: wic
|
||
2026-03-05 03:19:01 [INFO] Phase 2 (5-shot) complete: 6 succeeded, 0 failed
|
||
2026-03-05 03:19:01 [INFO] Phase 2 results saved: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/phase2_results.json
|
||
2026-03-05 03:19:01 [INFO] Phase 2 complete in 11s.
|
||
2026-03-05 03:19:01 [INFO]
|
||
2026-03-05 03:19:01 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:19:01 [INFO] PHASE 3 — Report Generation
|
||
2026-03-05 03:19:01 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:19:01 [INFO] Report saved: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/full_eval_report.md
|
||
2026-03-05 03:19:01 [INFO] Phase 3 complete in 0s.
|
||
2026-03-05 03:19:01 [INFO] ========================================================================
|
||
2026-03-05 03:19:01 [INFO] PIPELINE COMPLETE
|
||
2026-03-05 03:19:01 [INFO] ========================================================================
|
||
2026-03-05 03:19:01 [INFO] Total time : 47s
|
||
2026-03-05 03:19:01 [INFO] Output dir : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318
|
||
2026-03-05 03:19:01 [INFO] Phase 1 results : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/phase1_results.json
|
||
2026-03-05 03:19:01 [INFO] Phase 2 results : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/phase2_results.json
|
||
2026-03-05 03:19:01 [INFO] Gen samples : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/generation_samples.json
|
||
2026-03-05 03:19:01 [INFO] Report : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/3b_full_eval_20260305_0318/full_eval_report.md
|
||
2026-03-05 03:19:01 [INFO] Phase 1 tasks : 0 OK / 6 failed
|
||
2026-03-05 03:19:01 [INFO] Phase 2 tasks : 6 OK / 0 failed
|
||
2026-03-05 03:19:01 [INFO] ========================================================================
|
||
/usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 60 leaked semaphore objects to clean up at shutdown
|
||
warnings.warn('resource_tracker: There appear to be %d '
|