90 lines
7.1 KiB
Plaintext
90 lines
7.1 KiB
Plaintext
2026-03-05 03:35:28 [INFO] ========================================================================
|
||
2026-03-05 03:35:28 [INFO] FRANKENSTALLM 3B — Full Evaluation Pipeline
|
||
2026-03-05 03:35:28 [INFO] ========================================================================
|
||
2026-03-05 03:35:28 [INFO] Checkpoint : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/checkpoints/korean_3b_fp8_run1/checkpoint-0057000
|
||
2026-03-05 03:35:28 [INFO] Tokenizer : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/tokenizer/korean_sp/tokenizer.json
|
||
2026-03-05 03:35:28 [INFO] Data dir : /PROJECT/0325120031_A/ghong/taketimes/llm-bang/data
|
||
2026-03-05 03:35:28 [INFO] Output dir : eval/outputs/3b_full_eval_20260305_0318
|
||
2026-03-05 03:35:28 [INFO] GPUs : [2, 3, 4, 5, 6, 7]
|
||
2026-03-05 03:35:28 [INFO] SEQ_LEN : 2048 STRIDE: 512 BATCH_SIZE: 32
|
||
2026-03-05 03:35:28 [INFO] Phases : phase0=skip phase1=run phase2=run
|
||
2026-03-05 03:35:28 [INFO]
|
||
2026-03-05 03:35:28 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:35:28 [INFO] PHASE 0 — HF Checkpoint Conversion
|
||
2026-03-05 03:35:28 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:35:28 [INFO] Skipping Phase 0 — reusing: eval/outputs/3b_full_eval_20260305_0318/hf_3b_checkpoint-0057000
|
||
2026-03-05 03:35:28 [INFO]
|
||
2026-03-05 03:35:28 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:35:28 [INFO] PHASE 1 — Internal Evaluation — 6 GPU Parallel
|
||
2026-03-05 03:35:28 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 03:35:28 [INFO] Spawning: GPU 5 — Calibration + Token NLL (GPU 5)
|
||
2026-03-05 03:35:28 [INFO] Spawning: GPU 6 — Generation (15 prompts × 4 temps) (GPU 6)
|
||
2026-03-05 03:35:28 [INFO] Spawning: GPU 7 — Repetition grid (12 × 5) (GPU 7)
|
||
2026-03-05 03:35:28 [INFO] Spawning: GPU 2 — PPL: 3b_val.bin (GPU 2)
|
||
2026-03-05 03:35:28 [INFO] Spawning: GPU 3 — PPL: korean_c4 + korean_val (GPU 3)
|
||
2026-03-05 03:35:28 [INFO] Spawning: GPU 4 — PPL: hplt_ko + cc100_ko + PPL: 7 cosmo files + PPL: 7 remaining files (GPU 4)
|
||
2026-03-05 03:35:52 [INFO] [DONE] GPU 5 — Calibration + Token NLL
|
||
2026-03-05 03:38:04 [INFO] [DONE] GPU 6 — Generation (15 prompts × 4 temps)
|
||
2026-03-05 03:38:22 [INFO] [DONE] GPU 7 — Repetition grid (12 × 5)
|
||
2026-03-05 03:51:46 [INFO] [DONE] GPU 3 — PPL: korean_c4 + korean_val
|
||
2026-03-05 04:01:10 [INFO] [DONE] GPU 4 — PPL: hplt_ko + cc100_ko + PPL: 7 cosmo files + PPL: 7 remaining files
|
||
2026-03-05 04:12:48 [INFO] [DONE] GPU 2 — PPL: 3b_val.bin
|
||
2026-03-05 04:12:48 [INFO] Complete: 6 succeeded, 0 failed
|
||
2026-03-05 04:12:48 [INFO] Phase 1 results saved: eval/outputs/3b_full_eval_20260305_0318/phase1_results.json
|
||
2026-03-05 04:12:48 [INFO] Generation samples saved: eval/outputs/3b_full_eval_20260305_0318/generation_samples.json
|
||
2026-03-05 04:12:48 [INFO] Phase 1 complete in 37m 20s.
|
||
2026-03-05 04:12:48 [INFO]
|
||
2026-03-05 04:12:48 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 04:12:48 [INFO] PHASE 2 — lm-eval Benchmarks — 6 GPU Parallel
|
||
2026-03-05 04:12:48 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 04:12:48 [INFO] Running 0-shot benchmarks on 6 GPUs ...
|
||
2026-03-05 04:12:48 [INFO] Spawning: [0-shot] GPU 2 — KoBEST: boolq + copa (GPU 2)
|
||
2026-03-05 04:12:48 [INFO] Spawning: [0-shot] GPU 3 — KoBEST: hellaswag + sentineg (GPU 3)
|
||
2026-03-05 04:12:48 [INFO] Spawning: [0-shot] GPU 4 — KoBEST: wic (GPU 4)
|
||
2026-03-05 04:12:48 [INFO] Spawning: [0-shot] GPU 5 — HAE-RAE (all subtasks) (GPU 5)
|
||
2026-03-05 04:12:48 [INFO] Spawning: [0-shot] GPU 6 — MMLU-KO part 1/2 (GPU 6)
|
||
2026-03-05 04:12:48 [INFO] Spawning: [0-shot] GPU 7 — MMLU-KO part 2/2 (GPU 7)
|
||
2026-03-05 04:12:52 [INFO] [DONE] [0-shot] GPU 6 — MMLU-KO part 1/2
|
||
2026-03-05 04:12:52 [INFO] [DONE] [0-shot] GPU 7 — MMLU-KO part 2/2
|
||
2026-03-05 04:13:10 [INFO] [DONE] [0-shot] GPU 4 — KoBEST: wic
|
||
2026-03-05 04:13:18 [INFO] [DONE] [0-shot] GPU 3 — KoBEST: hellaswag + sentineg
|
||
2026-03-05 04:13:20 [INFO] [DONE] [0-shot] GPU 2 — KoBEST: boolq + copa
|
||
2026-03-05 04:13:28 [INFO] [DONE] [0-shot] GPU 5 — HAE-RAE (all subtasks)
|
||
2026-03-05 04:13:28 [INFO] Complete: 6 succeeded, 0 failed
|
||
2026-03-05 04:13:28 [INFO] Phase 2 (0-shot) complete.
|
||
2026-03-05 04:13:28 [INFO] Attempting 5-shot benchmarks ...
|
||
2026-03-05 04:13:28 [INFO] Spawning: [5-shot] GPU 2 — KoBEST: boolq + copa (GPU 2)
|
||
2026-03-05 04:13:28 [INFO] Spawning: [5-shot] GPU 3 — KoBEST: hellaswag + sentineg (GPU 3)
|
||
2026-03-05 04:13:28 [INFO] Spawning: [5-shot] GPU 4 — KoBEST: wic (GPU 4)
|
||
2026-03-05 04:13:28 [INFO] Spawning: [5-shot] GPU 5 — HAE-RAE (all subtasks) (GPU 5)
|
||
2026-03-05 04:13:28 [INFO] Spawning: [5-shot] GPU 6 — MMLU-KO part 1/2 (GPU 6)
|
||
2026-03-05 04:13:28 [INFO] Spawning: [5-shot] GPU 7 — MMLU-KO part 2/2 (GPU 7)
|
||
2026-03-05 04:13:32 [INFO] [DONE] [5-shot] GPU 6 — MMLU-KO part 1/2
|
||
2026-03-05 04:13:32 [INFO] [DONE] [5-shot] GPU 7 — MMLU-KO part 2/2
|
||
2026-03-05 04:14:24 [INFO] [DONE] [5-shot] GPU 4 — KoBEST: wic
|
||
2026-03-05 04:14:36 [INFO] [DONE] [5-shot] GPU 3 — KoBEST: hellaswag + sentineg
|
||
2026-03-05 04:14:58 [INFO] [DONE] [5-shot] GPU 2 — KoBEST: boolq + copa
|
||
2026-03-05 04:15:04 [INFO] [DONE] [5-shot] GPU 5 — HAE-RAE (all subtasks)
|
||
2026-03-05 04:15:04 [INFO] Complete: 6 succeeded, 0 failed
|
||
2026-03-05 04:15:04 [INFO] Phase 2 (5-shot) complete.
|
||
2026-03-05 04:15:04 [INFO] Phase 2 results saved: eval/outputs/3b_full_eval_20260305_0318/phase2_results.json
|
||
2026-03-05 04:15:04 [INFO] Phase 2 complete in 2m 16s.
|
||
2026-03-05 04:15:04 [INFO]
|
||
2026-03-05 04:15:04 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 04:15:04 [INFO] PHASE 3 — Report Generation
|
||
2026-03-05 04:15:04 [INFO] ------------------------------------------------------------------------
|
||
2026-03-05 04:15:04 [INFO] Report saved: eval/outputs/3b_full_eval_20260305_0318/full_eval_report.md
|
||
2026-03-05 04:15:04 [INFO] Phase 3 complete in 0s.
|
||
2026-03-05 04:15:04 [INFO] ========================================================================
|
||
2026-03-05 04:15:04 [INFO] PIPELINE COMPLETE
|
||
2026-03-05 04:15:04 [INFO] ========================================================================
|
||
2026-03-05 04:15:04 [INFO] Total time : 39m 36s
|
||
2026-03-05 04:15:04 [INFO] Output dir : eval/outputs/3b_full_eval_20260305_0318
|
||
2026-03-05 04:15:04 [INFO] Phase 1 results : eval/outputs/3b_full_eval_20260305_0318/phase1_results.json
|
||
2026-03-05 04:15:04 [INFO] Phase 2 results : eval/outputs/3b_full_eval_20260305_0318/phase2_results.json
|
||
2026-03-05 04:15:04 [INFO] Gen samples : eval/outputs/3b_full_eval_20260305_0318/generation_samples.json
|
||
2026-03-05 04:15:04 [INFO] Report : eval/outputs/3b_full_eval_20260305_0318/full_eval_report.md
|
||
2026-03-05 04:15:04 [INFO] Phase 1 tasks : 6 OK / 0 failed
|
||
2026-03-05 04:15:04 [INFO] Phase 2 tasks : 6 OK / 0 failed
|
||
2026-03-05 04:15:04 [INFO] ========================================================================
|