61 lines
79 KiB
Plaintext
61 lines
79 KiB
Plaintext
|
2026-02-27:11:42:00 INFO [_cli.run:376] Selected Tasks: ['kobest_boolq', 'kobest_copa', 'haerae_general_knowledge', 'haerae_history', 'paws_ko']
|
||
|
|
2026-02-27:11:42:00 INFO [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
|
|
2026-02-27:11:42:00 INFO [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/PROJECT/0325120031_A/ghong/taketimes/llm-bang/outputs/hf_checkpoint-best', 'dtype': 'float16'}
|
||
|
|
2026-02-27:11:42:02 INFO [models.huggingface:161] Using device 'cuda:0'
|
||
|
|
2026-02-27:11:42:03 INFO [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
|
||
|
|
Loading weights: 0%| | 0/219 [00:00<?, ?it/s]
Loading weights: 0%| | 1/219 [00:00<00:00, 9939.11it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/219 [00:00<00:00, 6168.09it/s, Materializing param=lm_head.weight]
Loading weights: 1%| | 2/219 [00:03<06:14, 1.73s/it, Materializing param=lm_head.weight]
Loading weights: 1%| | 2/219 [00:03<06:14, 1.73s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 1%| | 2/219 [00:03<06:14, 1.73s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 1%|▏ | 3/219 [00:03<03:50, 1.07s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 1%|▏ | 3/219 [00:03<03:50, 1.07s/it, Materializing param=model.layers.0.input_layernorm.weight]
Loading weights: 1%|▏ | 3/219 [00:03<03:50, 1.07s/it, Materializing param=model.layers.0.input_layernorm.weight]
Loading weights: 2%|▏ | 4/219 [00:03<03:49, 1.07s/it, Materializing param=model.layers.0.mlp.down_proj.weight]
Loading weights: 2%|▏ | 4/219 [00:03<03:49, 1.07s/it, Materializing param=model.layers.0.mlp.down_proj.weight]
Loading weights: 2%|▏ | 5/219 [00:03<03:48, 1.07s/it, Materializing param=model.layers.0.mlp.gate_proj.weight]
Loading weights: 2%|▏ | 5/219 [00:03<03:48, 1.07s/it, Materializing param=model.layers.0.mlp.gate_proj.weight]
Loading weights: 3%|▎ | 6/219 [00:03<03:47, 1.07s/it, Materializing param=model.layers.0.mlp.up_proj.weight]
Loading weights: 3%|▎ | 6/219 [00:03<03:47, 1.07s/it, Materializing param=model.layers.0.mlp.up_proj.weight]
Loading weights: 3%|▎ | 7/219 [00:03<03:46, 1.07s/it, Materializing param=model.layers.0.post_attention_layernorm.weight]
Loading weights: 3%|▎ | 7/219 [00:03<03:46, 1.07s/it, Materializing param=model.layers.0.post_attention_layernorm.weight]
Loading weights: 4%|▎ | 8/219 [00:03<03:45, 1.07s/it, Materializing param=model.layers.0.self_attn.k_proj.weight]
Loading weights: 4%|▎ | 8/219 [00:03<03:45, 1.07s/it, Materializing param=model.layers.0.self_attn.k_proj.weight]
Loading weights: 4%|▍ | 9/219 [00:03<03:44, 1.07s/it, Materializing param=model.layers.0.self_attn.o_proj.weight]
Loading weights: 4%|▍ | 9/219 [00:03<03:44, 1.07s/it, Materializing param=model.layers.0.self_attn.o_proj.weight]
Loading weights: 5%|▍ | 10/219 [00:03<03:43, 1.07s/it, Materializing param=model.layers.0.self_attn.q_proj.weight]
Loading weights: 5%|▍ | 10/219 [00:03<03:43, 1.07s/it, Materializing param=model.layers.0.self_attn.q_proj.weight]
Loading weights: 5%|▌ | 11/219 [00:03<03:41, 1.07s/it, Materializing param=model.layers.0.self_attn.v_proj.weight]
Loading weights: 5%|▌ | 11/219 [00:03<03:41, 1.07s/it, Materializing param=model.layers.0.self_attn.v_proj.weight]
Loading weights: 5%|▌ | 12/219 [00:03<00:38, 5.35it/s, Materializing param=model.layers.0.self_attn.v_proj.weight]
Loading weights: 5%|▌ | 12/219 [00:03<00:38, 5.35it/s, Materializing param=model.layers.1.input_layernorm.weight]
Loading weights: 5%|▌ | 12/219 [00:03<00:38, 5.35it/s, Materializing param=model.layers.1.input_layernorm.weight]
Loading weights: 6%|▌ | 13/219 [00:03<00:38, 5.35it/s, Materializing param=model.layers.1.mlp.down_proj.weight]
Loading weights: 6%|▌ | 13/219 [00:03<00:38, 5.35it/s, Materializing param=model.layers.1.mlp.down_proj.weight]
Loading weights: 6%|▋ | 14/219 [00:04<00:38, 5.35it/s, Materializing param=model.layers.1.mlp.gate_proj.weight]
Loading weights: 6%|▋ | 14/219 [00:04<00:38, 5.35it/s, Materializing param=model.layers.1.mlp.gate_proj.weight]
Loading weights: 7%|▋ | 15/219 [00:04<00:43, 4.71it/s, Materializing param=model.layers.1.mlp.gate_proj.weight]
Loading weights: 7%|▋ | 15/219 [00:04<00:43, 4.71it/s, Materializing para
|
||
|
|
The tied weights mapping and config for this model specifies to tie model.embed_tokens.weight to lm_head.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning
|
||
|
|
Generating train split: 0%| | 0/3665 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 3665/3665 [00:00<00:00, 831689.89 examples/s]
|
||
|
|
Generating test split: 0%| | 0/1404 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1404/1404 [00:00<00:00, 592315.71 examples/s]
|
||
|
|
Generating validation split: 0%| | 0/700 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 700/700 [00:00<00:00, 393831.36 examples/s]
|
||
|
|
Generating train split: 0%| | 0/3076 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 3076/3076 [00:00<00:00, 1183531.70 examples/s]
|
||
|
|
Generating test split: 0%| | 0/1000 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1000/1000 [00:00<00:00, 576774.48 examples/s]
|
||
|
|
Generating validation split: 0%| | 0/500 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 500/500 [00:00<00:00, 404075.53 examples/s]
|
||
|
|
Generating test split: 0 examples [00:00, ? examples/s]
Generating test split: 176 examples [00:00, 77241.55 examples/s]
|
||
|
|
Generating test split: 0 examples [00:00, ? examples/s]
Generating test split: 188 examples [00:00, 93383.37 examples/s]
|
||
|
|
Generating train split: 0%| | 0/49401 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 49401/49401 [00:00<00:00, 2001244.11 examples/s]
|
||
|
|
Generating test split: 0%| | 0/2000 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 2000/2000 [00:00<00:00, 899582.63 examples/s]
|
||
|
|
Generating validation split: 0%| | 0/2000 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 2000/2000 [00:00<00:00, 1037423.70 examples/s]
|
||
|
|
2026-02-27:11:43:03 INFO [tasks:700] Selected tasks:
|
||
|
|
2026-02-27:11:43:03 INFO [tasks:691] Task: paws_ko (paws-x/paws_ko.yaml)
|
||
|
|
2026-02-27:11:43:03 INFO [tasks:691] Task: haerae_history (haerae/haerae_hi.yaml)
|
||
|
|
2026-02-27:11:43:03 INFO [tasks:691] Task: haerae_general_knowledge (haerae/haerae_gk.yaml)
|
||
|
|
2026-02-27:11:43:03 INFO [tasks:691] Task: kobest_copa (kobest/kobest_copa.yaml)
|
||
|
|
2026-02-27:11:43:03 INFO [tasks:691] Task: kobest_boolq (kobest/kobest_boolq.yaml)
|
||
|
|
2026-02-27:11:43:03 WARNING [evaluator:333] Overwriting default num_fewshot of paws_ko from None to 0
|
||
|
|
2026-02-27:11:43:03 WARNING [evaluator:333] Overwriting default num_fewshot of haerae_history from None to 0
|
||
|
|
2026-02-27:11:43:03 WARNING [evaluator:333] Overwriting default num_fewshot of haerae_general_knowledge from None to 0
|
||
|
|
2026-02-27:11:43:03 WARNING [evaluator:333] Overwriting default num_fewshot of kobest_copa from None to 0
|
||
|
|
2026-02-27:11:43:03 WARNING [evaluator:333] Overwriting default num_fewshot of kobest_boolq from None to 0
|
||
|
|
2026-02-27:11:43:03 INFO [api.task:311] Building contexts for paws_ko on rank 0...
|
||
|
|
0%| | 0/2000 [00:00<?, ?it/s]
6%|▋ | 125/2000 [00:00<00:01, 1245.64it/s]
12%|█▎ | 250/2000 [00:00<00:01, 1244.84it/s]
19%|█▉ | 375/2000 [00:00<00:01, 1245.80it/s]
25%|██▌ | 501/2000 [00:00<00:01, 1249.00it/s]
31%|███▏ | 628/2000 [00:00<00:01, 1253.42it/s]
38%|███▊ | 755/2000 [00:00<00:00, 1258.99it/s]
44%|████▍ | 883/2000 [00:00<00:00, 1262.94it/s]
50%|█████ | 1010/2000 [00:00<00:00, 1264.62it/s]
57%|█████▋ | 1137/2000 [00:00<00:00, 1265.37it/s]
63%|██████▎ | 1264/2000 [00:01<00:00, 1250.69it/s]
70%|██████▉ | 1391/2000 [00:01<00:00, 1254.63it/s]
76%|███████▌ | 1519/2000 [00:01<00:00, 1260.47it/s]
82%|████████▏ | 1647/2000 [00:01<00:00, 1265.42it/s]
89%|████████▉ | 1775/2000 [00:01<00:00, 1267.53it/s]
95%|█████████▌| 1902/2000 [00:01<00:00, 1266.73it/s]
100%|██████████| 2000/2000 [00:01<00:00, 1101.53it/s]
|
||
|
|
2026-02-27:11:43:04 INFO [api.task:311] Building contexts for haerae_history on rank 0...
|
||
|
|
0%| | 0/188 [00:00<?, ?it/s]
100%|██████████| 188/188 [00:00<00:00, 3762.73it/s]
|
||
|
|
2026-02-27:11:43:05 INFO [api.task:311] Building contexts for haerae_general_knowledge on rank 0...
|
||
|
|
0%| | 0/176 [00:00<?, ?it/s]
100%|██████████| 176/176 [00:00<00:00, 3825.31it/s]
|
||
|
|
2026-02-27:11:43:05 INFO [api.task:311] Building contexts for kobest_copa on rank 0...
|
||
|
|
0%| | 0/1000 [00:00<?, ?it/s]
100%|██████████| 1000/1000 [00:00<00:00, 195229.19it/s]
|
||
|
|
2026-02-27:11:43:05 INFO [api.task:311] Building contexts for kobest_boolq on rank 0...
|
||
|
|
0%| | 0/1404 [00:00<?, ?it/s]
21%|██ | 298/1404 [00:00<00:00, 2978.72it/s]
43%|████▎ | 599/1404 [00:00<00:00, 2992.01it/s]
64%|██████▍ | 899/1404 [00:00<00:00, 2995.38it/s]
85%|████████▌ | 1199/1404 [00:00<00:00, 2992.41it/s]
100%|██████████| 1404/1404 [00:00<00:00, 2990.89it/s]
|
||
|
|
2026-02-27:11:43:05 INFO [evaluator:584] Running loglikelihood requests
|
||
|
|
Running loglikelihood requests: 0%| | 0/10628 [00:00<?, ?it/s]
Running loglikelihood requests: 0%| | 1/10628 [00:01<5:23:23, 1.83s/it]
Running loglikelihood requests: 1%| | 129/10628 [00:01<01:58, 88.80it/s]
Running loglikelihood requests: 2%|▏ | 257/10628 [00:02<00:55, 187.27it/s]
Running loglikelihood requests: 4%|▎ | 385/10628 [00:02<00:35, 292.11it/s]
Running loglikelihood requests: 4%|▍ | 458/10628 [00:02<00:35, 289.81it/s]
Running loglikelihood requests: 5%|▌ | 577/10628 [00:02<00:25, 395.42it/s]
Running loglikelihood requests: 7%|▋ | 705/10628 [00:02<00:19, 515.12it/s]
Running loglikelihood requests: 8%|▊ | 833/10628 [00:02<00:15, 629.77it/s]
Running loglikelihood requests: 9%|▉ | 961/10628 [00:03<00:13, 736.53it/s]
Running loglikelihood requests: 10%|█ | 1089/10628 [00:03<00:11, 829.21it/s]
Running loglikelihood requests: 11%|█▏ | 1217/10628 [00:03<00:10, 904.16it/s]
Running loglikelihood requests: 13%|█▎ | 1345/10628 [00:03<00:09, 966.04it/s]
Running loglikelihood requests: 14%|█▍ | 1473/10628 [00:03<00:09, 1015.77it/s]
Running loglikelihood requests: 15%|█▌ | 1601/10628 [00:03<00:08, 1026.68it/s]
Running loglikelihood requests: 17%|█▋ | 1793/10628 [00:03<00:07, 1182.59it/s]
Running loglikelihood requests: 18%|█▊ | 1921/10628 [00:03<00:07, 1180.18it/s]
Running loglikelihood requests: 20%|█▉ | 2113/10628 [00:03<00:06, 1297.26it/s]
Running loglikelihood requests: 22%|██▏ | 2305/10628 [00:04<00:06, 1314.13it/s]
Running loglikelihood requests: 23%|██▎ | 2497/10628 [00:04<00:06, 1331.10it/s]
Running loglikelihood requests: 25%|██▌ | 2689/10628 [00:04<00:05, 1412.99it/s]
Running loglikelihood requests: 27%|██▋ | 2881/10628 [00:04<00:05, 1398.27it/s]
Running loglikelihood requests: 29%|██▉ | 3073/10628 [00:04<00:05, 1466.01it/s]
Running loglikelihood requests: 31%|███ | 3265/10628 [00:04<00:05, 1449.78it/s]
Running loglikelihood requests: 33%|███▎ | 3457/10628 [00:04<00:04, 1509.37it/s]
Running loglikelihood requests: 34%|███▍ | 3649/10628 [00:05<00:04, 1474.97it/s]
Running loglikelihood requests: 36%|███▌ | 3841/10628 [00:05<00:04, 1464.36it/s]
Running loglikelihood requests: 38%|███▊ | 4033/10628 [00:05<00:04, 1455.68it/s]
Running loglikelihood requests: 40%|███▉ | 4225/10628 [00:05<00:04, 1460.91it/s]
Running loglikelihood requests: 42%|████▏ | 4417/10628 [00:05<00:04, 1464.49it/s]
Running loglikelihood requests: 43%|████▎ | 4565/10628 [00:05<00:04, 1468.23it/s]
Running loglikelihood requests: 45%|████▍ | 4737/10628 [00:05<00:04, 1439.85it/s]
Running loglikelihood requests: 46%|████▋ | 4929/10628 [00:05<00:03, 1467.82it/s]
Running loglikelihood requests: 48%|████▊ | 5121/10628 [00:06<00:03, 1495.31it/s]
Running loglikelihood requests: 50%|████▉ | 5313/10628 [00:06<00:03, 1602.51it/s]
Running loglikelihood requests: 52%|█████▏ | 5505/10628 [00:06<00:03, 1601.03it/s]
Running loglikelihood requests: 54%|█████▎ | 5697/10628 [00:06<00:03, 1610.21it/s]
Running loglikelihood requests: 56%|█████▌ | 5928/10628 [00:06<00:02, 1797.07it/s]
Running loglikelihood requests: 58%|█████▊ | 6145/10628 [00:06<00:02, 1718.71it/s]
Running loglikelihood requests: 60%|██████ | 6392/10628 [00:06<00:02, 1916.17it/s]
Running loglikelihood requests: 62%|██████▏ | 6589/10628 [00:06<00:02, 1840.17it/s]
Running loglikelihood requests: 64%|██████▍ | 6785/10628 [00:06<00:02, 1725.73it/s]
Running loglikelihood requests: 66%|██████▌ | 6977/10628 [00:07<00:02, 1733.70it/s]
Running loglikelihood requests: 67%|██████▋ | 7169/10628 [00:07<00:01, 1735.14it/s]
Runnin
|
||
|
|
fatal: (현재 폴더 또는 상위 폴더 부터 마운트 위치 /PROJECT 까지 일부가) 깃 저장소가 아닙니다
|
||
|
|
파일 시스템 경계에서 중지합니다. (GIT_DISCOVERY_ACROSS_FILESYSTEM 설정되지 않음)
|
||
|
|
2026-02-27:11:43:18 INFO [loggers.evaluation_tracker:247] Saving results aggregated
|
||
|
|
2026-02-27:11:43:18 INFO [loggers.evaluation_tracker:119] Saving per-task samples to /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/quick_sft_v2_best/__PROJECT__0325120031_A__ghong__taketimes__llm-bang__outputs__hf_checkpoint-best/*.jsonl
|
||
|
|
Passed argument batch_size = auto:1. Detecting largest batch size
|
||
|
|
Determined largest batch size: 64
|
||
|
|
hf ({'pretrained': '/PROJECT/0325120031_A/ghong/taketimes/llm-bang/outputs/hf_checkpoint-best', 'dtype': 'float16'}), gen_kwargs: ({}), limit: None, num_fewshot: 0, batch_size: auto (64)
|
||
|
|
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|
||
|
|
|------------------------|------:|------|-----:|--------|---|-----:|---|------|
|
||
|
|
|haerae_general_knowledge| 1|none | 0|acc |↑ |0.2273|± |0.0317|
|
||
|
|
| | |none | 0|acc_norm|↑ |0.2273|± |0.0317|
|
||
|
|
|haerae_history | 1|none | 0|acc |↑ |0.1543|± |0.0264|
|
||
|
|
| | |none | 0|acc_norm|↑ |0.1543|± |0.0264|
|
||
|
|
|kobest_boolq | 1|none | 0|acc |↑ |0.5000|± |0.0133|
|
||
|
|
| | |none | 0|f1 |↑ |0.3346|± | N/A|
|
||
|
|
|kobest_copa | 1|none | 0|acc |↑ |0.6460|± |0.0151|
|
||
|
|
| | |none | 0|f1 |↑ |0.6455|± | N/A|
|
||
|
|
|paws_ko | 1|none | 0|acc |↑ |0.4900|± |0.0112|
|
||
|
|
|