394 lines
136 KiB
Plaintext
394 lines
136 KiB
Plaintext
|
2026-03-05:01:45:40 INFO [_cli.run:376] Selected Tasks: ['belebele_kor_Hang', 'global_mmlu_full_ko']
|
|||
|
|
2026-03-05:01:45:40 INFO [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
|||
|
|
2026-03-05:01:45:40 INFO [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/hf_3b_base', 'dtype': 'bfloat16'}
|
|||
|
|
2026-03-05:01:45:42 INFO [models.huggingface:161] Using device 'cuda:2'
|
|||
|
|
2026-03-05:01:45:43 INFO [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:2'}
|
|||
|
|
Loading weights: 0%| | 0/171 [00:00<?, ?it/s]
Loading weights: 1%| | 1/171 [00:00<00:00, 11037.64it/s, Materializing param=lm_head.weight]
Loading weights: 1%| | 1/171 [00:00<00:00, 6967.28it/s, Materializing param=lm_head.weight]
Loading weights: 1%| | 2/171 [00:00<01:14, 2.28it/s, Materializing param=lm_head.weight]
Loading weights: 1%| | 2/171 [00:00<01:14, 2.28it/s, Materializing param=model.embed_tokens.weight]
Loading weights: 1%| | 2/171 [00:00<01:14, 2.28it/s, Materializing param=model.embed_tokens.weight]
Loading weights: 2%|▏ | 3/171 [00:00<01:13, 2.28it/s, Materializing param=model.layers.0.input_layernorm.weight]
Loading weights: 2%|▏ | 3/171 [00:00<01:13, 2.28it/s, Materializing param=model.layers.0.input_layernorm.weight]
Loading weights: 2%|▏ | 4/171 [00:00<01:13, 2.28it/s, Materializing param=model.layers.0.mlp.down_proj.weight]
Loading weights: 2%|▏ | 4/171 [00:00<01:13, 2.28it/s, Materializing param=model.layers.0.mlp.down_proj.weight]
Loading weights: 3%|▎ | 5/171 [00:00<01:12, 2.28it/s, Materializing param=model.layers.0.mlp.gate_proj.weight]
Loading weights: 3%|▎ | 5/171 [00:00<01:12, 2.28it/s, Materializing param=model.layers.0.mlp.gate_proj.weight]
Loading weights: 4%|▎ | 6/171 [00:00<01:12, 2.28it/s, Materializing param=model.layers.0.mlp.up_proj.weight]
Loading weights: 4%|▎ | 6/171 [00:00<01:12, 2.28it/s, Materializing param=model.layers.0.mlp.up_proj.weight]
Loading weights: 4%|▍ | 7/171 [00:00<01:11, 2.28it/s, Materializing param=model.layers.0.post_attention_layernorm.weight]
Loading weights: 4%|▍ | 7/171 [00:00<01:11, 2.28it/s, Materializing param=model.layers.0.post_attention_layernorm.weight]
Loading weights: 5%|▍ | 8/171 [00:00<01:11, 2.28it/s, Materializing param=model.layers.0.self_attn.o_proj.weight]
Loading weights: 5%|▍ | 8/171 [00:00<01:11, 2.28it/s, Materializing param=model.layers.0.self_attn.o_proj.weight]
Loading weights: 5%|▌ | 9/171 [00:00<01:11, 2.28it/s, Materializing param=model.layers.1.input_layernorm.weight]
Loading weights: 5%|▌ | 9/171 [00:00<01:11, 2.28it/s, Materializing param=model.layers.1.input_layernorm.weight]
Loading weights: 6%|▌ | 10/171 [00:00<01:10, 2.28it/s, Materializing param=model.layers.1.mlp.down_proj.weight]
Loading weights: 6%|▌ | 10/171 [00:00<01:10, 2.28it/s, Materializing param=model.layers.1.mlp.down_proj.weight]
Loading weights: 6%|▋ | 11/171 [00:00<01:10, 2.28it/s, Materializing param=model.layers.1.mlp.gate_proj.weight]
Loading weights: 6%|▋ | 11/171 [00:00<01:10, 2.28it/s, Materializing param=model.layers.1.mlp.gate_proj.weight]
Loading weights: 7%|▋ | 12/171 [00:00<01:09, 2.28it/s, Materializing param=model.layers.1.mlp.up_proj.weight]
Loading weights: 7%|▋ | 12/171 [00:00<01:09, 2.28it/s, Materializing param=model.layers.1.mlp.up_proj.weight]
Loading weights: 8%|▊ | 13/171 [00:00<01:09, 2.28it/s, Materializing param=model.layers.1.post_attention_layernorm.weight]
Loading weights: 8%|▊ | 13/171 [00:00<01:09, 2.28it/s, Materializing param=model.layers.1.post_attention_layernorm.weight]
Loading weights: 8%|▊ | 14/171 [00:00<01:08, 2.28it/s, Materializing param=model.layers.1.self_attn.o_proj.weight]
Loading weights: 8%|▊ | 14/171 [00:00<01:08, 2.28it/s, Materializing param=model.layers.1.self_attn.o_proj.weight]
Loading weights: 9%|▉ | 15/171 [00:01<00:08, 19.46it/s, Materializing param=model.layers.1.self_attn.o_proj.weight]
Loading weights: 9%|▉ | 15/171 [00:01<00:08, 19.46it/s, Materializing param=model.layers.2.input_layernorm.weight]
Loading weights: 9%|▉ | 15/171 [00:01<00:08, 19.46it/s, Materializing param=model.layers.2.input_layernorm.weight]
Loading weights: 9%|▉ | 16/171 [00:01<00:
|
|||
|
|
The tied weights mapping and config for this model specifies to tie model.embed_tokens.weight to lm_head.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning
|
|||
|
|
[1mLlamaForCausalLM LOAD REPORT[0m from: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/hf_3b_base
|
|||
|
|
Key | Status |
|
|||
|
|
----------------------------------------------+---------+-
|
|||
|
|
model.layers.{0...27}.self_attn.q_proj.weight | MISSING |
|
|||
|
|
model.layers.{0...27}.self_attn.v_proj.weight | MISSING |
|
|||
|
|
model.layers.{0...27}.self_attn.k_proj.weight | MISSING |
|
|||
|
|
|
|||
|
|
[3mNotes:
|
|||
|
|
- MISSING[3m :those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.[0m
|
|||
|
|
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
|
|||
|
|
Generating test split: 0%| | 0/900 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 900/900 [00:00<00:00, 178793.80 examples/s]
|
|||
|
|
Generating test split: 0%| | 0/14042 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 14042/14042 [00:00<00:00, 785484.55 examples/s]
|
|||
|
|
Generating dev split: 0%| | 0/285 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 285/285 [00:00<00:00, 120453.11 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 48144.37 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 112478.49 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 111205.05 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 60907.81 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111323.89 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110428.16 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 65886.38 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 109308.51 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 108639.32 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 62591.72 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 112723.85 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 111911.01 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 65604.34 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 113015.18 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 112324.43 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 63364.78 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 112597.25 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 111818.59 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 61408.44 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 112774.87 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 112018.08 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 61782.96 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 112190.41 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 111453.58 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 58425.06 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 110765.31 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110063.31 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 57121.26 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111143.70 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110434.78 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 60178.04 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 110261.08 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 109508.51 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 62759.31 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111994.45 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110993.38 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 60311.64 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111576.60 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110818.99 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 61544.39 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111468.37 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110296.63 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 65196.44 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 109774.35 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 108936.31 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 65737.83 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111618.17 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110485.33 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 58772.64 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 109977.03 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 108169.03 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 59990.80 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111575.86 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110240.89 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 67061.80 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111448.63 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110664.49 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 67303.45 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111353.70 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110545.68 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 58333.82 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 110372.55 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 109775.08 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 64545.17 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111225.49 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110424.02 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 59715.09 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 109399.98 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 108948.40 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 63334.57 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111402.50 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110625.83 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 58827.59 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111825.49 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 111037.74 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 64054.05 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 112605.82 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 111475.09 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 64828.71 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111729.43 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110601.94 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 62914.56 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111393.62 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110595.50 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 60845.80 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111298.79 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110502.33 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 65356.84 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111734.63 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110976.23 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 61601.48 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111176.85 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110279.08 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 63381.58 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111445.42 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110702.76 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 56263.61 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 110895.88 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110072.98 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 59521.82 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111488.12 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110697.98 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 63814.68 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111674.89 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110903.72 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 62934.43 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111610.25 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110767.51 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 66024.67 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111281.31 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110496.94 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 60275.14 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111207.06 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110354.71 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 65111.21 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111713.55 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110931.92 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 62604.83 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 108212.64 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 107652.21 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 58445.05 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111081.15 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110315.64 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 60982.38 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111409.89 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110515.81 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 62418.50 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 109277.89 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 108826.81 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 64426.90 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 110805.79 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110026.71 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 61943.03 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 107998.56 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 107156.68 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 63479.19 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 110504.13 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 109694.53 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 60329.90 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111334.98 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110584.91 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 61257.39 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 110938.17 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110219.85 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 59080.54 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111107.64 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110281.35 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 60470.29 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111578.83 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110718.58 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 64277.93 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111285.74 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110500.05 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 65792.10 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111430.37 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110545.68 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 66554.01 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111931.18 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 111083.82 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 60966.83 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 110460.48 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 109693.50 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 63773.83 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111404.72 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110578.68 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 64146.85 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 111702.15 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 110885.97 examples/s]
|
|||
|
|
Filter: 0%| | 0/285 [00:00<?, ? examples/s]
Filter: 100%|██████████| 285/285 [00:00<00:00, 57809.10 examples/s]
|
|||
|
|
Filter: 0%| | 0/14042 [00:00<?, ? examples/s]
Filter: 85%|████████▌ | 12000/14042 [00:00<00:00, 110140.44 examples/s]
Filter: 100%|██████████| 14042/14042 [00:00<00:00, 109313.39 examples/s]
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:700] Selected tasks:
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:703] Group: global_mmlu_full_ko
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:711] Subgroup: global_mmlu_full_ko_stem
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_abstract_algebra (global_mmlu/full/ko/global_mmlu_full_ko_abstract_algebra.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_anatomy (global_mmlu/full/ko/global_mmlu_full_ko_anatomy.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_astronomy (global_mmlu/full/ko/global_mmlu_full_ko_astronomy.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_college_biology (global_mmlu/full/ko/global_mmlu_full_ko_college_biology.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_college_chemistry (global_mmlu/full/ko/global_mmlu_full_ko_college_chemistry.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_college_computer_science (global_mmlu/full/ko/global_mmlu_full_ko_college_computer_science.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_college_mathematics (global_mmlu/full/ko/global_mmlu_full_ko_college_mathematics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_college_physics (global_mmlu/full/ko/global_mmlu_full_ko_college_physics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_computer_security (global_mmlu/full/ko/global_mmlu_full_ko_computer_security.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_conceptual_physics (global_mmlu/full/ko/global_mmlu_full_ko_conceptual_physics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_electrical_engineering (global_mmlu/full/ko/global_mmlu_full_ko_electrical_engineering.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_elementary_mathematics (global_mmlu/full/ko/global_mmlu_full_ko_elementary_mathematics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_biology (global_mmlu/full/ko/global_mmlu_full_ko_high_school_biology.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_chemistry (global_mmlu/full/ko/global_mmlu_full_ko_high_school_chemistry.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_computer_science (global_mmlu/full/ko/global_mmlu_full_ko_high_school_computer_science.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_mathematics (global_mmlu/full/ko/global_mmlu_full_ko_high_school_mathematics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_physics (global_mmlu/full/ko/global_mmlu_full_ko_high_school_physics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_statistics (global_mmlu/full/ko/global_mmlu_full_ko_high_school_statistics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_machine_learning (global_mmlu/full/ko/global_mmlu_full_ko_machine_learning.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:711] Subgroup: global_mmlu_full_ko_other
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_business_ethics (global_mmlu/full/ko/global_mmlu_full_ko_business_ethics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_clinical_knowledge (global_mmlu/full/ko/global_mmlu_full_ko_clinical_knowledge.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_college_medicine (global_mmlu/full/ko/global_mmlu_full_ko_college_medicine.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_global_facts (global_mmlu/full/ko/global_mmlu_full_ko_global_facts.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_human_aging (global_mmlu/full/ko/global_mmlu_full_ko_human_aging.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_management (global_mmlu/full/ko/global_mmlu_full_ko_management.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_marketing (global_mmlu/full/ko/global_mmlu_full_ko_marketing.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_medical_genetics (global_mmlu/full/ko/global_mmlu_full_ko_medical_genetics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_miscellaneous (global_mmlu/full/ko/global_mmlu_full_ko_miscellaneous.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_nutrition (global_mmlu/full/ko/global_mmlu_full_ko_nutrition.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_professional_accounting (global_mmlu/full/ko/global_mmlu_full_ko_professional_accounting.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_professional_medicine (global_mmlu/full/ko/global_mmlu_full_ko_professional_medicine.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_virology (global_mmlu/full/ko/global_mmlu_full_ko_virology.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:711] Subgroup: global_mmlu_full_ko_social_sciences
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_econometrics (global_mmlu/full/ko/global_mmlu_full_ko_econometrics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_geography (global_mmlu/full/ko/global_mmlu_full_ko_high_school_geography.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_government_and_politics (global_mmlu/full/ko/global_mmlu_full_ko_high_school_government_and_politics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_macroeconomics (global_mmlu/full/ko/global_mmlu_full_ko_high_school_macroeconomics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_microeconomics (global_mmlu/full/ko/global_mmlu_full_ko_high_school_microeconomics.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_psychology (global_mmlu/full/ko/global_mmlu_full_ko_high_school_psychology.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_human_sexuality (global_mmlu/full/ko/global_mmlu_full_ko_human_sexuality.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_professional_psychology (global_mmlu/full/ko/global_mmlu_full_ko_professional_psychology.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_public_relations (global_mmlu/full/ko/global_mmlu_full_ko_public_relations.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_security_studies (global_mmlu/full/ko/global_mmlu_full_ko_security_studies.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_sociology (global_mmlu/full/ko/global_mmlu_full_ko_sociology.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_us_foreign_policy (global_mmlu/full/ko/global_mmlu_full_ko_us_foreign_policy.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:711] Subgroup: global_mmlu_full_ko_humanities
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_formal_logic (global_mmlu/full/ko/global_mmlu_full_ko_formal_logic.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_european_history (global_mmlu/full/ko/global_mmlu_full_ko_high_school_european_history.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_us_history (global_mmlu/full/ko/global_mmlu_full_ko_high_school_us_history.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_high_school_world_history (global_mmlu/full/ko/global_mmlu_full_ko_high_school_world_history.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_international_law (global_mmlu/full/ko/global_mmlu_full_ko_international_law.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_jurisprudence (global_mmlu/full/ko/global_mmlu_full_ko_jurisprudence.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_logical_fallacies (global_mmlu/full/ko/global_mmlu_full_ko_logical_fallacies.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_moral_disputes (global_mmlu/full/ko/global_mmlu_full_ko_moral_disputes.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_moral_scenarios (global_mmlu/full/ko/global_mmlu_full_ko_moral_scenarios.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_philosophy (global_mmlu/full/ko/global_mmlu_full_ko_philosophy.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_prehistory (global_mmlu/full/ko/global_mmlu_full_ko_prehistory.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_professional_law (global_mmlu/full/ko/global_mmlu_full_ko_professional_law.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: global_mmlu_full_ko_world_religions (global_mmlu/full/ko/global_mmlu_full_ko_world_religions.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [tasks:691] Task: belebele_kor_Hang (belebele/belebele_kor_Hang.yaml)
|
|||
|
|
2026-03-05:01:48:08 INFO [api.task:311] Building contexts for global_mmlu_full_ko_abstract_algebra on rank 0...
|
|||
|
|
0%| | 0/100 [00:00<?, ?it/s]
100%|██████████| 100/100 [00:00<00:00, 2567.98it/s]
|
|||
|
|
2026-03-05:01:48:08 INFO [api.task:311] Building contexts for global_mmlu_full_ko_anatomy on rank 0...
|
|||
|
|
0%| | 0/135 [00:00<?, ?it/s]
100%|██████████| 135/135 [00:00<00:00, 2610.31it/s]
|
|||
|
|
2026-03-05:01:48:08 INFO [api.task:311] Building contexts for global_mmlu_full_ko_astronomy on rank 0...
|
|||
|
|
0%| | 0/152 [00:00<?, ?it/s]
100%|██████████| 152/152 [00:00<00:00, 2645.68it/s]
|
|||
|
|
2026-03-05:01:48:08 INFO [api.task:311] Building contexts for global_mmlu_full_ko_college_biology on rank 0...
|
|||
|
|
0%| | 0/144 [00:00<?, ?it/s]
100%|██████████| 144/144 [00:00<00:00, 2652.07it/s]
|
|||
|
|
2026-03-05:01:48:08 INFO [api.task:311] Building contexts for global_mmlu_full_ko_college_chemistry on rank 0...
|
|||
|
|
0%| | 0/100 [00:00<?, ?it/s]
100%|██████████| 100/100 [00:00<00:00, 2622.73it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_college_computer_science on rank 0...
|
|||
|
|
0%| | 0/100 [00:00<?, ?it/s]
100%|██████████| 100/100 [00:00<00:00, 2591.04it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_college_mathematics on rank 0...
|
|||
|
|
0%| | 0/100 [00:00<?, ?it/s]
100%|██████████| 100/100 [00:00<00:00, 2609.23it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_college_physics on rank 0...
|
|||
|
|
0%| | 0/102 [00:00<?, ?it/s]
100%|██████████| 102/102 [00:00<00:00, 2601.85it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_computer_security on rank 0...
|
|||
|
|
0%| | 0/100 [00:00<?, ?it/s]
100%|██████████| 100/100 [00:00<00:00, 2626.59it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_conceptual_physics on rank 0...
|
|||
|
|
0%| | 0/235 [00:00<?, ?it/s]
100%|██████████| 235/235 [00:00<00:00, 2642.05it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_electrical_engineering on rank 0...
|
|||
|
|
0%| | 0/145 [00:00<?, ?it/s]
100%|██████████| 145/145 [00:00<00:00, 2634.73it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_elementary_mathematics on rank 0...
|
|||
|
|
0%| | 0/378 [00:00<?, ?it/s]
71%|███████ | 268/378 [00:00<00:00, 2678.07it/s]
100%|██████████| 378/378 [00:00<00:00, 2685.30it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_biology on rank 0...
|
|||
|
|
0%| | 0/310 [00:00<?, ?it/s]
87%|████████▋ | 269/310 [00:00<00:00, 2686.67it/s]
100%|██████████| 310/310 [00:00<00:00, 2682.42it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_chemistry on rank 0...
|
|||
|
|
0%| | 0/203 [00:00<?, ?it/s]
100%|██████████| 203/203 [00:00<00:00, 2653.44it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_computer_science on rank 0...
|
|||
|
|
0%| | 0/100 [00:00<?, ?it/s]
100%|██████████| 100/100 [00:00<00:00, 2633.93it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_mathematics on rank 0...
|
|||
|
|
0%| | 0/270 [00:00<?, ?it/s]
99%|█████████▉| 267/270 [00:00<00:00, 2661.86it/s]
100%|██████████| 270/270 [00:00<00:00, 2659.17it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_physics on rank 0...
|
|||
|
|
0%| | 0/151 [00:00<?, ?it/s]
100%|██████████| 151/151 [00:00<00:00, 2633.09it/s]
|
|||
|
|
2026-03-05:01:48:09 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_statistics on rank 0...
|
|||
|
|
0%| | 0/216 [00:00<?, ?it/s]
100%|██████████| 216/216 [00:00<00:00, 2570.98it/s]
|
|||
|
|
2026-03-05:01:48:10 INFO [api.task:311] Building contexts for global_mmlu_full_ko_machine_learning on rank 0...
|
|||
|
|
0%| | 0/112 [00:00<?, ?it/s]
100%|██████████| 112/112 [00:00<00:00, 2615.28it/s]
|
|||
|
|
2026-03-05:01:48:10 INFO [api.task:311] Building contexts for global_mmlu_full_ko_business_ethics on rank 0...
|
|||
|
|
0%| | 0/100 [00:00<?, ?it/s]
100%|██████████| 100/100 [00:00<00:00, 2617.87it/s]
|
|||
|
|
2026-03-05:01:48:10 INFO [api.task:311] Building contexts for global_mmlu_full_ko_clinical_knowledge on rank 0...
|
|||
|
|
0%| | 0/265 [00:00<?, ?it/s]
100%|██████████| 265/265 [00:00<00:00, 2649.04it/s]
100%|██████████| 265/265 [00:00<00:00, 2647.16it/s]
|
|||
|
|
2026-03-05:01:48:10 INFO [api.task:311] Building contexts for global_mmlu_full_ko_college_medicine on rank 0...
|
|||
|
|
0%| | 0/173 [00:00<?, ?it/s]
100%|██████████| 173/173 [00:00<00:00, 2643.10it/s]
|
|||
|
|
2026-03-05:01:48:10 INFO [api.task:311] Building contexts for global_mmlu_full_ko_global_facts on rank 0...
|
|||
|
|
0%| | 0/100 [00:00<?, ?it/s]
100%|██████████| 100/100 [00:00<00:00, 2626.84it/s]
|
|||
|
|
2026-03-05:01:48:10 INFO [api.task:311] Building contexts for global_mmlu_full_ko_human_aging on rank 0...
|
|||
|
|
0%| | 0/223 [00:00<?, ?it/s]
100%|██████████| 223/223 [00:00<00:00, 2658.30it/s]
|
|||
|
|
2026-03-05:01:48:10 INFO [api.task:311] Building contexts for global_mmlu_full_ko_management on rank 0...
|
|||
|
|
0%| | 0/103 [00:00<?, ?it/s]
100%|██████████| 103/103 [00:00<00:00, 2635.03it/s]
|
|||
|
|
2026-03-05:01:48:10 INFO [api.task:311] Building contexts for global_mmlu_full_ko_marketing on rank 0...
|
|||
|
|
0%| | 0/234 [00:00<?, ?it/s]
100%|██████████| 234/234 [00:00<00:00, 2645.74it/s]
|
|||
|
|
2026-03-05:01:48:10 INFO [api.task:311] Building contexts for global_mmlu_full_ko_medical_genetics on rank 0...
|
|||
|
|
0%| | 0/100 [00:00<?, ?it/s]
100%|██████████| 100/100 [00:00<00:00, 2616.70it/s]
|
|||
|
|
2026-03-05:01:48:10 INFO [api.task:311] Building contexts for global_mmlu_full_ko_miscellaneous on rank 0...
|
|||
|
|
0%| | 0/783 [00:00<?, ?it/s]
34%|███▍ | 266/783 [00:00<00:00, 2657.08it/s]
68%|██████▊ | 534/783 [00:00<00:00, 2665.01it/s]
100%|██████████| 783/783 [00:00<00:00, 2668.39it/s]
|
|||
|
|
2026-03-05:01:48:11 INFO [api.task:311] Building contexts for global_mmlu_full_ko_nutrition on rank 0...
|
|||
|
|
0%| | 0/306 [00:00<?, ?it/s]
87%|████████▋ | 265/306 [00:00<00:00, 2648.78it/s]
100%|██████████| 306/306 [00:00<00:00, 2642.07it/s]
|
|||
|
|
2026-03-05:01:48:11 INFO [api.task:311] Building contexts for global_mmlu_full_ko_professional_accounting on rank 0...
|
|||
|
|
0%| | 0/282 [00:00<?, ?it/s]
95%|█████████▍| 267/282 [00:00<00:00, 2663.42it/s]
100%|██████████| 282/282 [00:00<00:00, 2662.57it/s]
|
|||
|
|
2026-03-05:01:48:11 INFO [api.task:311] Building contexts for global_mmlu_full_ko_professional_medicine on rank 0...
|
|||
|
|
0%| | 0/272 [00:00<?, ?it/s]
98%|█████████▊| 266/272 [00:00<00:00, 2659.76it/s]
100%|██████████| 272/272 [00:00<00:00, 2658.26it/s]
|
|||
|
|
2026-03-05:01:48:11 INFO [api.task:311] Building contexts for global_mmlu_full_ko_virology on rank 0...
|
|||
|
|
0%| | 0/166 [00:00<?, ?it/s]
100%|██████████| 166/166 [00:00<00:00, 2659.39it/s]
|
|||
|
|
2026-03-05:01:48:11 INFO [api.task:311] Building contexts for global_mmlu_full_ko_econometrics on rank 0...
|
|||
|
|
0%| | 0/114 [00:00<?, ?it/s]
100%|██████████| 114/114 [00:00<00:00, 2651.84it/s]
|
|||
|
|
2026-03-05:01:48:11 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_geography on rank 0...
|
|||
|
|
0%| | 0/198 [00:00<?, ?it/s]
100%|██████████| 198/198 [00:00<00:00, 2631.49it/s]
|
|||
|
|
2026-03-05:01:48:11 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_government_and_politics on rank 0...
|
|||
|
|
0%| | 0/193 [00:00<?, ?it/s]
100%|██████████| 193/193 [00:00<00:00, 2658.01it/s]
|
|||
|
|
2026-03-05:01:48:11 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_macroeconomics on rank 0...
|
|||
|
|
0%| | 0/390 [00:00<?, ?it/s]
68%|██████▊ | 266/390 [00:00<00:00, 2658.97it/s]
100%|██████████| 390/390 [00:00<00:00, 2662.06it/s]
|
|||
|
|
2026-03-05:01:48:11 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_microeconomics on rank 0...
|
|||
|
|
0%| | 0/238 [00:00<?, ?it/s]
100%|██████████| 238/238 [00:00<00:00, 2673.17it/s]
|
|||
|
|
2026-03-05:01:48:12 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_psychology on rank 0...
|
|||
|
|
0%| | 0/545 [00:00<?, ?it/s]
49%|████▉ | 266/545 [00:00<00:00, 2659.89it/s]
98%|█████████▊| 533/545 [00:00<00:00, 2664.62it/s]
100%|██████████| 545/545 [00:00<00:00, 2663.47it/s]
|
|||
|
|
2026-03-05:01:48:12 INFO [api.task:311] Building contexts for global_mmlu_full_ko_human_sexuality on rank 0...
|
|||
|
|
0%| | 0/131 [00:00<?, ?it/s]
100%|██████████| 131/131 [00:00<00:00, 2643.12it/s]
|
|||
|
|
2026-03-05:01:48:12 INFO [api.task:311] Building contexts for global_mmlu_full_ko_professional_psychology on rank 0...
|
|||
|
|
0%| | 0/612 [00:00<?, ?it/s]
44%|████▎ | 267/612 [00:00<00:00, 2664.93it/s]
88%|████████▊ | 537/612 [00:00<00:00, 2685.39it/s]
100%|██████████| 612/612 [00:00<00:00, 2681.13it/s]
|
|||
|
|
2026-03-05:01:48:12 INFO [api.task:311] Building contexts for global_mmlu_full_ko_public_relations on rank 0...
|
|||
|
|
0%| | 0/110 [00:00<?, ?it/s]
100%|██████████| 110/110 [00:00<00:00, 2629.52it/s]
|
|||
|
|
2026-03-05:01:48:12 INFO [api.task:311] Building contexts for global_mmlu_full_ko_security_studies on rank 0...
|
|||
|
|
0%| | 0/245 [00:00<?, ?it/s]
100%|██████████| 245/245 [00:00<00:00, 2660.38it/s]
|
|||
|
|
2026-03-05:01:48:12 INFO [api.task:311] Building contexts for global_mmlu_full_ko_sociology on rank 0...
|
|||
|
|
0%| | 0/201 [00:00<?, ?it/s]
100%|██████████| 201/201 [00:00<00:00, 2659.71it/s]
|
|||
|
|
2026-03-05:01:48:12 INFO [api.task:311] Building contexts for global_mmlu_full_ko_us_foreign_policy on rank 0...
|
|||
|
|
0%| | 0/100 [00:00<?, ?it/s]
100%|██████████| 100/100 [00:00<00:00, 2654.17it/s]
|
|||
|
|
2026-03-05:01:48:12 INFO [api.task:311] Building contexts for global_mmlu_full_ko_formal_logic on rank 0...
|
|||
|
|
0%| | 0/126 [00:00<?, ?it/s]
100%|██████████| 126/126 [00:00<00:00, 2607.54it/s]
|
|||
|
|
2026-03-05:01:48:12 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_european_history on rank 0...
|
|||
|
|
0%| | 0/165 [00:00<?, ?it/s]
100%|██████████| 165/165 [00:00<00:00, 2635.66it/s]
|
|||
|
|
2026-03-05:01:48:13 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_us_history on rank 0...
|
|||
|
|
0%| | 0/204 [00:00<?, ?it/s]
100%|██████████| 204/204 [00:00<00:00, 2639.29it/s]
|
|||
|
|
2026-03-05:01:48:13 INFO [api.task:311] Building contexts for global_mmlu_full_ko_high_school_world_history on rank 0...
|
|||
|
|
0%| | 0/237 [00:00<?, ?it/s]
100%|██████████| 237/237 [00:00<00:00, 2654.20it/s]
|
|||
|
|
2026-03-05:01:48:13 INFO [api.task:311] Building contexts for global_mmlu_full_ko_international_law on rank 0...
|
|||
|
|
0%| | 0/121 [00:00<?, ?it/s]
100%|██████████| 121/121 [00:00<00:00, 2629.51it/s]
|
|||
|
|
2026-03-05:01:48:13 INFO [api.task:311] Building contexts for global_mmlu_full_ko_jurisprudence on rank 0...
|
|||
|
|
0%| | 0/108 [00:00<?, ?it/s]
100%|██████████| 108/108 [00:00<00:00, 2619.73it/s]
|
|||
|
|
2026-03-05:01:48:13 INFO [api.task:311] Building contexts for global_mmlu_full_ko_logical_fallacies on rank 0...
|
|||
|
|
0%| | 0/163 [00:00<?, ?it/s]
100%|██████████| 163/163 [00:00<00:00, 2664.95it/s]
|
|||
|
|
2026-03-05:01:48:13 INFO [api.task:311] Building contexts for global_mmlu_full_ko_moral_disputes on rank 0...
|
|||
|
|
0%| | 0/346 [00:00<?, ?it/s]
77%|███████▋ | 267/346 [00:00<00:00, 2663.62it/s]
100%|██████████| 346/346 [00:00<00:00, 2667.41it/s]
|
|||
|
|
2026-03-05:01:48:13 INFO [api.task:311] Building contexts for global_mmlu_full_ko_moral_scenarios on rank 0...
|
|||
|
|
0%| | 0/895 [00:00<?, ?it/s]
30%|██▉ | 267/895 [00:00<00:00, 2667.60it/s]
60%|█████▉ | 535/895 [00:00<00:00, 2674.03it/s]
90%|████████▉ | 803/895 [00:00<00:00, 2673.21it/s]
100%|██████████| 895/895 [00:00<00:00, 2672.90it/s]
|
|||
|
|
2026-03-05:01:48:13 INFO [api.task:311] Building contexts for global_mmlu_full_ko_philosophy on rank 0...
|
|||
|
|
0%| | 0/311 [00:00<?, ?it/s]
86%|████████▌ | 266/311 [00:00<00:00, 2650.97it/s]
100%|██████████| 311/311 [00:00<00:00, 2649.39it/s]
|
|||
|
|
2026-03-05:01:48:14 INFO [api.task:311] Building contexts for global_mmlu_full_ko_prehistory on rank 0...
|
|||
|
|
0%| | 0/324 [00:00<?, ?it/s]
83%|████████▎ | 269/324 [00:00<00:00, 2682.21it/s]
100%|██████████| 324/324 [00:00<00:00, 2676.35it/s]
|
|||
|
|
2026-03-05:01:48:14 INFO [api.task:311] Building contexts for global_mmlu_full_ko_professional_law on rank 0...
|
|||
|
|
0%| | 0/1534 [00:00<?, ?it/s]
18%|█▊ | 269/1534 [00:00<00:00, 2683.96it/s]
35%|███▌ | 539/1534 [00:00<00:00, 2688.33it/s]
53%|█████▎ | 808/1534 [00:00<00:00, 2685.68it/s]
70%|███████ | 1077/1534 [00:00<00:00, 2683.27it/s]
88%|████████▊ | 1346/1534 [00:00<00:00, 2681.83it/s]
100%|██████████| 1534/1534 [00:00<00:00, 2683.84it/s]
|
|||
|
|
2026-03-05:01:48:14 INFO [api.task:311] Building contexts for global_mmlu_full_ko_world_religions on rank 0...
|
|||
|
|
0%| | 0/171 [00:00<?, ?it/s]
100%|██████████| 171/171 [00:00<00:00, 2544.60it/s]
|
|||
|
|
2026-03-05:01:48:14 INFO [api.task:311] Building contexts for belebele_kor_Hang on rank 0...
|
|||
|
|
0%| | 0/900 [00:00<?, ?it/s]
11%|█ | 100/900 [00:00<00:02, 270.80it/s]
25%|██▌ | 226/900 [00:00<00:01, 546.15it/s]
39%|███▉ | 354/900 [00:00<00:00, 753.67it/s]
54%|█████▍ | 485/900 [00:00<00:00, 913.87it/s]
69%|██████▊ | 617/900 [00:00<00:00, 1030.62it/s]
83%|████████▎ | 749/900 [00:00<00:00, 1114.43it/s]
98%|█████████▊| 881/900 [00:00<00:00, 1174.47it/s]
100%|██████████| 900/900 [00:00<00:00, 912.40it/s]
|
|||
|
|
2026-03-05:01:48:15 INFO [evaluator:584] Running loglikelihood requests
|
|||
|
|
Running loglikelihood requests: 0%| | 0/59768 [00:00<?, ?it/s]
Running loglikelihood requests: 0%| | 1/59768 [00:01<17:47:28, 1.07s/it]
Running loglikelihood requests: 0%| | 65/59768 [00:01<13:57, 71.28it/s]
Running loglikelihood requests: 0%| | 129/59768 [00:01<07:02, 141.04it/s]
Running loglikelihood requests: 0%| | 193/59768 [00:01<04:47, 207.33it/s]
Running loglikelihood requests: 0%| | 257/59768 [00:01<03:42, 267.98it/s]
Running loglikelihood requests: 1%| | 321/59768 [00:01<03:05, 319.97it/s]
Running loglikelihood requests: 1%| | 385/59768 [00:01<02:43, 362.69it/s]
Running loglikelihood requests: 1%| | 449/59768 [00:02<02:28, 399.14it/s]
Running loglikelihood requests: 1%| | 513/59768 [00:02<02:17, 432.35it/s]
Running loglikelihood requests: 1%| | 577/59768 [00:02<02:09, 457.33it/s]
Running loglikelihood requests: 1%| | 641/59768 [00:02<02:03, 477.92it/s]
Running loglikelihood requests: 1%| | 705/59768 [00:02<02:00, 491.75it/s]
Running loglikelihood requests: 1%|▏ | 769/59768 [00:02<01:58, 497.98it/s]
Running loglikelihood requests: 1%|▏ | 833/59768 [00:02<01:55, 509.62it/s]
Running loglikelihood requests: 2%|▏ | 897/59768 [00:02<01:53, 520.43it/s]
Running loglikelihood requests: 2%|▏ | 961/59768 [00:02<01:51, 528.98it/s]
Running loglikelihood requests: 2%|▏ | 1025/59768 [00:03<01:49, 536.65it/s]
Running loglikelihood requests: 2%|▏ | 1089/59768 [00:03<01:48, 540.03it/s]
Running loglikelihood requests: 2%|▏ | 1153/59768 [00:03<01:47, 544.61it/s]
Running loglikelihood requests: 2%|▏ | 1217/59768 [00:03<01:47, 545.19it/s]
Running loglikelihood requests: 2%|▏ | 1281/59768 [00:03<01:46, 547.38it/s]
Running loglikelihood requests: 2%|▏ | 1345/59768 [00:03<01:46, 547.67it/s]
Running loglikelihood requests: 2%|▏ | 1409/59768 [00:03<01:45, 552.93it/s]
Running loglikelihood requests: 2%|▏ | 1473/59768 [00:03<01:44, 557.23it/s]
Running loglikelihood requests: 3%|▎ | 1537/59768 [00:04<01:43, 560.44it/s]
Running loglikelihood requests: 3%|▎ | 1601/59768 [00:04<01:43, 563.55it/s]
Running loglikelihood requests: 3%|▎ | 1697/59768 [00:04<01:36, 601.76it/s]
Running loglikelihood requests: 3%|▎ | 1761/59768 [00:04<01:38, 589.81it/s]
Running loglikelihood requests: 3%|▎ | 1825/59768 [00:04<01:38, 585.57it/s]
Running loglikelihood requests: 3%|▎ | 1889/59768 [00:04<01:39, 581.46it/s]
Running loglikelihood requests: 3%|▎ | 1953/59768 [00:04<01:39, 580.57it/s]
Running loglikelihood requests: 3%|▎ | 2017/59768 [00:04<01:39, 579.90it/s]
Running loglikelihood requests: 3%|▎ | 2081/59768 [00:04<01:39, 576.95it/s]
Running loglikelihood requests: 4%|▎ | 2177/59768 [00:05<01:32, 620.28it/s]
Running loglikelihood requests: 4%|▍ | 2273/59768 [00:05<01:28, 649.23it/s]
Running loglikelihood requests: 4%|▍ | 2338/59768 [00:05<01:30, 633.97it/s]
Running loglikelihood requests: 4%|▍ | 2401/59768 [00:05<01:33, 615.79it/s]
Running loglikelihood requests: 4%|▍ | 2497/59768 [00:05<01:28, 647.91it/s]
Running loglikelihood requests: 4%|▍ | 2562/59768 [00:05<01:30, 633.16it/s]
Running loglikelihood requests: 4%|▍ | 2657/59768 [00:05<01:26, 657.08it/s]
Running loglikelihood requests: 5%|▍ | 2753/59768 [00:05<01:25, 667.85it/s]
Running loglikelihood requests: 5%|▍ | 2849/59768 [00:06<01:23, 681.08it/s]
Running loglikelihood requests: 5%|▍ | 2945/59768 [00:06<01:16, 745.98it/s]
Running loglikelihood requests: 5%|▌ | 3021/59768 [00:06<01:17, 734.44it/s]
Running loglikelihood requests: 5%|▌ | 3105/59768 [00:06<01:19, 708.38it/s]
Running loglikelihood requests: 5%|▌ | 3201/59768 [00:06<01:18, 718.32it/s]
Running logl
|
|||
|
|
2026-03-05:01:49:09 INFO [loggers.evaluation_tracker:247] Saving results aggregated
|
|||
|
|
hf ({'pretrained': '/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/outputs/hf_3b_base', 'dtype': 'bfloat16'}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 8
|
|||
|
|
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|
|||
|
|
|-----------------------------------------------------------|------:|------|-----:|--------|---|-----:|---|-----:|
|
|||
|
|
|belebele_kor_Hang | 0|none | 0|acc |↑ |0.2189|± |0.0138|
|
|||
|
|
| | |none | 0|acc_norm|↑ |0.2189|± |0.0138|
|
|||
|
|
|global_mmlu_full_ko | 0|none | |acc |↑ |0.2339|± |0.0036|
|
|||
|
|
| - global_mmlu_full_ko_humanities | 0|none | |acc |↑ |0.2389|± |0.0062|
|
|||
|
|
| - global_mmlu_full_ko_formal_logic | 0|none | 0|acc |↑ |0.2857|± |0.0404|
|
|||
|
|
| - global_mmlu_full_ko_high_school_european_history | 0|none | 0|acc |↑ |0.2000|± |0.0312|
|
|||
|
|
| - global_mmlu_full_ko_high_school_us_history | 0|none | 0|acc |↑ |0.2500|± |0.0304|
|
|||
|
|
| - global_mmlu_full_ko_high_school_world_history | 0|none | 0|acc |↑ |0.2447|± |0.0280|
|
|||
|
|
| - global_mmlu_full_ko_international_law | 0|none | 0|acc |↑ |0.1653|± |0.0339|
|
|||
|
|
| - global_mmlu_full_ko_jurisprudence | 0|none | 0|acc |↑ |0.2130|± |0.0396|
|
|||
|
|
| - global_mmlu_full_ko_logical_fallacies | 0|none | 0|acc |↑ |0.2147|± |0.0323|
|
|||
|
|
| - global_mmlu_full_ko_moral_disputes | 0|none | 0|acc |↑ |0.2312|± |0.0227|
|
|||
|
|
| - global_mmlu_full_ko_moral_scenarios | 0|none | 0|acc |↑ |0.2536|± |0.0146|
|
|||
|
|
| - global_mmlu_full_ko_philosophy | 0|none | 0|acc |↑ |0.2219|± |0.0236|
|
|||
|
|
| - global_mmlu_full_ko_prehistory | 0|none | 0|acc |↑ |0.2500|± |0.0241|
|
|||
|
|
| - global_mmlu_full_ko_professional_law | 0|none | 0|acc |↑ |0.2373|± |0.0109|
|
|||
|
|
| - global_mmlu_full_ko_world_religions | 0|none | 0|acc |↑ |0.2749|± |0.0342|
|
|||
|
|
| - global_mmlu_full_ko_other | 0|none | |acc |↑ |0.2327|± |0.0076|
|
|||
|
|
| - global_mmlu_full_ko_business_ethics | 0|none | 0|acc |↑ |0.3000|± |0.0461|
|
|||
|
|
| - global_mmlu_full_ko_clinical_knowledge | 0|none | 0|acc |↑ |0.2340|± |0.0261|
|
|||
|
|
| - global_mmlu_full_ko_college_medicine | 0|none | 0|acc |↑ |0.2543|± |0.0332|
|
|||
|
|
| - global_mmlu_full_ko_global_facts | 0|none | 0|acc |↑ |0.2200|± |0.0416|
|
|||
|
|
| - global_mmlu_full_ko_human_aging | 0|none | 0|acc |↑ |0.2511|± |0.0291|
|
|||
|
|
| - global_mmlu_full_ko_management | 0|none | 0|acc |↑ |0.2718|± |0.0441|
|
|||
|
|
| - global_mmlu_full_ko_marketing | 0|none | 0|acc |↑ |0.2991|± |0.0300|
|
|||
|
|
| - global_mmlu_full_ko_medical_genetics | 0|none | 0|acc |↑ |0.2500|± |0.0435|
|
|||
|
|
| - global_mmlu_full_ko_miscellaneous | 0|none | 0|acc |↑ |0.2031|± |0.0144|
|
|||
|
|
| - global_mmlu_full_ko_nutrition | 0|none | 0|acc |↑ |0.2222|± |0.0238|
|
|||
|
|
| - global_mmlu_full_ko_professional_accounting | 0|none | 0|acc |↑ |0.2057|± |0.0241|
|
|||
|
|
| - global_mmlu_full_ko_professional_medicine | 0|none | 0|acc |↑ |0.2059|± |0.0246|
|
|||
|
|
| - global_mmlu_full_ko_virology | 0|none | 0|acc |↑ |0.2711|± |0.0346|
|
|||
|
|
| - global_mmlu_full_ko_social_sciences | 0|none | |acc |↑ |0.2301|± |0.0076|
|
|||
|
|
| - global_mmlu_full_ko_econometrics | 0|none | 0|acc |↑ |0.2456|± |0.0405|
|
|||
|
|
| - global_mmlu_full_ko_high_school_geography | 0|none | 0|acc |↑ |0.2828|± |0.0321|
|
|||
|
|
| - global_mmlu_full_ko_high_school_government_and_politics| 0|none | 0|acc |↑ |0.1762|± |0.0275|
|
|||
|
|
| - global_mmlu_full_ko_high_school_macroeconomics | 0|none | 0|acc |↑ |0.2128|± |0.0208|
|
|||
|
|
| - global_mmlu_full_ko_high_school_microeconomics | 0|none | 0|acc |↑ |0.2269|± |0.0272|
|
|||
|
|
| - global_mmlu_full_ko_high_school_psychology | 0|none | 0|acc |↑ |0.2440|± |0.0184|
|
|||
|
|
| - global_mmlu_full_ko_human_sexuality | 0|none | 0|acc |↑ |0.1832|± |0.0339|
|
|||
|
|
| - global_mmlu_full_ko_professional_psychology | 0|none | 0|acc |↑ |0.2337|± |0.0171|
|
|||
|
|
| - global_mmlu_full_ko_public_relations | 0|none | 0|acc |↑ |0.2636|± |0.0422|
|
|||
|
|
| - global_mmlu_full_ko_security_studies | 0|none | 0|acc |↑ |0.2000|± |0.0256|
|
|||
|
|
| - global_mmlu_full_ko_sociology | 0|none | 0|acc |↑ |0.2289|± |0.0297|
|
|||
|
|
| - global_mmlu_full_ko_us_foreign_policy | 0|none | 0|acc |↑ |0.2900|± |0.0456|
|
|||
|
|
| - global_mmlu_full_ko_stem | 0|none | |acc |↑ |0.2312|± |0.0075|
|
|||
|
|
| - global_mmlu_full_ko_abstract_algebra | 0|none | 0|acc |↑ |0.2100|± |0.0409|
|
|||
|
|
| - global_mmlu_full_ko_anatomy | 0|none | 0|acc |↑ |0.2296|± |0.0363|
|
|||
|
|
| - global_mmlu_full_ko_astronomy | 0|none | 0|acc |↑ |0.2237|± |0.0339|
|
|||
|
|
| - global_mmlu_full_ko_college_biology | 0|none | 0|acc |↑ |0.2569|± |0.0365|
|
|||
|
|
| - global_mmlu_full_ko_college_chemistry | 0|none | 0|acc |↑ |0.2100|± |0.0409|
|
|||
|
|
| - global_mmlu_full_ko_college_computer_science | 0|none | 0|acc |↑ |0.2100|± |0.0409|
|
|||
|
|
| - global_mmlu_full_ko_college_mathematics | 0|none | 0|acc |↑ |0.2700|± |0.0446|
|
|||
|
|
| - global_mmlu_full_ko_college_physics | 0|none | 0|acc |↑ |0.2451|± |0.0428|
|
|||
|
|
| - global_mmlu_full_ko_computer_security | 0|none | 0|acc |↑ |0.3100|± |0.0465|
|
|||
|
|
| - global_mmlu_full_ko_conceptual_physics | 0|none | 0|acc |↑ |0.2511|± |0.0283|
|
|||
|
|
| - global_mmlu_full_ko_electrical_engineering | 0|none | 0|acc |↑ |0.2138|± |0.0342|
|
|||
|
|
| - global_mmlu_full_ko_elementary_mathematics | 0|none | 0|acc |↑ |0.2090|± |0.0209|
|
|||
|
|
| - global_mmlu_full_ko_high_school_biology | 0|none | 0|acc |↑ |0.2258|± |0.0238|
|
|||
|
|
| - global_mmlu_full_ko_high_school_chemistry | 0|none | 0|acc |↑ |0.1823|± |0.0272|
|
|||
|
|
| - global_mmlu_full_ko_high_school_computer_science | 0|none | 0|acc |↑ |0.2300|± |0.0423|
|
|||
|
|
| - global_mmlu_full_ko_high_school_mathematics | 0|none | 0|acc |↑ |0.2593|± |0.0267|
|
|||
|
|
| - global_mmlu_full_ko_high_school_physics | 0|none | 0|acc |↑ |0.2384|± |0.0348|
|
|||
|
|
| - global_mmlu_full_ko_high_school_statistics | 0|none | 0|acc |↑ |0.1898|± |0.0267|
|
|||
|
|
| - global_mmlu_full_ko_machine_learning | 0|none | 0|acc |↑ |0.3125|± |0.0440|
|
|||
|
|
|
|||
|
|
| Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
|
|||
|
|
|--------------------------------------|------:|------|------|------|---|-----:|---|-----:|
|
|||
|
|
|global_mmlu_full_ko | 0|none | |acc |↑ |0.2339|± |0.0036|
|
|||
|
|
| - global_mmlu_full_ko_humanities | 0|none | |acc |↑ |0.2389|± |0.0062|
|
|||
|
|
| - global_mmlu_full_ko_other | 0|none | |acc |↑ |0.2327|± |0.0076|
|
|||
|
|
| - global_mmlu_full_ko_social_sciences| 0|none | |acc |↑ |0.2301|± |0.0076|
|
|||
|
|
| - global_mmlu_full_ko_stem | 0|none | |acc |↑ |0.2312|± |0.0075|
|
|||
|
|
|