lomahony/eleuther-pythia410m-hh-sft
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
eleuther-pythia410m-hh-sft/sft-410m-eval-files/sft-pythia-410m-5shot-shelloutput.txt

515 lines
330 KiB
Plaintext
Raw Permalink Normal View History

初始化项目，由ModelHub XC社区提供模型 Model: lomahony/eleuther-pythia410m-hh-sft Source: Original Platform
2026-07-26 01:14:12 +08:00
2023-07-29:11:33:11,881 INFO [utils.py:148] Note: NumExpr detected 56 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-07-29:11:33:11,881 INFO [utils.py:160] NumExpr defaulting to 8 threads.
2023-07-29:11:33:13,402 WARNING [__init__.py:52] Failed to load config in
/home/laura/lm-evaluation-harness_big_refactor/lm_eval/tasks/hendrycks_ethics/utilitarianism_original.yaml
Config will not be added to registry
Error: argument of type 'NoneType' is not iterable
2023-07-29:11:33:13,462 INFO [main.py:135] Selected Tasks: ['arc_challenge', 'arc_easy', 'boolq', 'hellaswag', 'lambada_openai', 'openbookqa', 'piqa', 'sciq', 'wikitext', 'winogrande']
2023-07-29:11:33:13,525 INFO [huggingface.py:109] Using device 'cuda:0'
2023-07-29:11:33:17,358 WARNING [huggingface.py:246] WARNING: The number of total system GPUs does not match the number of spawned processes. If you would like to use data parallelism, please launch the script with 'accelerate launch *script*'. Current run will proceed with 1 devices.
2023-07-29:11:33:20,762 WARNING [builder.py:816] Found cached dataset ai2_arc (/home/laura/.cache/huggingface/datasets/ai2_arc/ARC-Challenge/1.0.0/1569c2591ea2683779581d9fb467203d9aa95543bb9b75dcfde5da92529fd7f6)
0%| | 0/3 [00:00<?, ?it/s] 100%|██████████| 3/3 [00:00<00:00, 834.19it/s]
2023-07-29:11:33:20,767 WARNING [task.py:701] Task 'arc_challenge': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:20,767 WARNING [task.py:701] Task 'arc_challenge': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:22,667 WARNING [builder.py:816] Found cached dataset ai2_arc (/home/laura/.cache/huggingface/datasets/ai2_arc/ARC-Easy/1.0.0/1569c2591ea2683779581d9fb467203d9aa95543bb9b75dcfde5da92529fd7f6)
0%| | 0/3 [00:00<?, ?it/s] 100%|██████████| 3/3 [00:00<00:00, 1093.12it/s]
2023-07-29:11:33:22,671 WARNING [task.py:701] Task 'arc_easy': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:22,671 WARNING [task.py:701] Task 'arc_easy': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:22,766 WARNING [task.py:571] metric acc is defined, but aggregation is not. using default aggregation=mean
2023-07-29:11:33:22,766 WARNING [task.py:583] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2023-07-29:11:33:24,624 WARNING [builder.py:816] Found cached dataset super_glue (/home/laura/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
0%| | 0/3 [00:00<?, ?it/s] 100%|██████████| 3/3 [00:00<00:00, 1106.97it/s]
2023-07-29:11:33:24,627 WARNING [task.py:701] Task 'boolq': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:24,627 WARNING [task.py:701] Task 'boolq': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:26,729 WARNING [builder.py:816] Found cached dataset hellaswag (/home/laura/.cache/huggingface/datasets/hellaswag/default/0.1.0/512a66dd8b1b1643ab4a48aa4f150d04c91680da6a4096498a5e5f799623d5ae)
0%| | 0/3 [00:00<?, ?it/s] 100%|██████████| 3/3 [00:00<00:00, 665.94it/s]
2023-07-29:11:33:26,734 WARNING [task.py:701] Task 'hellaswag': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:26,734 WARNING [task.py:701] Task 'hellaswag': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:31,046 WARNING [builder.py:816] Found cached dataset lambada_openai (/home/laura/.cache/huggingface/datasets/EleutherAI___lambada_openai/default/1.0.0/57baddecfa09d1790541ef07274c5666abfbe9d2ccd0cd46013cd557b0343095)
0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 925.49it/s]
2023-07-29:11:33:31,048 WARNING [task.py:701] Task 'lambada_openai': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:31,048 WARNING [task.py:302] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2023-07-29:11:33:31,048 WARNING [task.py:701] Task 'lambada_openai': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:31,048 WARNING [task.py:302] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2023-07-29:11:33:32,772 WARNING [builder.py:816] Found cached dataset openbookqa (/home/laura/.cache/huggingface/datasets/openbookqa/main/1.0.1/f338ccacfbc86fb8c2de3aa1c06d2ce686933de3bca284dba97d32592c52b33f)
0%| | 0/3 [00:00<?, ?it/s] 100%|██████████| 3/3 [00:00<00:00, 1027.26it/s]
2023-07-29:11:33:32,775 WARNING [task.py:701] Task 'openbookqa': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:32,775 WARNING [task.py:701] Task 'openbookqa': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:34,773 WARNING [builder.py:816] Found cached dataset piqa (/home/laura/.cache/huggingface/datasets/piqa/plain_text/1.1.0/6c611c1a9bf220943c4174e117d3b660859665baf1d43156230116185312d011)
0%| | 0/3 [00:00<?, ?it/s] 100%|██████████| 3/3 [00:00<00:00, 1058.63it/s]
2023-07-29:11:33:34,776 WARNING [task.py:701] Task 'piqa': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:34,776 WARNING [task.py:701] Task 'piqa': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:37,258 WARNING [builder.py:816] Found cached dataset sciq (/home/laura/.cache/huggingface/datasets/sciq/default/0.1.0/50e5c6e3795b55463819d399ec417bfd4c3c621105e00295ddb5f3633d708493)
0%| | 0/3 [00:00<?, ?it/s] 100%|██████████| 3/3 [00:00<00:00, 1016.31it/s]
2023-07-29:11:33:37,262 WARNING [task.py:701] Task 'sciq': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:37,262 WARNING [task.py:701] Task 'sciq': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:37,682 WARNING [task.py:571] metric word_perplexity is defined, but aggregation is not. using default aggregation=weighted_perplexity
2023-07-29:11:33:37,682 WARNING [task.py:583] metric word_perplexity is defined, but higher_is_better is not. using default higher_is_better=False
2023-07-29:11:33:37,682 WARNING [task.py:571] metric byte_perplexity is defined, but aggregation is not. using default aggregation=weighted_perplexity
2023-07-29:11:33:37,682 WARNING [task.py:583] metric byte_perplexity is defined, but higher_is_better is not. using default higher_is_better=False
2023-07-29:11:33:37,682 WARNING [task.py:571] metric bits_per_byte is defined, but aggregation is not. using default aggregation=bits_per_byte
2023-07-29:11:33:37,682 WARNING [task.py:583] metric bits_per_byte is defined, but higher_is_better is not. using default higher_is_better=False
2023-07-29:11:33:39,704 WARNING [builder.py:816] Found cached dataset wikitext_document_level (/home/laura/.cache/huggingface/datasets/EleutherAI___wikitext_document_level/wikitext-2-raw-v1/1.0.0/c7f10a7786444f898dd236db33d4bee9b130f8cbcac690e7bde9b0d027e19fc1)
0%| | 0/3 [00:00<?, ?it/s] 100%|██████████| 3/3 [00:00<00:00, 1333.36it/s]
2023-07-29:11:33:39,707 WARNING [task.py:701] Task 'wikitext': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:39,707 WARNING [task.py:701] Task 'wikitext': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:41,552 WARNING [builder.py:816] Found cached dataset winogrande (/home/laura/.cache/huggingface/datasets/winogrande/winogrande_xl/1.1.0/a826c3d3506aefe0e9e9390dcb53271070536586bab95849876b2c1743df56e2)
0%| | 0/3 [00:00<?, ?it/s] 100%|██████████| 3/3 [00:00<00:00, 1003.34it/s]
2023-07-29:11:33:41,556 WARNING [task.py:701] Task 'winogrande': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:41,556 WARNING [task.py:701] Task 'winogrande': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2023-07-29:11:33:42,560 INFO [task.py:357] Building contexts for task 'arc_challenge' on rank 0...
2023-07-29:11:33:54,836 INFO [evaluator.py:210] Task: arc_challenge; number of requests on this rank: 4687
2023-07-29:11:33:54,837 INFO [task.py:357] Building contexts for task 'arc_easy' on rank 0...
2023-07-29:11:34:19,682 INFO [evaluator.py:210] Task: arc_easy; number of requests on this rank: 9501
2023-07-29:11:34:19,682 INFO [task.py:357] Building contexts for task 'boolq' on rank 0...
2023-07-29:11:34:34,039 INFO [evaluator.py:210] Task: boolq; number of requests on this rank: 6540
2023-07-29:11:34:34,040 INFO [task.py:357] Building contexts for task 'hellaswag' on rank 0...
2023-07-29:11:38:46,649 INFO [evaluator.py:210] Task: hellaswag; number of requests on this rank: 40168
2023-07-29:11:38:46,651 INFO [task.py:357] Building contexts for task 'lambada_openai' on rank 0...
2023-07-29:11:39:29,240 INFO [evaluator.py:210] Task: lambada_openai; number of requests on this rank: 5153
2023-07-29:11:39:29,241 INFO [task.py:357] Building contexts for task 'openbookqa' on rank 0...
2023-07-29:11:39:33,131 INFO [evaluator.py:210] Task: openbookqa; number of requests on this rank: 2000
2023-07-29:11:39:33,131 INFO [task.py:357] Building contexts for task 'piqa' on rank 0...
2023-07-29:11:39:43,911 INFO [evaluator.py:210] Task: piqa; number of requests on this rank: 3676
2023-07-29:11:39:43,911 INFO [task.py:357] Building contexts for task 'sciq' on rank 0...
2023-07-29:11:39:52,155 INFO [evaluator.py:210] Task: sciq; number of requests on this rank: 4000
2023-07-29:11:39:52,156 INFO [task.py:357] Building contexts for task 'wikitext' on rank 0...
2023-07-29:11:39:52,638 INFO [evaluator.py:210] Task: wikitext; number of requests on this rank: 62
2023-07-29:11:39:52,638 INFO [task.py:357] Building contexts for task 'winogrande' on rank 0...
2023-07-29:11:39:52,740 INFO [evaluator.py:210] Task: winogrande; number of requests on this rank: 2534
2023-07-29:11:39:52,740 INFO [evaluator.py:243] Running loglikelihood requests
0%| | 0/78259 [00:00<?, ?it/s] 0%| | 16/78259 [00:04<5:52:27, 3.70it/s] 0%| | 32/78259 [00:07<5:00:06, 4.34it/s] 0%| | 48/78259 [00:10<4:19:19, 5.03it/s] 0%| | 64/78259 [00:12<3:55:39, 5.53it/s] 0%| | 80/78259 [00:14<3:38:21, 5.97it/s] 0%| | 96/78259 [00:17<3:27:00, 6.29it/s] 0%| | 112/78259 [00:19<3:18:18, 6.57it/s] 0%| | 128/78259 [00:21<3:11:27, 6.80it/s] 0%| | 144/78259 [00:23<3:06:26, 6.98it/s] 0%| | 160/78259 [00:25<3:02:48, 7.12it/s] 0%| | 176/78259 [00:27<2:59:14, 7.26it/s] 0%| | 192/78259 [00:30<2:56:12, 7.38it/s] 0%| | 208/78259 [00:32<2:53:51, 7.48it/s] 0%| | 224/78259 [00:34<2:51:39, 7.58it/s] 0%| | 240/78259 [00:36<2:49:49, 7.66it/s] 0%| | 256/78259 [00:38<2:46:40, 7.80it/s] 0%| | 272/78259 [00:40<2:44:06, 7.92it/s] 0%| | 288/78259 [00:42<2:42:00, 8.02it/s] 0%| | 304/78259 [00:43<2:39:45, 8.13it/s] 0%| | 320/78259 [00:45<2:38:15, 8.21it/s] 0%| | 336/78259 [00:47<2:36:26, 8.30it/s] 0%| | 352/78259 [00:49<2:35:13, 8.36it/s] 0%| | 368/78259 [00:51<2:33:53, 8.44it/s] 0%| | 384/78259 [00:53<2:33:12, 8.47it/s] 1%| | 400/78259 [00:55<2:32:33, 8.51it/s] 1%| | 416/78259 [00:57<2:32:02, 8.53it/s] 1%| | 432/78259 [00:58<2:31:28, 8.56it/s] 1%| | 448/78259 [01:00<2:30:48, 8.60it/s] 1%| | 464/78259 [01:02<2:30:09, 8.64it/s] 1%| | 480/78259 [01:04<2:29:16, 8.68it/s] 1%| | 496/78259 [01:06<2:28:38, 8.72it/s] 1%| | 512/78259 [01:08<2:27:40, 8.77it/s] 1%| | 528/78259 [01:09<2:26:47, 8.83it/s] 1%| | 544/78259 [01:11<2:26:08, 8.86it/s] 1%| | 560/78259 [01:13<2:25:20, 8.91it/s] 1%| | 576/78259 [01:15<2:24:41, 8.95it/s] 1%| | 592/78259 [01:16<2:22:14, 9.10it/s] 1%| | 608/78259 [01:18<2:20:55, 9.18it/s] 1%| | 624/78259 [01:20<2:19:53, 9.25it/s] 1%| | 640/78259 [01:21<2:19:09, 9.30it/s] 1%| | 656/78259 [01:23<2:18:54, 9.31it/s] 1%| | 672/78259 [01:25<2:18:24, 9.34it/s] 1%| | 688/78259 [01:27<2:17:52, 9.38it/s] 1%| | 704/78259 [01:28<2:17:05, 9.43it/s] 1%| | 720/78259 [01:30<2:16:37, 9.46it/s] 1%| | 736/78259 [01:32<2:16:04, 9.49it/s] 1%| | 752/78259 [01:33<2:15:52, 9.51it/s] 1%| | 768/78259 [01:35<2:15:31, 9.53it/s] 1%| | 784/78259 [01:37<2:15:18, 9.54it/s] 1%| | 800/78259 [01:38<2:14:48, 9.58it/s] 1%| | 816/78259 [01:40<2:14:43, 9.58it/s] 1%| | 832/78259 [01:42<2:14:27, 9.60it/s] 1%| | 848/78259 [01:43<2:14:26, 9.60it/s] 1%| | 864/78259 [01:45<2:14:06, 9.62it/s] 1%| | 880/78259 [01:47<2:13:49, 9.64it/s] 1%| | 896/78259 [01:48<2:13:32, 9.66it/s] 1%| | 912/78259 [01:50<2:13:01, 9.69it/s] 1%| | 928/78259 [01:52<2:13:15, 9.67it/s] 1%| | 944/78259 [01:53<2:12:47, 9.70it/s] 1%| | 960/78259 [01:55<2:12:57, 9.69it/s] 1%| | 976/78259 [01:56<2:12:46, 9.70it/s] 1%|▏ | 992/78259 [01:58<2:11:47, 9.77it/s] 1%|▏ | 1008/78259 [02:00<2:10:55, 9.83it/s] 1%|▏ | 1024/78259 [02:01<2:09:55, 9.91it/s] 1%|▏ | 1040/78259 [02:03<2:09:21, 9.95it/s] 1%|▏ | 1056/78259 [02:04<2:09:10, 9.96it/s] 1%|▏ | 1072/78259 [02:06<2:08:52, 9.98it/s] 1%|▏ | 1088/78259 [02:08<2:08:35, 10.00it/s] 1%|▏ | 1104/78259 [02:09<2:08:21, 10.02it/s] 1%|▏ | 1120/78259 [02:11<2:07:53, 10.05it/s] 1%|▏ | 1136/78259 [02:12<2:06:56, 10.13it/s] 1%|▏ | 1152/78259 [02:14<2:06:23, 10.17it/s] 1%|▏ | 1168/78259 [02:15<2:06:11, 10.18it/s] 2%|▏ | 1184/78259 [02:17<2:04:58, 10.28it/s] 2%|▏ | 1200/
2023-07-29:12:27:55,766 INFO [evaluator.py:243] Running loglikelihood_rolling requests
0%| | 0/62 [00:00<?, ?it/s] 2%|▏ | 1/62 [00:00<00:08, 7.11it/s] 3%|▎ | 2/62 [00:00<00:28, 2.09it/s] 5%|▍ | 3/62 [00:01<00:28, 2.09it/s] 6%|▋ | 4/62 [00:02<00:38, 1.50it/s] 8%|▊ | 5/62 [00:02<00:34, 1.67it/s] 10%|▉ | 6/62 [00:03<00:31, 1.80it/s] 11%|█▏ | 7/62 [00:04<00:45, 1.21it/s] 13%|█▎ | 8/62 [00:05<00:38, 1.40it/s] 15%|█▍ | 9/62 [00:06<00:52, 1.00it/s] 16%|█▌ | 10/62 [00:06<00:39, 1.31it/s] 18%|█▊ | 11/62 [00:07<00:34, 1.48it/s] 19%|█▉ | 12/62 [00:08<00:44, 1.12it/s] 21%|██ | 13/62 [00:08<00:32, 1.51it/s] 23%|██▎ | 14/62 [00:09<00:28, 1.66it/s] 26%|██▌ | 16/62 [00:09<00:18, 2.55it/s] 27%|██▋ | 17/62 [00:10<00:23, 1.88it/s] 29%|██▉ | 18/62 [00:10<00:18, 2.34it/s] 31%|███ | 19/62 [00:11<00:21, 1.98it/s] 32%|███▏ | 20/62 [00:12<00:23, 1.78it/s] 34%|███▍ | 21/62 [00:12<00:24, 1.65it/s] 35%|███▌ | 22/62 [00:13<00:19, 2.08it/s] 37%|███▋ | 23/62 [00:13<00:21, 1.82it/s] 39%|███▊ | 24/62 [00:15<00:32, 1.16it/s] 40%|████ | 25/62 [00:15<00:27, 1.33it/s] 42%|████▏ | 26/62 [00:16<00:29, 1.23it/s] 44%|████▎ | 27/62 [00:18<00:34, 1.02it/s] 45%|████▌ | 28/62 [00:18<00:25, 1.35it/s] 50%|█████ | 31/62 [00:19<00:14, 2.12it/s] 52%|█████▏ | 32/62 [00:19<00:12, 2.40it/s] 53%|█████▎ | 33/62 [00:20<00:15, 1.85it/s] 55%|█████▍ | 34/62 [00:20<00:13, 2.14it/s] 56%|█████▋ | 35/62 [00:21<00:17, 1.54it/s] 58%|█████▊ | 36/62 [00:22<00:19, 1.36it/s] 60%|█████▉ | 37/62 [00:22<00:14, 1.70it/s] 61%|██████▏ | 38/62 [00:24<00:22, 1.05it/s] 63%|██████▎ | 39/62 [00:25<00:20, 1.13it/s] 65%|██████▍ | 40/62 [00:26<00:22, 1.03s/it] 66%|██████▌ | 41/62 [00:28<00:23, 1.14s/it] 68%|██████▊ | 42/62 [00:28<00:18, 1.06it/s] 69%|██████▉ | 43/62 [00:28<00:13, 1.42it/s] 71%|███████ | 44/62 [00:29<00:12, 1.41it/s] 73%|███████▎ | 45/62 [00:30<00:10, 1.56it/s] 74%|███████▍ | 46/62 [00:30<00:07, 2.05it/s] 76%|███████▌ | 47/62 [00:30<00:05, 2.56it/s] 77%|███████▋ | 48/62 [00:31<00:07, 1.78it/s] 79%|███████▉ | 49/62 [00:31<00:05, 2.23it/s] 81%|████████ | 50/62 [00:31<00:04, 2.64it/s] 82%|████████▏ | 51/62 [00:32<00:04, 2.45it/s] 84%|████████▍ | 52/62 [00:32<00:04, 2.34it/s] 85%|████████▌ | 53/62 [00:32<00:03, 2.96it/s] 87%|████████▋ | 54/62 [00:33<00:02, 3.63it/s] 89%|████████▊ | 55/62 [00:33<00:01, 4.35it/s] 90%|█████████ | 56/62 [00:33<00:01, 5.12it/s] 92%|█████████▏| 57/62 [00:34<00:02, 2.36it/s] 95%|█████████▌| 59/62 [00:34<00:01, 2.83it/s] 97%|█████████▋| 60/62 [00:35<00:01, 1.99it/s] 98%|█████████▊| 61/62 [00:36<00:00, 1.79it/s] 100%|██████████| 62/62 [00:36<00:00, 1.86it/s] 100%|██████████| 62/62 [00:36<00:00, 1.68it/s]
bootstrapping for stddev: perplexity
0%| | 0/100 [00:00<?, ?it/s] 1%| | 1/100 [00:01<02:05, 1.27s/it] 36%|███▌ | 36/100 [00:01<00:01, 35.57it/s] 57%|█████▋ | 57/100 [00:02<00:01, 29.43it/s] 68%|██████▊ | 68/100 [00:02<00:00, 32.12it/s] 100%|██████████| 100/100 [00:02<00:00, 39.45it/s]
{
"results": {
"arc_challenge": {
"acc,none": 0.2363481228668942,
"acc_stderr,none": 0.012414960524301842,
"acc_norm,none": 0.2696245733788396,
"acc_norm_stderr,none": 0.01296804068686915
},
"arc_easy": {
"acc,none": 0.5505050505050505,
"acc_stderr,none": 0.010207308833916047,
"acc_norm,none": 0.5290404040404041,
"acc_norm_stderr,none": 0.01024246382639562
},
"boolq": {
"acc,none": 0.5015290519877675,
"acc_stderr,none": 0.008745014084425922
},
"hellaswag": {
"acc,none": 0.33808006373232424,
"acc_stderr,none": 0.004720891597174752,
"acc_norm,none": 0.40111531567416847,
"acc_norm_stderr,none": 0.004891226138578059
},
"lambada_openai": {
"perplexity,none": 14.092313022096198,
"perplexity_stderr,none": 0.4548197909283815,
"acc,none": 0.44343101106151755,
"acc_stderr,none": 0.006921251108304399
},
"openbookqa": {
"acc,none": 0.178,
"acc_stderr,none": 0.017123622189062257,
"acc_norm,none": 0.288,
"acc_norm_stderr,none": 0.020271503835075217
},
"piqa": {
"acc,none": 0.6860718171926007,
"acc_stderr,none": 0.010827928134189646,
"acc_norm,none": 0.6844396082698585,
"acc_norm_stderr,none": 0.010843119201758934
},
"sciq": {
"acc,none": 0.886,
"acc_stderr,none": 0.01005510343582333,
"acc_norm,none": 0.879,
"acc_norm_stderr,none": 0.010318210380946094
},
"wikitext": {
"word_perplexity,none": 35.96785758155289,
"byte_perplexity,none": 1.8050958522509775,
"bits_per_byte,none": 0.8520754475808856
},
"winogrande": {
"acc,none": 0.5327545382794001,
"acc_stderr,none": 0.014022300570434137
}
},
"configs": {
"arc_challenge": {
"task": "arc_challenge",
"group": [
"ai2_arc",
"multiple_choice"
],
"dataset_path": "ai2_arc",
"dataset_name": "ARC-Challenge",
"training_split": "train",
"validation_split": "validation",
"test_split": "test",
"doc_to_text": "Question: {{question}}\nAnswer:",
"doc_to_target": "{{choices.label.index(answerKey)}}",
"doc_to_choice": "{{choices.text}}",
"description": "",
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"num_fewshot": 5,
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
},
{
"metric": "acc_norm",
"aggregation": "mean",
"higher_is_better": true
}
],
"output_type": "multiple_choice",
"repeats": 1,
"should_decontaminate": true,
"doc_to_decontamination_query": "Question: {{question}}\nAnswer:"
},
"arc_easy": {
"task": "arc_easy",
"group": [
"ai2_arc",
"multiple_choice"
],
"dataset_path": "ai2_arc",
"dataset_name": "ARC-Easy",
"training_split": "train",
"validation_split": "validation",
"test_split": "test",
"doc_to_text": "Question: {{question}}\nAnswer:",
"doc_to_target": "{{choices.label.index(answerKey)}}",
"doc_to_choice": "{{choices.text}}",
"description": "",
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"num_fewshot": 5,
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
},
{
"metric": "acc_norm",
"aggregation": "mean",
"higher_is_better": true
}
],
"output_type": "multiple_choice",
"repeats": 1,
"should_decontaminate": true,
"doc_to_decontamination_query": "Question: {{question}}\nAnswer:"
},
"boolq": {
"task": "boolq",
"group": [
"super-glue-lm-eval-v1"
],
"dataset_path": "super_glue",
"dataset_name": "boolq",
"training_split": "train",
"validation_split": "validation",
"doc_to_text": "{{passage}}\nQuestion: {{question}}?\nAnswer:",
"doc_to_target": "label",
"doc_to_choice": [
"no",
"yes"
],
"description": "",
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"num_fewshot": 5,
"metric_list": [
{
"metric": "acc"
}
],
"output_type": "multiple_choice",
"repeats": 1,
"should_decontaminate": true,
"doc_to_decontamination_query": "passage"
},
"hellaswag": {
"task": "hellaswag",
"group": [
"multiple_choice"
],
"dataset_path": "hellaswag",
"training_split": "train",
"validation_split": "validation",
"doc_to_text": "{% set text = activity_label ~ ': ' ~ ctx_a ~ ' ' ~ ctx_b.capitalize() %}{{text|trim|replace(' [title]', '. ')|regex_replace('\\[.*?\\]', '')|replace(' ', ' ')}}",
"doc_to_target": "{{label}}",
"doc_to_choice": "{{endings|map('trim')|map('replace', ' [title]', '. ')|map('regex_replace', '\\[.*?\\]', '')|map('replace', ' ', ' ')|list}}",
"description": "",
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"num_fewshot": 5,
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
},
{
"metric": "acc_norm",
"aggregation": "mean",
"higher_is_better": true
}
],
"output_type": "multiple_choice",
"repeats": 1,
"should_decontaminate": false
},
"lambada_openai": {
"task": "lambada_openai",
"group": [
"lambada",
"loglikelihood",
"perplexity"
],
"dataset_path": "EleutherAI/lambada_openai",
"dataset_name": "default",
"test_split": "test",
"template_aliases": "",
"doc_to_text": "{{text.split(' ')[:-1]|join(' ')}}",
"doc_to_target": "{{' '+text.split(' ')[-1]}}",
"description": "",
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"num_fewshot": 5,
"metric_list": [
{
"metric": "perplexity",
"aggregation": "perplexity",
"higher_is_better": false
},
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
}
],
"output_type": "loglikelihood",
"repeats": 1,
"should_decontaminate": true,
"doc_to_decontamination_query": "{{text}}"
},
"openbookqa": {
"task": "openbookqa",
"group": [
"multiple_choice"
],
"dataset_path": "openbookqa",
"dataset_name": "main",
"training_split": "train",
"validation_split": "validation",
"test_split": "test",
"doc_to_text": "question_stem",
"doc_to_target": "{{choices.label.index(answerKey.lstrip())}}",
"doc_to_choice": "{{choices.text}}",
"description": "",
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"num_fewshot": 5,
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
},
{
"metric": "acc_norm",
"aggregation": "mean",
"higher_is_better": true
}
],
"output_type": "multiple_choice",
"repeats": 1,
"should_decontaminate": true,
"doc_to_decontamination_query": "question_stem"
},
"piqa": {
"task": "piqa",
"group": [
"multiple_choice"
],
"dataset_path": "piqa",
"training_split": "train",
"validation_split": "validation",
"doc_to_text": "Question: {{goal}}\nAnswer:",
"doc_to_target": "label",
"doc_to_choice": "{{[sol1, sol2]}}",
"description": "",
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"num_fewshot": 5,
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
},
{
"metric": "acc_norm",
"aggregation": "mean",
"higher_is_better": true
}
],
"output_type": "multiple_choice",
"repeats": 1,
"should_decontaminate": true,
"doc_to_decontamination_query": "goal"
},
"sciq": {
"task": "sciq",
"group": [
"multiple_choice"
],
"dataset_path": "sciq",
"training_split": "train",
"validation_split": "validation",
"test_split": "test",
"doc_to_text": "{{support.lstrip()}}\nQuestion: {{question}}\nAnswer:",
"doc_to_target": 3,
"doc_to_choice": "{{[distractor1, distractor2, distractor3, correct_answer]}}",
"description": "",
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"num_fewshot": 5,
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
},
{
"metric": "acc_norm",
"aggregation": "mean",
"higher_is_better": true
}
],
"output_type": "multiple_choice",
"repeats": 1,
"should_decontaminate": true,
"doc_to_decontamination_query": "{{support}} {{question}}"
},
"wikitext": {
"task": "wikitext",
"group": [
"perplexity",
"loglikelihood_rolling"
],
"dataset_path": "EleutherAI/wikitext_document_level",
"dataset_name": "wikitext-2-raw-v1",
"training_split": "train",
"validation_split": "validation",
"test_split": "test",
"template_aliases": "",
"doc_to_text": "",
"doc_to_target": "<function wikitext_detokenizer at 0x7f9fa4a64040>",
"description": "",
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"num_fewshot": 5,
"metric_list": [
{
"metric": "word_perplexity"
},
{
"metric": "byte_perplexity"
},
{
"metric": "bits_per_byte"
}
],
"output_type": "loglikelihood_rolling",
"repeats": 1,
"should_decontaminate": true,
"doc_to_decontamination_query": "{{page}}"
},
"winogrande": {
"task": "winogrande",
"dataset_path": "winogrande",
"dataset_name": "winogrande_xl",
"training_split": "train",
"validation_split": "validation",
"doc_to_text": "<function doc_to_text at 0x7f9fa4a36ef0>",
"doc_to_target": "<function doc_to_target at 0x7f9fa4a37370>",
"doc_to_choice": "<function doc_to_choice at 0x7f9fa4a375b0>",
"description": "",
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"num_fewshot": 5,
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
}
],
"output_type": "multiple_choice",
"repeats": 1,
"should_decontaminate": false
}
},
"versions": {
"arc_challenge": "Yaml",
"arc_easy": "Yaml",
"boolq": "Yaml",
"hellaswag": "Yaml",
"lambada_openai": "Yaml",
"openbookqa": "Yaml",
"piqa": "Yaml",
"sciq": "Yaml",
"wikitext": "Yaml",
"winogrande": "Yaml"
},
"config": {
"model": "hf",
"model_args": "pretrained=lomahony/eleuther-pythia410m-hh-sft",
"num_fewshot": 5,
"batch_size": 16,
"batch_sizes": [],
"device": "cuda:0",
"use_cache": null,
"limit": null,
"bootstrap_iters": 100000
},
"git_hash": "4e44f0a"
}
hf (pretrained=lomahony/eleuther-pythia410m-hh-sft), limit: None, num_fewshot: 5, batch_size: 16
| Task |Version|Filter| Metric | Value | |Stderr|
|--------------|-------|------|---------------|------:|---|-----:|
|arc_challenge |Yaml |none |acc | 0.2363|± |0.0124|
| | |none |acc_norm | 0.2696|± |0.0130|
|arc_easy |Yaml |none |acc | 0.5505|± |0.0102|
| | |none |acc_norm | 0.5290|± |0.0102|
|boolq |Yaml |none |acc | 0.5015|± |0.0087|
|hellaswag |Yaml |none |acc | 0.3381|± |0.0047|
| | |none |acc_norm | 0.4011|± |0.0049|
|lambada_openai|Yaml |none |perplexity |14.0923|± |0.4548|
| | |none |acc | 0.4434|± |0.0069|
|openbookqa |Yaml |none |acc | 0.1780|± |0.0171|
| | |none |acc_norm | 0.2880|± |0.0203|
|piqa |Yaml |none |acc | 0.6861|± |0.0108|
| | |none |acc_norm | 0.6844|± |0.0108|
|sciq |Yaml |none |acc | 0.8860|± |0.0101|
| | |none |acc_norm | 0.8790|± |0.0103|
|wikitext |Yaml |none |word_perplexity|35.9679| | |
| | |none |byte_perplexity| 1.8051| | |
| | |none |bits_per_byte | 0.8521| | |
|winogrande |Yaml |none |acc | 0.5328|± |0.0140|
Reference in New Issue Copy Permalink