95 lines
12 KiB
Markdown
95 lines
12 KiB
Markdown
| Tasks |Version|Filter|n-shot| Metric | | Value | | Stderr |
|
|
|------------------------------------------------------------|------:|------|-----:|---------------|---|--------:|---|--------|
|
|
|arc_challenge | 1|none | 0|acc |↑ | 0.2287|± | 0.0123|
|
|
| | |none | 0|acc_norm |↑ | 0.2594|± | 0.0128|
|
|
|arc_easy | 1|none | 0|acc |↑ | 0.4592|± | 0.0102|
|
|
| | |none | 0|acc_norm |↑ | 0.4415|± | 0.0102|
|
|
|blimp | 2|none | |acc |↑ | 0.6741|± | 0.0016|
|
|
| - blimp_adjunct_island | 1|none | 0|acc |↑ | 0.5530|± | 0.0157|
|
|
| - blimp_anaphor_gender_agreement | 1|none | 0|acc |↑ | 0.5740|± | 0.0156|
|
|
| - blimp_anaphor_number_agreement | 1|none | 0|acc |↑ | 0.7990|± | 0.0127|
|
|
| - blimp_animate_subject_passive | 1|none | 0|acc |↑ | 0.7090|± | 0.0144|
|
|
| - blimp_animate_subject_trans | 1|none | 0|acc |↑ | 0.6680|± | 0.0149|
|
|
| - blimp_causative | 1|none | 0|acc |↑ | 0.6220|± | 0.0153|
|
|
| - blimp_complex_NP_island | 1|none | 0|acc |↑ | 0.5080|± | 0.0158|
|
|
| - blimp_coordinate_structure_constraint_complex_left_branch| 1|none | 0|acc |↑ | 0.4180|± | 0.0156|
|
|
| - blimp_coordinate_structure_constraint_object_extraction | 1|none | 0|acc |↑ | 0.5930|± | 0.0155|
|
|
| - blimp_determiner_noun_agreement_1 | 1|none | 0|acc |↑ | 0.9170|± | 0.0087|
|
|
| - blimp_determiner_noun_agreement_2 | 1|none | 0|acc |↑ | 0.8680|± | 0.0107|
|
|
| - blimp_determiner_noun_agreement_irregular_1 | 1|none | 0|acc |↑ | 0.8490|± | 0.0113|
|
|
| - blimp_determiner_noun_agreement_irregular_2 | 1|none | 0|acc |↑ | 0.8160|± | 0.0123|
|
|
| - blimp_determiner_noun_agreement_with_adj_2 | 1|none | 0|acc |↑ | 0.8030|± | 0.0126|
|
|
| - blimp_determiner_noun_agreement_with_adj_irregular_1 | 1|none | 0|acc |↑ | 0.7980|± | 0.0127|
|
|
| - blimp_determiner_noun_agreement_with_adj_irregular_2 | 1|none | 0|acc |↑ | 0.7770|± | 0.0132|
|
|
| - blimp_determiner_noun_agreement_with_adjective_1 | 1|none | 0|acc |↑ | 0.8670|± | 0.0107|
|
|
| - blimp_distractor_agreement_relational_noun | 1|none | 0|acc |↑ | 0.6860|± | 0.0147|
|
|
| - blimp_distractor_agreement_relative_clause | 1|none | 0|acc |↑ | 0.5830|± | 0.0156|
|
|
| - blimp_drop_argument | 1|none | 0|acc |↑ | 0.7320|± | 0.0140|
|
|
| - blimp_ellipsis_n_bar_1 | 1|none | 0|acc |↑ | 0.6780|± | 0.0148|
|
|
| - blimp_ellipsis_n_bar_2 | 1|none | 0|acc |↑ | 0.8170|± | 0.0122|
|
|
| - blimp_existential_there_object_raising | 1|none | 0|acc |↑ | 0.8300|± | 0.0119|
|
|
| - blimp_existential_there_quantifiers_1 | 1|none | 0|acc |↑ | 0.8730|± | 0.0105|
|
|
| - blimp_existential_there_quantifiers_2 | 1|none | 0|acc |↑ | 0.4750|± | 0.0158|
|
|
| - blimp_existential_there_subject_raising | 1|none | 0|acc |↑ | 0.7880|± | 0.0129|
|
|
| - blimp_expletive_it_object_raising | 1|none | 0|acc |↑ | 0.6840|± | 0.0147|
|
|
| - blimp_inchoative | 1|none | 0|acc |↑ | 0.4720|± | 0.0158|
|
|
| - blimp_intransitive | 1|none | 0|acc |↑ | 0.6270|± | 0.0153|
|
|
| - blimp_irregular_past_participle_adjectives | 1|none | 0|acc |↑ | 0.8550|± | 0.0111|
|
|
| - blimp_irregular_past_participle_verbs | 1|none | 0|acc |↑ | 0.6420|± | 0.0152|
|
|
| - blimp_irregular_plural_subject_verb_agreement_1 | 1|none | 0|acc |↑ | 0.7010|± | 0.0145|
|
|
| - blimp_irregular_plural_subject_verb_agreement_2 | 1|none | 0|acc |↑ | 0.7560|± | 0.0136|
|
|
| - blimp_left_branch_island_echo_question | 1|none | 0|acc |↑ | 0.6110|± | 0.0154|
|
|
| - blimp_left_branch_island_simple_question | 1|none | 0|acc |↑ | 0.4140|± | 0.0156|
|
|
| - blimp_matrix_question_npi_licensor_present | 1|none | 0|acc |↑ | 0.0510|± | 0.0070|
|
|
| - blimp_npi_present_1 | 1|none | 0|acc |↑ | 0.5360|± | 0.0158|
|
|
| - blimp_npi_present_2 | 1|none | 0|acc |↑ | 0.5310|± | 0.0158|
|
|
| - blimp_only_npi_licensor_present | 1|none | 0|acc |↑ | 0.5590|± | 0.0157|
|
|
| - blimp_only_npi_scope | 1|none | 0|acc |↑ | 0.5270|± | 0.0158|
|
|
| - blimp_passive_1 | 1|none | 0|acc |↑ | 0.7820|± | 0.0131|
|
|
| - blimp_passive_2 | 1|none | 0|acc |↑ | 0.8350|± | 0.0117|
|
|
| - blimp_principle_A_c_command | 1|none | 0|acc |↑ | 0.5920|± | 0.0155|
|
|
| - blimp_principle_A_case_1 | 1|none | 0|acc |↑ | 1.0000|± | 0|
|
|
| - blimp_principle_A_case_2 | 1|none | 0|acc |↑ | 0.8780|± | 0.0104|
|
|
| - blimp_principle_A_domain_1 | 1|none | 0|acc |↑ | 0.9730|± | 0.0051|
|
|
| - blimp_principle_A_domain_2 | 1|none | 0|acc |↑ | 0.7930|± | 0.0128|
|
|
| - blimp_principle_A_domain_3 | 1|none | 0|acc |↑ | 0.5450|± | 0.0158|
|
|
| - blimp_principle_A_reconstruction | 1|none | 0|acc |↑ | 0.4330|± | 0.0157|
|
|
| - blimp_regular_plural_subject_verb_agreement_1 | 1|none | 0|acc |↑ | 0.7320|± | 0.0140|
|
|
| - blimp_regular_plural_subject_verb_agreement_2 | 1|none | 0|acc |↑ | 0.7870|± | 0.0130|
|
|
| - blimp_sentential_negation_npi_licensor_present | 1|none | 0|acc |↑ | 0.9520|± | 0.0068|
|
|
| - blimp_sentential_negation_npi_scope | 1|none | 0|acc |↑ | 0.5610|± | 0.0157|
|
|
| - blimp_sentential_subject_island | 1|none | 0|acc |↑ | 0.5320|± | 0.0158|
|
|
| - blimp_superlative_quantifiers_1 | 1|none | 0|acc |↑ | 0.4650|± | 0.0158|
|
|
| - blimp_superlative_quantifiers_2 | 1|none | 0|acc |↑ | 0.3300|± | 0.0149|
|
|
| - blimp_tough_vs_raising_1 | 1|none | 0|acc |↑ | 0.4270|± | 0.0156|
|
|
| - blimp_tough_vs_raising_2 | 1|none | 0|acc |↑ | 0.8320|± | 0.0118|
|
|
| - blimp_transitive | 1|none | 0|acc |↑ | 0.7440|± | 0.0138|
|
|
| - blimp_wh_island | 1|none | 0|acc |↑ | 0.6050|± | 0.0155|
|
|
| - blimp_wh_questions_object_gap | 1|none | 0|acc |↑ | 0.6520|± | 0.0151|
|
|
| - blimp_wh_questions_subject_gap | 1|none | 0|acc |↑ | 0.8930|± | 0.0098|
|
|
| - blimp_wh_questions_subject_gap_long_distance | 1|none | 0|acc |↑ | 0.8520|± | 0.0112|
|
|
| - blimp_wh_vs_that_no_gap | 1|none | 0|acc |↑ | 0.9610|± | 0.0061|
|
|
| - blimp_wh_vs_that_no_gap_long_distance | 1|none | 0|acc |↑ | 0.9750|± | 0.0049|
|
|
| - blimp_wh_vs_that_with_gap | 1|none | 0|acc |↑ | 0.3050|± | 0.0146|
|
|
| - blimp_wh_vs_that_with_gap_long_distance | 1|none | 0|acc |↑ | 0.1600|± | 0.0116|
|
|
|boolq | 2|none | 0|acc |↑ | 0.5080|± | 0.0087|
|
|
|hellaswag | 1|none | 0|acc |↑ | 0.2791|± | 0.0045|
|
|
| | |none | 0|acc_norm |↑ | 0.2932|± | 0.0045|
|
|
|lambada_openai | 1|none | 0|acc |↑ | 0.1566|± | 0.0051|
|
|
| | |none | 0|perplexity |↓ | 674.5031|± | 40.5885|
|
|
|lambada_standard | 1|none | 0|acc |↑ | 0.0790|± | 0.0038|
|
|
| | |none | 0|perplexity |↓ |6783.2040|± |434.8107|
|
|
|openbookqa | 1|none | 0|acc |↑ | 0.1780|± | 0.0171|
|
|
| | |none | 0|acc_norm |↑ | 0.2660|± | 0.0198|
|
|
|piqa | 1|none | 0|acc |↑ | 0.5958|± | 0.0114|
|
|
| | |none | 0|acc_norm |↑ | 0.5930|± | 0.0115|
|
|
|sciq | 1|none | 0|acc |↑ | 0.6090|± | 0.0154|
|
|
| | |none | 0|acc_norm |↑ | 0.5740|± | 0.0156|
|
|
|wikitext | 2|none | 0|bits_per_byte |↓ | 1.4107|± | N/A |
|
|
| | |none | 0|byte_perplexity|↓ | 2.6587|± | N/A |
|
|
| | |none | 0|word_perplexity|↓ | 186.6040|± | N/A |
|
|
|winogrande | 1|none | 0|acc |↑ | 0.4996|± | 0.0141|
|
|
|
|
|Groups|Version|Filter|n-shot|Metric| |Value | |Stderr|
|
|
|------|------:|------|------|------|---|-----:|---|-----:|
|
|
|blimp | 2|none | |acc |↑ |0.6741|± |0.0016| |