multilingual-e5-large-instruct/README.md

tags, model-index, language, license

tags	model-index	language	license
mteb sentence-transformers transformers	name results multilingual-e5-large-instruct task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en) en test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 76.23880597014924 type value ap 39.07351965022687 type value f1 70.04836733862683 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (de) de test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 66.71306209850107 type value ap 79.01499914759529 type value f1 64.81951817560703 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en-ext) en-ext test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 73.85307346326837 type value ap 22.447519885878737 type value f1 61.0162730745633 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (ja) ja test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 76.04925053533191 type value ap 23.44983217128922 type value f1 62.5723230907759 task dataset metrics type Classification type name config split revision mteb/amazon_polarity MTEB AmazonPolarityClassification default test e2d317d38cd51312af73b3d32a06d1a08b442046 type value accuracy 96.28742500000001 type value ap 94.8449918887462 type value f1 96.28680923610432 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (en) en test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 56.716 type value f1 55.76510398266401 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (de) de test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 52.99999999999999 type value f1 52.00829994765178 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (es) es test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 48.806000000000004 type value f1 48.082345914983634 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (fr) fr test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 48.507999999999996 type value f1 47.68752844642045 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (ja) ja test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 47.709999999999994 type value f1 47.05870376637181 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (zh) zh test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 44.662000000000006 type value f1 43.42371965372771 task dataset metrics type Retrieval type name config split revision arguana MTEB ArguAna default test None type value map_at_1 31.721 type value map_at_10 49.221 type value map_at_100 49.884 type value map_at_1000 49.888 type value map_at_3 44.31 type value map_at_5 47.276 type value mrr_at_1 32.432 type value mrr_at_10 49.5 type value mrr_at_100 50.163000000000004 type value mrr_at_1000 50.166 type value mrr_at_3 44.618 type value mrr_at_5 47.541 type value ndcg_at_1 31.721 type value ndcg_at_10 58.384 type value ndcg_at_100 61.111000000000004 type value ndcg_at_1000 61.187999999999995 type value ndcg_at_3 48.386 type value ndcg_at_5 53.708999999999996 type value precision_at_1 31.721 type value precision_at_10 8.741 type value precision_at_100 0.991 type value precision_at_1000 0.1 type value precision_at_3 20.057 type value precision_at_5 14.609 type value recall_at_1 31.721 type value recall_at_10 87.411 type value recall_at_100 99.075 type value recall_at_1000 99.644 type value recall_at_3 60.171 type value recall_at_5 73.044 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-p2p MTEB ArxivClusteringP2P default test a122ad7f3f0291bf49cc6f4d32aa80929df69d5d type value v_measure 46.40419580759799 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-s2s MTEB ArxivClusteringS2S default test f910caf1a6075f7329cdf8c1a6135696f37dbd53 type value v_measure 40.48593255007969 task dataset metrics type Reranking type name config split revision mteb/askubuntudupquestions-reranking MTEB AskUbuntuDupQuestions default test 2000358ca161889fa9c082cb41daa8dcfb161a54 type value map 63.889179122289995 type value mrr 77.61146286769556 task dataset metrics type STS type name config split revision mteb/biosses-sts MTEB BIOSSES default test d3fb88f8f02e40887cd149695127462bbcf29b4a type value cos_sim_pearson 88.15075203727929 type value cos_sim_spearman 86.9622224570873 type value euclidean_pearson 86.70473853624121 type value euclidean_spearman 86.9622224570873 type value manhattan_pearson 86.21089380980065 type value manhattan_spearman 86.75318154937008 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (de-en) de-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 99.65553235908142 type value f1 99.60681976339595 type value precision 99.58246346555325 type value recall 99.65553235908142 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (fr-en) fr-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 99.26260180497468 type value f1 99.14520507740848 type value precision 99.08650671362535 type value recall 99.26260180497468 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (ru-en) ru-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 98.07412538967787 type value f1 97.86629719431936 type value precision 97.76238309664012 type value recall 98.07412538967787 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (zh-en) zh-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 99.42074776197998 type value f1 99.38564156573635 type value precision 99.36808846761454 type value recall 99.42074776197998 task dataset metrics type Classification type name config split revision mteb/banking77 MTEB Banking77Classification default test 0fd18e25b25c072e09e0d92ab615fda904d66300 type value accuracy 85.73376623376623 type value f1 85.68480707214599 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-p2p MTEB BiorxivClusteringP2P default test 65b79d1d13f80053f67aca9498d9402c2d9f1f40 type value v_measure 40.935218072113855 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-s2s MTEB BiorxivClusteringS2S default test 258694dd0231531bc1fd9de6ceb52a0853c6d908 type value v_measure 36.276389017675264 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackRetrieval default test None type value map_at_1 27.764166666666668 type value map_at_10 37.298166666666674 type value map_at_100 38.530166666666666 type value map_at_1000 38.64416666666667 type value map_at_3 34.484833333333334 type value map_at_5 36.0385 type value mrr_at_1 32.93558333333333 type value mrr_at_10 41.589749999999995 type value mrr_at_100 42.425333333333334 type value mrr_at_1000 42.476333333333336 type value mrr_at_3 39.26825 type value mrr_at_5 40.567083333333336 type value ndcg_at_1 32.93558333333333 type value ndcg_at_10 42.706583333333334 type value ndcg_at_100 47.82483333333333 type value ndcg_at_1000 49.95733333333334 type value ndcg_at_3 38.064750000000004 type value ndcg_at_5 40.18158333333333 type value precision_at_1 32.93558333333333 type value precision_at_10 7.459833333333334 type value precision_at_100 1.1830833333333335 type value precision_at_1000 0.15608333333333332 type value precision_at_3 17.5235 type value precision_at_5 12.349833333333333 type value recall_at_1 27.764166666666668 type value recall_at_10 54.31775 type value recall_at_100 76.74350000000001 type value recall_at_1000 91.45208333333332 type value recall_at_3 41.23425 type value recall_at_5 46.73983333333334 task dataset metrics type Retrieval type name config split revision climate-fever MTEB ClimateFEVER default test None type value map_at_1 12.969 type value map_at_10 21.584999999999997 type value map_at_100 23.3 type value map_at_1000 23.5 type value map_at_3 18.218999999999998 type value map_at_5 19.983 type value mrr_at_1 29.316 type value mrr_at_10 40.033 type value mrr_at_100 40.96 type value mrr_at_1000 41.001 type value mrr_at_3 37.123 type value mrr_at_5 38.757999999999996 type value ndcg_at_1 29.316 type value ndcg_at_10 29.858 type value ndcg_at_100 36.756 type value ndcg_at_1000 40.245999999999995 type value ndcg_at_3 24.822 type value ndcg_at_5 26.565 type value precision_at_1 29.316 type value precision_at_10 9.186 type value precision_at_100 1.6549999999999998 type value precision_at_1000 0.22999999999999998 type value precision_at_3 18.436 type value precision_at_5 13.876 type value recall_at_1 12.969 type value recall_at_10 35.142 type value recall_at_100 59.143 type value recall_at_1000 78.594 type value recall_at_3 22.604 type value recall_at_5 27.883000000000003 task dataset metrics type Retrieval type name config split revision dbpedia-entity MTEB DBPedia default test None type value map_at_1 8.527999999999999 type value map_at_10 17.974999999999998 type value map_at_100 25.665 type value map_at_1000 27.406000000000002 type value map_at_3 13.017999999999999 type value map_at_5 15.137 type value mrr_at_1 62.5 type value mrr_at_10 71.891 type value mrr_at_100 72.294 type value mrr_at_1000 72.296 type value mrr_at_3 69.958 type value mrr_at_5 71.121 type value ndcg_at_1 50.875 type value ndcg_at_10 38.36 type value ndcg_at_100 44.235 type value ndcg_at_1000 52.154 type value ndcg_at_3 43.008 type value ndcg_at_5 40.083999999999996 type value precision_at_1 62.5 type value precision_at_10 30.0 type value precision_at_100 10.038 type value precision_at_1000 2.0869999999999997 type value precision_at_3 46.833000000000006 type value precision_at_5 38.800000000000004 type value recall_at_1 8.527999999999999 type value recall_at_10 23.828 type value recall_at_100 52.322 type value recall_at_1000 77.143 type value recall_at_3 14.136000000000001 type value recall_at_5 17.761 task dataset metrics type Classification type name config split revision mteb/emotion MTEB EmotionClassification default test 4f58c6b202a23cf9a4da393831edf4f9183cad37 type value accuracy 51.51 type value f1 47.632159862049896 task dataset metrics type Retrieval type name config split revision fever MTEB FEVER default test None type value map_at_1 60.734 type value map_at_10 72.442 type value map_at_100 72.735 type value map_at_1000 72.75 type value map_at_3 70.41199999999999 type value map_at_5 71.80499999999999 type value mrr_at_1 65.212 type value mrr_at_10 76.613 type value mrr_at_100 76.79899999999999 type value mrr_at_1000 76.801 type value mrr_at_3 74.8 type value mrr_at_5 76.12400000000001 type value ndcg_at_1 65.212 type value ndcg_at_10 77.988 type value ndcg_at_100 79.167 type value ndcg_at_1000 79.452 type value ndcg_at_3 74.362 type value ndcg_at_5 76.666 type value precision_at_1 65.212 type value precision_at_10 10.003 type value precision_at_100 1.077 type value precision_at_1000 0.11199999999999999 type value precision_at_3 29.518 type value precision_at_5 19.016 type value recall_at_1 60.734 type value recall_at_10 90.824 type value recall_at_100 95.71600000000001 type value recall_at_1000 97.577 type value recall_at_3 81.243 type value recall_at_5 86.90299999999999 task dataset metrics type Retrieval type name config split revision fiqa MTEB FiQA2018 default test None type value map_at_1 23.845 type value map_at_10 39.281 type value map_at_100 41.422 type value map_at_1000 41.593 type value map_at_3 34.467 type value map_at_5 37.017 type value mrr_at_1 47.531 type value mrr_at_10 56.204 type value mrr_at_100 56.928999999999995 type value mrr_at_1000 56.962999999999994 type value mrr_at_3 54.115 type value mrr_at_5 55.373000000000005 type value ndcg_at_1 47.531 type value ndcg_at_10 47.711999999999996 type value ndcg_at_100 54.510999999999996 type value ndcg_at_1000 57.103 type value ndcg_at_3 44.145 type value ndcg_at_5 45.032 type value precision_at_1 47.531 type value precision_at_10 13.194 type value precision_at_100 2.045 type value precision_at_1000 0.249 type value precision_at_3 29.424 type value precision_at_5 21.451 type value recall_at_1 23.845 type value recall_at_10 54.967 type value recall_at_100 79.11399999999999 type value recall_at_1000 94.56700000000001 type value recall_at_3 40.256 type value recall_at_5 46.215 task dataset metrics type Retrieval type name config split revision hotpotqa MTEB HotpotQA default test None type value map_at_1 37.819 type value map_at_10 60.889 type value map_at_100 61.717999999999996 type value map_at_1000 61.778 type value map_at_3 57.254000000000005 type value map_at_5 59.541 type value mrr_at_1 75.638 type value mrr_at_10 82.173 type value mrr_at_100 82.362 type value mrr_at_1000 82.37 type value mrr_at_3 81.089 type value mrr_at_5 81.827 type value ndcg_at_1 75.638 type value ndcg_at_10 69.317 type value ndcg_at_100 72.221 type value ndcg_at_1000 73.382 type value ndcg_at_3 64.14 type value ndcg_at_5 67.07600000000001 type value precision_at_1 75.638 type value precision_at_10 14.704999999999998 type value precision_at_100 1.698 type value precision_at_1000 0.185 type value precision_at_3 41.394999999999996 type value precision_at_5 27.162999999999997 type value recall_at_1 37.819 type value recall_at_10 73.52499999999999 type value recall_at_100 84.875 type value recall_at_1000 92.559 type value recall_at_3 62.092999999999996 type value recall_at_5 67.907 task dataset metrics type Classification type name config split revision mteb/imdb MTEB ImdbClassification default test 3d86128a09e091d6018b6d26cad27f2739fc2db7 type value accuracy 94.60079999999999 type value ap 92.67396345347356 type value f1 94.5988098167121 task dataset metrics type Retrieval type name config split revision msmarco MTEB MSMARCO default dev None type value map_at_1 21.285 type value map_at_10 33.436 type value map_at_100 34.63 type value map_at_1000 34.681 type value map_at_3 29.412 type value map_at_5 31.715 type value mrr_at_1 21.848 type value mrr_at_10 33.979 type value mrr_at_100 35.118 type value mrr_at_1000 35.162 type value mrr_at_3 30.036 type value mrr_at_5 32.298 type value ndcg_at_1 21.862000000000002 type value ndcg_at_10 40.43 type value ndcg_at_100 46.17 type value ndcg_at_1000 47.412 type value ndcg_at_3 32.221 type value ndcg_at_5 36.332 type value precision_at_1 21.862000000000002 type value precision_at_10 6.491 type value precision_at_100 0.935 type value precision_at_1000 0.104 type value precision_at_3 13.744 type value precision_at_5 10.331999999999999 type value recall_at_1 21.285 type value recall_at_10 62.083 type value recall_at_100 88.576 type value recall_at_1000 98.006 type value recall_at_3 39.729 type value recall_at_5 49.608000000000004 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (en) en test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 93.92612859097127 type value f1 93.82370333372853 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (de) de test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 92.67681036911807 type value f1 92.14191382411472 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (es) es test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 92.26817878585723 type value f1 91.92824250337878 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (fr) fr test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 89.96554963983714 type value f1 90.02859329630792 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (hi) hi test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 90.02509860164935 type value f1 89.30665159182062 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (th) th test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 87.55515370705244 type value f1 87.94449232331907 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (en) en test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 82.4623803009576 type value f1 66.06738378772725 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (de) de test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 79.3716539870386 type value f1 60.37614033396853 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (es) es test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 80.34022681787857 type value f1 58.302008026952 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (fr) fr test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 76.72095208268087 type value f1 59.64524724009049 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (hi) hi test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 77.87020437432773 type value f1 57.80202694670567 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (th) th test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 77.73598553345387 type value f1 58.19628250675031 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (af) af test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 67.6630800268998 type value f1 65.00996668051691 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (am) am test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 60.7128446536651 type value f1 57.95860594874963 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ar) ar test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 63.61129791526563 type value f1 59.75328290206483 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (az) az test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.00134498991257 type value f1 67.0230483991802 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (bn) bn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 68.54068594485541 type value f1 65.54604628946976 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (cy) cy test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 63.032952252858095 type value f1 58.715741857057104 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (da) da test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 71.80901143241427 type value f1 68.33963989243877 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (de) de test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 72.47141896435777 type value f1 69.56765020308262 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (el) el test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 71.2373907195696 type value f1 69.04529836036467 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (en) en test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 77.05783456624076 type value f1 74.69430584708174 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (es) es test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 72.82111634162744 type value f1 70.77228952803762 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fa) fa test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 74.25353059852051 type value f1 71.05310103416411 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fi) fi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 72.28648285137861 type value f1 69.08020473732226 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fr) fr test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 73.31540013449899 type value f1 70.9426355465791 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (he) he test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 70.2151983860121 type value f1 67.52541755908858 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hi) hi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 71.58372562205784 type value f1 69.49769064229827 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hu) hu test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 71.9233355749832 type value f1 69.36311548259593 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hy) hy test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 68.07330195023538 type value f1 64.99882022345572 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (id) id test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 72.62273032952253 type value f1 70.6394885471001 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (is) is test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 65.77000672494957 type value f1 62.9368944815065 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (it) it test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 73.453261600538 type value f1 70.85069934666681 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ja) ja test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 74.6906523201076 type value f1 72.03249740074217 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (jv) jv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 63.03631472763953 type value f1 59.3165215571852 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ka) ka test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 58.913920645595155 type value f1 57.367337711611285 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (km) km test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 54.42837928715535 type value f1 52.60527294970906 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (kn) kn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 66.33490248823135 type value f1 63.213340969404065 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ko) ko test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 70.58507061197041 type value f1 68.40256628040486 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (lv) lv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.11230665770006 type value f1 66.44863577842305 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ml) ml test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.70073974445192 type value f1 67.21291337273702 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (mn) mn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 66.43913920645595 type value f1 64.09838087422806 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ms) ms test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 70.80026899798251 type value f1 68.76986742962444 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (my) my test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 64.78816408876934 type value f1 62.18781873428972 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nb) nb test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 71.6577000672495 type value f1 68.75171511133003 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nl) nl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 74.42501681237391 type value f1 71.18434963451544 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pl) pl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 73.64828513786146 type value f1 70.67741914007422 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pt) pt test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 73.62811028917284 type value f1 71.36402039740959 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ro) ro test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 71.88634835238736 type value f1 69.23701923480677 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ru) ru test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 74.15938130464022 type value f1 71.87792218993388 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sl) sl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.96301277740416 type value f1 67.29584200202983 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sq) sq test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.49562878278412 type value f1 66.91716685679431 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sv) sv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 74.6805648957633 type value f1 72.02723592594374 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sw) sw test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 63.00605245460659 type value f1 60.16716669482932 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ta) ta test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 66.90988567585742 type value f1 63.99405488777784 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (te) te test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 67.62273032952253 type value f1 65.17213906909481 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (th) th test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.50907868190988 type value f1 69.15165697194853 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tl) tl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.30733019502352 type value f1 66.69024007380474 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tr) tr test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 72.24277067921989 type value f1 68.80515408492947 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ur) ur test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 67.49831876260929 type value f1 64.83778567111116 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (vi) vi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 71.28782784129119 type value f1 69.3294186700733 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-CN) zh-CN test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 73.315400134499 type value f1 71.22674385243207 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-TW) zh-TW test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.37794216543377 type value f1 68.96962492838232 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (af) af test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.33557498318764 type value f1 72.28949738478356 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (am) am test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 65.84398117014123 type value f1 64.71026362091463 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ar) ar test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 69.76462676529925 type value f1 69.8229667407667 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (az) az test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.02420981842636 type value f1 71.76576384895898 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (bn) bn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.7572293207801 type value f1 72.76840765295256 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (cy) cy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 68.02286482851379 type value f1 66.17237947327872 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (da) da test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.60928043039678 type value f1 77.27094731234773 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (de) de test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.68325487558843 type value f1 77.97530399082261 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (el) el test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 76.13315400134498 type value f1 75.97558584796424 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (en) en test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 80.47410894418292 type value f1 80.52244841473792 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (es) es test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 76.9670477471419 type value f1 77.37318805793146 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fa) fa test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 78.09683927370544 type value f1 77.69773737430847 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fi) fi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 75.20847343644922 type value f1 75.17071738727348 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fr) fr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.07464694014796 type value f1 77.16136207698571 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (he) he test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.53396099529255 type value f1 73.58296404484122 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hi) hi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 75.75319435104237 type value f1 75.24674707850833 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hu) hu test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.0948217888366 type value f1 76.47559490205028 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hy) hy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.07599193006052 type value f1 70.76028043093511 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (id) id test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.10490921318089 type value f1 77.01215275283272 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (is) is test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.25756556825824 type value f1 70.20605314648762 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (it) it test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.08137188971082 type value f1 77.3899269057439 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ja) ja test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 79.35440484196369 type value f1 79.58964690002772 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (jv) jv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 68.42299932750504 type value f1 68.07844356925413 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ka) ka test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 66.15669132481507 type value f1 65.89383352608513 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (km) km test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 60.11432414256894 type value f1 57.69910594559806 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (kn) kn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.24747814391392 type value f1 70.42455553830918 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ko) ko test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 76.46267652992603 type value f1 76.8854559308316 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (lv) lv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.24815063887021 type value f1 72.77805034658074 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ml) ml test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.11566913248151 type value f1 73.86147988001356 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (mn) mn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.0168123739072 type value f1 69.38515920054571 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ms) ms test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.41156691324814 type value f1 73.43474953408237 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (my) my test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 68.39609952925353 type value f1 67.29731681109291 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nb) nb test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.20914593140552 type value f1 77.07066497935367 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nl) nl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 78.52387357094821 type value f1 78.5259569473291 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pl) pl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 76.6913248150639 type value f1 76.91201656350455 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pt) pt test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.1217215870881 type value f1 77.41179937912504 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ro) ro test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 75.25891055817083 type value f1 75.8089244542887 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ru) ru test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.70679219905851 type value f1 78.21459594517711 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sl) sl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.83523873570948 type value f1 74.86847028401978 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sq) sq test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.71755211835911 type value f1 74.0214326485662 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sv) sv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 79.06523201075991 type value f1 79.10545620325138 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sw) sw test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 67.91862811028918 type value f1 66.50386121217983 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ta) ta test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.93140551445865 type value f1 70.755435928495 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (te) te test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.40753194351042 type value f1 71.61816115782923 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (th) th test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 75.1815736381977 type value f1 75.08016717887205 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tl) tl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.86482851378614 type value f1 72.39521180006291 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tr) tr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 76.46940147948891 type value f1 76.70044085362349 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ur) ur test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.89307330195024 type value f1 71.5721825332298 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (vi) vi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.7511768661735 type value f1 75.17918654541515 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-CN) zh-CN test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 78.69535978480162 type value f1 78.90019070153316 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-TW) zh-TW test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 75.45729657027572 type value f1 76.19578371794672 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-p2p MTEB MedrxivClusteringP2P default test e7a26af6f3ae46b30dde8737f02c07b1505bcc73 type value v_measure 36.92715354123554 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-s2s MTEB MedrxivClusteringS2S default test 35191c8c0dca72d8ff3efcd72aa802307d469663 type value v_measure 35.53536244162518 task dataset metrics type Reranking type name config split revision mteb/mind_small MTEB MindSmallReranking default test 3bdac13927fdc888b903db93b2ffdbd90b295a69 type value map 33.08507884504006 type value mrr 34.32436977159129 task dataset metrics type Retrieval type name config split revision nfcorpus MTEB NFCorpus default test None type value map_at_1 5.935 type value map_at_10 13.297 type value map_at_100 16.907 type value map_at_1000 18.391 type value map_at_3 9.626999999999999 type value map_at_5 11.190999999999999 type value mrr_at_1 46.129999999999995 type value mrr_at_10 54.346000000000004 type value mrr_at_100 55.067 type value mrr_at_1000 55.1 type value mrr_at_3 51.961 type value mrr_at_5 53.246 type value ndcg_at_1 44.118 type value ndcg_at_10 35.534 type value ndcg_at_100 32.946999999999996 type value ndcg_at_1000 41.599000000000004 type value ndcg_at_3 40.25 type value ndcg_at_5 37.978 type value precision_at_1 46.129999999999995 type value precision_at_10 26.842 type value precision_at_100 8.427 type value precision_at_1000 2.128 type value precision_at_3 37.977 type value precision_at_5 32.879000000000005 type value recall_at_1 5.935 type value recall_at_10 17.211000000000002 type value recall_at_100 34.33 type value recall_at_1000 65.551 type value recall_at_3 10.483 type value recall_at_5 13.078999999999999 task dataset metrics type Retrieval type name config split revision nq MTEB NQ default test None type value map_at_1 35.231 type value map_at_10 50.202000000000005 type value map_at_100 51.154999999999994 type value map_at_1000 51.181 type value map_at_3 45.774 type value map_at_5 48.522 type value mrr_at_1 39.687 type value mrr_at_10 52.88 type value mrr_at_100 53.569 type value mrr_at_1000 53.58500000000001 type value mrr_at_3 49.228 type value mrr_at_5 51.525 type value ndcg_at_1 39.687 type value ndcg_at_10 57.754000000000005 type value ndcg_at_100 61.597 type value ndcg_at_1000 62.18900000000001 type value ndcg_at_3 49.55 type value ndcg_at_5 54.11899999999999 type value precision_at_1 39.687 type value precision_at_10 9.313 type value precision_at_100 1.146 type value precision_at_1000 0.12 type value precision_at_3 22.229 type value precision_at_5 15.939 type value recall_at_1 35.231 type value recall_at_10 78.083 type value recall_at_100 94.42099999999999 type value recall_at_1000 98.81 type value recall_at_3 57.047000000000004 type value recall_at_5 67.637 task dataset metrics type Retrieval type name config split revision quora MTEB QuoraRetrieval default test None type value map_at_1 71.241 type value map_at_10 85.462 type value map_at_100 86.083 type value map_at_1000 86.09700000000001 type value map_at_3 82.49499999999999 type value map_at_5 84.392 type value mrr_at_1 82.09 type value mrr_at_10 88.301 type value mrr_at_100 88.383 type value mrr_at_1000 88.384 type value mrr_at_3 87.37 type value mrr_at_5 88.035 type value ndcg_at_1 82.12 type value ndcg_at_10 89.149 type value ndcg_at_100 90.235 type value ndcg_at_1000 90.307 type value ndcg_at_3 86.37599999999999 type value ndcg_at_5 87.964 type value precision_at_1 82.12 type value precision_at_10 13.56 type value precision_at_100 1.539 type value precision_at_1000 0.157 type value precision_at_3 37.88 type value precision_at_5 24.92 type value recall_at_1 71.241 type value recall_at_10 96.128 type value recall_at_100 99.696 type value recall_at_1000 99.994 type value recall_at_3 88.181 type value recall_at_5 92.694 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering MTEB RedditClustering default test 24640382cdbf8abc73003fb0fa6d111a705499eb type value v_measure 56.59757799655151 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering-p2p MTEB RedditClusteringP2P default test 282350215ef01743dc01b456c7f5241fa8937f16 type value v_measure 64.27391998854624 task dataset metrics type Retrieval type name config split revision scidocs MTEB SCIDOCS default test None type value map_at_1 4.243 type value map_at_10 10.965 type value map_at_100 12.934999999999999 type value map_at_1000 13.256 type value map_at_3 7.907 type value map_at_5 9.435 type value mrr_at_1 20.9 type value mrr_at_10 31.849 type value mrr_at_100 32.964 type value mrr_at_1000 33.024 type value mrr_at_3 28.517 type value mrr_at_5 30.381999999999998 type value ndcg_at_1 20.9 type value ndcg_at_10 18.723 type value ndcg_at_100 26.384999999999998 type value ndcg_at_1000 32.114 type value ndcg_at_3 17.753 type value ndcg_at_5 15.558 type value precision_at_1 20.9 type value precision_at_10 9.8 type value precision_at_100 2.078 type value precision_at_1000 0.345 type value precision_at_3 16.900000000000002 type value precision_at_5 13.88 type value recall_at_1 4.243 type value recall_at_10 19.885 type value recall_at_100 42.17 type value recall_at_1000 70.12 type value recall_at_3 10.288 type value recall_at_5 14.072000000000001 task dataset metrics type STS type name config split revision mteb/sickr-sts MTEB SICK-R default test a6ea5a8cab320b040a23452cc28066d9beae2cee type value cos_sim_pearson 85.84209174935282 type value cos_sim_spearman 81.73248048438833 type value euclidean_pearson 83.02810070308149 type value euclidean_spearman 81.73248295679514 type value manhattan_pearson 82.95368060376002 type value manhattan_spearman 81.60277910998718 task dataset metrics type STS type name config split revision mteb/sts12-sts MTEB STS12 default test a0d554a64d88156834ff5ae9920b964011b16384 type value cos_sim_pearson 88.52628804556943 type value cos_sim_spearman 82.5713913555672 type value euclidean_pearson 85.8796774746988 type value euclidean_spearman 82.57137506803424 type value manhattan_pearson 85.79671002960058 type value manhattan_spearman 82.49445981618027 task dataset metrics type STS type name config split revision mteb/sts13-sts MTEB STS13 default test 7e90230a92c190f1bf69ae9002b8cea547a64cca type value cos_sim_pearson 86.23682503505542 type value cos_sim_spearman 87.15008956711806 type value euclidean_pearson 86.79805401524959 type value euclidean_spearman 87.15008956711806 type value manhattan_pearson 86.65298502699244 type value manhattan_spearman 86.97677821948562 task dataset metrics type STS type name config split revision mteb/sts14-sts MTEB STS14 default test 6031580fec1f6af667f0bd2da0a551cf4f0b2375 type value cos_sim_pearson 85.63370304677802 type value cos_sim_spearman 84.97105553540318 type value euclidean_pearson 85.28896108687721 type value euclidean_spearman 84.97105553540318 type value manhattan_pearson 85.09663190337331 type value manhattan_spearman 84.79126831644619 task dataset metrics type STS type name config split revision mteb/sts15-sts MTEB STS15 default test ae752c7c21bf194d8b67fd573edf7ae58183cbe3 type value cos_sim_pearson 90.2614838800733 type value cos_sim_spearman 91.0509162991835 type value euclidean_pearson 90.33098317533373 type value euclidean_spearman 91.05091625871644 type value manhattan_pearson 90.26250435151107 type value manhattan_spearman 90.97999594417519 task dataset metrics type STS type name config split revision mteb/sts16-sts MTEB STS16 default test 4d8694f8f0e0100860b497b999b3dbed754a0513 type value cos_sim_pearson 85.80480973335091 type value cos_sim_spearman 87.313695492969 type value euclidean_pearson 86.49267251576939 type value euclidean_spearman 87.313695492969 type value manhattan_pearson 86.44019901831935 type value manhattan_spearman 87.24205395460392 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-en) en-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 90.05662789380672 type value cos_sim_spearman 90.02759424426651 type value euclidean_pearson 90.4042483422981 type value euclidean_spearman 90.02759424426651 type value manhattan_pearson 90.51446975000226 type value manhattan_spearman 90.08832889933616 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (en) en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 67.5975528273532 type value cos_sim_spearman 67.62969861411354 type value euclidean_pearson 69.224275734323 type value euclidean_spearman 67.62969861411354 type value manhattan_pearson 69.3761447059927 type value manhattan_spearman 67.90921005611467 task dataset metrics type STS type name config split revision mteb/stsbenchmark-sts MTEB STSBenchmark default test b0fddb56ed78048fa8b90373c8a3cfc37b684831 type value cos_sim_pearson 87.11244327231684 type value cos_sim_spearman 88.37902438979035 type value euclidean_pearson 87.86054279847336 type value euclidean_spearman 88.37902438979035 type value manhattan_pearson 87.77257757320378 type value manhattan_spearman 88.25208966098123 task dataset metrics type Reranking type name config split revision mteb/scidocs-reranking MTEB SciDocsRR default test d3c5e1fc0b855ab6097bf1cda04dd73947d7caab type value map 85.87174608143563 type value mrr 96.12836872640794 task dataset metrics type Retrieval type name config split revision scifact MTEB SciFact default test None type value map_at_1 57.760999999999996 type value map_at_10 67.258 type value map_at_100 67.757 type value map_at_1000 67.78800000000001 type value map_at_3 64.602 type value map_at_5 65.64 type value mrr_at_1 60.667 type value mrr_at_10 68.441 type value mrr_at_100 68.825 type value mrr_at_1000 68.853 type value mrr_at_3 66.444 type value mrr_at_5 67.26100000000001 type value ndcg_at_1 60.667 type value ndcg_at_10 71.852 type value ndcg_at_100 73.9 type value ndcg_at_1000 74.628 type value ndcg_at_3 67.093 type value ndcg_at_5 68.58 type value precision_at_1 60.667 type value precision_at_10 9.6 type value precision_at_100 1.0670000000000002 type value precision_at_1000 0.11199999999999999 type value precision_at_3 26.111 type value precision_at_5 16.733 type value recall_at_1 57.760999999999996 type value recall_at_10 84.967 type value recall_at_100 93.833 type value recall_at_1000 99.333 type value recall_at_3 71.589 type value recall_at_5 75.483 task dataset metrics type PairClassification type name config split revision mteb/sprintduplicatequestions-pairclassification MTEB SprintDuplicateQuestions default test d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 type value cos_sim_accuracy 99.66633663366336 type value cos_sim_ap 91.17685358899108 type value cos_sim_f1 82.16818642350559 type value cos_sim_precision 83.26488706365504 type value cos_sim_recall 81.10000000000001 type value dot_accuracy 99.66633663366336 type value dot_ap 91.17663411119032 type value dot_f1 82.16818642350559 type value dot_precision 83.26488706365504 type value dot_recall 81.10000000000001 type value euclidean_accuracy 99.66633663366336 type value euclidean_ap 91.17685189882275 type value euclidean_f1 82.16818642350559 type value euclidean_precision 83.26488706365504 type value euclidean_recall 81.10000000000001 type value manhattan_accuracy 99.66633663366336 type value manhattan_ap 91.2241619496737 type value manhattan_f1 82.20472440944883 type value manhattan_precision 86.51933701657458 type value manhattan_recall 78.3 type value max_accuracy 99.66633663366336 type value max_ap 91.2241619496737 type value max_f1 82.20472440944883 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering MTEB StackExchangeClustering default test 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 type value v_measure 66.85101268897951 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering-p2p MTEB StackExchangeClusteringP2P default test 815ca46b2622cec33ccafc3735d572c266efdb44 type value v_measure 42.461184054706905 task dataset metrics type Reranking type name config split revision mteb/stackoverflowdupquestions-reranking MTEB StackOverflowDupQuestions default test e185fbe320c72810689fc5848eb6114e1ef5ec69 type value map 51.44542568873886 type value mrr 52.33656151854681 task dataset metrics type Summarization type name config split revision mteb/summeval MTEB SummEval default test cda12ad7615edc362dbf25a00fdd61d3b1eaf93c type value cos_sim_pearson 30.75982974997539 type value cos_sim_spearman 30.385405026539914 type value dot_pearson 30.75982433546523 type value dot_spearman 30.385405026539914 task dataset metrics type Retrieval type name config split revision trec-covid MTEB TRECCOVID default test None type value map_at_1 0.22799999999999998 type value map_at_10 2.064 type value map_at_100 13.056000000000001 type value map_at_1000 31.747999999999998 type value map_at_3 0.67 type value map_at_5 1.097 type value mrr_at_1 90.0 type value mrr_at_10 94.667 type value mrr_at_100 94.667 type value mrr_at_1000 94.667 type value mrr_at_3 94.667 type value mrr_at_5 94.667 type value ndcg_at_1 86.0 type value ndcg_at_10 82.0 type value ndcg_at_100 64.307 type value ndcg_at_1000 57.023999999999994 type value ndcg_at_3 85.816 type value ndcg_at_5 84.904 type value precision_at_1 90.0 type value precision_at_10 85.8 type value precision_at_100 66.46 type value precision_at_1000 25.202 type value precision_at_3 90.0 type value precision_at_5 89.2 type value recall_at_1 0.22799999999999998 type value recall_at_10 2.235 type value recall_at_100 16.185 type value recall_at_1000 53.620999999999995 type value recall_at_3 0.7040000000000001 type value recall_at_5 1.172 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (sqi-eng) sqi-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.39999999999999 type value f1 96.75 type value precision 96.45 type value recall 97.39999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fry-eng) fry-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 85.54913294797689 type value f1 82.46628131021194 type value precision 81.1175337186898 type value recall 85.54913294797689 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kur-eng) kur-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 81.21951219512195 type value f1 77.33333333333334 type value precision 75.54878048780488 type value recall 81.21951219512195 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tur-eng) tur-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 98.6 type value f1 98.26666666666665 type value precision 98.1 type value recall 98.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (deu-eng) deu-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 99.5 type value f1 99.33333333333333 type value precision 99.25 type value recall 99.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nld-eng) nld-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.8 type value f1 97.2 type value precision 96.89999999999999 type value recall 97.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ron-eng) ron-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.8 type value f1 97.18333333333334 type value precision 96.88333333333333 type value recall 97.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ang-eng) ang-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 77.61194029850746 type value f1 72.81094527363183 type value precision 70.83333333333333 type value recall 77.61194029850746 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ido-eng) ido-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.7 type value f1 91.91666666666667 type value precision 91.08333333333334 type value recall 93.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (jav-eng) jav-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 88.29268292682927 type value f1 85.27642276422765 type value precision 84.01277584204414 type value recall 88.29268292682927 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (isl-eng) isl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.1 type value f1 95.0 type value precision 94.46666666666668 type value recall 96.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (slv-eng) slv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.681652490887 type value f1 91.90765492102065 type value precision 91.05913325232888 type value recall 93.681652490887 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cym-eng) cym-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.17391304347827 type value f1 89.97101449275361 type value precision 88.96811594202899 type value recall 92.17391304347827 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kaz-eng) kaz-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 90.43478260869566 type value f1 87.72173913043478 type value precision 86.42028985507245 type value recall 90.43478260869566 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (est-eng) est-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 90.4 type value f1 88.03 type value precision 86.95 type value recall 90.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (heb-eng) heb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.4 type value f1 91.45666666666666 type value precision 90.525 type value recall 93.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gla-eng) gla-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 81.9059107358263 type value f1 78.32557872364869 type value precision 76.78260286824823 type value recall 81.9059107358263 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mar-eng) mar-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.3 type value f1 92.58333333333333 type value precision 91.73333333333332 type value recall 94.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lat-eng) lat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 79.10000000000001 type value f1 74.50500000000001 type value precision 72.58928571428571 type value recall 79.10000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bel-eng) bel-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.6 type value f1 95.55 type value precision 95.05 type value recall 96.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pms-eng) pms-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 82.0952380952381 type value f1 77.98458049886621 type value precision 76.1968253968254 type value recall 82.0952380952381 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gle-eng) gle-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 87.9 type value f1 84.99190476190476 type value precision 83.65 type value recall 87.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pes-eng) pes-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.7 type value f1 94.56666666666666 type value precision 94.01666666666667 type value recall 95.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nob-eng) nob-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 98.6 type value f1 98.2 type value precision 98.0 type value recall 98.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bul-eng) bul-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.6 type value f1 94.38333333333334 type value precision 93.78333333333335 type value recall 95.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cbk-eng) cbk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 87.4 type value f1 84.10380952380952 type value precision 82.67 type value recall 87.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hun-eng) hun-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.5 type value f1 94.33333333333334 type value precision 93.78333333333333 type value recall 95.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (uig-eng) uig-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 89.4 type value f1 86.82000000000001 type value precision 85.64500000000001 type value recall 89.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (rus-eng) rus-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.1 type value f1 93.56666666666668 type value precision 92.81666666666666 type value recall 95.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (spa-eng) spa-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 98.9 type value f1 98.6 type value precision 98.45 type value recall 98.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hye-eng) hye-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.01347708894879 type value f1 93.51752021563343 type value precision 92.82794249775381 type value recall 95.01347708894879 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tel-eng) tel-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.00854700854701 type value f1 96.08262108262107 type value precision 95.65527065527067 type value recall 97.00854700854701 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (afr-eng) afr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.5 type value f1 95.39999999999999 type value precision 94.88333333333333 type value recall 96.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mon-eng) mon-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.5909090909091 type value f1 95.49242424242425 type value precision 94.9621212121212 type value recall 96.5909090909091 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (arz-eng) arz-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 84.90566037735849 type value f1 81.85883997204752 type value precision 80.54507337526205 type value recall 84.90566037735849 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hrv-eng) hrv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.5 type value f1 96.75 type value precision 96.38333333333333 type value recall 97.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nov-eng) nov-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 86.7704280155642 type value f1 82.99610894941635 type value precision 81.32295719844358 type value recall 86.7704280155642 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gsw-eng) gsw-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 67.52136752136752 type value f1 61.89662189662191 type value precision 59.68660968660969 type value recall 67.52136752136752 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nds-eng) nds-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 89.2 type value f1 86.32 type value precision 85.015 type value recall 89.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ukr-eng) ukr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.0 type value f1 94.78333333333333 type value precision 94.18333333333334 type value recall 96.0 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (uzb-eng) uzb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 83.8785046728972 type value f1 80.54517133956385 type value precision 79.154984423676 type value recall 83.8785046728972 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lit-eng) lit-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.60000000000001 type value f1 92.01333333333334 type value precision 91.28333333333333 type value recall 93.60000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ina-eng) ina-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.1 type value f1 96.26666666666667 type value precision 95.85000000000001 type value recall 97.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lfn-eng) lfn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 84.3 type value f1 80.67833333333333 type value precision 79.03928571428571 type value recall 84.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (zsm-eng) zsm-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.3 type value f1 96.48333333333332 type value precision 96.08333333333331 type value recall 97.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ita-eng) ita-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.7 type value f1 94.66666666666667 type value precision 94.16666666666667 type value recall 95.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cmn-eng) cmn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.2 type value f1 96.36666666666667 type value precision 95.96666666666668 type value recall 97.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lvs-eng) lvs-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.3 type value f1 92.80666666666667 type value precision 92.12833333333333 type value recall 94.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (glg-eng) glg-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.0 type value f1 96.22333333333334 type value precision 95.875 type value recall 97.0 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ceb-eng) ceb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 74.33333333333333 type value f1 70.78174603174602 type value precision 69.28333333333332 type value recall 74.33333333333333 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bre-eng) bre-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 37.6 type value f1 32.938348952090365 type value precision 31.2811038961039 type value recall 37.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ben-eng) ben-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 91.5 type value f1 89.13333333333333 type value precision 88.03333333333333 type value recall 91.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swg-eng) swg-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 82.14285714285714 type value f1 77.67857142857143 type value precision 75.59523809523809 type value recall 82.14285714285714 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (arq-eng) arq-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 69.0450054884742 type value f1 63.070409283362075 type value precision 60.58992781824835 type value recall 69.0450054884742 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kab-eng) kab-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 63.1 type value f1 57.848333333333336 type value precision 55.69500000000001 type value recall 63.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fra-eng) fra-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.1 type value f1 95.01666666666667 type value precision 94.5 type value recall 96.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (por-eng) por-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.89999999999999 type value f1 94.90666666666667 type value precision 94.425 type value recall 95.89999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tat-eng) tat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 87.6 type value f1 84.61333333333333 type value precision 83.27 type value recall 87.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (oci-eng) oci-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 76.4 type value f1 71.90746031746032 type value precision 70.07027777777778 type value recall 76.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pol-eng) pol-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.89999999999999 type value f1 97.26666666666667 type value precision 96.95 type value recall 97.89999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (war-eng) war-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 78.8 type value f1 74.39555555555555 type value precision 72.59416666666667 type value recall 78.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (aze-eng) aze-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.19999999999999 type value f1 93.78999999999999 type value precision 93.125 type value recall 95.19999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (vie-eng) vie-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.8 type value f1 97.1 type value precision 96.75 type value recall 97.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nno-eng) nno-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.6 type value f1 94.25666666666666 type value precision 93.64166666666668 type value recall 95.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cha-eng) cha-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 56.934306569343065 type value f1 51.461591936044485 type value precision 49.37434827945776 type value recall 56.934306569343065 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mhr-eng) mhr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 20.200000000000003 type value f1 16.91799284049284 type value precision 15.791855158730158 type value recall 20.200000000000003 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dan-eng) dan-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.2 type value f1 95.3 type value precision 94.85 type value recall 96.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ell-eng) ell-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.3 type value f1 95.11666666666667 type value precision 94.53333333333333 type value recall 96.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (amh-eng) amh-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 89.88095238095238 type value f1 87.14285714285714 type value precision 85.96230158730161 type value recall 89.88095238095238 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pam-eng) pam-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 24.099999999999998 type value f1 19.630969083349783 type value precision 18.275094905094907 type value recall 24.099999999999998 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hsb-eng) hsb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 83.4368530020704 type value f1 79.45183870649709 type value precision 77.7432712215321 type value recall 83.4368530020704 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (srp-eng) srp-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.8 type value f1 94.53333333333333 type value precision 93.91666666666666 type value recall 95.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (epo-eng) epo-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 98.8 type value f1 98.48333333333332 type value precision 98.33333333333334 type value recall 98.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kzj-eng) kzj-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 17.5 type value f1 14.979285714285714 type value precision 14.23235060690943 type value recall 17.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (awa-eng) awa-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.93939393939394 type value f1 91.991341991342 type value precision 91.05339105339105 type value recall 93.93939393939394 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fao-eng) fao-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 89.31297709923665 type value f1 86.76844783715012 type value precision 85.63613231552164 type value recall 89.31297709923665 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mal-eng) mal-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 99.12663755458514 type value f1 98.93255701115964 type value precision 98.83551673944687 type value recall 99.12663755458514 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ile-eng) ile-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.0 type value f1 89.77999999999999 type value precision 88.78333333333333 type value recall 92.0 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bos-eng) bos-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.89265536723164 type value f1 95.85687382297553 type value precision 95.33898305084746 type value recall 96.89265536723164 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cor-eng) cor-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 14.6 type value f1 11.820611790170615 type value precision 11.022616224355355 type value recall 14.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cat-eng) cat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.89999999999999 type value f1 94.93333333333334 type value precision 94.48666666666666 type value recall 95.89999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (eus-eng) eus-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 87.6 type value f1 84.72333333333334 type value precision 83.44166666666666 type value recall 87.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (yue-eng) yue-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.8 type value f1 93.47333333333333 type value precision 92.875 type value recall 94.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swe-eng) swe-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.6 type value f1 95.71666666666665 type value precision 95.28333333333335 type value recall 96.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dtp-eng) dtp-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 17.8 type value f1 14.511074040901628 type value precision 13.503791000666002 type value recall 17.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kat-eng) kat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.10187667560321 type value f1 92.46648793565683 type value precision 91.71134941912423 type value recall 94.10187667560321 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (jpn-eng) jpn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.0 type value f1 96.11666666666666 type value precision 95.68333333333334 type value recall 97.0 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (csb-eng) csb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 72.72727272727273 type value f1 66.58949745906267 type value precision 63.86693017127799 type value recall 72.72727272727273 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (xho-eng) xho-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 90.14084507042254 type value f1 88.26291079812206 type value precision 87.32394366197182 type value recall 90.14084507042254 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (orv-eng) orv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 64.67065868263472 type value f1 58.2876627696987 type value precision 55.79255774165953 type value recall 64.67065868263472 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ind-eng) ind-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.6 type value f1 94.41666666666667 type value precision 93.85 type value recall 95.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tuk-eng) tuk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 55.172413793103445 type value f1 49.63992493549144 type value precision 47.71405113769646 type value recall 55.172413793103445 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (max-eng) max-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 77.46478873239437 type value f1 73.4417616811983 type value precision 71.91607981220658 type value recall 77.46478873239437 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swh-eng) swh-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 84.61538461538461 type value f1 80.91452991452994 type value precision 79.33760683760683 type value recall 84.61538461538461 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hin-eng) hin-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 98.2 type value f1 97.6 type value precision 97.3 type value recall 98.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dsb-eng) dsb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 75.5741127348643 type value f1 72.00417536534445 type value precision 70.53467872883321 type value recall 75.5741127348643 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ber-eng) ber-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 62.2 type value f1 55.577460317460314 type value precision 52.98583333333333 type value recall 62.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tam-eng) tam-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.18241042345277 type value f1 90.6468124709167 type value precision 89.95656894679696 type value recall 92.18241042345277 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (slk-eng) slk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.1 type value f1 95.13333333333333 type value precision 94.66666666666667 type value recall 96.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tgl-eng) tgl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.8 type value f1 95.85000000000001 type value precision 95.39999999999999 type value recall 96.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ast-eng) ast-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.1259842519685 type value f1 89.76377952755905 type value precision 88.71391076115485 type value recall 92.1259842519685 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mkd-eng) mkd-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.1 type value f1 92.49 type value precision 91.725 type value recall 94.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (khm-eng) khm-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 77.5623268698061 type value f1 73.27364463791058 type value precision 71.51947852086357 type value recall 77.5623268698061 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ces-eng) ces-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.39999999999999 type value f1 96.56666666666666 type value precision 96.16666666666667 type value recall 97.39999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tzl-eng) tzl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 66.34615384615384 type value f1 61.092032967032964 type value precision 59.27197802197802 type value recall 66.34615384615384 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (urd-eng) urd-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.89999999999999 type value f1 93.41190476190476 type value precision 92.7 type value recall 94.89999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ara-eng) ara-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.10000000000001 type value f1 91.10000000000001 type value precision 90.13333333333333 type value recall 93.10000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kor-eng) kor-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.7 type value f1 91.97333333333334 type value precision 91.14166666666667 type value recall 93.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (yid-eng) yid-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.21698113207547 type value f1 90.3796046720575 type value precision 89.56367924528303 type value recall 92.21698113207547 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fin-eng) fin-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.6 type value f1 96.91666666666667 type value precision 96.6 type value recall 97.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tha-eng) tha-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.44525547445255 type value f1 96.71532846715328 type value precision 96.35036496350365 type value recall 97.44525547445255 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (wuu-eng) wuu-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.1 type value f1 92.34000000000002 type value precision 91.49166666666667 type value recall 94.1 task dataset metrics type Retrieval type name config split revision webis-touche2020 MTEB Touche2020 default test None type value map_at_1 3.2910000000000004 type value map_at_10 10.373000000000001 type value map_at_100 15.612 type value map_at_1000 17.06 type value map_at_3 6.119 type value map_at_5 7.917000000000001 type value mrr_at_1 44.897999999999996 type value mrr_at_10 56.054 type value mrr_at_100 56.82000000000001 type value mrr_at_1000 56.82000000000001 type value mrr_at_3 52.381 type value mrr_at_5 53.81 type value ndcg_at_1 42.857 type value ndcg_at_10 27.249000000000002 type value ndcg_at_100 36.529 type value ndcg_at_1000 48.136 type value ndcg_at_3 33.938 type value ndcg_at_5 29.951 type value precision_at_1 44.897999999999996 type value precision_at_10 22.653000000000002 type value precision_at_100 7.000000000000001 type value precision_at_1000 1.48 type value precision_at_3 32.653 type value precision_at_5 27.755000000000003 type value recall_at_1 3.2910000000000004 type value recall_at_10 16.16 type value recall_at_100 43.908 type value recall_at_1000 79.823 type value recall_at_3 7.156 type value recall_at_5 10.204 task dataset metrics type Classification type name config split revision mteb/toxic_conversations_50k MTEB ToxicConversationsClassification default test d7c0de2777da35d6aae2200a62c6e0e5af397c4c type value accuracy 71.05879999999999 type value ap 14.609748142799111 type value f1 54.878956295843096 task dataset metrics type Classification type name config split revision mteb/tweet_sentiment_extraction MTEB TweetSentimentExtractionClassification default test d604517c81ca91fe16a244d1248fc021f9ecee7a type value accuracy 64.61799660441426 type value f1 64.8698191961434 task dataset metrics type Clustering type name config split revision mteb/twentynewsgroups-clustering MTEB TwentyNewsgroupsClustering default test 6125ec4e24fa026cec8a478383ee943acfbd5449 type value v_measure 51.32860036611885 task dataset metrics type PairClassification type name config split revision mteb/twittersemeval2015-pairclassification MTEB TwitterSemEval2015 default test 70970daeab8776df92f5ea462b6173c0b46fd2d1 type value cos_sim_accuracy 88.34714192048638 type value cos_sim_ap 80.26732975975634 type value cos_sim_f1 73.53415148134374 type value cos_sim_precision 69.34767360299276 type value cos_sim_recall 78.25857519788919 type value dot_accuracy 88.34714192048638 type value dot_ap 80.26733698491206 type value dot_f1 73.53415148134374 type value dot_precision 69.34767360299276 type value dot_recall 78.25857519788919 type value euclidean_accuracy 88.34714192048638 type value euclidean_ap 80.26734337771738 type value euclidean_f1 73.53415148134374 type value euclidean_precision 69.34767360299276 type value euclidean_recall 78.25857519788919 type value manhattan_accuracy 88.30541813196639 type value manhattan_ap 80.19415808104145 type value manhattan_f1 73.55143870713441 type value manhattan_precision 73.25307511122743 type value manhattan_recall 73.85224274406332 type value max_accuracy 88.34714192048638 type value max_ap 80.26734337771738 type value max_f1 73.55143870713441 task dataset metrics type PairClassification type name config split revision mteb/twitterurlcorpus-pairclassification MTEB TwitterURLCorpus default test 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf type value cos_sim_accuracy 89.81061047075717 type value cos_sim_ap 87.11747055081017 type value cos_sim_f1 80.04355498817256 type value cos_sim_precision 78.1165262000733 type value cos_sim_recall 82.06806282722513 type value dot_accuracy 89.81061047075717 type value dot_ap 87.11746902745236 type value dot_f1 80.04355498817256 type value dot_precision 78.1165262000733 type value dot_recall 82.06806282722513 type value euclidean_accuracy 89.81061047075717 type value euclidean_ap 87.11746919324248 type value euclidean_f1 80.04355498817256 type value euclidean_precision 78.1165262000733 type value euclidean_recall 82.06806282722513 type value manhattan_accuracy 89.79508673885202 type value manhattan_ap 87.11074390832218 type value manhattan_f1 80.13002540726349 type value manhattan_precision 77.83826945412311 type value manhattan_recall 82.56082537727133 type value max_accuracy 89.81061047075717 type value max_ap 87.11747055081017 type value max_f1 80.13002540726349	multilingual af am ar as az be bg bn br bs ca cs cy da de el en eo es et eu fa fi fr fy ga gd gl gu ha he hi hr hu hy id is it ja jv ka kk km kn ko ku ky la lo lt lv mg mk ml mn mr ms my ne nl no om or pa pl ps pt ro ru sa sd si sk sl so sq sr su sv sw ta te th tl tr ug uk ur uz vi xh yi zh	mit

Multilingual-E5-large-instruct

Multilingual E5 Text Embeddings: A Technical Report. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024

This model has 24 layers and the embedding size is 1024.

Usage

Below are examples to encode queries and passages from the MS-MARCO passage ranking dataset.

Transformers

import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def average_pool(last_hidden_states: Tensor,
                 attention_mask: Tensor) -> Tensor:
    last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
    return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]

def get_detailed_instruct(task_description: str, query: str) -> str:
    return f'Instruct: {task_description}\nQuery: {query}'

# Each query must come with a one-sentence instruction that describes the task
task = 'Given a web search query, retrieve relevant passages that answer the query'
queries = [
    get_detailed_instruct(task, 'how much protein should a female eat'),
    get_detailed_instruct(task, '南瓜的家常做法')
]
# No need to add instruction for retrieval documents
documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"
]
input_texts = queries + documents

tokenizer = AutoTokenizer.from_pretrained('intfloat/multilingual-e5-large-instruct')
model = AutoModel.from_pretrained('intfloat/multilingual-e5-large-instruct')

# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')

outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
# => [[91.92852783203125, 67.580322265625], [70.3814468383789, 92.1330795288086]]

Sentence Transformers

from sentence_transformers import SentenceTransformer

def get_detailed_instruct(task_description: str, query: str) -> str:
    return f'Instruct: {task_description}\nQuery: {query}'

# Each query must come with a one-sentence instruction that describes the task
task = 'Given a web search query, retrieve relevant passages that answer the query'
queries = [
    get_detailed_instruct(task, 'how much protein should a female eat'),
    get_detailed_instruct(task, '南瓜的家常做法')
]
# No need to add instruction for retrieval documents
documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"
]
input_texts = queries + documents

model = SentenceTransformer('intfloat/multilingual-e5-large-instruct')

embeddings = model.encode(input_texts, convert_to_tensor=True, normalize_embeddings=True)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
# [[91.92853546142578, 67.5802993774414], [70.38143157958984, 92.13307189941406]]

Infinity

Usage with Infinity:

docker run --gpus all -v $PWD/data:/app/.cache -e HF_TOKEN=$HF_TOKEN -p "7997":"7997" \
michaelf34/infinity:0.0.68 \
v2 --model-id intfloat/multilingual-e5-large-instruct --revision "main" --dtype float16 --batch-size 32 --engine torch --port 7997

Supported Languages

This model is initialized from xlm-roberta-large and continually trained on a mixture of multilingual datasets. It supports 100 languages from xlm-roberta, but low-resource languages may see performance degradation.

Training Details

Initialization: xlm-roberta-large

First stage: contrastive pre-training with 1 billion weakly supervised text pairs.

Second stage: fine-tuning on datasets from the E5-mistral paper.

MTEB Benchmark Evaluation

Check out unilm/e5 to reproduce evaluation results on the BEIR and MTEB benchmark.

FAQ

1. Do I need to add instructions to the query?

Yes, this is how the model is trained, otherwise you will see a performance degradation. The task definition should be a one-sentence instruction that describes the task. This is a way to customize text embeddings for different scenarios through natural language instructions.

Please check out unilm/e5/utils.py for instructions we used for evaluation.

On the other hand, there is no need to add instructions to the document side.

2. Why are my reproduced results slightly different from reported in the model card?

Different versions of transformers and pytorch could cause negligible but non-zero performance differences.

3. Why does the cosine similarity scores distribute around 0.7 to 1.0?

This is a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss.

For text embedding tasks like text retrieval or semantic similarity, what matters is the relative order of the scores instead of the absolute values, so this should not be an issue.

Citation

If you find our paper or models helpful, please consider cite as follows:

@article{wang2024multilingual,
  title={Multilingual E5 Text Embeddings: A Technical Report},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2402.05672},
  year={2024}
}

Limitations

Long texts will be truncated to at most 512 tokens.