multilingual-e5-large/README.md

tags, model-index, language, license

tags	model-index	language	license
mteb Sentence Transformers sentence-similarity feature-extraction sentence-transformers	name results multilingual-e5-large task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en) en test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 79.05970149253731 type value ap 43.486574390835635 type value f1 73.32700092140148 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (de) de test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 71.22055674518201 type value ap 81.55756710830498 type value f1 69.28271787752661 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en-ext) en-ext test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 80.41979010494754 type value ap 29.34879922376344 type value f1 67.62475449011278 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (ja) ja test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 77.8372591006424 type value ap 26.557560591210738 type value f1 64.96619417368707 task dataset metrics type Classification type name config split revision mteb/amazon_polarity MTEB AmazonPolarityClassification default test e2d317d38cd51312af73b3d32a06d1a08b442046 type value accuracy 93.489875 type value ap 90.98758636917603 type value f1 93.48554819717332 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (en) en test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 47.564 type value f1 46.75122173518047 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (de) de test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 45.400000000000006 type value f1 44.17195682400632 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (es) es test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 43.068 type value f1 42.38155696855596 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (fr) fr test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 41.89 type value f1 40.84407321682663 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (ja) ja test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 40.120000000000005 type value f1 39.522976223819114 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (zh) zh test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 38.832 type value f1 38.0392533394713 task dataset metrics type Retrieval type name config split revision arguana MTEB ArguAna default test None type value map_at_1 30.725 type value map_at_10 46.055 type value map_at_100 46.900999999999996 type value map_at_1000 46.911 type value map_at_3 41.548 type value map_at_5 44.297 type value mrr_at_1 31.152 type value mrr_at_10 46.231 type value mrr_at_100 47.07 type value mrr_at_1000 47.08 type value mrr_at_3 41.738 type value mrr_at_5 44.468999999999994 type value ndcg_at_1 30.725 type value ndcg_at_10 54.379999999999995 type value ndcg_at_100 58.138 type value ndcg_at_1000 58.389 type value ndcg_at_3 45.156 type value ndcg_at_5 50.123 type value precision_at_1 30.725 type value precision_at_10 8.087 type value precision_at_100 0.9769999999999999 type value precision_at_1000 0.1 type value precision_at_3 18.54 type value precision_at_5 13.542000000000002 type value recall_at_1 30.725 type value recall_at_10 80.868 type value recall_at_100 97.653 type value recall_at_1000 99.57300000000001 type value recall_at_3 55.619 type value recall_at_5 67.71000000000001 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-p2p MTEB ArxivClusteringP2P default test a122ad7f3f0291bf49cc6f4d32aa80929df69d5d type value v_measure 44.30960650674069 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-s2s MTEB ArxivClusteringS2S default test f910caf1a6075f7329cdf8c1a6135696f37dbd53 type value v_measure 38.427074197498996 task dataset metrics type Reranking type name config split revision mteb/askubuntudupquestions-reranking MTEB AskUbuntuDupQuestions default test 2000358ca161889fa9c082cb41daa8dcfb161a54 type value map 60.28270056031872 type value mrr 74.38332673789738 task dataset metrics type STS type name config split revision mteb/biosses-sts MTEB BIOSSES default test d3fb88f8f02e40887cd149695127462bbcf29b4a type value cos_sim_pearson 84.05942144105269 type value cos_sim_spearman 82.51212105850809 type value euclidean_pearson 81.95639829909122 type value euclidean_spearman 82.3717564144213 type value manhattan_pearson 81.79273425468256 type value manhattan_spearman 82.20066817871039 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (de-en) de-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 99.46764091858039 type value f1 99.37717466945023 type value precision 99.33194154488518 type value recall 99.46764091858039 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (fr-en) fr-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 98.29407880255337 type value f1 98.11248073959938 type value precision 98.02443319392472 type value recall 98.29407880255337 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (ru-en) ru-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 97.79009352268791 type value f1 97.5176076665512 type value precision 97.38136473848286 type value recall 97.79009352268791 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (zh-en) zh-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 99.26276987888363 type value f1 99.20133403545726 type value precision 99.17500438827453 type value recall 99.26276987888363 task dataset metrics type Classification type name config split revision mteb/banking77 MTEB Banking77Classification default test 0fd18e25b25c072e09e0d92ab615fda904d66300 type value accuracy 84.72727272727273 type value f1 84.67672206031433 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-p2p MTEB BiorxivClusteringP2P default test 65b79d1d13f80053f67aca9498d9402c2d9f1f40 type value v_measure 35.34220182511161 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-s2s MTEB BiorxivClusteringS2S default test 258694dd0231531bc1fd9de6ceb52a0853c6d908 type value v_measure 33.4987096128766 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackRetrieval default test None type value map_at_1 25.558249999999997 type value map_at_10 34.44425000000001 type value map_at_100 35.59833333333333 type value map_at_1000 35.706916666666665 type value map_at_3 31.691749999999995 type value map_at_5 33.252916666666664 type value mrr_at_1 30.252666666666666 type value mrr_at_10 38.60675 type value mrr_at_100 39.42666666666666 type value mrr_at_1000 39.48408333333334 type value mrr_at_3 36.17441666666665 type value mrr_at_5 37.56275 type value ndcg_at_1 30.252666666666666 type value ndcg_at_10 39.683 type value ndcg_at_100 44.68541666666667 type value ndcg_at_1000 46.94316666666668 type value ndcg_at_3 34.961749999999995 type value ndcg_at_5 37.215666666666664 type value precision_at_1 30.252666666666666 type value precision_at_10 6.904166666666667 type value precision_at_100 1.0989999999999995 type value precision_at_1000 0.14733333333333334 type value precision_at_3 16.037666666666667 type value precision_at_5 11.413583333333333 type value recall_at_1 25.558249999999997 type value recall_at_10 51.13341666666666 type value recall_at_100 73.08366666666667 type value recall_at_1000 88.79483333333334 type value recall_at_3 37.989083333333326 type value recall_at_5 43.787833333333325 task dataset metrics type Retrieval type name config split revision climate-fever MTEB ClimateFEVER default test None type value map_at_1 10.338 type value map_at_10 18.360000000000003 type value map_at_100 19.942 type value map_at_1000 20.134 type value map_at_3 15.174000000000001 type value map_at_5 16.830000000000002 type value mrr_at_1 23.257 type value mrr_at_10 33.768 type value mrr_at_100 34.707 type value mrr_at_1000 34.766000000000005 type value mrr_at_3 30.977 type value mrr_at_5 32.528 type value ndcg_at_1 23.257 type value ndcg_at_10 25.733 type value ndcg_at_100 32.288 type value ndcg_at_1000 35.992000000000004 type value ndcg_at_3 20.866 type value ndcg_at_5 22.612 type value precision_at_1 23.257 type value precision_at_10 8.124 type value precision_at_100 1.518 type value precision_at_1000 0.219 type value precision_at_3 15.679000000000002 type value precision_at_5 12.117 type value recall_at_1 10.338 type value recall_at_10 31.154 type value recall_at_100 54.161 type value recall_at_1000 75.21900000000001 type value recall_at_3 19.427 type value recall_at_5 24.214 task dataset metrics type Retrieval type name config split revision dbpedia-entity MTEB DBPedia default test None type value map_at_1 8.498 type value map_at_10 19.103 type value map_at_100 27.375 type value map_at_1000 28.981 type value map_at_3 13.764999999999999 type value map_at_5 15.950000000000001 type value mrr_at_1 65.5 type value mrr_at_10 74.53800000000001 type value mrr_at_100 74.71799999999999 type value mrr_at_1000 74.725 type value mrr_at_3 72.792 type value mrr_at_5 73.554 type value ndcg_at_1 53.37499999999999 type value ndcg_at_10 41.286 type value ndcg_at_100 45.972 type value ndcg_at_1000 53.123 type value ndcg_at_3 46.172999999999995 type value ndcg_at_5 43.033 type value precision_at_1 65.5 type value precision_at_10 32.725 type value precision_at_100 10.683 type value precision_at_1000 1.978 type value precision_at_3 50 type value precision_at_5 41.349999999999994 type value recall_at_1 8.498 type value recall_at_10 25.070999999999998 type value recall_at_100 52.383 type value recall_at_1000 74.91499999999999 type value recall_at_3 15.207999999999998 type value recall_at_5 18.563 task dataset metrics type Classification type name config split revision mteb/emotion MTEB EmotionClassification default test 4f58c6b202a23cf9a4da393831edf4f9183cad37 type value accuracy 46.5 type value f1 41.93833713984145 task dataset metrics type Retrieval type name config split revision fever MTEB FEVER default test None type value map_at_1 67.914 type value map_at_10 78.10000000000001 type value map_at_100 78.333 type value map_at_1000 78.346 type value map_at_3 76.626 type value map_at_5 77.627 type value mrr_at_1 72.74199999999999 type value mrr_at_10 82.414 type value mrr_at_100 82.511 type value mrr_at_1000 82.513 type value mrr_at_3 81.231 type value mrr_at_5 82.065 type value ndcg_at_1 72.74199999999999 type value ndcg_at_10 82.806 type value ndcg_at_100 83.677 type value ndcg_at_1000 83.917 type value ndcg_at_3 80.305 type value ndcg_at_5 81.843 type value precision_at_1 72.74199999999999 type value precision_at_10 10.24 type value precision_at_100 1.089 type value precision_at_1000 0.11299999999999999 type value precision_at_3 31.268 type value precision_at_5 19.706000000000003 type value recall_at_1 67.914 type value recall_at_10 92.889 type value recall_at_100 96.42699999999999 type value recall_at_1000 97.92 type value recall_at_3 86.21 type value recall_at_5 90.036 task dataset metrics type Retrieval type name config split revision fiqa MTEB FiQA2018 default test None type value map_at_1 22.166 type value map_at_10 35.57 type value map_at_100 37.405 type value map_at_1000 37.564 type value map_at_3 30.379 type value map_at_5 33.324 type value mrr_at_1 43.519000000000005 type value mrr_at_10 51.556000000000004 type value mrr_at_100 52.344 type value mrr_at_1000 52.373999999999995 type value mrr_at_3 48.868 type value mrr_at_5 50.319 type value ndcg_at_1 43.519000000000005 type value ndcg_at_10 43.803 type value ndcg_at_100 50.468999999999994 type value ndcg_at_1000 53.111 type value ndcg_at_3 38.893 type value ndcg_at_5 40.653 type value precision_at_1 43.519000000000005 type value precision_at_10 12.253 type value precision_at_100 1.931 type value precision_at_1000 0.242 type value precision_at_3 25.617 type value precision_at_5 19.383 type value recall_at_1 22.166 type value recall_at_10 51.6 type value recall_at_100 76.574 type value recall_at_1000 92.192 type value recall_at_3 34.477999999999994 type value recall_at_5 41.835 task dataset metrics type Retrieval type name config split revision hotpotqa MTEB HotpotQA default test None type value map_at_1 39.041 type value map_at_10 62.961999999999996 type value map_at_100 63.79899999999999 type value map_at_1000 63.854 type value map_at_3 59.399 type value map_at_5 61.669 type value mrr_at_1 78.082 type value mrr_at_10 84.321 type value mrr_at_100 84.49600000000001 type value mrr_at_1000 84.502 type value mrr_at_3 83.421 type value mrr_at_5 83.977 type value ndcg_at_1 78.082 type value ndcg_at_10 71.229 type value ndcg_at_100 74.10900000000001 type value ndcg_at_1000 75.169 type value ndcg_at_3 66.28699999999999 type value ndcg_at_5 69.084 type value precision_at_1 78.082 type value precision_at_10 14.993 type value precision_at_100 1.7239999999999998 type value precision_at_1000 0.186 type value precision_at_3 42.737 type value precision_at_5 27.843 type value recall_at_1 39.041 type value recall_at_10 74.96300000000001 type value recall_at_100 86.199 type value recall_at_1000 93.228 type value recall_at_3 64.105 type value recall_at_5 69.608 task dataset metrics type Classification type name config split revision mteb/imdb MTEB ImdbClassification default test 3d86128a09e091d6018b6d26cad27f2739fc2db7 type value accuracy 90.23160000000001 type value ap 85.5674856808308 type value f1 90.18033354786317 task dataset metrics type Retrieval type name config split revision msmarco MTEB MSMARCO default dev None type value map_at_1 24.091 type value map_at_10 36.753 type value map_at_100 37.913000000000004 type value map_at_1000 37.958999999999996 type value map_at_3 32.818999999999996 type value map_at_5 35.171 type value mrr_at_1 24.742 type value mrr_at_10 37.285000000000004 type value mrr_at_100 38.391999999999996 type value mrr_at_1000 38.431 type value mrr_at_3 33.440999999999995 type value mrr_at_5 35.75 type value ndcg_at_1 24.742 type value ndcg_at_10 43.698 type value ndcg_at_100 49.145 type value ndcg_at_1000 50.23800000000001 type value ndcg_at_3 35.769 type value ndcg_at_5 39.961999999999996 type value precision_at_1 24.742 type value precision_at_10 6.7989999999999995 type value precision_at_100 0.95 type value precision_at_1000 0.104 type value precision_at_3 15.096000000000002 type value precision_at_5 11.183 type value recall_at_1 24.091 type value recall_at_10 65.068 type value recall_at_100 89.899 type value recall_at_1000 98.16 type value recall_at_3 43.68 type value recall_at_5 53.754999999999995 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (en) en test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 93.66621067031465 type value f1 93.49622853272142 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (de) de test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 91.94702733164272 type value f1 91.17043441745282 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (es) es test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 92.20146764509674 type value f1 91.98359080555608 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (fr) fr test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 88.99780770435328 type value f1 89.19746342724068 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (hi) hi test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 89.78486912871998 type value f1 89.24578823628642 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (th) th test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 88.74502712477394 type value f1 89.00297573881542 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (en) en test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 77.9046967624259 type value f1 59.36787125785957 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (de) de test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 74.5280360664976 type value f1 57.17723440888718 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (es) es test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 75.44029352901934 type value f1 54.052855531072964 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (fr) fr test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 70.5606013153774 type value f1 52.62215934386531 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (hi) hi test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 73.11581211903908 type value f1 52.341291845645465 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (th) th test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 74.28933092224233 type value f1 57.07918745504911 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (af) af test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 62.38063214525892 type value f1 59.46463723443009 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (am) am test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 56.06926698049766 type value f1 52.49084283283562 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ar) ar test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 60.74983187626093 type value f1 56.960640620165904 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (az) az test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 64.86550100874243 type value f1 62.47370548140688 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (bn) bn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 63.971082716879636 type value f1 61.03812421957381 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (cy) cy test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 54.98318762609282 type value f1 51.51207916008392 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (da) da test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.45527908540686 type value f1 66.16631905400318 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (de) de test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.32750504371216 type value f1 66.16755288646591 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (el) el test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.09213180901143 type value f1 66.95654394661507 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (en) en test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 73.75588433086752 type value f1 71.79973779656923 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (es) es test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 70.49428379287154 type value f1 68.37494379215734 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fa) fa test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.90921318090115 type value f1 66.79517376481645 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fi) fi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 70.12104909213181 type value f1 67.29448842879584 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fr) fr test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.34095494283793 type value f1 67.01134288992947 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (he) he test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 67.61264290517822 type value f1 64.68730512660757 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hi) hi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 67.79757901815738 type value f1 65.24938539425598 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hu) hu test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.68728984532616 type value f1 67.0487169762553 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hy) hy test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 62.07464694014795 type value f1 59.183532276789286 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (id) id test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 70.04707464694015 type value f1 67.66829629003848 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (is) is test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 62.42434431741762 type value f1 59.01617226544757 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (it) it test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 70.53127101546738 type value f1 68.10033760906255 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ja) ja test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 72.50504371217215 type value f1 69.74931103158923 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (jv) jv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 57.91190316072628 type value f1 54.05551136648796 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ka) ka test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 51.78211163416275 type value f1 49.874888544058535 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (km) km test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 47.017484868863484 type value f1 44.53364263352014 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (kn) kn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 62.16207128446537 type value f1 59.01185692320829 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ko) ko test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.42501681237391 type value f1 67.13169450166086 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (lv) lv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 67.0780094149294 type value f1 64.41720167850707 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ml) ml test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 65.57162071284466 type value f1 62.414138683804424 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (mn) mn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 61.71149966375252 type value f1 58.594805125087234 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ms) ms test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 66.03900470746471 type value f1 63.87937257883887 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (my) my test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 60.8776059179556 type value f1 57.48587618059131 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nb) nb test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.87895090786819 type value f1 66.8141299430347 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nl) nl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 70.45057162071285 type value f1 67.46444039673516 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pl) pl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 71.546738399462 type value f1 68.63640876702655 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pt) pt test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 70.72965702757229 type value f1 68.54119560379115 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ro) ro test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 68.35574983187625 type value f1 65.88844917691927 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ru) ru test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 71.70477471418964 type value f1 69.19665697061978 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sl) sl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 67.0880968392737 type value f1 64.76962317666086 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sq) sq test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 65.18493611297916 type value f1 62.49984559035371 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sv) sv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 71.75857431069265 type value f1 69.20053687623418 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sw) sw test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 58.500336247478145 type value f1 55.2972398687929 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ta) ta test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 62.68997982515132 type value f1 59.36848202755348 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (te) te test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 63.01950235373235 type value f1 60.09351954625423 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (th) th test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 68.29186281102892 type value f1 67.57860496703447 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tl) tl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 64.77471418964357 type value f1 61.913983147713836 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tr) tr test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.87222595830532 type value f1 66.03679033708141 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ur) ur test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 64.04505716207127 type value f1 61.28569169817908 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (vi) vi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.38466711499663 type value f1 67.20532357036844 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-CN) zh-CN test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 71.12306657700067 type value f1 68.91251226588182 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-TW) zh-TW test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 66.20040349697378 type value f1 66.02657347714175 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (af) af test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 68.73907195696032 type value f1 66.98484521791418 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (am) am test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 60.58843308675185 type value f1 58.95591723092005 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ar) ar test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 66.22730329522528 type value f1 66.0894499712115 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (az) az test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 66.48285137861465 type value f1 65.21963176785157 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (bn) bn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 67.74714189643578 type value f1 66.8212192745412 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (cy) cy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 59.09213180901143 type value f1 56.70735546356339 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (da) da test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 75.05716207128448 type value f1 74.8413712365364 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (de) de test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.69737726967047 type value f1 74.7664341963 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (el) el test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.90383322125084 type value f1 73.59201554448323 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (en) en test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.51176866173503 type value f1 77.46104434577758 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (es) es test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.31069266980496 type value f1 74.61048660675635 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fa) fa test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.95225285810356 type value f1 72.33160006574627 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fi) fi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.12373907195696 type value f1 73.20921012557481 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fr) fr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.86684599865501 type value f1 73.82348774610831 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (he) he test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.40215198386012 type value f1 71.11945183971858 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hi) hi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.12844653665098 type value f1 71.34450495911766 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hu) hu test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.52252858103566 type value f1 73.98878711342999 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hy) hy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 64.93611297915265 type value f1 63.723200467653385 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (id) id test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.11903160726295 type value f1 73.82138439467096 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (is) is test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 67.15198386012105 type value f1 66.02172193802167 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (it) it test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.32414256893072 type value f1 74.30943421170574 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ja) ja test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.46805648957633 type value f1 77.62808409298209 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (jv) jv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 63.318762609280434 type value f1 62.094284066075076 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ka) ka test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 58.34902488231338 type value f1 57.12893860987984 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (km) km test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 50.88433086751849 type value f1 48.2272350802058 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (kn) kn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 66.4425016812374 type value f1 64.61463095996173 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ko) ko test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 75.04707464694015 type value f1 75.05099199098998 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (lv) lv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.50437121721586 type value f1 69.83397721096314 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ml) ml test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 69.94283792871553 type value f1 68.8704663703913 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (mn) mn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 64.79488903833222 type value f1 63.615424063345436 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ms) ms test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 69.88231338264963 type value f1 68.57892302593237 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (my) my test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 63.248150638870214 type value f1 61.06680605338809 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nb) nb test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.84196368527236 type value f1 74.52566464968763 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nl) nl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.8285137861466 type value f1 74.8853197608802 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pl) pl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.13248150638869 type value f1 74.3982040999179 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pt) pt test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.49024882313383 type value f1 73.82153848368573 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ro) ro test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.72158708809684 type value f1 71.85049433180541 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ru) ru test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 75.137861466039 type value f1 75.37628348188467 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sl) sl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.86953597848016 type value f1 71.87537624521661 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sq) sq test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.27572293207801 type value f1 68.80017302344231 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sv) sv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 76.09952925353059 type value f1 76.07992707688408 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sw) sw test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 63.140551445864155 type value f1 61.73855010331415 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ta) ta test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 66.27774041694687 type value f1 64.83664868894539 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (te) te test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 66.69468728984533 type value f1 64.76239666920868 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (th) th test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.44653665097512 type value f1 73.14646052013873 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tl) tl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 67.71351714862139 type value f1 66.67212180163382 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tr) tr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.9946200403497 type value f1 73.87348793725525 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ur) ur test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 68.15400134498992 type value f1 67.09433241421094 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (vi) vi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.11365164761264 type value f1 73.59502539433753 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-CN) zh-CN test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 76.82582380632145 type value f1 76.89992945316313 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-TW) zh-TW test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.81237390719569 type value f1 72.36499770986265 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-p2p MTEB MedrxivClusteringP2P default test e7a26af6f3ae46b30dde8737f02c07b1505bcc73 type value v_measure 31.480506569594695 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-s2s MTEB MedrxivClusteringS2S default test 35191c8c0dca72d8ff3efcd72aa802307d469663 type value v_measure 29.71252128004552 task dataset metrics type Reranking type name config split revision mteb/mind_small MTEB MindSmallReranking default test 3bdac13927fdc888b903db93b2ffdbd90b295a69 type value map 31.421396787056548 type value mrr 32.48155274872267 task dataset metrics type Retrieval type name config split revision nfcorpus MTEB NFCorpus default test None type value map_at_1 5.595 type value map_at_10 12.642000000000001 type value map_at_100 15.726 type value map_at_1000 17.061999999999998 type value map_at_3 9.125 type value map_at_5 10.866000000000001 type value mrr_at_1 43.344 type value mrr_at_10 52.227999999999994 type value mrr_at_100 52.898999999999994 type value mrr_at_1000 52.944 type value mrr_at_3 49.845 type value mrr_at_5 51.115 type value ndcg_at_1 41.949999999999996 type value ndcg_at_10 33.995 type value ndcg_at_100 30.869999999999997 type value ndcg_at_1000 39.487 type value ndcg_at_3 38.903999999999996 type value ndcg_at_5 37.236999999999995 type value precision_at_1 43.344 type value precision_at_10 25.480000000000004 type value precision_at_100 7.672 type value precision_at_1000 2.028 type value precision_at_3 36.636 type value precision_at_5 32.632 type value recall_at_1 5.595 type value recall_at_10 16.466 type value recall_at_100 31.226 type value recall_at_1000 62.778999999999996 type value recall_at_3 9.931 type value recall_at_5 12.884 task dataset metrics type Retrieval type name config split revision nq MTEB NQ default test None type value map_at_1 40.414 type value map_at_10 56.754000000000005 type value map_at_100 57.457 type value map_at_1000 57.477999999999994 type value map_at_3 52.873999999999995 type value map_at_5 55.175 type value mrr_at_1 45.278 type value mrr_at_10 59.192 type value mrr_at_100 59.650000000000006 type value mrr_at_1000 59.665 type value mrr_at_3 56.141 type value mrr_at_5 57.998000000000005 type value ndcg_at_1 45.278 type value ndcg_at_10 64.056 type value ndcg_at_100 66.89 type value ndcg_at_1000 67.364 type value ndcg_at_3 56.97 type value ndcg_at_5 60.719 type value precision_at_1 45.278 type value precision_at_10 9.994 type value precision_at_100 1.165 type value precision_at_1000 0.121 type value precision_at_3 25.512 type value precision_at_5 17.509 type value recall_at_1 40.414 type value recall_at_10 83.596 type value recall_at_100 95.72 type value recall_at_1000 99.24 type value recall_at_3 65.472 type value recall_at_5 74.039 task dataset metrics type Retrieval type name config split revision quora MTEB QuoraRetrieval default test None type value map_at_1 70.352 type value map_at_10 84.369 type value map_at_100 85.02499999999999 type value map_at_1000 85.04 type value map_at_3 81.42399999999999 type value map_at_5 83.279 type value mrr_at_1 81.05 type value mrr_at_10 87.401 type value mrr_at_100 87.504 type value mrr_at_1000 87.505 type value mrr_at_3 86.443 type value mrr_at_5 87.10799999999999 type value ndcg_at_1 81.04 type value ndcg_at_10 88.181 type value ndcg_at_100 89.411 type value ndcg_at_1000 89.507 type value ndcg_at_3 85.28099999999999 type value ndcg_at_5 86.888 type value precision_at_1 81.04 type value precision_at_10 13.406 type value precision_at_100 1.5350000000000001 type value precision_at_1000 0.157 type value precision_at_3 37.31 type value precision_at_5 24.54 type value recall_at_1 70.352 type value recall_at_10 95.358 type value recall_at_100 99.541 type value recall_at_1000 99.984 type value recall_at_3 87.111 type value recall_at_5 91.643 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering MTEB RedditClustering default test 24640382cdbf8abc73003fb0fa6d111a705499eb type value v_measure 46.54068723291946 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering-p2p MTEB RedditClusteringP2P default test 282350215ef01743dc01b456c7f5241fa8937f16 type value v_measure 63.216287629895994 task dataset metrics type Retrieval type name config split revision scidocs MTEB SCIDOCS default test None type value map_at_1 4.023000000000001 type value map_at_10 10.071 type value map_at_100 11.892 type value map_at_1000 12.196 type value map_at_3 7.234 type value map_at_5 8.613999999999999 type value mrr_at_1 19.900000000000002 type value mrr_at_10 30.516 type value mrr_at_100 31.656000000000002 type value mrr_at_1000 31.723000000000003 type value mrr_at_3 27.400000000000002 type value mrr_at_5 29.270000000000003 type value ndcg_at_1 19.900000000000002 type value ndcg_at_10 17.474 type value ndcg_at_100 25.020999999999997 type value ndcg_at_1000 30.728 type value ndcg_at_3 16.588 type value ndcg_at_5 14.498 type value precision_at_1 19.900000000000002 type value precision_at_10 9.139999999999999 type value precision_at_100 2.011 type value precision_at_1000 0.33899999999999997 type value precision_at_3 15.667 type value precision_at_5 12.839999999999998 type value recall_at_1 4.023000000000001 type value recall_at_10 18.497 type value recall_at_100 40.8 type value recall_at_1000 68.812 type value recall_at_3 9.508 type value recall_at_5 12.983 task dataset metrics type STS type name config split revision mteb/sickr-sts MTEB SICK-R default test a6ea5a8cab320b040a23452cc28066d9beae2cee type value cos_sim_pearson 83.967008785134 type value cos_sim_spearman 80.23142141101837 type value euclidean_pearson 81.20166064704539 type value euclidean_spearman 80.18961335654585 type value manhattan_pearson 81.13925443187625 type value manhattan_spearman 80.07948723044424 task dataset metrics type STS type name config split revision mteb/sts12-sts MTEB STS12 default test a0d554a64d88156834ff5ae9920b964011b16384 type value cos_sim_pearson 86.94262461316023 type value cos_sim_spearman 80.01596278563865 type value euclidean_pearson 83.80799622922581 type value euclidean_spearman 79.94984954947103 type value manhattan_pearson 83.68473841756281 type value manhattan_spearman 79.84990707951822 task dataset metrics type STS type name config split revision mteb/sts13-sts MTEB STS13 default test 7e90230a92c190f1bf69ae9002b8cea547a64cca type value cos_sim_pearson 80.57346443146068 type value cos_sim_spearman 81.54689837570866 type value euclidean_pearson 81.10909881516007 type value euclidean_spearman 81.56746243261762 type value manhattan_pearson 80.87076036186582 type value manhattan_spearman 81.33074987964402 task dataset metrics type STS type name config split revision mteb/sts14-sts MTEB STS14 default test 6031580fec1f6af667f0bd2da0a551cf4f0b2375 type value cos_sim_pearson 79.54733787179849 type value cos_sim_spearman 77.72202105610411 type value euclidean_pearson 78.9043595478849 type value euclidean_spearman 77.93422804309435 type value manhattan_pearson 78.58115121621368 type value manhattan_spearman 77.62508135122033 task dataset metrics type STS type name config split revision mteb/sts15-sts MTEB STS15 default test ae752c7c21bf194d8b67fd573edf7ae58183cbe3 type value cos_sim_pearson 88.59880017237558 type value cos_sim_spearman 89.31088630824758 type value euclidean_pearson 88.47069261564656 type value euclidean_spearman 89.33581971465233 type value manhattan_pearson 88.40774264100956 type value manhattan_spearman 89.28657485627835 task dataset metrics type STS type name config split revision mteb/sts16-sts MTEB STS16 default test 4d8694f8f0e0100860b497b999b3dbed754a0513 type value cos_sim_pearson 84.08055117917084 type value cos_sim_spearman 85.78491813080304 type value euclidean_pearson 84.99329155500392 type value euclidean_spearman 85.76728064677287 type value manhattan_pearson 84.87947428989587 type value manhattan_spearman 85.62429454917464 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (ko-ko) ko-ko test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 82.14190939287384 type value cos_sim_spearman 82.27331573306041 type value euclidean_pearson 81.891896953716 type value euclidean_spearman 82.37695542955998 type value manhattan_pearson 81.73123869460504 type value manhattan_spearman 82.19989168441421 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (ar-ar) ar-ar test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 76.84695301843362 type value cos_sim_spearman 77.87790986014461 type value euclidean_pearson 76.91981583106315 type value euclidean_spearman 77.88154772749589 type value manhattan_pearson 76.94953277451093 type value manhattan_spearman 77.80499230728604 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-ar) en-ar test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 75.44657840482016 type value cos_sim_spearman 75.05531095119674 type value euclidean_pearson 75.88161755829299 type value euclidean_spearman 74.73176238219332 type value manhattan_pearson 75.63984765635362 type value manhattan_spearman 74.86476440770737 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-de) en-de test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 85.64700140524133 type value cos_sim_spearman 86.16014210425672 type value euclidean_pearson 86.49086860843221 type value euclidean_spearman 86.09729326815614 type value manhattan_pearson 86.43406265125513 type value manhattan_spearman 86.17740150939994 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-en) en-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 87.91170098764921 type value cos_sim_spearman 88.12437004058931 type value euclidean_pearson 88.81828254494437 type value euclidean_spearman 88.14831794572122 type value manhattan_pearson 88.93442183448961 type value manhattan_spearman 88.15254630778304 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-tr) en-tr test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 72.91390577997292 type value cos_sim_spearman 71.22979457536074 type value euclidean_pearson 74.40314008106749 type value euclidean_spearman 72.54972136083246 type value manhattan_pearson 73.85687539530218 type value manhattan_spearman 72.09500771742637 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (es-en) es-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 80.9301067983089 type value cos_sim_spearman 80.74989828346473 type value euclidean_pearson 81.36781301814257 type value euclidean_spearman 80.9448819964426 type value manhattan_pearson 81.0351322685609 type value manhattan_spearman 80.70192121844177 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (es-es) es-es test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 87.13820465980005 type value cos_sim_spearman 86.73532498758757 type value euclidean_pearson 87.21329451846637 type value euclidean_spearman 86.57863198601002 type value manhattan_pearson 87.06973713818554 type value manhattan_spearman 86.47534918791499 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (fr-en) fr-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 85.48720108904415 type value cos_sim_spearman 85.62221757068387 type value euclidean_pearson 86.1010129512749 type value euclidean_spearman 85.86580966509942 type value manhattan_pearson 86.26800938808971 type value manhattan_spearman 85.88902721678429 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (it-en) it-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 83.98021347333516 type value cos_sim_spearman 84.53806553803501 type value euclidean_pearson 84.61483347248364 type value euclidean_spearman 85.14191408011702 type value manhattan_pearson 84.75297588825967 type value manhattan_spearman 85.33176753669242 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (nl-en) nl-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 84.51856644893233 type value cos_sim_spearman 85.27510748506413 type value euclidean_pearson 85.09886861540977 type value euclidean_spearman 85.62579245860887 type value manhattan_pearson 84.93017860464607 type value manhattan_spearman 85.5063988898453 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (en) en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 62.581573200584195 type value cos_sim_spearman 63.05503590247928 type value euclidean_pearson 63.652564812602094 type value euclidean_spearman 62.64811520876156 type value manhattan_pearson 63.506842893061076 type value manhattan_spearman 62.51289573046917 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de) de test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 48.2248801729127 type value cos_sim_spearman 56.5936604678561 type value euclidean_pearson 43.98149464089 type value euclidean_spearman 56.108561882423615 type value manhattan_pearson 43.86880305903564 type value manhattan_spearman 56.04671150510166 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es) es test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 55.17564527009831 type value cos_sim_spearman 64.57978560979488 type value euclidean_pearson 58.8818330154583 type value euclidean_spearman 64.99214839071281 type value manhattan_pearson 58.72671436121381 type value manhattan_spearman 65.10713416616109 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (pl) pl test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 26.772131864023297 type value cos_sim_spearman 34.68200792408681 type value euclidean_pearson 16.68082419005441 type value euclidean_spearman 34.83099932652166 type value manhattan_pearson 16.52605949659529 type value manhattan_spearman 34.82075801399475 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (tr) tr test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 54.42415189043831 type value cos_sim_spearman 63.54594264576758 type value euclidean_pearson 57.36577498297745 type value euclidean_spearman 63.111466379158074 type value manhattan_pearson 57.584543715873885 type value manhattan_spearman 63.22361054139183 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (ar) ar test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 47.55216762405518 type value cos_sim_spearman 56.98670142896412 type value euclidean_pearson 50.15318757562699 type value euclidean_spearman 56.524941926541906 type value manhattan_pearson 49.955618528674904 type value manhattan_spearman 56.37102209240117 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (ru) ru test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 49.20540980338571 type value cos_sim_spearman 59.9009453504406 type value euclidean_pearson 49.557749853620535 type value euclidean_spearman 59.76631621172456 type value manhattan_pearson 49.62340591181147 type value manhattan_spearman 59.94224880322436 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh) zh test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 51.508169956576985 type value cos_sim_spearman 66.82461565306046 type value euclidean_pearson 56.2274426480083 type value euclidean_spearman 66.6775323848333 type value manhattan_pearson 55.98277796300661 type value manhattan_spearman 66.63669848497175 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (fr) fr test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 72.86478788045507 type value cos_sim_spearman 76.7946552053193 type value euclidean_pearson 75.01598530490269 type value euclidean_spearman 76.83618917858281 type value manhattan_pearson 74.68337628304332 type value manhattan_spearman 76.57480204017773 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-en) de-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 55.922619099401984 type value cos_sim_spearman 56.599362477240774 type value euclidean_pearson 56.68307052369783 type value euclidean_spearman 54.28760436777401 type value manhattan_pearson 56.67763566500681 type value manhattan_spearman 53.94619541711359 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es-en) es-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 66.74357206710913 type value cos_sim_spearman 72.5208244925311 type value euclidean_pearson 67.49254562186032 type value euclidean_spearman 72.02469076238683 type value manhattan_pearson 67.45251772238085 type value manhattan_spearman 72.05538819984538 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (it) it test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 71.25734330033191 type value cos_sim_spearman 76.98349083946823 type value euclidean_pearson 73.71642838667736 type value euclidean_spearman 77.01715504651384 type value manhattan_pearson 73.61712711868105 type value manhattan_spearman 77.01392571153896 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (pl-en) pl-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 63.18215462781212 type value cos_sim_spearman 65.54373266117607 type value euclidean_pearson 64.54126095439005 type value euclidean_spearman 65.30410369102711 type value manhattan_pearson 63.50332221148234 type value manhattan_spearman 64.3455878104313 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh-en) zh-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 62.30509221440029 type value cos_sim_spearman 65.99582704642478 type value euclidean_pearson 63.43818859884195 type value euclidean_spearman 66.83172582815764 type value manhattan_pearson 63.055779168508764 type value manhattan_spearman 65.49585020501449 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es-it) es-it test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 59.587830825340404 type value cos_sim_spearman 68.93467614588089 type value euclidean_pearson 62.3073527367404 type value euclidean_spearman 69.69758171553175 type value manhattan_pearson 61.9074580815789 type value manhattan_spearman 69.57696375597865 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-fr) de-fr test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 57.143220125577066 type value cos_sim_spearman 67.78857859159226 type value euclidean_pearson 55.58225107923733 type value euclidean_spearman 67.80662907184563 type value manhattan_pearson 56.24953502726514 type value manhattan_spearman 67.98262125431616 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-pl) de-pl test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 21.826928900322066 type value cos_sim_spearman 49.578506634400405 type value euclidean_pearson 27.939890138843214 type value euclidean_spearman 52.71950519136242 type value manhattan_pearson 26.39878683847546 type value manhattan_spearman 47.54609580342499 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (fr-pl) fr-pl test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 57.27603854632001 type value cos_sim_spearman 50.709255283710995 type value euclidean_pearson 59.5419024445929 type value euclidean_spearman 50.709255283710995 type value manhattan_pearson 59.03256832438492 type value manhattan_spearman 61.97797868009122 task dataset metrics type STS type name config split revision mteb/stsbenchmark-sts MTEB STSBenchmark default test b0fddb56ed78048fa8b90373c8a3cfc37b684831 type value cos_sim_pearson 85.00757054859712 type value cos_sim_spearman 87.29283629622222 type value euclidean_pearson 86.54824171775536 type value euclidean_spearman 87.24364730491402 type value manhattan_pearson 86.5062156915074 type value manhattan_spearman 87.15052170378574 task dataset metrics type Reranking type name config split revision mteb/scidocs-reranking MTEB SciDocsRR default test d3c5e1fc0b855ab6097bf1cda04dd73947d7caab type value map 82.03549357197389 type value mrr 95.05437645143527 task dataset metrics type Retrieval type name config split revision scifact MTEB SciFact default test None type value map_at_1 57.260999999999996 type value map_at_10 66.259 type value map_at_100 66.884 type value map_at_1000 66.912 type value map_at_3 63.685 type value map_at_5 65.35499999999999 type value mrr_at_1 60.333000000000006 type value mrr_at_10 67.5 type value mrr_at_100 68.013 type value mrr_at_1000 68.038 type value mrr_at_3 65.61099999999999 type value mrr_at_5 66.861 type value ndcg_at_1 60.333000000000006 type value ndcg_at_10 70.41 type value ndcg_at_100 73.10600000000001 type value ndcg_at_1000 73.846 type value ndcg_at_3 66.133 type value ndcg_at_5 68.499 type value precision_at_1 60.333000000000006 type value precision_at_10 9.232999999999999 type value precision_at_100 1.0630000000000002 type value precision_at_1000 0.11299999999999999 type value precision_at_3 25.667 type value precision_at_5 17.067 type value recall_at_1 57.260999999999996 type value recall_at_10 81.94399999999999 type value recall_at_100 93.867 type value recall_at_1000 99.667 type value recall_at_3 70.339 type value recall_at_5 76.25 task dataset metrics type PairClassification type name config split revision mteb/sprintduplicatequestions-pairclassification MTEB SprintDuplicateQuestions default test d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 type value cos_sim_accuracy 99.74356435643564 type value cos_sim_ap 93.13411948212683 type value cos_sim_f1 86.80521991300147 type value cos_sim_precision 84.00374181478017 type value cos_sim_recall 89.8 type value dot_accuracy 99.67920792079208 type value dot_ap 89.27277565444479 type value dot_f1 83.9276990718124 type value dot_precision 82.04393505253104 type value dot_recall 85.9 type value euclidean_accuracy 99.74257425742574 type value euclidean_ap 93.17993008259062 type value euclidean_f1 86.69396110542476 type value euclidean_precision 88.78406708595388 type value euclidean_recall 84.7 type value manhattan_accuracy 99.74257425742574 type value manhattan_ap 93.14413755550099 type value manhattan_f1 86.82483594144371 type value manhattan_precision 87.66564729867483 type value manhattan_recall 86 type value max_accuracy 99.74356435643564 type value max_ap 93.17993008259062 type value max_f1 86.82483594144371 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering MTEB StackExchangeClustering default test 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 type value v_measure 57.525863806168566 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering-p2p MTEB StackExchangeClusteringP2P default test 815ca46b2622cec33ccafc3735d572c266efdb44 type value v_measure 32.68850574423839 task dataset metrics type Reranking type name config split revision mteb/stackoverflowdupquestions-reranking MTEB StackOverflowDupQuestions default test e185fbe320c72810689fc5848eb6114e1ef5ec69 type value map 49.71580650644033 type value mrr 50.50971903913081 task dataset metrics type Summarization type name config split revision mteb/summeval MTEB SummEval default test cda12ad7615edc362dbf25a00fdd61d3b1eaf93c type value cos_sim_pearson 29.152190498799484 type value cos_sim_spearman 29.686180371952727 type value dot_pearson 27.248664793816342 type value dot_spearman 28.37748983721745 task dataset metrics type Retrieval type name config split revision trec-covid MTEB TRECCOVID default test None type value map_at_1 0.20400000000000001 type value map_at_10 1.6209999999999998 type value map_at_100 9.690999999999999 type value map_at_1000 23.733 type value map_at_3 0.575 type value map_at_5 0.885 type value mrr_at_1 78 type value mrr_at_10 86.56700000000001 type value mrr_at_100 86.56700000000001 type value mrr_at_1000 86.56700000000001 type value mrr_at_3 85.667 type value mrr_at_5 86.56700000000001 type value ndcg_at_1 76 type value ndcg_at_10 71.326 type value ndcg_at_100 54.208999999999996 type value ndcg_at_1000 49.252 type value ndcg_at_3 74.235 type value ndcg_at_5 73.833 type value precision_at_1 78 type value precision_at_10 74.8 type value precision_at_100 55.50000000000001 type value precision_at_1000 21.836 type value precision_at_3 78 type value precision_at_5 78 type value recall_at_1 0.20400000000000001 type value recall_at_10 1.894 type value recall_at_100 13.245999999999999 type value recall_at_1000 46.373 type value recall_at_3 0.613 type value recall_at_5 0.991 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (sqi-eng) sqi-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.89999999999999 type value f1 94.69999999999999 type value precision 94.11666666666667 type value recall 95.89999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fry-eng) fry-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 68.20809248554913 type value f1 63.431048720066066 type value precision 61.69143958161298 type value recall 68.20809248554913 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kur-eng) kur-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 71.21951219512195 type value f1 66.82926829268293 type value precision 65.1260162601626 type value recall 71.21951219512195 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tur-eng) tur-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.2 type value f1 96.26666666666667 type value precision 95.8 type value recall 97.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (deu-eng) deu-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 99.3 type value f1 99.06666666666666 type value precision 98.95 type value recall 99.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nld-eng) nld-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.39999999999999 type value f1 96.63333333333333 type value precision 96.26666666666668 type value recall 97.39999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ron-eng) ron-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96 type value f1 94.86666666666666 type value precision 94.31666666666668 type value recall 96 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ang-eng) ang-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 47.01492537313433 type value f1 40.178867566927266 type value precision 38.179295828549556 type value recall 47.01492537313433 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ido-eng) ido-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 86.5 type value f1 83.62537480063796 type value precision 82.44555555555554 type value recall 86.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (jav-eng) jav-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 80.48780487804879 type value f1 75.45644599303138 type value precision 73.37398373983739 type value recall 80.48780487804879 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (isl-eng) isl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.7 type value f1 91.95666666666666 type value precision 91.125 type value recall 93.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (slv-eng) slv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 91.73754556500607 type value f1 89.65168084244632 type value precision 88.73025516403402 type value recall 91.73754556500607 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cym-eng) cym-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 81.04347826086956 type value f1 76.2128364389234 type value precision 74.2 type value recall 81.04347826086956 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kaz-eng) kaz-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 83.65217391304348 type value f1 79.4376811594203 type value precision 77.65797101449274 type value recall 83.65217391304348 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (est-eng) est-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 87.5 type value f1 85.02690476190476 type value precision 83.96261904761904 type value recall 87.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (heb-eng) heb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 89.3 type value f1 86.52333333333333 type value precision 85.22833333333332 type value recall 89.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gla-eng) gla-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 65.01809408926418 type value f1 59.00594446432805 type value precision 56.827215807915444 type value recall 65.01809408926418 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mar-eng) mar-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 91.2 type value f1 88.58 type value precision 87.33333333333334 type value recall 91.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lat-eng) lat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 59.199999999999996 type value f1 53.299166276284915 type value precision 51.3383908045977 type value recall 59.199999999999996 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bel-eng) bel-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.2 type value f1 91.2 type value precision 90.25 type value recall 93.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pms-eng) pms-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 64.76190476190476 type value f1 59.867110667110666 type value precision 58.07390192653351 type value recall 64.76190476190476 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gle-eng) gle-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 76.2 type value f1 71.48147546897547 type value precision 69.65409090909091 type value recall 76.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pes-eng) pes-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.8 type value f1 92.14 type value precision 91.35833333333333 type value recall 93.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nob-eng) nob-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.89999999999999 type value f1 97.2 type value precision 96.85000000000001 type value recall 97.89999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bul-eng) bul-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.6 type value f1 92.93333333333334 type value precision 92.13333333333333 type value recall 94.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cbk-eng) cbk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 74.1 type value f1 69.14817460317461 type value precision 67.2515873015873 type value recall 74.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hun-eng) hun-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.19999999999999 type value f1 94.01333333333335 type value precision 93.46666666666667 type value recall 95.19999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (uig-eng) uig-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 76.9 type value f1 72.07523809523809 type value precision 70.19777777777779 type value recall 76.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (rus-eng) rus-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.1 type value f1 92.31666666666666 type value precision 91.43333333333332 type value recall 94.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (spa-eng) spa-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.8 type value f1 97.1 type value precision 96.76666666666668 type value recall 97.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hye-eng) hye-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.85714285714286 type value f1 90.92093441150045 type value precision 90.00449236298293 type value recall 92.85714285714286 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tel-eng) tel-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.16239316239316 type value f1 91.33903133903132 type value precision 90.56267806267806 type value recall 93.16239316239316 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (afr-eng) afr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.4 type value f1 90.25666666666666 type value precision 89.25833333333334 type value recall 92.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mon-eng) mon-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 90.22727272727272 type value f1 87.53030303030303 type value precision 86.37121212121211 type value recall 90.22727272727272 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (arz-eng) arz-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 79.03563941299791 type value f1 74.7349505840072 type value precision 72.9035639412998 type value recall 79.03563941299791 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hrv-eng) hrv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97 type value f1 96.15 type value precision 95.76666666666668 type value recall 97 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nov-eng) nov-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 76.26459143968872 type value f1 71.55642023346303 type value precision 69.7544932369835 type value recall 76.26459143968872 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gsw-eng) gsw-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 58.119658119658126 type value f1 51.65242165242165 type value precision 49.41768108434775 type value recall 58.119658119658126 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nds-eng) nds-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 74.3 type value f1 69.52055555555555 type value precision 67.7574938949939 type value recall 74.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ukr-eng) ukr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.8 type value f1 93.31666666666666 type value precision 92.60000000000001 type value recall 94.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (uzb-eng) uzb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 76.63551401869158 type value f1 72.35202492211837 type value precision 70.60358255451713 type value recall 76.63551401869158 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lit-eng) lit-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 90.4 type value f1 88.4811111111111 type value precision 87.7452380952381 type value recall 90.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ina-eng) ina-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95 type value f1 93.60666666666667 type value precision 92.975 type value recall 95 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lfn-eng) lfn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 67.2 type value f1 63.01595782872099 type value precision 61.596587301587306 type value recall 67.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (zsm-eng) zsm-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.7 type value f1 94.52999999999999 type value precision 94 type value recall 95.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ita-eng) ita-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.6 type value f1 93.28999999999999 type value precision 92.675 type value recall 94.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cmn-eng) cmn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.39999999999999 type value f1 95.28333333333333 type value precision 94.75 type value recall 96.39999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lvs-eng) lvs-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 91.9 type value f1 89.83 type value precision 88.92 type value recall 91.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (glg-eng) glg-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.69999999999999 type value f1 93.34222222222223 type value precision 92.75416666666668 type value recall 94.69999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ceb-eng) ceb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 60.333333333333336 type value f1 55.31203703703703 type value precision 53.39971108326371 type value recall 60.333333333333336 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bre-eng) bre-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 12.9 type value f1 11.099861903031458 type value precision 10.589187932631877 type value recall 12.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ben-eng) ben-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 86.7 type value f1 83.0152380952381 type value precision 81.37833333333333 type value recall 86.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swg-eng) swg-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 63.39285714285714 type value f1 56.832482993197274 type value precision 54.56845238095237 type value recall 63.39285714285714 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (arq-eng) arq-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 48.73765093304062 type value f1 41.555736920720456 type value precision 39.06874531737319 type value recall 48.73765093304062 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kab-eng) kab-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 41.099999999999994 type value f1 36.540165945165946 type value precision 35.05175685425686 type value recall 41.099999999999994 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fra-eng) fra-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.89999999999999 type value f1 93.42333333333333 type value precision 92.75833333333333 type value recall 94.89999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (por-eng) por-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.89999999999999 type value f1 93.63333333333334 type value precision 93.01666666666665 type value recall 94.89999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tat-eng) tat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 77.9 type value f1 73.64833333333334 type value precision 71.90282106782105 type value recall 77.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (oci-eng) oci-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 59.4 type value f1 54.90521367521367 type value precision 53.432840025471606 type value recall 59.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pol-eng) pol-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.39999999999999 type value f1 96.6 type value precision 96.2 type value recall 97.39999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (war-eng) war-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 67.2 type value f1 62.25926129426129 type value precision 60.408376623376626 type value recall 67.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (aze-eng) aze-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 90.2 type value f1 87.60666666666667 type value precision 86.45277777777778 type value recall 90.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (vie-eng) vie-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.7 type value f1 97 type value precision 96.65 type value recall 97.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nno-eng) nno-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.2 type value f1 91.39746031746031 type value precision 90.6125 type value recall 93.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cha-eng) cha-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 32.11678832116788 type value f1 27.210415386260234 type value precision 26.20408990846947 type value recall 32.11678832116788 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mhr-eng) mhr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 8.5 type value f1 6.787319277832475 type value precision 6.3452094433344435 type value recall 8.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dan-eng) dan-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.1 type value f1 95.08 type value precision 94.61666666666667 type value recall 96.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ell-eng) ell-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.3 type value f1 93.88333333333333 type value precision 93.18333333333332 type value recall 95.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (amh-eng) amh-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 85.11904761904762 type value f1 80.69444444444444 type value precision 78.72023809523809 type value recall 85.11904761904762 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pam-eng) pam-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 11.1 type value f1 9.276381801735853 type value precision 8.798174603174601 type value recall 11.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hsb-eng) hsb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 63.56107660455487 type value f1 58.70433569191332 type value precision 56.896926581464015 type value recall 63.56107660455487 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (srp-eng) srp-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.69999999999999 type value f1 93.10000000000001 type value precision 92.35 type value recall 94.69999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (epo-eng) epo-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.8 type value f1 96.01222222222222 type value precision 95.67083333333332 type value recall 96.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kzj-eng) kzj-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 9.2 type value f1 7.911555250305249 type value precision 7.631246556216846 type value recall 9.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (awa-eng) awa-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 77.48917748917748 type value f1 72.27375798804371 type value precision 70.14430014430013 type value recall 77.48917748917748 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fao-eng) fao-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 77.09923664122137 type value f1 72.61541257724463 type value precision 70.8998380754106 type value recall 77.09923664122137 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mal-eng) mal-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 98.2532751091703 type value f1 97.69529354682193 type value precision 97.42843279961184 type value recall 98.2532751091703 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ile-eng) ile-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 82.8 type value f1 79.14672619047619 type value precision 77.59489247311828 type value recall 82.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bos-eng) bos-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.35028248587571 type value f1 92.86252354048965 type value precision 92.2080979284369 type value recall 94.35028248587571 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cor-eng) cor-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 8.5 type value f1 6.282429263935621 type value precision 5.783274240739785 type value recall 8.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cat-eng) cat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.7 type value f1 91.025 type value precision 90.30428571428571 type value recall 92.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (eus-eng) eus-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 81 type value f1 77.8232380952381 type value precision 76.60194444444444 type value recall 81 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (yue-eng) yue-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 91 type value f1 88.70857142857142 type value precision 87.7 type value recall 91 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swe-eng) swe-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.39999999999999 type value f1 95.3 type value precision 94.76666666666667 type value recall 96.39999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dtp-eng) dtp-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 8.1 type value f1 7.001008218834307 type value precision 6.708329562594269 type value recall 8.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kat-eng) kat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 87.1313672922252 type value f1 84.09070598748882 type value precision 82.79171454104429 type value recall 87.1313672922252 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (jpn-eng) jpn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.39999999999999 type value f1 95.28333333333333 type value precision 94.73333333333332 type value recall 96.39999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (csb-eng) csb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 42.29249011857708 type value f1 36.981018542283365 type value precision 35.415877813576024 type value recall 42.29249011857708 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (xho-eng) xho-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 83.80281690140845 type value f1 80.86854460093896 type value precision 79.60093896713614 type value recall 83.80281690140845 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (orv-eng) orv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 45.26946107784431 type value f1 39.80235464678088 type value precision 38.14342660001342 type value recall 45.26946107784431 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ind-eng) ind-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.3 type value f1 92.9 type value precision 92.26666666666668 type value recall 94.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tuk-eng) tuk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 37.93103448275862 type value f1 33.15192743764172 type value precision 31.57456528146183 type value recall 37.93103448275862 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (max-eng) max-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 69.01408450704226 type value f1 63.41549295774648 type value precision 61.342778895595806 type value recall 69.01408450704226 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swh-eng) swh-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 76.66666666666667 type value f1 71.60705960705961 type value precision 69.60683760683762 type value recall 76.66666666666667 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hin-eng) hin-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.8 type value f1 94.48333333333333 type value precision 93.83333333333333 type value recall 95.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dsb-eng) dsb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 52.81837160751566 type value f1 48.435977731384824 type value precision 47.11291973845539 type value recall 52.81837160751566 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ber-eng) ber-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 44.9 type value f1 38.88962621607783 type value precision 36.95936507936508 type value recall 44.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tam-eng) tam-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 90.55374592833876 type value f1 88.22553125484721 type value precision 87.26927252985884 type value recall 90.55374592833876 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (slk-eng) slk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.6 type value f1 93.13333333333333 type value precision 92.45333333333333 type value recall 94.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tgl-eng) tgl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.7 type value f1 91.99666666666667 type value precision 91.26666666666668 type value recall 93.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ast-eng) ast-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 85.03937007874016 type value f1 81.75853018372703 type value precision 80.34120734908137 type value recall 85.03937007874016 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mkd-eng) mkd-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 88.3 type value f1 85.5 type value precision 84.25833333333334 type value recall 88.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (khm-eng) khm-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 65.51246537396122 type value f1 60.02297410192148 type value precision 58.133467727289236 type value recall 65.51246537396122 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ces-eng) ces-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96 type value f1 94.89 type value precision 94.39166666666667 type value recall 96 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tzl-eng) tzl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 57.692307692307686 type value f1 53.162393162393165 type value precision 51.70673076923077 type value recall 57.692307692307686 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (urd-eng) urd-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 91.60000000000001 type value f1 89.21190476190475 type value precision 88.08666666666667 type value recall 91.60000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ara-eng) ara-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 88 type value f1 85.47 type value precision 84.43266233766234 type value recall 88 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kor-eng) kor-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.7 type value f1 90.64999999999999 type value precision 89.68333333333332 type value recall 92.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (yid-eng) yid-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 80.30660377358491 type value f1 76.33044137466307 type value precision 74.78970125786164 type value recall 80.30660377358491 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fin-eng) fin-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.39999999999999 type value f1 95.44 type value precision 94.99166666666666 type value recall 96.39999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tha-eng) tha-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.53284671532847 type value f1 95.37712895377129 type value precision 94.7992700729927 type value recall 96.53284671532847 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (wuu-eng) wuu-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 89 type value f1 86.23190476190476 type value precision 85.035 type value recall 89 task dataset metrics type Retrieval type name config split revision webis-touche2020 MTEB Touche2020 default test None type value map_at_1 2.585 type value map_at_10 9.012 type value map_at_100 14.027000000000001 type value map_at_1000 15.565000000000001 type value map_at_3 5.032 type value map_at_5 6.657 type value mrr_at_1 28.571 type value mrr_at_10 45.377 type value mrr_at_100 46.119 type value mrr_at_1000 46.127 type value mrr_at_3 41.156 type value mrr_at_5 42.585 type value ndcg_at_1 27.551 type value ndcg_at_10 23.395 type value ndcg_at_100 33.342 type value ndcg_at_1000 45.523 type value ndcg_at_3 25.158 type value ndcg_at_5 23.427 type value precision_at_1 28.571 type value precision_at_10 21.429000000000002 type value precision_at_100 6.714 type value precision_at_1000 1.473 type value precision_at_3 27.211000000000002 type value precision_at_5 24.490000000000002 type value recall_at_1 2.585 type value recall_at_10 15.418999999999999 type value recall_at_100 42.485 type value recall_at_1000 79.536 type value recall_at_3 6.239999999999999 type value recall_at_5 8.996 task dataset metrics type Classification type name config split revision mteb/toxic_conversations_50k MTEB ToxicConversationsClassification default test d7c0de2777da35d6aae2200a62c6e0e5af397c4c type value accuracy 71.3234 type value ap 14.361688653847423 type value f1 54.819068624319044 task dataset metrics type Classification type name config split revision mteb/tweet_sentiment_extraction MTEB TweetSentimentExtractionClassification default test d604517c81ca91fe16a244d1248fc021f9ecee7a type value accuracy 61.97792869269949 type value f1 62.28965628513728 task dataset metrics type Clustering type name config split revision mteb/twentynewsgroups-clustering MTEB TwentyNewsgroupsClustering default test 6125ec4e24fa026cec8a478383ee943acfbd5449 type value v_measure 38.90540145385218 task dataset metrics type PairClassification type name config split revision mteb/twittersemeval2015-pairclassification MTEB TwitterSemEval2015 default test 70970daeab8776df92f5ea462b6173c0b46fd2d1 type value cos_sim_accuracy 86.53513739047506 type value cos_sim_ap 75.27741586677557 type value cos_sim_f1 69.18792902473774 type value cos_sim_precision 67.94708725515136 type value cos_sim_recall 70.47493403693932 type value dot_accuracy 84.7052512368123 type value dot_ap 69.36075482849378 type value dot_f1 64.44688376631296 type value dot_precision 59.92288500793831 type value dot_recall 69.70976253298153 type value euclidean_accuracy 86.60666388508076 type value euclidean_ap 75.47512772621097 type value euclidean_f1 69.413872536473 type value euclidean_precision 67.39562624254472 type value euclidean_recall 71.55672823218997 type value manhattan_accuracy 86.52917684925792 type value manhattan_ap 75.34000110496703 type value manhattan_f1 69.28489190226429 type value manhattan_precision 67.24608889992551 type value manhattan_recall 71.45118733509234 type value max_accuracy 86.60666388508076 type value max_ap 75.47512772621097 type value max_f1 69.413872536473 task dataset metrics type PairClassification type name config split revision mteb/twitterurlcorpus-pairclassification MTEB TwitterURLCorpus default test 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf type value cos_sim_accuracy 89.01695967710637 type value cos_sim_ap 85.8298270742901 type value cos_sim_f1 78.46988128389272 type value cos_sim_precision 74.86017897091722 type value cos_sim_recall 82.44533415460425 type value dot_accuracy 88.19420188613343 type value dot_ap 83.82679165901324 type value dot_f1 76.55833777304208 type value dot_precision 75.6884875846501 type value dot_recall 77.44841392054204 type value euclidean_accuracy 89.03054294252338 type value euclidean_ap 85.89089555185325 type value euclidean_f1 78.62997658079624 type value euclidean_precision 74.92329149232914 type value euclidean_recall 82.72251308900523 type value manhattan_accuracy 89.0266620095471 type value manhattan_ap 85.86458997929147 type value manhattan_f1 78.50685331000291 type value manhattan_precision 74.5499861534201 type value manhattan_recall 82.90729904527257 type value max_accuracy 89.03054294252338 type value max_ap 85.89089555185325 type value max_f1 78.62997658079624	multilingual af am ar as az be bg bn br bs ca cs cy da de el en eo es et eu fa fi fr fy ga gd gl gu ha he hi hr hu hy id is it ja jv ka kk km kn ko ku ky la lo lt lv mg mk ml mn mr ms my ne nl no om or pa pl ps pt ro ru sa sd si sk sl so sq sr su sv sw ta te th tl tr ug uk ur uz vi xh yi zh	mit

Multilingual-E5-large

Multilingual E5 Text Embeddings: A Technical Report. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024

This model has 24 layers and the embedding size is 1024.

Usage

Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset.

import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def average_pool(last_hidden_states: Tensor,
                 attention_mask: Tensor) -> Tensor:
    last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
    return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]


# Each input text should start with "query: " or "passage: ", even for non-English texts.
# For tasks other than retrieval, you can simply use the "query: " prefix.
input_texts = ['query: how much protein should a female eat',
               'query: 南瓜的家常做法',
               "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
               "passage: 1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"]

tokenizer = AutoTokenizer.from_pretrained('intfloat/multilingual-e5-large')
model = AutoModel.from_pretrained('intfloat/multilingual-e5-large')

# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')

outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())

Supported Languages

This model is initialized from xlm-roberta-large and continually trained on a mixture of multilingual datasets. It supports 100 languages from xlm-roberta, but low-resource languages may see performance degradation.

Training Details

Initialization: xlm-roberta-large

First stage: contrastive pre-training with weak supervision

Dataset	Weak supervision	# of text pairs
Filtered mC4	(title, page content)	1B
CC News	(title, news content)	400M
NLLB	translation pairs	2.4B
Wikipedia	(hierarchical section title, passage)	150M
Filtered Reddit	(comment, response)	800M
S2ORC	(title, abstract) and citation pairs	100M
Stackexchange	(question, answer)	50M
xP3	(input prompt, response)	80M
Miscellaneous unsupervised SBERT data	-	10M

Second stage: supervised fine-tuning

Dataset	Language	# of text pairs
MS MARCO	English	500k
NQ	English	70k
Trivia QA	English	60k
NLI from SimCSE	English	<300k
ELI5	English	500k
DuReader Retrieval	Chinese	86k
KILT Fever	English	70k
KILT HotpotQA	English	70k
SQuAD	English	87k
Quora	English	150k
Mr. TyDi	11 languages	50k
MIRACL	16 languages	40k

For all labeled datasets, we only use its training set for fine-tuning.

For other training details, please refer to our paper at https://arxiv.org/pdf/2402.05672.

Benchmark Results on Mr. TyDi

Model	Avg MRR@10		ar	bn	en	fi	id	ja	ko	ru	sw	te	th
BM25	33.3		36.7	41.3	15.1	28.8	38.2	21.7	28.1	32.9	39.6	42.4	41.7
mDPR	16.7		26.0	25.8	16.2	11.3	14.6	18.1	21.9	18.5	7.3	10.6	13.5
BM25 + mDPR	41.7		49.1	53.5	28.4	36.5	45.5	35.5	36.2	42.7	40.5	42.0	49.2
multilingual-e5-small	64.4		71.5	66.3	54.5	57.7	63.2	55.4	54.3	60.8	65.4	89.1	70.1
multilingual-e5-base	65.9		72.3	65.0	58.5	60.8	64.9	56.6	55.8	62.7	69.0	86.6	72.7
multilingual-e5-large	70.5		77.5	73.2	60.8	66.8	68.5	62.5	61.6	65.8	72.7	90.2	76.2

MTEB Benchmark Evaluation

Check out unilm/e5 to reproduce evaluation results on the BEIR and MTEB benchmark.

Support for Sentence Transformers

Below is an example for usage with sentence_transformers.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('intfloat/multilingual-e5-large')
input_texts = [
    'query: how much protein should a female eat',
    'query: 南瓜的家常做法',
    "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 i     s 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or traini     ng for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "passage: 1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮     ,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,     放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油     锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀      6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"
]
embeddings = model.encode(input_texts, normalize_embeddings=True)

Package requirements

pip install sentence_transformers~=2.2.2

Contributors: michaelfeil

FAQ

1. Do I need to add the prefix "query: " and "passage: " to input texts?

Yes, this is how the model is trained, otherwise you will see a performance degradation.

Here are some rules of thumb:

• Use "query: " and "passage: " correspondingly for asymmetric tasks such as passage retrieval in open QA, ad-hoc information retrieval.

• Use "query: " prefix for symmetric tasks such as semantic similarity, bitext mining, paraphrase retrieval.

• Use "query: " prefix if you want to use embeddings as features, such as linear probing classification, clustering.

2. Why are my reproduced results slightly different from reported in the model card?

Different versions of transformers and pytorch could cause negligible but non-zero performance differences.

3. Why does the cosine similarity scores distribute around 0.7 to 1.0?

This is a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss.

For text embedding tasks like text retrieval or semantic similarity, what matters is the relative order of the scores instead of the absolute values, so this should not be an issue.

Citation

If you find our paper or models helpful, please consider cite as follows:

@article{wang2024multilingual,
  title={Multilingual E5 Text Embeddings: A Technical Report},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2402.05672},
  year={2024}
}

Limitations

Long texts will be truncated to at most 512 tokens.