multilingual-e5-base/README.md

tags, model-index, language, license

tags	model-index	language	license
mteb Sentence Transformers sentence-similarity sentence-transformers	name results multilingual-e5-base task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en) en test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 78.97014925373135 type value ap 43.69351129103008 type value f1 73.38075030070492 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (de) de test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 71.7237687366167 type value ap 82.22089859962671 type value f1 69.95532758884401 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en-ext) en-ext test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 79.65517241379312 type value ap 28.507918657094738 type value f1 66.84516013726119 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (ja) ja test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 73.32976445396146 type value ap 20.720481637566014 type value f1 59.78002763416003 task dataset metrics type Classification type name config split revision mteb/amazon_polarity MTEB AmazonPolarityClassification default test e2d317d38cd51312af73b3d32a06d1a08b442046 type value accuracy 90.63775 type value ap 87.22277903861716 type value f1 90.60378636386807 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (en) en test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 44.546 type value f1 44.05666638370923 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (de) de test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 41.828 type value f1 41.2710255644252 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (es) es test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 40.534 type value f1 39.820743174270326 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (fr) fr test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 39.684 type value f1 39.11052682815307 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (ja) ja test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 37.436 type value f1 37.07082931930871 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (zh) zh test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 37.226000000000006 type value f1 36.65372077739185 task dataset metrics type Retrieval type name config split revision arguana MTEB ArguAna default test None type value map_at_1 22.831000000000003 type value map_at_10 36.42 type value map_at_100 37.699 type value map_at_1000 37.724000000000004 type value map_at_3 32.207 type value map_at_5 34.312 type value mrr_at_1 23.257 type value mrr_at_10 36.574 type value mrr_at_100 37.854 type value mrr_at_1000 37.878 type value mrr_at_3 32.385000000000005 type value mrr_at_5 34.48 type value ndcg_at_1 22.831000000000003 type value ndcg_at_10 44.230000000000004 type value ndcg_at_100 49.974000000000004 type value ndcg_at_1000 50.522999999999996 type value ndcg_at_3 35.363 type value ndcg_at_5 39.164 type value precision_at_1 22.831000000000003 type value precision_at_10 6.935 type value precision_at_100 0.9520000000000001 type value precision_at_1000 0.099 type value precision_at_3 14.841 type value precision_at_5 10.754 type value recall_at_1 22.831000000000003 type value recall_at_10 69.346 type value recall_at_100 95.235 type value recall_at_1000 99.36 type value recall_at_3 44.523 type value recall_at_5 53.769999999999996 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-p2p MTEB ArxivClusteringP2P default test a122ad7f3f0291bf49cc6f4d32aa80929df69d5d type value v_measure 40.27789869854063 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-s2s MTEB ArxivClusteringS2S default test f910caf1a6075f7329cdf8c1a6135696f37dbd53 type value v_measure 35.41979463347428 task dataset metrics type Reranking type name config split revision mteb/askubuntudupquestions-reranking MTEB AskUbuntuDupQuestions default test 2000358ca161889fa9c082cb41daa8dcfb161a54 type value map 58.22752045109304 type value mrr 71.51112430198303 task dataset metrics type STS type name config split revision mteb/biosses-sts MTEB BIOSSES default test d3fb88f8f02e40887cd149695127462bbcf29b4a type value cos_sim_pearson 84.71147646622866 type value cos_sim_spearman 85.059167046486 type value euclidean_pearson 75.88421613600647 type value euclidean_spearman 75.12821787150585 type value manhattan_pearson 75.22005646957604 type value manhattan_spearman 74.42880434453272 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (de-en) de-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 99.23799582463465 type value f1 99.12665274878218 type value precision 99.07098121085595 type value recall 99.23799582463465 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (fr-en) fr-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 97.88685890380806 type value f1 97.59336708489249 type value precision 97.44662117543473 type value recall 97.88685890380806 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (ru-en) ru-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 97.47142362313821 type value f1 97.1989377670015 type value precision 97.06384944001847 type value recall 97.47142362313821 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (zh-en) zh-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 98.4728804634018 type value f1 98.2973494821836 type value precision 98.2095839915745 type value recall 98.4728804634018 task dataset metrics type Classification type name config split revision mteb/banking77 MTEB Banking77Classification default test 0fd18e25b25c072e09e0d92ab615fda904d66300 type value accuracy 82.74025974025975 type value f1 82.67420447730439 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-p2p MTEB BiorxivClusteringP2P default test 65b79d1d13f80053f67aca9498d9402c2d9f1f40 type value v_measure 35.0380848063507 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-s2s MTEB BiorxivClusteringS2S default test 258694dd0231531bc1fd9de6ceb52a0853c6d908 type value v_measure 29.45956405670166 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackAndroidRetrieval default test None type value map_at_1 32.122 type value map_at_10 42.03 type value map_at_100 43.364000000000004 type value map_at_1000 43.474000000000004 type value map_at_3 38.804 type value map_at_5 40.585 type value mrr_at_1 39.914 type value mrr_at_10 48.227 type value mrr_at_100 49.018 type value mrr_at_1000 49.064 type value mrr_at_3 45.994 type value mrr_at_5 47.396 type value ndcg_at_1 39.914 type value ndcg_at_10 47.825 type value ndcg_at_100 52.852 type value ndcg_at_1000 54.891 type value ndcg_at_3 43.517 type value ndcg_at_5 45.493 type value precision_at_1 39.914 type value precision_at_10 8.956 type value precision_at_100 1.388 type value precision_at_1000 0.182 type value precision_at_3 20.791999999999998 type value precision_at_5 14.821000000000002 type value recall_at_1 32.122 type value recall_at_10 58.294999999999995 type value recall_at_100 79.726 type value recall_at_1000 93.099 type value recall_at_3 45.017 type value recall_at_5 51.002 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackEnglishRetrieval default test None type value map_at_1 29.677999999999997 type value map_at_10 38.684000000000005 type value map_at_100 39.812999999999995 type value map_at_1000 39.945 type value map_at_3 35.831 type value map_at_5 37.446 type value mrr_at_1 37.771 type value mrr_at_10 44.936 type value mrr_at_100 45.583 type value mrr_at_1000 45.634 type value mrr_at_3 42.771 type value mrr_at_5 43.994 type value ndcg_at_1 37.771 type value ndcg_at_10 44.059 type value ndcg_at_100 48.192 type value ndcg_at_1000 50.375 type value ndcg_at_3 40.172000000000004 type value ndcg_at_5 41.899 type value precision_at_1 37.771 type value precision_at_10 8.286999999999999 type value precision_at_100 1.322 type value precision_at_1000 0.178 type value precision_at_3 19.406000000000002 type value precision_at_5 13.745 type value recall_at_1 29.677999999999997 type value recall_at_10 53.071 type value recall_at_100 70.812 type value recall_at_1000 84.841 type value recall_at_3 41.016000000000005 type value recall_at_5 46.22 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGamingRetrieval default test None type value map_at_1 42.675000000000004 type value map_at_10 53.93599999999999 type value map_at_100 54.806999999999995 type value map_at_1000 54.867 type value map_at_3 50.934000000000005 type value map_at_5 52.583 type value mrr_at_1 48.339 type value mrr_at_10 57.265 type value mrr_at_100 57.873 type value mrr_at_1000 57.906 type value mrr_at_3 55.193000000000005 type value mrr_at_5 56.303000000000004 type value ndcg_at_1 48.339 type value ndcg_at_10 59.19799999999999 type value ndcg_at_100 62.743 type value ndcg_at_1000 63.99399999999999 type value ndcg_at_3 54.367 type value ndcg_at_5 56.548 type value precision_at_1 48.339 type value precision_at_10 9.216000000000001 type value precision_at_100 1.1809999999999998 type value precision_at_1000 0.134 type value precision_at_3 23.72 type value precision_at_5 16.025 type value recall_at_1 42.675000000000004 type value recall_at_10 71.437 type value recall_at_100 86.803 type value recall_at_1000 95.581 type value recall_at_3 58.434 type value recall_at_5 63.754 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGisRetrieval default test None type value map_at_1 23.518 type value map_at_10 30.648999999999997 type value map_at_100 31.508999999999997 type value map_at_1000 31.604 type value map_at_3 28.247 type value map_at_5 29.65 type value mrr_at_1 25.650000000000002 type value mrr_at_10 32.771 type value mrr_at_100 33.554 type value mrr_at_1000 33.629999999999995 type value mrr_at_3 30.433 type value mrr_at_5 31.812 type value ndcg_at_1 25.650000000000002 type value ndcg_at_10 34.929 type value ndcg_at_100 39.382 type value ndcg_at_1000 41.913 type value ndcg_at_3 30.292 type value ndcg_at_5 32.629999999999995 type value precision_at_1 25.650000000000002 type value precision_at_10 5.311 type value precision_at_100 0.792 type value precision_at_1000 0.105 type value precision_at_3 12.58 type value precision_at_5 8.994 type value recall_at_1 23.518 type value recall_at_10 46.19 type value recall_at_100 67.123 type value recall_at_1000 86.442 type value recall_at_3 33.678000000000004 type value recall_at_5 39.244 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackMathematicaRetrieval default test None type value map_at_1 15.891 type value map_at_10 22.464000000000002 type value map_at_100 23.483 type value map_at_1000 23.613 type value map_at_3 20.080000000000002 type value map_at_5 21.526 type value mrr_at_1 20.025000000000002 type value mrr_at_10 26.712999999999997 type value mrr_at_100 27.650000000000002 type value mrr_at_1000 27.737000000000002 type value mrr_at_3 24.274 type value mrr_at_5 25.711000000000002 type value ndcg_at_1 20.025000000000002 type value ndcg_at_10 27.028999999999996 type value ndcg_at_100 32.064 type value ndcg_at_1000 35.188 type value ndcg_at_3 22.512999999999998 type value ndcg_at_5 24.89 type value precision_at_1 20.025000000000002 type value precision_at_10 4.776 type value precision_at_100 0.8500000000000001 type value precision_at_1000 0.125 type value precision_at_3 10.531 type value precision_at_5 7.811 type value recall_at_1 15.891 type value recall_at_10 37.261 type value recall_at_100 59.12 type value recall_at_1000 81.356 type value recall_at_3 24.741 type value recall_at_5 30.753999999999998 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackPhysicsRetrieval default test None type value map_at_1 27.544 type value map_at_10 36.283 type value map_at_100 37.467 type value map_at_1000 37.574000000000005 type value map_at_3 33.528999999999996 type value map_at_5 35.028999999999996 type value mrr_at_1 34.166999999999994 type value mrr_at_10 41.866 type value mrr_at_100 42.666 type value mrr_at_1000 42.716 type value mrr_at_3 39.541 type value mrr_at_5 40.768 type value ndcg_at_1 34.166999999999994 type value ndcg_at_10 41.577 type value ndcg_at_100 46.687 type value ndcg_at_1000 48.967 type value ndcg_at_3 37.177 type value ndcg_at_5 39.097 type value precision_at_1 34.166999999999994 type value precision_at_10 7.420999999999999 type value precision_at_100 1.165 type value precision_at_1000 0.154 type value precision_at_3 17.291999999999998 type value precision_at_5 12.166 type value recall_at_1 27.544 type value recall_at_10 51.99399999999999 type value recall_at_100 73.738 type value recall_at_1000 89.33 type value recall_at_3 39.179 type value recall_at_5 44.385999999999996 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackProgrammersRetrieval default test None type value map_at_1 26.661 type value map_at_10 35.475 type value map_at_100 36.626999999999995 type value map_at_1000 36.741 type value map_at_3 32.818000000000005 type value map_at_5 34.397 type value mrr_at_1 32.647999999999996 type value mrr_at_10 40.784 type value mrr_at_100 41.602 type value mrr_at_1000 41.661 type value mrr_at_3 38.68 type value mrr_at_5 39.838 type value ndcg_at_1 32.647999999999996 type value ndcg_at_10 40.697 type value ndcg_at_100 45.799 type value ndcg_at_1000 48.235 type value ndcg_at_3 36.516 type value ndcg_at_5 38.515 type value precision_at_1 32.647999999999996 type value precision_at_10 7.202999999999999 type value precision_at_100 1.1360000000000001 type value precision_at_1000 0.151 type value precision_at_3 17.314 type value precision_at_5 12.145999999999999 type value recall_at_1 26.661 type value recall_at_10 50.995000000000005 type value recall_at_100 73.065 type value recall_at_1000 89.781 type value recall_at_3 39.073 type value recall_at_5 44.395 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackRetrieval default test None type value map_at_1 25.946583333333333 type value map_at_10 33.79725 type value map_at_100 34.86408333333333 type value map_at_1000 34.9795 type value map_at_3 31.259999999999998 type value map_at_5 32.71541666666666 type value mrr_at_1 30.863749999999996 type value mrr_at_10 37.99183333333333 type value mrr_at_100 38.790499999999994 type value mrr_at_1000 38.85575000000001 type value mrr_at_3 35.82083333333333 type value mrr_at_5 37.07533333333333 type value ndcg_at_1 30.863749999999996 type value ndcg_at_10 38.52141666666667 type value ndcg_at_100 43.17966666666667 type value ndcg_at_1000 45.64608333333333 type value ndcg_at_3 34.333000000000006 type value ndcg_at_5 36.34975 type value precision_at_1 30.863749999999996 type value precision_at_10 6.598999999999999 type value precision_at_100 1.0502500000000001 type value precision_at_1000 0.14400000000000002 type value precision_at_3 15.557583333333334 type value precision_at_5 11.020000000000001 type value recall_at_1 25.946583333333333 type value recall_at_10 48.36991666666666 type value recall_at_100 69.02408333333334 type value recall_at_1000 86.43858333333331 type value recall_at_3 36.4965 type value recall_at_5 41.76258333333334 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackStatsRetrieval default test None type value map_at_1 22.431 type value map_at_10 28.889 type value map_at_100 29.642000000000003 type value map_at_1000 29.742 type value map_at_3 26.998 type value map_at_5 28.172000000000004 type value mrr_at_1 25.307000000000002 type value mrr_at_10 31.763 type value mrr_at_100 32.443 type value mrr_at_1000 32.531 type value mrr_at_3 29.959000000000003 type value mrr_at_5 31.063000000000002 type value ndcg_at_1 25.307000000000002 type value ndcg_at_10 32.586999999999996 type value ndcg_at_100 36.5 type value ndcg_at_1000 39.133 type value ndcg_at_3 29.25 type value ndcg_at_5 31.023 type value precision_at_1 25.307000000000002 type value precision_at_10 4.954 type value precision_at_100 0.747 type value precision_at_1000 0.104 type value precision_at_3 12.577 type value precision_at_5 8.741999999999999 type value recall_at_1 22.431 type value recall_at_10 41.134 type value recall_at_100 59.28600000000001 type value recall_at_1000 78.857 type value recall_at_3 31.926 type value recall_at_5 36.335 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackTexRetrieval default test None type value map_at_1 17.586 type value map_at_10 23.304 type value map_at_100 24.159 type value map_at_1000 24.281 type value map_at_3 21.316 type value map_at_5 22.383 type value mrr_at_1 21.645 type value mrr_at_10 27.365000000000002 type value mrr_at_100 28.108 type value mrr_at_1000 28.192 type value mrr_at_3 25.482 type value mrr_at_5 26.479999999999997 type value ndcg_at_1 21.645 type value ndcg_at_10 27.306 type value ndcg_at_100 31.496000000000002 type value ndcg_at_1000 34.53 type value ndcg_at_3 23.73 type value ndcg_at_5 25.294 type value precision_at_1 21.645 type value precision_at_10 4.797 type value precision_at_100 0.8059999999999999 type value precision_at_1000 0.121 type value precision_at_3 10.850999999999999 type value precision_at_5 7.736 type value recall_at_1 17.586 type value recall_at_10 35.481 type value recall_at_100 54.534000000000006 type value recall_at_1000 76.456 type value recall_at_3 25.335 type value recall_at_5 29.473 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackUnixRetrieval default test None type value map_at_1 25.095 type value map_at_10 32.374 type value map_at_100 33.537 type value map_at_1000 33.634 type value map_at_3 30.089 type value map_at_5 31.433 type value mrr_at_1 29.198 type value mrr_at_10 36.01 type value mrr_at_100 37.022 type value mrr_at_1000 37.083 type value mrr_at_3 33.94 type value mrr_at_5 35.148 type value ndcg_at_1 29.198 type value ndcg_at_10 36.729 type value ndcg_at_100 42.114000000000004 type value ndcg_at_1000 44.592 type value ndcg_at_3 32.644 type value ndcg_at_5 34.652 type value precision_at_1 29.198 type value precision_at_10 5.970000000000001 type value precision_at_100 0.967 type value precision_at_1000 0.129 type value precision_at_3 14.396999999999998 type value precision_at_5 10.093 type value recall_at_1 25.095 type value recall_at_10 46.392 type value recall_at_100 69.706 type value recall_at_1000 87.738 type value recall_at_3 35.303000000000004 type value recall_at_5 40.441 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWebmastersRetrieval default test None type value map_at_1 26.857999999999997 type value map_at_10 34.066 type value map_at_100 35.671 type value map_at_1000 35.881 type value map_at_3 31.304 type value map_at_5 32.885 type value mrr_at_1 32.411 type value mrr_at_10 38.987 type value mrr_at_100 39.894 type value mrr_at_1000 39.959 type value mrr_at_3 36.626999999999995 type value mrr_at_5 38.011 type value ndcg_at_1 32.411 type value ndcg_at_10 39.208 type value ndcg_at_100 44.626 type value ndcg_at_1000 47.43 type value ndcg_at_3 35.091 type value ndcg_at_5 37.119 type value precision_at_1 32.411 type value precision_at_10 7.51 type value precision_at_100 1.486 type value precision_at_1000 0.234 type value precision_at_3 16.14 type value precision_at_5 11.976 type value recall_at_1 26.857999999999997 type value recall_at_10 47.407 type value recall_at_100 72.236 type value recall_at_1000 90.77 type value recall_at_3 35.125 type value recall_at_5 40.522999999999996 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWordpressRetrieval default test None type value map_at_1 21.3 type value map_at_10 27.412999999999997 type value map_at_100 28.29 type value map_at_1000 28.398 type value map_at_3 25.169999999999998 type value map_at_5 26.496 type value mrr_at_1 23.29 type value mrr_at_10 29.215000000000003 type value mrr_at_100 30.073 type value mrr_at_1000 30.156 type value mrr_at_3 26.956000000000003 type value mrr_at_5 28.38 type value ndcg_at_1 23.29 type value ndcg_at_10 31.113000000000003 type value ndcg_at_100 35.701 type value ndcg_at_1000 38.505 type value ndcg_at_3 26.727 type value ndcg_at_5 29.037000000000003 type value precision_at_1 23.29 type value precision_at_10 4.787 type value precision_at_100 0.763 type value precision_at_1000 0.11100000000000002 type value precision_at_3 11.091 type value precision_at_5 7.985 type value recall_at_1 21.3 type value recall_at_10 40.782000000000004 type value recall_at_100 62.13999999999999 type value recall_at_1000 83.012 type value recall_at_3 29.131 type value recall_at_5 34.624 task dataset metrics type Retrieval type name config split revision climate-fever MTEB ClimateFEVER default test None type value map_at_1 9.631 type value map_at_10 16.634999999999998 type value map_at_100 18.23 type value map_at_1000 18.419 type value map_at_3 13.66 type value map_at_5 15.173 type value mrr_at_1 21.368000000000002 type value mrr_at_10 31.56 type value mrr_at_100 32.58 type value mrr_at_1000 32.633 type value mrr_at_3 28.241 type value mrr_at_5 30.225 type value ndcg_at_1 21.368000000000002 type value ndcg_at_10 23.855999999999998 type value ndcg_at_100 30.686999999999998 type value ndcg_at_1000 34.327000000000005 type value ndcg_at_3 18.781 type value ndcg_at_5 20.73 type value precision_at_1 21.368000000000002 type value precision_at_10 7.564 type value precision_at_100 1.496 type value precision_at_1000 0.217 type value precision_at_3 13.876 type value precision_at_5 11.062 type value recall_at_1 9.631 type value recall_at_10 29.517 type value recall_at_100 53.452 type value recall_at_1000 74.115 type value recall_at_3 17.605999999999998 type value recall_at_5 22.505 task dataset metrics type Retrieval type name config split revision dbpedia-entity MTEB DBPedia default test None type value map_at_1 8.885 type value map_at_10 18.798000000000002 type value map_at_100 26.316 type value map_at_1000 27.869 type value map_at_3 13.719000000000001 type value map_at_5 15.716 type value mrr_at_1 66 type value mrr_at_10 74.263 type value mrr_at_100 74.519 type value mrr_at_1000 74.531 type value mrr_at_3 72.458 type value mrr_at_5 73.321 type value ndcg_at_1 53.87499999999999 type value ndcg_at_10 40.355999999999995 type value ndcg_at_100 44.366 type value ndcg_at_1000 51.771 type value ndcg_at_3 45.195 type value ndcg_at_5 42.187000000000005 type value precision_at_1 66 type value precision_at_10 31.75 type value precision_at_100 10.11 type value precision_at_1000 1.9800000000000002 type value precision_at_3 48.167 type value precision_at_5 40.050000000000004 type value recall_at_1 8.885 type value recall_at_10 24.471999999999998 type value recall_at_100 49.669000000000004 type value recall_at_1000 73.383 type value recall_at_3 14.872 type value recall_at_5 18.262999999999998 task dataset metrics type Classification type name config split revision mteb/emotion MTEB EmotionClassification default test 4f58c6b202a23cf9a4da393831edf4f9183cad37 type value accuracy 45.18 type value f1 40.26878691789978 task dataset metrics type Retrieval type name config split revision fever MTEB FEVER default test None type value map_at_1 62.751999999999995 type value map_at_10 74.131 type value map_at_100 74.407 type value map_at_1000 74.423 type value map_at_3 72.329 type value map_at_5 73.555 type value mrr_at_1 67.282 type value mrr_at_10 78.292 type value mrr_at_100 78.455 type value mrr_at_1000 78.458 type value mrr_at_3 76.755 type value mrr_at_5 77.839 type value ndcg_at_1 67.282 type value ndcg_at_10 79.443 type value ndcg_at_100 80.529 type value ndcg_at_1000 80.812 type value ndcg_at_3 76.281 type value ndcg_at_5 78.235 type value precision_at_1 67.282 type value precision_at_10 10.078 type value precision_at_100 1.082 type value precision_at_1000 0.11199999999999999 type value precision_at_3 30.178 type value precision_at_5 19.232 type value recall_at_1 62.751999999999995 type value recall_at_10 91.521 type value recall_at_100 95.997 type value recall_at_1000 97.775 type value recall_at_3 83.131 type value recall_at_5 87.93299999999999 task dataset metrics type Retrieval type name config split revision fiqa MTEB FiQA2018 default test None type value map_at_1 18.861 type value map_at_10 30.252000000000002 type value map_at_100 32.082 type value map_at_1000 32.261 type value map_at_3 25.909 type value map_at_5 28.296 type value mrr_at_1 37.346000000000004 type value mrr_at_10 45.802 type value mrr_at_100 46.611999999999995 type value mrr_at_1000 46.659 type value mrr_at_3 43.056 type value mrr_at_5 44.637 type value ndcg_at_1 37.346000000000004 type value ndcg_at_10 38.169 type value ndcg_at_100 44.864 type value ndcg_at_1000 47.974 type value ndcg_at_3 33.619 type value ndcg_at_5 35.317 type value precision_at_1 37.346000000000004 type value precision_at_10 10.693999999999999 type value precision_at_100 1.775 type value precision_at_1000 0.231 type value precision_at_3 22.325 type value precision_at_5 16.852 type value recall_at_1 18.861 type value recall_at_10 45.672000000000004 type value recall_at_100 70.60499999999999 type value recall_at_1000 89.216 type value recall_at_3 30.361 type value recall_at_5 36.998999999999995 task dataset metrics type Retrieval type name config split revision hotpotqa MTEB HotpotQA default test None type value map_at_1 37.852999999999994 type value map_at_10 59.961 type value map_at_100 60.78 type value map_at_1000 60.843 type value map_at_3 56.39999999999999 type value map_at_5 58.646 type value mrr_at_1 75.70599999999999 type value mrr_at_10 82.321 type value mrr_at_100 82.516 type value mrr_at_1000 82.525 type value mrr_at_3 81.317 type value mrr_at_5 81.922 type value ndcg_at_1 75.70599999999999 type value ndcg_at_10 68.557 type value ndcg_at_100 71.485 type value ndcg_at_1000 72.71600000000001 type value ndcg_at_3 63.524 type value ndcg_at_5 66.338 type value precision_at_1 75.70599999999999 type value precision_at_10 14.463000000000001 type value precision_at_100 1.677 type value precision_at_1000 0.184 type value precision_at_3 40.806 type value precision_at_5 26.709 type value recall_at_1 37.852999999999994 type value recall_at_10 72.316 type value recall_at_100 83.842 type value recall_at_1000 91.999 type value recall_at_3 61.209 type value recall_at_5 66.77199999999999 task dataset metrics type Classification type name config split revision mteb/imdb MTEB ImdbClassification default test 3d86128a09e091d6018b6d26cad27f2739fc2db7 type value accuracy 85.46039999999999 type value ap 79.9812521351881 type value f1 85.31722909702084 task dataset metrics type Retrieval type name config split revision msmarco MTEB MSMARCO default dev None type value map_at_1 22.704 type value map_at_10 35.329 type value map_at_100 36.494 type value map_at_1000 36.541000000000004 type value map_at_3 31.476 type value map_at_5 33.731 type value mrr_at_1 23.294999999999998 type value mrr_at_10 35.859 type value mrr_at_100 36.968 type value mrr_at_1000 37.008 type value mrr_at_3 32.085 type value mrr_at_5 34.299 type value ndcg_at_1 23.324 type value ndcg_at_10 42.274 type value ndcg_at_100 47.839999999999996 type value ndcg_at_1000 48.971 type value ndcg_at_3 34.454 type value ndcg_at_5 38.464 type value precision_at_1 23.324 type value precision_at_10 6.648 type value precision_at_100 0.9440000000000001 type value precision_at_1000 0.104 type value precision_at_3 14.674999999999999 type value precision_at_5 10.850999999999999 type value recall_at_1 22.704 type value recall_at_10 63.660000000000004 type value recall_at_100 89.29899999999999 type value recall_at_1000 97.88900000000001 type value recall_at_3 42.441 type value recall_at_5 52.04 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (en) en test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 93.1326949384405 type value f1 92.89743579612082 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (de) de test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 89.62524654832347 type value f1 88.65106082263151 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (es) es test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 90.59039359573046 type value f1 90.31532892105662 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (fr) fr test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 86.21046038208581 type value f1 86.41459529813113 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (hi) hi test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 87.3180351380423 type value f1 86.71383078226444 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (th) th test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 86.24231464737792 type value f1 86.31845567592403 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (en) en test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 75.27131782945736 type value f1 57.52079940417103 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (de) de test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 71.2341504649197 type value f1 51.349951558039244 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (es) es test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 71.27418278852569 type value f1 50.1714985749095 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (fr) fr test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 67.68243031631694 type value f1 50.1066160836192 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (hi) hi test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 69.2362854069559 type value f1 48.821279948766424 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (th) th test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 71.71428571428571 type value f1 53.94611389496195 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (af) af test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 59.97646267652992 type value f1 57.26797883561521 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (am) am test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 53.65501008742435 type value f1 50.416258382177034 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ar) ar test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 57.45796906523201 type value f1 53.306690547422185 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (az) az test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 62.59246805648957 type value f1 59.818381969051494 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (bn) bn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 61.126429051782104 type value f1 58.25993593933026 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (cy) cy test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 50.057162071284466 type value f1 46.96095728790911 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (da) da test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 66.64425016812375 type value f1 62.858291698755764 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (de) de test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 66.08944182918628 type value f1 62.44639030604241 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (el) el test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 64.68056489576328 type value f1 61.775326758789504 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (en) en test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 72.11163416274377 type value f1 69.70789096927015 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (es) es test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 68.40282447881641 type value f1 66.38492065671895 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fa) fa test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 67.24613315400134 type value f1 64.3348019501336 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fi) fi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 65.78345662407531 type value f1 62.21279452354622 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fr) fr test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 67.9455279085407 type value f1 65.48193124964094 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (he) he test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 62.05110961667788 type value f1 58.097856564684534 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hi) hi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 64.95292535305985 type value f1 62.09182174767901 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hu) hu test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 64.97310020174848 type value f1 61.14252567730396 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hy) hy test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 60.08069939475453 type value f1 57.044041742492034 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (id) id test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 66.63752521856085 type value f1 63.889340907205316 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (is) is test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 56.385339609952936 type value f1 53.449033750088304 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (it) it test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 68.93073301950234 type value f1 65.9884357824104 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ja) ja test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 68.94418291862812 type value f1 66.48740222583132 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (jv) jv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 54.26025554808339 type value f1 50.19562815100793 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ka) ka test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 48.98789509078682 type value f1 46.65788438676836 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (km) km test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 44.68728984532616 type value f1 41.642419349541996 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (kn) kn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 59.19300605245461 type value f1 55.8626492442437 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ko) ko test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 66.33826496301278 type value f1 63.89499791648792 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (lv) lv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 60.33960995292536 type value f1 57.15242464180892 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ml) ml test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 63.09347679892402 type value f1 59.64733214063841 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (mn) mn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 58.75924680564896 type value f1 55.96585692366827 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ms) ms test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 62.48486886348352 type value f1 59.45143559032946 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (my) my test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 58.56422326832549 type value f1 54.96368702901926 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nb) nb test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 66.18022864828512 type value f1 63.05369805040634 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nl) nl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 67.30329522528581 type value f1 64.06084612020727 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pl) pl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 68.36919973100201 type value f1 65.12154124788887 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pt) pt test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 68.98117014122394 type value f1 66.41847559806962 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ro) ro test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 65.53799596503026 type value f1 62.17067330740817 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ru) ru test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.01815736381977 type value f1 66.24988369607843 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sl) sl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 62.34700739744452 type value f1 59.957933424941636 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sq) sq test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 61.23402824478815 type value f1 57.98836976018471 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sv) sv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 68.54068594485541 type value f1 65.43849680666855 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sw) sw test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 55.998655010087425 type value f1 52.83737515406804 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ta) ta test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 58.71217215870882 type value f1 55.051794977833026 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (te) te test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 59.724277067921996 type value f1 56.33485571838306 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (th) th test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 65.59515803631473 type value f1 64.96772366193588 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tl) tl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 60.860793544048406 type value f1 58.148845819115394 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tr) tr test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 67.40753194351043 type value f1 63.18903778054698 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ur) ur test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 61.52320107599194 type value f1 58.356144563398516 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (vi) vi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 66.17014122394083 type value f1 63.919964062638925 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-CN) zh-CN test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 69.15601882985878 type value f1 67.01451905761371 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-TW) zh-TW test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 64.65030262273034 type value f1 64.14420425129063 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (af) af test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 65.08742434431743 type value f1 63.044060042311756 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (am) am test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 58.52387357094821 type value f1 56.82398588814534 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ar) ar test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 62.239408204438476 type value f1 61.92570286170469 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (az) az test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 63.74915938130463 type value f1 62.130740689396276 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (bn) bn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 65.00336247478144 type value f1 63.71080635228055 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (cy) cy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 52.837928715534645 type value f1 50.390741680320836 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (da) da test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.42098184263618 type value f1 71.41355113538995 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (de) de test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.95359784801613 type value f1 71.42699340156742 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (el) el test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.18157363819772 type value f1 69.74836113037671 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (en) en test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 77.08137188971082 type value f1 76.78000685068261 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (es) es test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.5030262273033 type value f1 71.71620130425673 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fa) fa test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.24546065904505 type value f1 69.07638311730359 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fi) fi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 69.12911903160726 type value f1 68.32651736539815 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fr) fr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.89307330195025 type value f1 71.33986549860187 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (he) he test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 67.44451916610626 type value f1 66.90192664503866 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hi) hi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 69.16274377942166 type value f1 68.01090953775066 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hu) hu test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.75319435104237 type value f1 70.18035309201403 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hy) hy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 63.14391392064559 type value f1 61.48286540778145 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (id) id test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.70275722932078 type value f1 70.26164779846495 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (is) is test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 60.93813046402153 type value f1 58.8852862116525 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (it) it test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.320107599193 type value f1 72.19836409602924 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ja) ja test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 74.65366509751176 type value f1 74.55188288799579 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (jv) jv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 59.694014794889036 type value f1 58.11353311721067 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ka) ka test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 54.37457969065231 type value f1 52.81306134311697 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (km) km test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 48.3086751849361 type value f1 45.396449765419376 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (kn) kn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 62.151983860121064 type value f1 60.31762544281696 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ko) ko test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.44788164088769 type value f1 71.68150151736367 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (lv) lv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 62.81439139206455 type value f1 62.06735559105593 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ml) ml test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 68.04303967720242 type value f1 66.68298851670133 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (mn) mn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 61.43913920645595 type value f1 60.25605977560783 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ms) ms test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 66.90316072629456 type value f1 65.1325924692381 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (my) my test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 61.63752521856086 type value f1 59.14284778039585 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nb) nb test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.63080026899797 type value f1 70.89771864626877 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nl) nl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.10827168796234 type value f1 71.71954219691159 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pl) pl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.59515803631471 type value f1 70.05040128099003 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pt) pt test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.83389374579691 type value f1 70.84877936562735 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ro) ro test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 69.18628110289173 type value f1 68.97232927921841 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ru) ru test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.99260255548083 type value f1 72.85139492157732 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sl) sl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 65.26227303295225 type value f1 65.08833655469431 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sq) sq test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 66.48621385339611 type value f1 64.43483199071298 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sv) sv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.14391392064559 type value f1 72.2580822579741 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sw) sw test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 59.88567585743107 type value f1 58.3073765932569 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ta) ta test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 62.38399462004034 type value f1 60.82139544252606 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (te) te test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 62.58574310692671 type value f1 60.71443370385374 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (th) th test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.61398789509079 type value f1 70.99761812049401 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tl) tl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 62.73705447209146 type value f1 61.680849331794796 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tr) tr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.66778749159381 type value f1 71.17320646080115 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ur) ur test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 64.640215198386 type value f1 63.301805157015444 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (vi) vi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.00672494956288 type value f1 70.26005548582106 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-CN) zh-CN test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 75.42030934767989 type value f1 75.2074842882598 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-TW) zh-TW test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.69266980497646 type value f1 70.94103167391192 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-p2p MTEB MedrxivClusteringP2P default test e7a26af6f3ae46b30dde8737f02c07b1505bcc73 type value v_measure 28.91697191169135 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-s2s MTEB MedrxivClusteringS2S default test 35191c8c0dca72d8ff3efcd72aa802307d469663 type value v_measure 28.434000079573313 task dataset metrics type Reranking type name config split revision mteb/mind_small MTEB MindSmallReranking default test 3bdac13927fdc888b903db93b2ffdbd90b295a69 type value map 30.96683513343383 type value mrr 31.967364078714834 task dataset metrics type Retrieval type name config split revision nfcorpus MTEB NFCorpus default test None type value map_at_1 5.5280000000000005 type value map_at_10 11.793 type value map_at_100 14.496999999999998 type value map_at_1000 15.783 type value map_at_3 8.838 type value map_at_5 10.07 type value mrr_at_1 43.653 type value mrr_at_10 51.531000000000006 type value mrr_at_100 52.205 type value mrr_at_1000 52.242999999999995 type value mrr_at_3 49.431999999999995 type value mrr_at_5 50.470000000000006 type value ndcg_at_1 42.415000000000006 type value ndcg_at_10 32.464999999999996 type value ndcg_at_100 28.927999999999997 type value ndcg_at_1000 37.629000000000005 type value ndcg_at_3 37.845 type value ndcg_at_5 35.147 type value precision_at_1 43.653 type value precision_at_10 23.932000000000002 type value precision_at_100 7.17 type value precision_at_1000 1.967 type value precision_at_3 35.397 type value precision_at_5 29.907 type value recall_at_1 5.5280000000000005 type value recall_at_10 15.568000000000001 type value recall_at_100 28.54 type value recall_at_1000 59.864 type value recall_at_3 9.822000000000001 type value recall_at_5 11.726 task dataset metrics type Retrieval type name config split revision nq MTEB NQ default test None type value map_at_1 37.041000000000004 type value map_at_10 52.664 type value map_at_100 53.477 type value map_at_1000 53.505 type value map_at_3 48.510999999999996 type value map_at_5 51.036 type value mrr_at_1 41.338 type value mrr_at_10 55.071000000000005 type value mrr_at_100 55.672 type value mrr_at_1000 55.689 type value mrr_at_3 51.82 type value mrr_at_5 53.852 type value ndcg_at_1 41.338 type value ndcg_at_10 60.01800000000001 type value ndcg_at_100 63.409000000000006 type value ndcg_at_1000 64.017 type value ndcg_at_3 52.44799999999999 type value ndcg_at_5 56.571000000000005 type value precision_at_1 41.338 type value precision_at_10 9.531 type value precision_at_100 1.145 type value precision_at_1000 0.12 type value precision_at_3 23.416 type value precision_at_5 16.46 type value recall_at_1 37.041000000000004 type value recall_at_10 79.76299999999999 type value recall_at_100 94.39 type value recall_at_1000 98.851 type value recall_at_3 60.465 type value recall_at_5 69.906 task dataset metrics type Retrieval type name config split revision quora MTEB QuoraRetrieval default test None type value map_at_1 69.952 type value map_at_10 83.758 type value map_at_100 84.406 type value map_at_1000 84.425 type value map_at_3 80.839 type value map_at_5 82.646 type value mrr_at_1 80.62 type value mrr_at_10 86.947 type value mrr_at_100 87.063 type value mrr_at_1000 87.064 type value mrr_at_3 85.96000000000001 type value mrr_at_5 86.619 type value ndcg_at_1 80.63 type value ndcg_at_10 87.64800000000001 type value ndcg_at_100 88.929 type value ndcg_at_1000 89.054 type value ndcg_at_3 84.765 type value ndcg_at_5 86.291 type value precision_at_1 80.63 type value precision_at_10 13.314 type value precision_at_100 1.525 type value precision_at_1000 0.157 type value precision_at_3 37.1 type value precision_at_5 24.372 type value recall_at_1 69.952 type value recall_at_10 94.955 type value recall_at_100 99.38 type value recall_at_1000 99.96000000000001 type value recall_at_3 86.60600000000001 type value recall_at_5 90.997 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering MTEB RedditClustering default test 24640382cdbf8abc73003fb0fa6d111a705499eb type value v_measure 42.41329517878427 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering-p2p MTEB RedditClusteringP2P default test 282350215ef01743dc01b456c7f5241fa8937f16 type value v_measure 55.171278362748666 task dataset metrics type Retrieval type name config split revision scidocs MTEB SCIDOCS default test None type value map_at_1 4.213 type value map_at_10 9.895 type value map_at_100 11.776 type value map_at_1000 12.084 type value map_at_3 7.2669999999999995 type value map_at_5 8.620999999999999 type value mrr_at_1 20.8 type value mrr_at_10 31.112000000000002 type value mrr_at_100 32.274 type value mrr_at_1000 32.35 type value mrr_at_3 28.133000000000003 type value mrr_at_5 29.892999999999997 type value ndcg_at_1 20.8 type value ndcg_at_10 17.163999999999998 type value ndcg_at_100 24.738 type value ndcg_at_1000 30.316 type value ndcg_at_3 16.665 type value ndcg_at_5 14.478 type value precision_at_1 20.8 type value precision_at_10 8.74 type value precision_at_100 1.963 type value precision_at_1000 0.33 type value precision_at_3 15.467 type value precision_at_5 12.6 type value recall_at_1 4.213 type value recall_at_10 17.698 type value recall_at_100 39.838 type value recall_at_1000 66.893 type value recall_at_3 9.418 type value recall_at_5 12.773000000000001 task dataset metrics type STS type name config split revision mteb/sickr-sts MTEB SICK-R default test a6ea5a8cab320b040a23452cc28066d9beae2cee type value cos_sim_pearson 82.90453315738294 type value cos_sim_spearman 78.51197850080254 type value euclidean_pearson 80.09647123597748 type value euclidean_spearman 78.63548011514061 type value manhattan_pearson 80.10645285675231 type value manhattan_spearman 78.57861806068901 task dataset metrics type STS type name config split revision mteb/sts12-sts MTEB STS12 default test a0d554a64d88156834ff5ae9920b964011b16384 type value cos_sim_pearson 84.2616156846401 type value cos_sim_spearman 76.69713867850156 type value euclidean_pearson 77.97948563800394 type value euclidean_spearman 74.2371211567807 type value manhattan_pearson 77.69697879669705 type value manhattan_spearman 73.86529778022278 task dataset metrics type STS type name config split revision mteb/sts13-sts MTEB STS13 default test 7e90230a92c190f1bf69ae9002b8cea547a64cca type value cos_sim_pearson 77.0293269315045 type value cos_sim_spearman 78.02555120584198 type value euclidean_pearson 78.25398100379078 type value euclidean_spearman 78.66963870599464 type value manhattan_pearson 78.14314682167348 type value manhattan_spearman 78.57692322969135 task dataset metrics type STS type name config split revision mteb/sts14-sts MTEB STS14 default test 6031580fec1f6af667f0bd2da0a551cf4f0b2375 type value cos_sim_pearson 79.16989925136942 type value cos_sim_spearman 76.5996225327091 type value euclidean_pearson 77.8319003279786 type value euclidean_spearman 76.42824009468998 type value manhattan_pearson 77.69118862737736 type value manhattan_spearman 76.25568104762812 task dataset metrics type STS type name config split revision mteb/sts15-sts MTEB STS15 default test ae752c7c21bf194d8b67fd573edf7ae58183cbe3 type value cos_sim_pearson 87.42012286935325 type value cos_sim_spearman 88.15654297884122 type value euclidean_pearson 87.34082819427852 type value euclidean_spearman 88.06333589547084 type value manhattan_pearson 87.25115596784842 type value manhattan_spearman 87.9559927695203 task dataset metrics type STS type name config split revision mteb/sts16-sts MTEB STS16 default test 4d8694f8f0e0100860b497b999b3dbed754a0513 type value cos_sim_pearson 82.88222044996712 type value cos_sim_spearman 84.28476589061077 type value euclidean_pearson 83.17399758058309 type value euclidean_spearman 83.85497357244542 type value manhattan_pearson 83.0308397703786 type value manhattan_spearman 83.71554539935046 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (ko-ko) ko-ko test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 80.20682986257339 type value cos_sim_spearman 79.94567120362092 type value euclidean_pearson 79.43122480368902 type value euclidean_spearman 79.94802077264987 type value manhattan_pearson 79.32653021527081 type value manhattan_spearman 79.80961146709178 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (ar-ar) ar-ar test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 74.46578144394383 type value cos_sim_spearman 74.52496637472179 type value euclidean_pearson 72.2903807076809 type value euclidean_spearman 73.55549359771645 type value manhattan_pearson 72.09324837709393 type value manhattan_spearman 73.36743103606581 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-ar) en-ar test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 71.37272335116 type value cos_sim_spearman 71.26702117766037 type value euclidean_pearson 67.114829954434 type value euclidean_spearman 66.37938893947761 type value manhattan_pearson 66.79688574095246 type value manhattan_spearman 66.17292828079667 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-de) en-de test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 80.61016770129092 type value cos_sim_spearman 82.08515426632214 type value euclidean_pearson 80.557340361131 type value euclidean_spearman 80.37585812266175 type value manhattan_pearson 80.6782873404285 type value manhattan_spearman 80.6678073032024 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-en) en-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 87.00150745350108 type value cos_sim_spearman 87.83441972211425 type value euclidean_pearson 87.94826702308792 type value euclidean_spearman 87.46143974860725 type value manhattan_pearson 87.97560344306105 type value manhattan_spearman 87.5267102829796 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-tr) en-tr test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 64.76325252267235 type value cos_sim_spearman 63.32615095463905 type value euclidean_pearson 64.07920669155716 type value euclidean_spearman 61.21409893072176 type value manhattan_pearson 64.26308625680016 type value manhattan_spearman 61.2438185254079 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (es-en) es-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 75.82644463022595 type value cos_sim_spearman 76.50381269945073 type value euclidean_pearson 75.1328548315934 type value euclidean_spearman 75.63761139408453 type value manhattan_pearson 75.18610101241407 type value manhattan_spearman 75.30669266354164 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (es-es) es-es test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 87.49994164686832 type value cos_sim_spearman 86.73743986245549 type value euclidean_pearson 86.8272894387145 type value euclidean_spearman 85.97608491000507 type value manhattan_pearson 86.74960140396779 type value manhattan_spearman 85.79285984190273 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (fr-en) fr-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 79.58172210788469 type value cos_sim_spearman 80.17516468334607 type value euclidean_pearson 77.56537843470504 type value euclidean_spearman 77.57264627395521 type value manhattan_pearson 78.09703521695943 type value manhattan_spearman 78.15942760916954 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (it-en) it-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 79.7589932931751 type value cos_sim_spearman 80.15210089028162 type value euclidean_pearson 77.54135223516057 type value euclidean_spearman 77.52697996368764 type value manhattan_pearson 77.65734439572518 type value manhattan_spearman 77.77702992016121 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (nl-en) nl-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 79.16682365511267 type value cos_sim_spearman 79.25311267628506 type value euclidean_pearson 77.54882036762244 type value euclidean_spearman 77.33212935194827 type value manhattan_pearson 77.98405516064015 type value manhattan_spearman 77.85075717865719 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (en) en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 59.10473294775917 type value cos_sim_spearman 61.82780474476838 type value euclidean_pearson 45.885111672377256 type value euclidean_spearman 56.88306351932454 type value manhattan_pearson 46.101218127323186 type value manhattan_spearman 56.80953694186333 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de) de test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 45.781923079584146 type value cos_sim_spearman 55.95098449691107 type value euclidean_pearson 25.4571031323205 type value euclidean_spearman 49.859978118078935 type value manhattan_pearson 25.624938455041384 type value manhattan_spearman 49.99546185049401 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es) es test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 60.00618133997907 type value cos_sim_spearman 66.57896677718321 type value euclidean_pearson 42.60118466388821 type value euclidean_spearman 62.8210759715209 type value manhattan_pearson 42.63446860604094 type value manhattan_spearman 62.73803068925271 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (pl) pl test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 28.460759121626943 type value cos_sim_spearman 34.13459007469131 type value euclidean_pearson 6.0917739325525195 type value euclidean_spearman 27.9947262664867 type value manhattan_pearson 6.16877864169911 type value manhattan_spearman 28.00664163971514 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (tr) tr test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 57.42546621771696 type value cos_sim_spearman 63.699663168970474 type value euclidean_pearson 38.12085278789738 type value euclidean_spearman 58.12329140741536 type value manhattan_pearson 37.97364549443335 type value manhattan_spearman 57.81545502318733 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (ar) ar test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 46.82241380954213 type value cos_sim_spearman 57.86569456006391 type value euclidean_pearson 31.80480070178813 type value euclidean_spearman 52.484000620130104 type value manhattan_pearson 31.952708554646097 type value manhattan_spearman 52.8560972356195 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (ru) ru test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 52.00447170498087 type value cos_sim_spearman 60.664116225735164 type value euclidean_pearson 33.87382555421702 type value euclidean_spearman 55.74649067458667 type value manhattan_pearson 33.99117246759437 type value manhattan_spearman 55.98749034923899 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh) zh test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 58.06497233105448 type value cos_sim_spearman 65.62968801135676 type value euclidean_pearson 47.482076613243905 type value euclidean_spearman 62.65137791498299 type value manhattan_pearson 47.57052626104093 type value manhattan_spearman 62.436916516613294 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (fr) fr test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 70.49397298562575 type value cos_sim_spearman 74.79604041187868 type value euclidean_pearson 49.661891561317795 type value euclidean_spearman 70.31535537621006 type value manhattan_pearson 49.553715741850006 type value manhattan_spearman 70.24779344636806 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-en) de-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 55.640574515348696 type value cos_sim_spearman 54.927959317689 type value euclidean_pearson 29.00139666967476 type value euclidean_spearman 41.86386566971605 type value manhattan_pearson 29.47411067730344 type value manhattan_spearman 42.337438424952786 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es-en) es-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 68.14095292259312 type value cos_sim_spearman 73.99017581234789 type value euclidean_pearson 46.46304297872084 type value euclidean_spearman 60.91834114800041 type value manhattan_pearson 47.07072666338692 type value manhattan_spearman 61.70415727977926 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (it) it test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 73.27184653359575 type value cos_sim_spearman 77.76070252418626 type value euclidean_pearson 62.30586577544778 type value euclidean_spearman 75.14246629110978 type value manhattan_pearson 62.328196884927046 type value manhattan_spearman 75.1282792981433 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (pl-en) pl-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 71.59448528829957 type value cos_sim_spearman 70.37277734222123 type value euclidean_pearson 57.63145565721123 type value euclidean_spearman 66.10113048304427 type value manhattan_pearson 57.18897811586808 type value manhattan_spearman 66.5595511215901 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh-en) zh-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 66.37520607720838 type value cos_sim_spearman 69.92282148997948 type value euclidean_pearson 40.55768770125291 type value euclidean_spearman 55.189128944669605 type value manhattan_pearson 41.03566433468883 type value manhattan_spearman 55.61251893174558 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es-it) es-it test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 57.791929533771835 type value cos_sim_spearman 66.45819707662093 type value euclidean_pearson 39.03686018511092 type value euclidean_spearman 56.01282695640428 type value manhattan_pearson 38.91586623619632 type value manhattan_spearman 56.69394943612747 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-fr) de-fr test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 47.82224468473866 type value cos_sim_spearman 59.467307194781164 type value euclidean_pearson 27.428459190256145 type value euclidean_spearman 60.83463107397519 type value manhattan_pearson 27.487391578496638 type value manhattan_spearman 61.281380460246496 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-pl) de-pl test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 16.306666792752644 type value cos_sim_spearman 39.35486427252405 type value euclidean_pearson -2.7887154897955435 type value euclidean_spearman 27.1296051831719 type value manhattan_pearson -3.202291270581297 type value manhattan_spearman 26.32895849218158 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (fr-pl) fr-pl test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 59.67006803805076 type value cos_sim_spearman 73.24670207647144 type value euclidean_pearson 46.91884681500483 type value euclidean_spearman 16.903085094570333 type value manhattan_pearson 46.88391675325812 type value manhattan_spearman 28.17180849095055 task dataset metrics type STS type name config split revision mteb/stsbenchmark-sts MTEB STSBenchmark default test b0fddb56ed78048fa8b90373c8a3cfc37b684831 type value cos_sim_pearson 83.79555591223837 type value cos_sim_spearman 85.63658602085185 type value euclidean_pearson 85.22080894037671 type value euclidean_spearman 85.54113580167038 type value manhattan_pearson 85.1639505960118 type value manhattan_spearman 85.43502665436196 task dataset metrics type Reranking type name config split revision mteb/scidocs-reranking MTEB SciDocsRR default test d3c5e1fc0b855ab6097bf1cda04dd73947d7caab type value map 80.73900991689766 type value mrr 94.81624131133934 task dataset metrics type Retrieval type name config split revision scifact MTEB SciFact default test None type value map_at_1 55.678000000000004 type value map_at_10 65.135 type value map_at_100 65.824 type value map_at_1000 65.852 type value map_at_3 62.736000000000004 type value map_at_5 64.411 type value mrr_at_1 58.333 type value mrr_at_10 66.5 type value mrr_at_100 67.053 type value mrr_at_1000 67.08 type value mrr_at_3 64.944 type value mrr_at_5 65.89399999999999 type value ndcg_at_1 58.333 type value ndcg_at_10 69.34700000000001 type value ndcg_at_100 72.32 type value ndcg_at_1000 73.014 type value ndcg_at_3 65.578 type value ndcg_at_5 67.738 type value precision_at_1 58.333 type value precision_at_10 9.033 type value precision_at_100 1.0670000000000002 type value precision_at_1000 0.11199999999999999 type value precision_at_3 25.444 type value precision_at_5 16.933 type value recall_at_1 55.678000000000004 type value recall_at_10 80.72200000000001 type value recall_at_100 93.93299999999999 type value recall_at_1000 99.333 type value recall_at_3 70.783 type value recall_at_5 75.978 task dataset metrics type PairClassification type name config split revision mteb/sprintduplicatequestions-pairclassification MTEB SprintDuplicateQuestions default test d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 type value cos_sim_accuracy 99.74653465346535 type value cos_sim_ap 93.01476369929063 type value cos_sim_f1 86.93009118541033 type value cos_sim_precision 88.09034907597535 type value cos_sim_recall 85.8 type value dot_accuracy 99.22970297029703 type value dot_ap 51.58725659485144 type value dot_f1 53.51351351351352 type value dot_precision 58.235294117647065 type value dot_recall 49.5 type value euclidean_accuracy 99.74356435643564 type value euclidean_ap 92.40332894384368 type value euclidean_f1 86.97838109602817 type value euclidean_precision 87.46208291203236 type value euclidean_recall 86.5 type value manhattan_accuracy 99.73069306930694 type value manhattan_ap 92.01320815721121 type value manhattan_f1 86.4135864135864 type value manhattan_precision 86.32734530938124 type value manhattan_recall 86.5 type value max_accuracy 99.74653465346535 type value max_ap 93.01476369929063 type value max_f1 86.97838109602817 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering MTEB StackExchangeClustering default test 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 type value v_measure 55.2660514302523 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering-p2p MTEB StackExchangeClusteringP2P default test 815ca46b2622cec33ccafc3735d572c266efdb44 type value v_measure 30.4637783572547 task dataset metrics type Reranking type name config split revision mteb/stackoverflowdupquestions-reranking MTEB StackOverflowDupQuestions default test e185fbe320c72810689fc5848eb6114e1ef5ec69 type value map 49.41377758357637 type value mrr 50.138451213818854 task dataset metrics type Summarization type name config split revision mteb/summeval MTEB SummEval default test cda12ad7615edc362dbf25a00fdd61d3b1eaf93c type value cos_sim_pearson 28.887846011166594 type value cos_sim_spearman 30.10823258355903 type value dot_pearson 12.888049550236385 type value dot_spearman 12.827495903098123 task dataset metrics type Retrieval type name config split revision trec-covid MTEB TRECCOVID default test None type value map_at_1 0.21 type value map_at_10 1.667 type value map_at_100 9.15 type value map_at_1000 22.927 type value map_at_3 0.573 type value map_at_5 0.915 type value mrr_at_1 80 type value mrr_at_10 87.167 type value mrr_at_100 87.167 type value mrr_at_1000 87.167 type value mrr_at_3 85.667 type value mrr_at_5 87.167 type value ndcg_at_1 76 type value ndcg_at_10 69.757 type value ndcg_at_100 52.402 type value ndcg_at_1000 47.737 type value ndcg_at_3 71.866 type value ndcg_at_5 72.225 type value precision_at_1 80 type value precision_at_10 75 type value precision_at_100 53.959999999999994 type value precision_at_1000 21.568 type value precision_at_3 76.667 type value precision_at_5 78 type value recall_at_1 0.21 type value recall_at_10 1.9189999999999998 type value recall_at_100 12.589 type value recall_at_1000 45.312000000000005 type value recall_at_3 0.61 type value recall_at_5 1.019 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (sqi-eng) sqi-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.10000000000001 type value f1 90.06 type value precision 89.17333333333333 type value recall 92.10000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fry-eng) fry-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 56.06936416184971 type value f1 50.87508028259473 type value precision 48.97398843930635 type value recall 56.06936416184971 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kur-eng) kur-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 57.3170731707317 type value f1 52.96080139372822 type value precision 51.67861124382864 type value recall 57.3170731707317 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tur-eng) tur-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.3 type value f1 92.67333333333333 type value precision 91.90833333333333 type value recall 94.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (deu-eng) deu-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.7 type value f1 97.07333333333332 type value precision 96.79500000000002 type value recall 97.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nld-eng) nld-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.69999999999999 type value f1 93.2 type value precision 92.48333333333333 type value recall 94.69999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ron-eng) ron-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.9 type value f1 91.26666666666667 type value precision 90.59444444444445 type value recall 92.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ang-eng) ang-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 34.32835820895522 type value f1 29.074180380150533 type value precision 28.068207322920596 type value recall 34.32835820895522 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ido-eng) ido-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 78.5 type value f1 74.3945115995116 type value precision 72.82967843459222 type value recall 78.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (jav-eng) jav-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 66.34146341463415 type value f1 61.2469400518181 type value precision 59.63977756660683 type value recall 66.34146341463415 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (isl-eng) isl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 80.9 type value f1 76.90349206349207 type value precision 75.32921568627451 type value recall 80.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (slv-eng) slv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 84.93317132442284 type value f1 81.92519105034295 type value precision 80.71283920615635 type value recall 84.93317132442284 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cym-eng) cym-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 71.1304347826087 type value f1 65.22394755003451 type value precision 62.912422360248435 type value recall 71.1304347826087 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kaz-eng) kaz-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 79.82608695652173 type value f1 75.55693581780538 type value precision 73.79420289855072 type value recall 79.82608695652173 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (est-eng) est-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 74 type value f1 70.51022222222223 type value precision 69.29673599347512 type value recall 74 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (heb-eng) heb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 78.7 type value f1 74.14238095238095 type value precision 72.27214285714285 type value recall 78.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gla-eng) gla-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 48.97466827503016 type value f1 43.080330405420874 type value precision 41.36505499593557 type value recall 48.97466827503016 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mar-eng) mar-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 89.60000000000001 type value f1 86.62333333333333 type value precision 85.225 type value recall 89.60000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lat-eng) lat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 45.2 type value f1 39.5761253006253 type value precision 37.991358436312 type value recall 45.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bel-eng) bel-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 89.5 type value f1 86.70333333333333 type value precision 85.53166666666667 type value recall 89.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pms-eng) pms-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 50.095238095238095 type value f1 44.60650460650461 type value precision 42.774116796477045 type value recall 50.095238095238095 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gle-eng) gle-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 63.4 type value f1 58.35967261904762 type value precision 56.54857142857143 type value recall 63.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pes-eng) pes-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 89.2 type value f1 87.075 type value precision 86.12095238095239 type value recall 89.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nob-eng) nob-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96.8 type value f1 95.90333333333334 type value precision 95.50833333333333 type value recall 96.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bul-eng) bul-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 90.9 type value f1 88.6288888888889 type value precision 87.61607142857142 type value recall 90.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cbk-eng) cbk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 65.2 type value f1 60.54377630539395 type value precision 58.89434482711381 type value recall 65.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hun-eng) hun-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 87 type value f1 84.32412698412699 type value precision 83.25527777777778 type value recall 87 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (uig-eng) uig-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 68.7 type value f1 63.07883541295306 type value precision 61.06117424242426 type value recall 68.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (rus-eng) rus-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.7 type value f1 91.78333333333335 type value precision 90.86666666666667 type value recall 93.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (spa-eng) spa-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.7 type value f1 96.96666666666667 type value precision 96.61666666666667 type value recall 97.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hye-eng) hye-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 88.27493261455525 type value f1 85.90745732255168 type value precision 84.91389637616052 type value recall 88.27493261455525 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tel-eng) tel-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 90.5982905982906 type value f1 88.4900284900285 type value precision 87.57122507122507 type value recall 90.5982905982906 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (afr-eng) afr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 89.5 type value f1 86.90769841269842 type value precision 85.80178571428571 type value recall 89.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mon-eng) mon-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 82.5 type value f1 78.36796536796538 type value precision 76.82196969696969 type value recall 82.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (arz-eng) arz-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 71.48846960167715 type value f1 66.78771089148448 type value precision 64.98302885095339 type value recall 71.48846960167715 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hrv-eng) hrv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.1 type value f1 92.50333333333333 type value precision 91.77499999999999 type value recall 94.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nov-eng) nov-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 71.20622568093385 type value f1 66.83278891450098 type value precision 65.35065777283677 type value recall 71.20622568093385 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gsw-eng) gsw-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 48.717948717948715 type value f1 43.53146853146853 type value precision 42.04721204721204 type value recall 48.717948717948715 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nds-eng) nds-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 58.5 type value f1 53.8564991863928 type value precision 52.40329436122275 type value recall 58.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ukr-eng) ukr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 90.8 type value f1 88.29 type value precision 87.09166666666667 type value recall 90.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (uzb-eng) uzb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 67.28971962616822 type value f1 62.63425307817832 type value precision 60.98065939771546 type value recall 67.28971962616822 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lit-eng) lit-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 78.7 type value f1 75.5264472455649 type value precision 74.38205086580086 type value recall 78.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ina-eng) ina-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 88.7 type value f1 86.10809523809525 type value precision 85.07602564102565 type value recall 88.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lfn-eng) lfn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 56.99999999999999 type value f1 52.85487521402737 type value precision 51.53985162713104 type value recall 56.99999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (zsm-eng) zsm-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94 type value f1 92.45333333333333 type value precision 91.79166666666667 type value recall 94 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ita-eng) ita-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.30000000000001 type value f1 90.61333333333333 type value precision 89.83333333333331 type value recall 92.30000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cmn-eng) cmn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.69999999999999 type value f1 93.34555555555555 type value precision 92.75416666666668 type value recall 94.69999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lvs-eng) lvs-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 80.2 type value f1 76.6563035113035 type value precision 75.3014652014652 type value recall 80.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (glg-eng) glg-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 84.7 type value f1 82.78689263765207 type value precision 82.06705086580087 type value recall 84.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ceb-eng) ceb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 50.33333333333333 type value f1 45.461523661523664 type value precision 43.93545574795575 type value recall 50.33333333333333 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bre-eng) bre-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 6.6000000000000005 type value f1 5.442121400446441 type value precision 5.146630385487529 type value recall 6.6000000000000005 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ben-eng) ben-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 85 type value f1 81.04666666666667 type value precision 79.25 type value recall 85 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swg-eng) swg-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 47.32142857142857 type value f1 42.333333333333336 type value precision 40.69196428571429 type value recall 47.32142857142857 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (arq-eng) arq-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 30.735455543358945 type value f1 26.73616790022338 type value precision 25.397823220451283 type value recall 30.735455543358945 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kab-eng) kab-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 25.1 type value f1 21.975989896371022 type value precision 21.059885632257203 type value recall 25.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fra-eng) fra-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.3 type value f1 92.75666666666666 type value precision 92.06166666666665 type value recall 94.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (por-eng) por-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.1 type value f1 92.74 type value precision 92.09166666666667 type value recall 94.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tat-eng) tat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 71.3 type value f1 66.922442002442 type value precision 65.38249567099568 type value recall 71.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (oci-eng) oci-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 40.300000000000004 type value f1 35.78682789299971 type value precision 34.66425128716588 type value recall 40.300000000000004 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pol-eng) pol-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 96 type value f1 94.82333333333334 type value precision 94.27833333333334 type value recall 96 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (war-eng) war-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 51.1 type value f1 47.179074753133584 type value precision 46.06461044702424 type value recall 51.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (aze-eng) aze-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 87.7 type value f1 84.71 type value precision 83.46166666666667 type value recall 87.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (vie-eng) vie-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.8 type value f1 94.68333333333334 type value precision 94.13333333333334 type value recall 95.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nno-eng) nno-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 85.39999999999999 type value f1 82.5577380952381 type value precision 81.36833333333334 type value recall 85.39999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cha-eng) cha-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 21.16788321167883 type value f1 16.948865627297987 type value precision 15.971932568647897 type value recall 21.16788321167883 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mhr-eng) mhr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 6.9 type value f1 5.515526831658907 type value precision 5.141966366966367 type value recall 6.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dan-eng) dan-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.2 type value f1 91.39666666666668 type value precision 90.58666666666667 type value recall 93.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ell-eng) ell-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.2 type value f1 89.95666666666666 type value precision 88.92833333333333 type value recall 92.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (amh-eng) amh-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 79.76190476190477 type value f1 74.93386243386244 type value precision 73.11011904761904 type value recall 79.76190476190477 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pam-eng) pam-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 8.799999999999999 type value f1 6.921439712248537 type value precision 6.489885109680683 type value recall 8.799999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hsb-eng) hsb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 45.75569358178054 type value f1 40.34699501312631 type value precision 38.57886764719063 type value recall 45.75569358178054 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (srp-eng) srp-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 91.4 type value f1 89.08333333333333 type value precision 88.01666666666668 type value recall 91.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (epo-eng) epo-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.60000000000001 type value f1 92.06690476190477 type value precision 91.45095238095239 type value recall 93.60000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kzj-eng) kzj-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 7.5 type value f1 6.200363129378736 type value precision 5.89115314822466 type value recall 7.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (awa-eng) awa-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 73.59307359307358 type value f1 68.38933553219267 type value precision 66.62698412698413 type value recall 73.59307359307358 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fao-eng) fao-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 69.8473282442748 type value f1 64.72373682297346 type value precision 62.82834214131924 type value recall 69.8473282442748 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mal-eng) mal-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 97.5254730713246 type value f1 96.72489082969432 type value precision 96.33672974284326 type value recall 97.5254730713246 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ile-eng) ile-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 75.6 type value f1 72.42746031746033 type value precision 71.14036630036631 type value recall 75.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bos-eng) bos-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 91.24293785310734 type value f1 88.86064030131826 type value precision 87.73540489642184 type value recall 91.24293785310734 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cor-eng) cor-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 6.2 type value f1 4.383083659794954 type value precision 4.027861324289673 type value recall 6.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cat-eng) cat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 86.8 type value f1 84.09428571428572 type value precision 83.00333333333333 type value recall 86.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (eus-eng) eus-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 60.699999999999996 type value f1 56.1584972394755 type value precision 54.713456330903135 type value recall 60.699999999999996 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (yue-eng) yue-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 84.2 type value f1 80.66190476190475 type value precision 79.19690476190476 type value recall 84.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swe-eng) swe-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 93.2 type value f1 91.33 type value precision 90.45 type value recall 93.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dtp-eng) dtp-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 6.3 type value f1 5.126828976748276 type value precision 4.853614328966668 type value recall 6.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kat-eng) kat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 81.76943699731903 type value f1 77.82873739308057 type value precision 76.27622452019234 type value recall 81.76943699731903 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (jpn-eng) jpn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.30000000000001 type value f1 90.29666666666665 type value precision 89.40333333333334 type value recall 92.30000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (csb-eng) csb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 29.249011857707508 type value f1 24.561866096392947 type value precision 23.356583740215456 type value recall 29.249011857707508 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (xho-eng) xho-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 77.46478873239437 type value f1 73.23943661971832 type value precision 71.66666666666667 type value recall 77.46478873239437 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (orv-eng) orv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 20.35928143712575 type value f1 15.997867865075824 type value precision 14.882104658301346 type value recall 20.35928143712575 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ind-eng) ind-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 92.2 type value f1 90.25999999999999 type value precision 89.45333333333335 type value recall 92.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tuk-eng) tuk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 23.15270935960591 type value f1 19.65673625772148 type value precision 18.793705293464992 type value recall 23.15270935960591 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (max-eng) max-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 59.154929577464785 type value f1 52.3868463305083 type value precision 50.14938113529662 type value recall 59.154929577464785 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swh-eng) swh-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 70.51282051282051 type value f1 66.8089133089133 type value precision 65.37645687645687 type value recall 70.51282051282051 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hin-eng) hin-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 94.6 type value f1 93 type value precision 92.23333333333333 type value recall 94.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dsb-eng) dsb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 38.62212943632568 type value f1 34.3278276962583 type value precision 33.07646935732408 type value recall 38.62212943632568 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ber-eng) ber-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 28.1 type value f1 23.579609223054604 type value precision 22.39622774921555 type value recall 28.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tam-eng) tam-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 88.27361563517914 type value f1 85.12486427795874 type value precision 83.71335504885994 type value recall 88.27361563517914 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (slk-eng) slk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 88.6 type value f1 86.39928571428571 type value precision 85.4947557997558 type value recall 88.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tgl-eng) tgl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 86.5 type value f1 83.77952380952381 type value precision 82.67602564102565 type value recall 86.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ast-eng) ast-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 79.52755905511812 type value f1 75.3055868016498 type value precision 73.81889763779527 type value recall 79.52755905511812 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mkd-eng) mkd-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 77.9 type value f1 73.76261904761905 type value precision 72.11670995670995 type value recall 77.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (khm-eng) khm-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 53.8781163434903 type value f1 47.25804051288816 type value precision 45.0603482390186 type value recall 53.8781163434903 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ces-eng) ces-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 91.10000000000001 type value f1 88.88 type value precision 87.96333333333334 type value recall 91.10000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tzl-eng) tzl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 38.46153846153847 type value f1 34.43978243978244 type value precision 33.429487179487175 type value recall 38.46153846153847 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (urd-eng) urd-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 88.9 type value f1 86.19888888888887 type value precision 85.07440476190476 type value recall 88.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ara-eng) ara-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 85.9 type value f1 82.58857142857143 type value precision 81.15666666666667 type value recall 85.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kor-eng) kor-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 86.8 type value f1 83.36999999999999 type value precision 81.86833333333333 type value recall 86.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (yid-eng) yid-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 68.51415094339622 type value f1 63.195000099481234 type value precision 61.394033442972116 type value recall 68.51415094339622 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fin-eng) fin-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 88.5 type value f1 86.14603174603175 type value precision 85.1162037037037 type value recall 88.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tha-eng) tha-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 95.62043795620438 type value f1 94.40389294403892 type value precision 93.7956204379562 type value recall 95.62043795620438 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (wuu-eng) wuu-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553 type value accuracy 81.8 type value f1 78.6532178932179 type value precision 77.46348795840176 type value recall 81.8 task dataset metrics type Retrieval type name config split revision webis-touche2020 MTEB Touche2020 default test None type value map_at_1 2.603 type value map_at_10 8.5 type value map_at_100 12.985 type value map_at_1000 14.466999999999999 type value map_at_3 4.859999999999999 type value map_at_5 5.817 type value mrr_at_1 28.571 type value mrr_at_10 42.331 type value mrr_at_100 43.592999999999996 type value mrr_at_1000 43.592999999999996 type value mrr_at_3 38.435 type value mrr_at_5 39.966 type value ndcg_at_1 26.531 type value ndcg_at_10 21.353 type value ndcg_at_100 31.087999999999997 type value ndcg_at_1000 43.163000000000004 type value ndcg_at_3 22.999 type value ndcg_at_5 21.451 type value precision_at_1 28.571 type value precision_at_10 19.387999999999998 type value precision_at_100 6.265 type value precision_at_1000 1.4160000000000001 type value precision_at_3 24.490000000000002 type value precision_at_5 21.224 type value recall_at_1 2.603 type value recall_at_10 14.474 type value recall_at_100 40.287 type value recall_at_1000 76.606 type value recall_at_3 5.978 type value recall_at_5 7.819 task dataset metrics type Classification type name config split revision mteb/toxic_conversations_50k MTEB ToxicConversationsClassification default test d7c0de2777da35d6aae2200a62c6e0e5af397c4c type value accuracy 69.7848 type value ap 13.661023167088224 type value f1 53.61686134460943 task dataset metrics type Classification type name config split revision mteb/tweet_sentiment_extraction MTEB TweetSentimentExtractionClassification default test d604517c81ca91fe16a244d1248fc021f9ecee7a type value accuracy 61.28183361629882 type value f1 61.55481034919965 task dataset metrics type Clustering type name config split revision mteb/twentynewsgroups-clustering MTEB TwentyNewsgroupsClustering default test 6125ec4e24fa026cec8a478383ee943acfbd5449 type value v_measure 35.972128420092396 task dataset metrics type PairClassification type name config split revision mteb/twittersemeval2015-pairclassification MTEB TwitterSemEval2015 default test 70970daeab8776df92f5ea462b6173c0b46fd2d1 type value cos_sim_accuracy 85.59933241938367 type value cos_sim_ap 72.20760361208136 type value cos_sim_f1 66.4447731755424 type value cos_sim_precision 62.35539102267469 type value cos_sim_recall 71.10817941952506 type value dot_accuracy 78.98313166835548 type value dot_ap 44.492521645493795 type value dot_f1 45.814889336016094 type value dot_precision 37.02439024390244 type value dot_recall 60.07915567282321 type value euclidean_accuracy 85.3907134767837 type value euclidean_ap 71.53847289080343 type value euclidean_f1 65.95952206778834 type value euclidean_precision 61.31006346328196 type value euclidean_recall 71.37203166226914 type value manhattan_accuracy 85.40859510043511 type value manhattan_ap 71.49664104395515 type value manhattan_f1 65.98569969356485 type value manhattan_precision 63.928748144482924 type value manhattan_recall 68.17941952506597 type value max_accuracy 85.59933241938367 type value max_ap 72.20760361208136 type value max_f1 66.4447731755424 task dataset metrics type PairClassification type name config split revision mteb/twitterurlcorpus-pairclassification MTEB TwitterURLCorpus default test 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf type value cos_sim_accuracy 88.83261536073273 type value cos_sim_ap 85.48178133644264 type value cos_sim_f1 77.87816307403935 type value cos_sim_precision 75.88953021114926 type value cos_sim_recall 79.97382198952879 type value dot_accuracy 79.76287499514883 type value dot_ap 59.17438838475084 type value dot_f1 56.34566667855996 type value dot_precision 52.50349092359864 type value dot_recall 60.794579611949494 type value euclidean_accuracy 88.76857996662397 type value euclidean_ap 85.22764834359887 type value euclidean_f1 77.65379751543554 type value euclidean_precision 75.11152683839401 type value euclidean_recall 80.37419156144134 type value manhattan_accuracy 88.6987231730508 type value manhattan_ap 85.18907981724007 type value manhattan_f1 77.51967028849757 type value manhattan_precision 75.49992701795358 type value manhattan_recall 79.65044656606098 type value max_accuracy 88.83261536073273 type value max_ap 85.48178133644264 type value max_f1 77.87816307403935	multilingual af am ar as az be bg bn br bs ca cs cy da de el en eo es et eu fa fi fr fy ga gd gl gu ha he hi hr hu hy id is it ja jv ka kk km kn ko ku ky la lo lt lv mg mk ml mn mr ms my ne nl no om or pa pl ps pt ro ru sa sd si sk sl so sq sr su sv sw ta te th tl tr ug uk ur uz vi xh yi zh	mit

Multilingual-E5-base

Multilingual E5 Text Embeddings: A Technical Report. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024

This model has 12 layers and the embedding size is 768.

Usage

Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset.

import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def average_pool(last_hidden_states: Tensor,
                 attention_mask: Tensor) -> Tensor:
    last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
    return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]


# Each input text should start with "query: " or "passage: ", even for non-English texts.
# For tasks other than retrieval, you can simply use the "query: " prefix.
input_texts = ['query: how much protein should a female eat',
               'query: 南瓜的家常做法',
               "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
               "passage: 1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"]

tokenizer = AutoTokenizer.from_pretrained('intfloat/multilingual-e5-base')
model = AutoModel.from_pretrained('intfloat/multilingual-e5-base')

# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')

outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())

Supported Languages

This model is initialized from xlm-roberta-base and continually trained on a mixture of multilingual datasets. It supports 100 languages from xlm-roberta, but low-resource languages may see performance degradation.

Training Details

Initialization: xlm-roberta-base

First stage: contrastive pre-training with weak supervision

Dataset	Weak supervision	# of text pairs
Filtered mC4	(title, page content)	1B
CC News	(title, news content)	400M
NLLB	translation pairs	2.4B
Wikipedia	(hierarchical section title, passage)	150M
Filtered Reddit	(comment, response)	800M
S2ORC	(title, abstract) and citation pairs	100M
Stackexchange	(question, answer)	50M
xP3	(input prompt, response)	80M
Miscellaneous unsupervised SBERT data	-	10M

Second stage: supervised fine-tuning

Dataset	Language	# of text pairs
MS MARCO	English	500k
NQ	English	70k
Trivia QA	English	60k
NLI from SimCSE	English	<300k
ELI5	English	500k
DuReader Retrieval	Chinese	86k
KILT Fever	English	70k
KILT HotpotQA	English	70k
SQuAD	English	87k
Quora	English	150k
Mr. TyDi	11 languages	50k
MIRACL	16 languages	40k

For all labeled datasets, we only use its training set for fine-tuning.

For other training details, please refer to our paper at https://arxiv.org/pdf/2402.05672.

Benchmark Results on Mr. TyDi

Model	Avg MRR@10		ar	bn	en	fi	id	ja	ko	ru	sw	te	th
BM25	33.3		36.7	41.3	15.1	28.8	38.2	21.7	28.1	32.9	39.6	42.4	41.7
mDPR	16.7		26.0	25.8	16.2	11.3	14.6	18.1	21.9	18.5	7.3	10.6	13.5
BM25 + mDPR	41.7		49.1	53.5	28.4	36.5	45.5	35.5	36.2	42.7	40.5	42.0	49.2
multilingual-e5-small	64.4		71.5	66.3	54.5	57.7	63.2	55.4	54.3	60.8	65.4	89.1	70.1
multilingual-e5-base	65.9		72.3	65.0	58.5	60.8	64.9	56.6	55.8	62.7	69.0	86.6	72.7
multilingual-e5-large	70.5		77.5	73.2	60.8	66.8	68.5	62.5	61.6	65.8	72.7	90.2	76.2

MTEB Benchmark Evaluation

Check out unilm/e5 to reproduce evaluation results on the BEIR and MTEB benchmark.

Support for Sentence Transformers

Below is an example for usage with sentence_transformers.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('intfloat/multilingual-e5-base')
input_texts = [
    'query: how much protein should a female eat',
    'query: 南瓜的家常做法',
    "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 i     s 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or traini     ng for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "passage: 1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮     ,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,     放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油     锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀      6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"
]
embeddings = model.encode(input_texts, normalize_embeddings=True)

Package requirements

pip install sentence_transformers~=2.2.2

Contributors: michaelfeil

FAQ

1. Do I need to add the prefix "query: " and "passage: " to input texts?

Yes, this is how the model is trained, otherwise you will see a performance degradation.

Here are some rules of thumb:

• Use "query: " and "passage: " correspondingly for asymmetric tasks such as passage retrieval in open QA, ad-hoc information retrieval.

• Use "query: " prefix for symmetric tasks such as semantic similarity, bitext mining, paraphrase retrieval.

• Use "query: " prefix if you want to use embeddings as features, such as linear probing classification, clustering.

2. Why are my reproduced results slightly different from reported in the model card?

Different versions of transformers and pytorch could cause negligible but non-zero performance differences.

3. Why does the cosine similarity scores distribute around 0.7 to 1.0?

This is a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss.

For text embedding tasks like text retrieval or semantic similarity, what matters is the relative order of the scores instead of the absolute values, so this should not be an issue.

Citation

If you find our paper or models helpful, please consider cite as follows:

@article{wang2024multilingual,
  title={Multilingual E5 Text Embeddings: A Technical Report},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2402.05672},
  year={2024}
}

Limitations

Long texts will be truncated to at most 512 tokens.