sgpt-bloom-7b1-msmarco/README.md

pipeline_tag, tags, model-index

pipeline_tag	tags	model-index
sentence-similarity	sentence-transformers feature-extraction sentence-similarity mteb	name results sgpt-bloom-7b1-msmarco task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en) en test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 68.05970149253731 type value ap 31.640363460776193 type value f1 62.50025574145796 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (de) de test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 61.34903640256959 type value ap 75.18797161500426 type value f1 59.04772570730417 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en-ext) en-ext test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 67.78110944527737 type value ap 19.218916023322706 type value f1 56.24477391445512 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (ja) ja test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 58.23340471092078 type value ap 13.20222967424681 type value f1 47.511718095460296 task dataset metrics type Classification type name config split revision mteb/amazon_polarity MTEB AmazonPolarityClassification default test 80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1 type value accuracy 68.97232499999998 type value ap 63.53632885535693 type value f1 68.62038513152868 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (en) en test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 33.855999999999995 type value f1 33.43468222830134 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (de) de test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 29.697999999999997 type value f1 29.39935388885501 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (es) es test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 35.974000000000004 type value f1 35.25910820714383 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (fr) fr test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 35.922 type value f1 35.38637028933444 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (ja) ja test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 27.636 type value f1 27.178349955978266 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (zh) zh test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 32.632 type value f1 32.08014766494587 task dataset metrics type Retrieval type name config split revision arguana MTEB ArguAna default test 5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3 type value map_at_1 23.684 type value map_at_10 38.507999999999996 type value map_at_100 39.677 type value map_at_1000 39.690999999999995 type value map_at_3 33.369 type value map_at_5 36.15 type value mrr_at_1 24.04 type value mrr_at_10 38.664 type value mrr_at_100 39.833 type value mrr_at_1000 39.847 type value mrr_at_3 33.476 type value mrr_at_5 36.306 type value ndcg_at_1 23.684 type value ndcg_at_10 47.282000000000004 type value ndcg_at_100 52.215 type value ndcg_at_1000 52.551 type value ndcg_at_3 36.628 type value ndcg_at_5 41.653 type value precision_at_1 23.684 type value precision_at_10 7.553 type value precision_at_100 0.97 type value precision_at_1000 0.1 type value precision_at_3 15.363 type value precision_at_5 11.664 type value recall_at_1 23.684 type value recall_at_10 75.533 type value recall_at_100 97.013 type value recall_at_1000 99.57300000000001 type value recall_at_3 46.088 type value recall_at_5 58.321 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-p2p MTEB ArxivClusteringP2P default test 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8 type value v_measure 44.59375023881131 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-s2s MTEB ArxivClusteringS2S default test b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3 type value v_measure 38.02921907752556 task dataset metrics type Reranking type name config split revision mteb/askubuntudupquestions-reranking MTEB AskUbuntuDupQuestions default test 4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c type value map 59.97321570342109 type value mrr 73.18284746955106 task dataset metrics type STS type name config split revision mteb/biosses-sts MTEB BIOSSES default test 9ee918f184421b6bd48b78f6c714d86546106103 type value cos_sim_pearson 89.09091435741429 type value cos_sim_spearman 85.31459455332202 type value euclidean_pearson 79.3587681410798 type value euclidean_spearman 76.8174129874685 type value manhattan_pearson 79.57051762121769 type value manhattan_spearman 76.75837549768094 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (de-en) de-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 54.27974947807933 type value f1 54.00144411132214 type value precision 53.87119374071357 type value recall 54.27974947807933 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (fr-en) fr-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 97.3365617433414 type value f1 97.06141316310809 type value precision 96.92567319685965 type value recall 97.3365617433414 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (ru-en) ru-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 46.05472809144441 type value f1 45.30319274690595 type value precision 45.00015469655234 type value recall 46.05472809144441 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (zh-en) zh-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 98.10426540284361 type value f1 97.96384061786905 type value precision 97.89362822538178 type value recall 98.10426540284361 task dataset metrics type Classification type name config split revision mteb/banking77 MTEB Banking77Classification default test 44fa15921b4c889113cc5df03dd4901b49161ab7 type value accuracy 84.33441558441558 type value f1 84.31653077470322 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-p2p MTEB BiorxivClusteringP2P default test 11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55 type value v_measure 36.025318694698086 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-s2s MTEB BiorxivClusteringS2S default test c0fab014e1bcb8d3a5e31b2088972a1e01547dc1 type value v_measure 32.484889034590346 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackAndroidRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 30.203999999999997 type value map_at_10 41.314 type value map_at_100 42.66 type value map_at_1000 42.775999999999996 type value map_at_3 37.614999999999995 type value map_at_5 39.643 type value mrr_at_1 37.482 type value mrr_at_10 47.075 type value mrr_at_100 47.845 type value mrr_at_1000 47.887 type value mrr_at_3 44.635000000000005 type value mrr_at_5 45.966 type value ndcg_at_1 37.482 type value ndcg_at_10 47.676 type value ndcg_at_100 52.915 type value ndcg_at_1000 54.82900000000001 type value ndcg_at_3 42.562 type value ndcg_at_5 44.852 type value precision_at_1 37.482 type value precision_at_10 9.142 type value precision_at_100 1.436 type value precision_at_1000 0.189 type value precision_at_3 20.458000000000002 type value precision_at_5 14.821000000000002 type value recall_at_1 30.203999999999997 type value recall_at_10 60.343 type value recall_at_100 82.58 type value recall_at_1000 94.813 type value recall_at_3 45.389 type value recall_at_5 51.800999999999995 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackEnglishRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 30.889 type value map_at_10 40.949999999999996 type value map_at_100 42.131 type value map_at_1000 42.253 type value map_at_3 38.346999999999994 type value map_at_5 39.782000000000004 type value mrr_at_1 38.79 type value mrr_at_10 46.944 type value mrr_at_100 47.61 type value mrr_at_1000 47.650999999999996 type value mrr_at_3 45.053 type value mrr_at_5 46.101 type value ndcg_at_1 38.79 type value ndcg_at_10 46.286 type value ndcg_at_100 50.637 type value ndcg_at_1000 52.649 type value ndcg_at_3 42.851 type value ndcg_at_5 44.311 type value precision_at_1 38.79 type value precision_at_10 8.516 type value precision_at_100 1.3679999999999999 type value precision_at_1000 0.183 type value precision_at_3 20.637 type value precision_at_5 14.318 type value recall_at_1 30.889 type value recall_at_10 55.327000000000005 type value recall_at_100 74.091 type value recall_at_1000 86.75500000000001 type value recall_at_3 44.557 type value recall_at_5 49.064 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGamingRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 39.105000000000004 type value map_at_10 50.928 type value map_at_100 51.958000000000006 type value map_at_1000 52.017 type value map_at_3 47.638999999999996 type value map_at_5 49.624 type value mrr_at_1 44.639 type value mrr_at_10 54.261 type value mrr_at_100 54.913999999999994 type value mrr_at_1000 54.945 type value mrr_at_3 51.681999999999995 type value mrr_at_5 53.290000000000006 type value ndcg_at_1 44.639 type value ndcg_at_10 56.678 type value ndcg_at_100 60.649 type value ndcg_at_1000 61.855000000000004 type value ndcg_at_3 51.092999999999996 type value ndcg_at_5 54.096999999999994 type value precision_at_1 44.639 type value precision_at_10 9.028 type value precision_at_100 1.194 type value precision_at_1000 0.135 type value precision_at_3 22.508 type value precision_at_5 15.661 type value recall_at_1 39.105000000000004 type value recall_at_10 70.367 type value recall_at_100 87.359 type value recall_at_1000 95.88 type value recall_at_3 55.581 type value recall_at_5 62.821000000000005 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGisRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 23.777 type value map_at_10 32.297 type value map_at_100 33.516 type value map_at_1000 33.592 type value map_at_3 30.001 type value map_at_5 31.209999999999997 type value mrr_at_1 25.989 type value mrr_at_10 34.472 type value mrr_at_100 35.518 type value mrr_at_1000 35.577 type value mrr_at_3 32.185 type value mrr_at_5 33.399 type value ndcg_at_1 25.989 type value ndcg_at_10 37.037 type value ndcg_at_100 42.699 type value ndcg_at_1000 44.725 type value ndcg_at_3 32.485 type value ndcg_at_5 34.549 type value precision_at_1 25.989 type value precision_at_10 5.718 type value precision_at_100 0.89 type value precision_at_1000 0.11 type value precision_at_3 14.049 type value precision_at_5 9.672 type value recall_at_1 23.777 type value recall_at_10 49.472 type value recall_at_100 74.857 type value recall_at_1000 90.289 type value recall_at_3 37.086000000000006 type value recall_at_5 42.065999999999995 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackMathematicaRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 13.377 type value map_at_10 21.444 type value map_at_100 22.663 type value map_at_1000 22.8 type value map_at_3 18.857 type value map_at_5 20.426 type value mrr_at_1 16.542 type value mrr_at_10 25.326999999999998 type value mrr_at_100 26.323 type value mrr_at_1000 26.406000000000002 type value mrr_at_3 22.823 type value mrr_at_5 24.340999999999998 type value ndcg_at_1 16.542 type value ndcg_at_10 26.479000000000003 type value ndcg_at_100 32.29 type value ndcg_at_1000 35.504999999999995 type value ndcg_at_3 21.619 type value ndcg_at_5 24.19 type value precision_at_1 16.542 type value precision_at_10 5.075 type value precision_at_100 0.9339999999999999 type value precision_at_1000 0.135 type value precision_at_3 10.697 type value precision_at_5 8.134 type value recall_at_1 13.377 type value recall_at_10 38.027 type value recall_at_100 63.439 type value recall_at_1000 86.354 type value recall_at_3 25.0 type value recall_at_5 31.306 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackPhysicsRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 28.368 type value map_at_10 39.305 type value map_at_100 40.637 type value map_at_1000 40.753 type value map_at_3 36.077999999999996 type value map_at_5 37.829 type value mrr_at_1 34.937000000000005 type value mrr_at_10 45.03 type value mrr_at_100 45.78 type value mrr_at_1000 45.827 type value mrr_at_3 42.348 type value mrr_at_5 43.807 type value ndcg_at_1 34.937000000000005 type value ndcg_at_10 45.605000000000004 type value ndcg_at_100 50.941 type value ndcg_at_1000 52.983000000000004 type value ndcg_at_3 40.366 type value ndcg_at_5 42.759 type value precision_at_1 34.937000000000005 type value precision_at_10 8.402 type value precision_at_100 1.2959999999999998 type value precision_at_1000 0.164 type value precision_at_3 19.217000000000002 type value precision_at_5 13.725000000000001 type value recall_at_1 28.368 type value recall_at_10 58.5 type value recall_at_100 80.67999999999999 type value recall_at_1000 93.925 type value recall_at_3 43.956 type value recall_at_5 50.065000000000005 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackProgrammersRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 24.851 type value map_at_10 34.758 type value map_at_100 36.081 type value map_at_1000 36.205999999999996 type value map_at_3 31.678 type value map_at_5 33.398 type value mrr_at_1 31.279 type value mrr_at_10 40.138 type value mrr_at_100 41.005 type value mrr_at_1000 41.065000000000005 type value mrr_at_3 37.519000000000005 type value mrr_at_5 38.986 type value ndcg_at_1 31.279 type value ndcg_at_10 40.534 type value ndcg_at_100 46.093 type value ndcg_at_1000 48.59 type value ndcg_at_3 35.473 type value ndcg_at_5 37.801 type value precision_at_1 31.279 type value precision_at_10 7.477 type value precision_at_100 1.2 type value precision_at_1000 0.159 type value precision_at_3 17.047 type value precision_at_5 12.306000000000001 type value recall_at_1 24.851 type value recall_at_10 52.528 type value recall_at_100 76.198 type value recall_at_1000 93.12 type value recall_at_3 38.257999999999996 type value recall_at_5 44.440000000000005 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 25.289833333333334 type value map_at_10 34.379333333333335 type value map_at_100 35.56916666666666 type value map_at_1000 35.68633333333333 type value map_at_3 31.63916666666666 type value map_at_5 33.18383333333334 type value mrr_at_1 30.081749999999996 type value mrr_at_10 38.53658333333333 type value mrr_at_100 39.37825 type value mrr_at_1000 39.43866666666666 type value mrr_at_3 36.19025 type value mrr_at_5 37.519749999999995 type value ndcg_at_1 30.081749999999996 type value ndcg_at_10 39.62041666666667 type value ndcg_at_100 44.74825 type value ndcg_at_1000 47.11366666666667 type value ndcg_at_3 35.000499999999995 type value ndcg_at_5 37.19283333333333 type value precision_at_1 30.081749999999996 type value precision_at_10 6.940249999999999 type value precision_at_100 1.1164166666666668 type value precision_at_1000 0.15025000000000002 type value precision_at_3 16.110416666666666 type value precision_at_5 11.474416666666668 type value recall_at_1 25.289833333333334 type value recall_at_10 51.01591666666667 type value recall_at_100 73.55275000000002 type value recall_at_1000 90.02666666666667 type value recall_at_3 38.15208333333334 type value recall_at_5 43.78458333333334 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackStatsRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 23.479 type value map_at_10 31.2 type value map_at_100 32.11 type value map_at_1000 32.214 type value map_at_3 29.093999999999998 type value map_at_5 30.415 type value mrr_at_1 26.840000000000003 type value mrr_at_10 34.153 type value mrr_at_100 34.971000000000004 type value mrr_at_1000 35.047 type value mrr_at_3 32.285000000000004 type value mrr_at_5 33.443 type value ndcg_at_1 26.840000000000003 type value ndcg_at_10 35.441 type value ndcg_at_100 40.150000000000006 type value ndcg_at_1000 42.74 type value ndcg_at_3 31.723000000000003 type value ndcg_at_5 33.71 type value precision_at_1 26.840000000000003 type value precision_at_10 5.552 type value precision_at_100 0.859 type value precision_at_1000 0.11499999999999999 type value precision_at_3 13.804 type value precision_at_5 9.600999999999999 type value recall_at_1 23.479 type value recall_at_10 45.442 type value recall_at_100 67.465 type value recall_at_1000 86.53 type value recall_at_3 35.315999999999995 type value recall_at_5 40.253 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackTexRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 16.887 type value map_at_10 23.805 type value map_at_100 24.804000000000002 type value map_at_1000 24.932000000000002 type value map_at_3 21.632 type value map_at_5 22.845 type value mrr_at_1 20.75 type value mrr_at_10 27.686 type value mrr_at_100 28.522 type value mrr_at_1000 28.605000000000004 type value mrr_at_3 25.618999999999996 type value mrr_at_5 26.723999999999997 type value ndcg_at_1 20.75 type value ndcg_at_10 28.233000000000004 type value ndcg_at_100 33.065 type value ndcg_at_1000 36.138999999999996 type value ndcg_at_3 24.361 type value ndcg_at_5 26.111 type value precision_at_1 20.75 type value precision_at_10 5.124 type value precision_at_100 0.8750000000000001 type value precision_at_1000 0.131 type value precision_at_3 11.539000000000001 type value precision_at_5 8.273 type value recall_at_1 16.887 type value recall_at_10 37.774 type value recall_at_100 59.587 type value recall_at_1000 81.523 type value recall_at_3 26.837 type value recall_at_5 31.456 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackUnixRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 25.534000000000002 type value map_at_10 33.495999999999995 type value map_at_100 34.697 type value map_at_1000 34.805 type value map_at_3 31.22 type value map_at_5 32.277 type value mrr_at_1 29.944 type value mrr_at_10 37.723 type value mrr_at_100 38.645 type value mrr_at_1000 38.712999999999994 type value mrr_at_3 35.665 type value mrr_at_5 36.681999999999995 type value ndcg_at_1 29.944 type value ndcg_at_10 38.407000000000004 type value ndcg_at_100 43.877 type value ndcg_at_1000 46.312 type value ndcg_at_3 34.211000000000006 type value ndcg_at_5 35.760999999999996 type value precision_at_1 29.944 type value precision_at_10 6.343 type value precision_at_100 1.023 type value precision_at_1000 0.133 type value precision_at_3 15.360999999999999 type value precision_at_5 10.428999999999998 type value recall_at_1 25.534000000000002 type value recall_at_10 49.204 type value recall_at_100 72.878 type value recall_at_1000 89.95 type value recall_at_3 37.533 type value recall_at_5 41.611 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWebmastersRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 26.291999999999998 type value map_at_10 35.245 type value map_at_100 36.762 type value map_at_1000 36.983 type value map_at_3 32.439 type value map_at_5 33.964 type value mrr_at_1 31.423000000000002 type value mrr_at_10 39.98 type value mrr_at_100 40.791 type value mrr_at_1000 40.854 type value mrr_at_3 37.451 type value mrr_at_5 38.854 type value ndcg_at_1 31.423000000000002 type value ndcg_at_10 40.848 type value ndcg_at_100 46.35 type value ndcg_at_1000 49.166 type value ndcg_at_3 36.344 type value ndcg_at_5 38.36 type value precision_at_1 31.423000000000002 type value precision_at_10 7.767 type value precision_at_100 1.498 type value precision_at_1000 0.23700000000000002 type value precision_at_3 16.733 type value precision_at_5 12.213000000000001 type value recall_at_1 26.291999999999998 type value recall_at_10 51.184 type value recall_at_100 76.041 type value recall_at_1000 94.11500000000001 type value recall_at_3 38.257000000000005 type value recall_at_5 43.68 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWordpressRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 20.715 type value map_at_10 27.810000000000002 type value map_at_100 28.810999999999996 type value map_at_1000 28.904999999999998 type value map_at_3 25.069999999999997 type value map_at_5 26.793 type value mrr_at_1 22.366 type value mrr_at_10 29.65 type value mrr_at_100 30.615 type value mrr_at_1000 30.686999999999998 type value mrr_at_3 27.017999999999997 type value mrr_at_5 28.644 type value ndcg_at_1 22.366 type value ndcg_at_10 32.221 type value ndcg_at_100 37.313 type value ndcg_at_1000 39.871 type value ndcg_at_3 26.918 type value ndcg_at_5 29.813000000000002 type value precision_at_1 22.366 type value precision_at_10 5.139 type value precision_at_100 0.8240000000000001 type value precision_at_1000 0.11199999999999999 type value precision_at_3 11.275 type value precision_at_5 8.540000000000001 type value recall_at_1 20.715 type value recall_at_10 44.023 type value recall_at_100 67.458 type value recall_at_1000 87.066 type value recall_at_3 30.055 type value recall_at_5 36.852000000000004 task dataset metrics type Retrieval type name config split revision climate-fever MTEB ClimateFEVER default test 392b78eb68c07badcd7c2cd8f39af108375dfcce type value map_at_1 11.859 type value map_at_10 20.625 type value map_at_100 22.5 type value map_at_1000 22.689 type value map_at_3 16.991 type value map_at_5 18.781 type value mrr_at_1 26.906000000000002 type value mrr_at_10 39.083 type value mrr_at_100 39.978 type value mrr_at_1000 40.014 type value mrr_at_3 35.44 type value mrr_at_5 37.619 type value ndcg_at_1 26.906000000000002 type value ndcg_at_10 29.386000000000003 type value ndcg_at_100 36.510999999999996 type value ndcg_at_1000 39.814 type value ndcg_at_3 23.558 type value ndcg_at_5 25.557999999999996 type value precision_at_1 26.906000000000002 type value precision_at_10 9.342 type value precision_at_100 1.6969999999999998 type value precision_at_1000 0.231 type value precision_at_3 17.503 type value precision_at_5 13.655000000000001 type value recall_at_1 11.859 type value recall_at_10 35.929 type value recall_at_100 60.21300000000001 type value recall_at_1000 78.606 type value recall_at_3 21.727 type value recall_at_5 27.349 task dataset metrics type Retrieval type name config split revision dbpedia-entity MTEB DBPedia default test f097057d03ed98220bc7309ddb10b71a54d667d6 type value map_at_1 8.627 type value map_at_10 18.248 type value map_at_100 25.19 type value map_at_1000 26.741 type value map_at_3 13.286000000000001 type value map_at_5 15.126000000000001 type value mrr_at_1 64.75 type value mrr_at_10 71.865 type value mrr_at_100 72.247 type value mrr_at_1000 72.255 type value mrr_at_3 69.958 type value mrr_at_5 71.108 type value ndcg_at_1 53.25 type value ndcg_at_10 39.035 type value ndcg_at_100 42.735 type value ndcg_at_1000 50.166 type value ndcg_at_3 43.857 type value ndcg_at_5 40.579 type value precision_at_1 64.75 type value precision_at_10 30.75 type value precision_at_100 9.54 type value precision_at_1000 2.035 type value precision_at_3 47.333 type value precision_at_5 39.0 type value recall_at_1 8.627 type value recall_at_10 23.413 type value recall_at_100 48.037 type value recall_at_1000 71.428 type value recall_at_3 14.158999999999999 type value recall_at_5 17.002 task dataset metrics type Classification type name config split revision mteb/emotion MTEB EmotionClassification default test 829147f8f75a25f005913200eb5ed41fae320aa1 type value accuracy 44.865 type value f1 41.56625743266997 task dataset metrics type Retrieval type name config split revision fever MTEB FEVER default test 1429cf27e393599b8b359b9b72c666f96b2525f9 type value map_at_1 57.335 type value map_at_10 68.29499999999999 type value map_at_100 68.69800000000001 type value map_at_1000 68.714 type value map_at_3 66.149 type value map_at_5 67.539 type value mrr_at_1 61.656 type value mrr_at_10 72.609 type value mrr_at_100 72.923 type value mrr_at_1000 72.928 type value mrr_at_3 70.645 type value mrr_at_5 71.938 type value ndcg_at_1 61.656 type value ndcg_at_10 73.966 type value ndcg_at_100 75.663 type value ndcg_at_1000 75.986 type value ndcg_at_3 69.959 type value ndcg_at_5 72.269 type value precision_at_1 61.656 type value precision_at_10 9.581000000000001 type value precision_at_100 1.054 type value precision_at_1000 0.11 type value precision_at_3 27.743000000000002 type value precision_at_5 17.939 type value recall_at_1 57.335 type value recall_at_10 87.24300000000001 type value recall_at_100 94.575 type value recall_at_1000 96.75399999999999 type value recall_at_3 76.44800000000001 type value recall_at_5 82.122 task dataset metrics type Retrieval type name config split revision fiqa MTEB FiQA2018 default test 41b686a7f28c59bcaaa5791efd47c67c8ebe28be type value map_at_1 17.014000000000003 type value map_at_10 28.469 type value map_at_100 30.178 type value map_at_1000 30.369 type value map_at_3 24.63 type value map_at_5 26.891 type value mrr_at_1 34.259 type value mrr_at_10 43.042 type value mrr_at_100 43.91 type value mrr_at_1000 43.963 type value mrr_at_3 40.483999999999995 type value mrr_at_5 42.135 type value ndcg_at_1 34.259 type value ndcg_at_10 35.836 type value ndcg_at_100 42.488 type value ndcg_at_1000 45.902 type value ndcg_at_3 32.131 type value ndcg_at_5 33.697 type value precision_at_1 34.259 type value precision_at_10 10.0 type value precision_at_100 1.699 type value precision_at_1000 0.22999999999999998 type value precision_at_3 21.502 type value precision_at_5 16.296 type value recall_at_1 17.014000000000003 type value recall_at_10 42.832 type value recall_at_100 67.619 type value recall_at_1000 88.453 type value recall_at_3 29.537000000000003 type value recall_at_5 35.886 task dataset metrics type Retrieval type name config split revision hotpotqa MTEB HotpotQA default test 766870b35a1b9ca65e67a0d1913899973551fc6c type value map_at_1 34.558 type value map_at_10 48.039 type value map_at_100 48.867 type value map_at_1000 48.941 type value map_at_3 45.403 type value map_at_5 46.983999999999995 type value mrr_at_1 69.11500000000001 type value mrr_at_10 75.551 type value mrr_at_100 75.872 type value mrr_at_1000 75.887 type value mrr_at_3 74.447 type value mrr_at_5 75.113 type value ndcg_at_1 69.11500000000001 type value ndcg_at_10 57.25599999999999 type value ndcg_at_100 60.417 type value ndcg_at_1000 61.976 type value ndcg_at_3 53.258 type value ndcg_at_5 55.374 type value precision_at_1 69.11500000000001 type value precision_at_10 11.689 type value precision_at_100 1.418 type value precision_at_1000 0.163 type value precision_at_3 33.018 type value precision_at_5 21.488 type value recall_at_1 34.558 type value recall_at_10 58.447 type value recall_at_100 70.91199999999999 type value recall_at_1000 81.31 type value recall_at_3 49.527 type value recall_at_5 53.72 task dataset metrics type Classification type name config split revision mteb/imdb MTEB ImdbClassification default test 8d743909f834c38949e8323a8a6ce8721ea6c7f4 type value accuracy 61.772000000000006 type value ap 57.48217702943605 type value f1 61.20495351356274 task dataset metrics type Retrieval type name config split revision msmarco MTEB MSMARCO default validation e6838a846e2408f22cf5cc337ebc83e0bcf77849 type value map_at_1 22.044 type value map_at_10 34.211000000000006 type value map_at_100 35.394 type value map_at_1000 35.443000000000005 type value map_at_3 30.318 type value map_at_5 32.535 type value mrr_at_1 22.722 type value mrr_at_10 34.842 type value mrr_at_100 35.954 type value mrr_at_1000 35.997 type value mrr_at_3 30.991000000000003 type value mrr_at_5 33.2 type value ndcg_at_1 22.722 type value ndcg_at_10 41.121 type value ndcg_at_100 46.841 type value ndcg_at_1000 48.049 type value ndcg_at_3 33.173 type value ndcg_at_5 37.145 type value precision_at_1 22.722 type value precision_at_10 6.516 type value precision_at_100 0.9400000000000001 type value precision_at_1000 0.104 type value precision_at_3 14.093 type value precision_at_5 10.473 type value recall_at_1 22.044 type value recall_at_10 62.382000000000005 type value recall_at_100 88.914 type value recall_at_1000 98.099 type value recall_at_3 40.782000000000004 type value recall_at_5 50.322 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (en) en test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 93.68217054263563 type value f1 93.25810075739523 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (de) de test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 82.05409974640745 type value f1 80.42814140324903 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (es) es test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 93.54903268845896 type value f1 92.8909878077932 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (fr) fr test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 90.98340119010334 type value f1 90.51522537281313 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (hi) hi test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 89.33309429903191 type value f1 88.60371305209185 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (th) th test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 60.4882459312839 type value f1 59.02590456131682 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (en) en test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 71.34290925672595 type value f1 54.44803151449109 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (de) de test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 61.92448577063963 type value f1 43.125939975781854 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (es) es test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 74.48965977318213 type value f1 51.855353687466696 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (fr) fr test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 69.11994989038521 type value f1 50.57872704171278 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (hi) hi test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 64.84761563284331 type value f1 43.61322970761394 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (th) th test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 49.35623869801085 type value f1 33.48547326952042 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (af) af test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 47.85474108944183 type value f1 46.50175016795915 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (am) am test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 33.29858776059179 type value f1 31.803027601259082 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ar) ar test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 59.24680564895763 type value f1 57.037691806846865 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (az) az test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 45.23537323470073 type value f1 44.81126398428613 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (bn) bn test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 61.590450571620714 type value f1 59.247442149977104 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (cy) cy test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 44.9226630800269 type value f1 44.076183379991654 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (da) da test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 51.23066577000672 type value f1 50.20719330417618 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (de) de test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 56.0995292535306 type value f1 53.29421532133969 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (el) el test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 46.12642905178211 type value f1 44.441530267639635 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (en) en test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 69.67047747141896 type value f1 68.38493366054783 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (es) es test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 66.3483523873571 type value f1 65.13046416817832 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fa) fa test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 51.20040349697378 type value f1 49.02889836601541 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fi) fi test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 45.33288500336248 type value f1 42.91893101970983 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fr) fr test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 66.95359784801613 type value f1 64.98788914810562 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (he) he test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 43.18090114324143 type value f1 41.31250407417542 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hi) hi test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 63.54068594485541 type value f1 61.94829361488948 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hu) hu test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 44.7343644922663 type value f1 43.23001702247849 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hy) hy test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.1271015467384 type value f1 36.94700198241727 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (id) id test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 64.05514458641561 type value f1 62.35033731674541 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (is) is test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 44.351042367182245 type value f1 43.13370397574502 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (it) it test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 60.77000672494955 type value f1 59.71546868957779 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ja) ja test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 61.22057834566241 type value f1 59.447639306287044 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (jv) jv test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 50.9448554135844 type value f1 48.524338247875214 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ka) ka test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 33.8399462004035 type value f1 33.518999997305535 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (km) km test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 37.34028244788165 type value f1 35.6156599064704 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (kn) kn test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 53.544048419636844 type value f1 51.29299915455352 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ko) ko test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 53.35574983187625 type value f1 51.463936565192945 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (lv) lv test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 46.503026227303295 type value f1 46.049497734375514 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ml) ml test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 58.268325487558826 type value f1 56.10849656896158 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (mn) mn test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 40.27572293207801 type value f1 40.20097238549224 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ms) ms test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 59.64694014794889 type value f1 58.39584148789066 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (my) my test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 37.41761936785474 type value f1 35.04551731363685 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nb) nb test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 49.408204438466704 type value f1 48.39369057638714 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nl) nl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 52.09482178883659 type value f1 49.91518031712698 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pl) pl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 50.477471418964356 type value f1 48.429495257184705 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pt) pt test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 66.69468728984532 type value f1 65.40306868707009 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ro) ro test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 50.52790854068594 type value f1 49.780400354514 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ru) ru test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 58.31540013449899 type value f1 56.144142926685134 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sl) sl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 47.74041694687289 type value f1 46.16767322761359 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sq) sq test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 48.94418291862811 type value f1 48.445352284756325 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sv) sv test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 50.78681909885676 type value f1 49.64882295494536 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sw) sw test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 49.811701412239415 type value f1 48.213234514449375 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ta) ta test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 56.39542703429725 type value f1 54.031981085233795 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (te) te test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 54.71082716879623 type value f1 52.513144113474596 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (th) th test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 44.425016812373904 type value f1 43.96016300057656 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tl) tl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 50.205110961667785 type value f1 48.86669996798709 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tr) tr test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 46.56355077336921 type value f1 45.18252022585022 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ur) ur test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 56.748486886348346 type value f1 54.29884570375382 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (vi) vi test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 64.52589105581708 type value f1 62.97947342861603 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-CN) zh-CN test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 67.06792199058508 type value f1 65.36025601634017 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-TW) zh-TW test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 62.89172831203766 type value f1 62.69803707054342 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (af) af test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 51.47276395427035 type value f1 49.37463208130799 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (am) am test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 34.86886348352387 type value f1 33.74178074349636 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ar) ar test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 65.20511096166778 type value f1 65.85812500602437 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (az) az test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 45.578345662407536 type value f1 44.44514917028003 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (bn) bn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 67.29657027572293 type value f1 67.24477523937466 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (cy) cy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 46.29455279085407 type value f1 43.8563839951935 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (da) da test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 53.52387357094821 type value f1 51.70977848027552 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (de) de test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 61.741761936785466 type value f1 60.219169644792295 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (el) el test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 48.957632817753876 type value f1 46.878428264460034 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (en) en test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 75.33624747814393 type value f1 75.9143846211171 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (es) es test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.34229993275049 type value f1 73.78165397558983 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fa) fa test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 53.174176193678555 type value f1 51.709679227778985 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fi) fi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 44.6906523201076 type value f1 41.54881682785664 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fr) fr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 72.9119031607263 type value f1 73.2742013056326 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (he) he test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 43.10356422326832 type value f1 40.8859122581252 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hi) hi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 69.27370544720914 type value f1 69.39544506405082 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hu) hu test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 45.16476126429052 type value f1 42.74022531579054 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hy) hy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 38.73234700739744 type value f1 37.40546754951026 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (id) id test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.12777404169468 type value f1 70.27219152812738 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (is) is test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 44.21318090114325 type value f1 41.934593213829366 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (it) it test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 65.57162071284466 type value f1 64.83341759045335 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ja) ja test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 65.75991930060525 type value f1 65.16549875504951 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (jv) jv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 54.79488903833223 type value f1 54.03616401426859 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ka) ka test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 32.992602555480836 type value f1 31.820068470018846 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (km) km test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 39.34431741761937 type value f1 36.436221665290105 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (kn) kn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 60.501008742434436 type value f1 60.051013712579085 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ko) ko test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 55.689307330195035 type value f1 53.94058032286942 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (lv) lv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 44.351042367182245 type value f1 42.05421666771541 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ml) ml test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 65.53127101546738 type value f1 65.98462024333497 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (mn) mn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 38.71553463349025 type value f1 37.44327037149584 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ms) ms test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 64.98991257565567 type value f1 63.87720198978004 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (my) my test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 36.839273705447205 type value f1 35.233967279698376 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nb) nb test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 51.79892400806993 type value f1 49.66926632125972 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nl) nl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 56.31809011432415 type value f1 53.832185336179826 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pl) pl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 49.979825151311374 type value f1 48.83013175441888 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pt) pt test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 71.45595158036315 type value f1 72.08708814699702 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ro) ro test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 53.68527236045729 type value f1 52.23278593929981 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ru) ru test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 61.60390047074647 type value f1 60.50391482195116 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sl) sl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 48.036314727639535 type value f1 46.43480413383716 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sq) sq test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 50.05716207128445 type value f1 48.85821859948888 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sv) sv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 51.728312037659705 type value f1 49.89292996950847 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sw) sw test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 54.21990585070613 type value f1 52.8711542984193 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ta) ta test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 62.770679219905844 type value f1 63.09441501491594 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (te) te test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 62.58574310692671 type value f1 61.61370697612978 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (th) th test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 45.17821116341628 type value f1 43.85143229183324 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tl) tl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 52.064559515803644 type value f1 50.94356892049626 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tr) tr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 47.205783456624076 type value f1 47.04223644120489 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ur) ur test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 64.25689307330195 type value f1 63.89944944984115 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (vi) vi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.60524546065905 type value f1 71.5634157334358 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-CN) zh-CN test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 73.95427034297242 type value f1 74.39706882311063 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-TW) zh-TW test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 70.29926025554808 type value f1 71.32045932560297 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-p2p MTEB MedrxivClusteringP2P default test dcefc037ef84348e49b0d29109e891c01067226b type value v_measure 31.054474964883806 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-s2s MTEB MedrxivClusteringS2S default test 3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc type value v_measure 29.259725940477523 task dataset metrics type Reranking type name config split revision mteb/mind_small MTEB MindSmallReranking default test 3bdac13927fdc888b903db93b2ffdbd90b295a69 type value map 31.785007883256572 type value mrr 32.983556622438456 task dataset metrics type Retrieval type name config split revision nfcorpus MTEB NFCorpus default test 7eb63cc0c1eb59324d709ebed25fcab851fa7610 type value map_at_1 5.742 type value map_at_10 13.074 type value map_at_100 16.716 type value map_at_1000 18.238 type value map_at_3 9.600999999999999 type value map_at_5 11.129999999999999 type value mrr_at_1 47.988 type value mrr_at_10 55.958 type value mrr_at_100 56.58800000000001 type value mrr_at_1000 56.620000000000005 type value mrr_at_3 54.025 type value mrr_at_5 55.31 type value ndcg_at_1 46.44 type value ndcg_at_10 35.776 type value ndcg_at_100 32.891999999999996 type value ndcg_at_1000 41.835 type value ndcg_at_3 41.812 type value ndcg_at_5 39.249 type value precision_at_1 48.297000000000004 type value precision_at_10 26.687 type value precision_at_100 8.511000000000001 type value precision_at_1000 2.128 type value precision_at_3 39.009 type value precision_at_5 33.994 type value recall_at_1 5.742 type value recall_at_10 16.993 type value recall_at_100 33.69 type value recall_at_1000 66.75 type value recall_at_3 10.817 type value recall_at_5 13.256 task dataset metrics type Retrieval type name config split revision nq MTEB NQ default test 6062aefc120bfe8ece5897809fb2e53bfe0d128c type value map_at_1 30.789 type value map_at_10 45.751999999999995 type value map_at_100 46.766000000000005 type value map_at_1000 46.798 type value map_at_3 41.746 type value map_at_5 44.046 type value mrr_at_1 34.618 type value mrr_at_10 48.288 type value mrr_at_100 49.071999999999996 type value mrr_at_1000 49.094 type value mrr_at_3 44.979 type value mrr_at_5 46.953 type value ndcg_at_1 34.589 type value ndcg_at_10 53.151 type value ndcg_at_100 57.537000000000006 type value ndcg_at_1000 58.321999999999996 type value ndcg_at_3 45.628 type value ndcg_at_5 49.474000000000004 type value precision_at_1 34.589 type value precision_at_10 8.731 type value precision_at_100 1.119 type value precision_at_1000 0.11900000000000001 type value precision_at_3 20.819 type value precision_at_5 14.728 type value recall_at_1 30.789 type value recall_at_10 73.066 type value recall_at_100 92.27 type value recall_at_1000 98.18 type value recall_at_3 53.632999999999996 type value recall_at_5 62.476 task dataset metrics type Retrieval type name config split revision quora MTEB QuoraRetrieval default test 6205996560df11e3a3da9ab4f926788fc30a7db4 type value map_at_1 54.993 type value map_at_10 69.07600000000001 type value map_at_100 70.05799999999999 type value map_at_1000 70.09 type value map_at_3 65.456 type value map_at_5 67.622 type value mrr_at_1 63.07000000000001 type value mrr_at_10 72.637 type value mrr_at_100 73.029 type value mrr_at_1000 73.033 type value mrr_at_3 70.572 type value mrr_at_5 71.86399999999999 type value ndcg_at_1 63.07000000000001 type value ndcg_at_10 74.708 type value ndcg_at_100 77.579 type value ndcg_at_1000 77.897 type value ndcg_at_3 69.69999999999999 type value ndcg_at_5 72.321 type value precision_at_1 63.07000000000001 type value precision_at_10 11.851 type value precision_at_100 1.481 type value precision_at_1000 0.156 type value precision_at_3 30.747000000000003 type value precision_at_5 20.830000000000002 type value recall_at_1 54.993 type value recall_at_10 87.18900000000001 type value recall_at_100 98.137 type value recall_at_1000 99.833 type value recall_at_3 73.654 type value recall_at_5 80.36 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering MTEB RedditClustering default test b2805658ae38990172679479369a78b86de8c390 type value v_measure 35.53178375429036 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering-p2p MTEB RedditClusteringP2P default test 385e3cb46b4cfa89021f56c4380204149d0efe33 type value v_measure 54.520782970558265 task dataset metrics type Retrieval type name config split revision scidocs MTEB SCIDOCS default test 5c59ef3e437a0a9651c8fe6fde943e7dce59fba5 type value map_at_1 4.3229999999999995 type value map_at_10 10.979999999999999 type value map_at_100 12.867 type value map_at_1000 13.147 type value map_at_3 7.973 type value map_at_5 9.513 type value mrr_at_1 21.3 type value mrr_at_10 32.34 type value mrr_at_100 33.428999999999995 type value mrr_at_1000 33.489999999999995 type value mrr_at_3 28.999999999999996 type value mrr_at_5 31.019999999999996 type value ndcg_at_1 21.3 type value ndcg_at_10 18.619 type value ndcg_at_100 26.108999999999998 type value ndcg_at_1000 31.253999999999998 type value ndcg_at_3 17.842 type value ndcg_at_5 15.673 type value precision_at_1 21.3 type value precision_at_10 9.55 type value precision_at_100 2.0340000000000003 type value precision_at_1000 0.327 type value precision_at_3 16.667 type value precision_at_5 13.76 type value recall_at_1 4.3229999999999995 type value recall_at_10 19.387 type value recall_at_100 41.307 type value recall_at_1000 66.475 type value recall_at_3 10.143 type value recall_at_5 14.007 task dataset metrics type STS type name config split revision mteb/sickr-sts MTEB SICK-R default test 20a6d6f312dd54037fe07a32d58e5e168867909d type value cos_sim_pearson 78.77975189382573 type value cos_sim_spearman 69.81522686267631 type value euclidean_pearson 71.37617936889518 type value euclidean_spearman 65.71738481148611 type value manhattan_pearson 71.58222165832424 type value manhattan_spearman 65.86851365286654 task dataset metrics type STS type name config split revision mteb/sts12-sts MTEB STS12 default test fdf84275bb8ce4b49c971d02e84dd1abc677a50f type value cos_sim_pearson 77.75509450443367 type value cos_sim_spearman 69.66180222442091 type value euclidean_pearson 74.98512779786111 type value euclidean_spearman 69.5997451409469 type value manhattan_pearson 75.50135090962459 type value manhattan_spearman 69.94984748475302 task dataset metrics type STS type name config split revision mteb/sts13-sts MTEB STS13 default test 1591bfcbe8c69d4bf7fe2a16e2451017832cafb9 type value cos_sim_pearson 79.42363892383264 type value cos_sim_spearman 79.66529244176742 type value euclidean_pearson 79.50429208135942 type value euclidean_spearman 80.44767586416276 type value manhattan_pearson 79.58563944997708 type value manhattan_spearman 80.51452267103 task dataset metrics type STS type name config split revision mteb/sts14-sts MTEB STS14 default test e2125984e7df8b7871f6ae9949cf6b6795e7c54b type value cos_sim_pearson 79.2749401478149 type value cos_sim_spearman 74.6076920702392 type value euclidean_pearson 73.3302002952881 type value euclidean_spearman 70.67029803077013 type value manhattan_pearson 73.52699344010296 type value manhattan_spearman 70.8517556194297 task dataset metrics type STS type name config split revision mteb/sts15-sts MTEB STS15 default test 1cd7298cac12a96a373b6a2f18738bb3e739a9b6 type value cos_sim_pearson 83.20884740785921 type value cos_sim_spearman 83.80600789090722 type value euclidean_pearson 74.9154089816344 type value euclidean_spearman 75.69243899592276 type value manhattan_pearson 75.0312832634451 type value manhattan_spearman 75.78324960357642 task dataset metrics type STS type name config split revision mteb/sts16-sts MTEB STS16 default test 360a0b2dff98700d09e634a01e1cc1624d3e42cd type value cos_sim_pearson 79.63194141000497 type value cos_sim_spearman 80.40118418350866 type value euclidean_pearson 72.07354384551088 type value euclidean_spearman 72.28819150373845 type value manhattan_pearson 72.08736119834145 type value manhattan_spearman 72.28347083261288 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (ko-ko) ko-ko test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 66.78512789499386 type value cos_sim_spearman 66.89125587193288 type value euclidean_pearson 58.74535708627959 type value euclidean_spearman 59.62103716794647 type value manhattan_pearson 59.00494529143961 type value manhattan_spearman 59.832257846799806 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (ar-ar) ar-ar test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 75.48960503523992 type value cos_sim_spearman 76.4223037534204 type value euclidean_pearson 64.93966381820944 type value euclidean_spearman 62.39697395373789 type value manhattan_pearson 65.54480770061505 type value manhattan_spearman 62.944204863043105 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-ar) en-ar test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 77.7331440643619 type value cos_sim_spearman 78.0748413292835 type value euclidean_pearson 38.533108233460304 type value euclidean_spearman 35.37638615280026 type value manhattan_pearson 41.0639726746513 type value manhattan_spearman 37.688161243671765 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-de) en-de test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 58.4628923720782 type value cos_sim_spearman 59.10093128795948 type value euclidean_pearson 30.422902393436836 type value euclidean_spearman 27.837806030497457 type value manhattan_pearson 32.51576984630963 type value manhattan_spearman 29.181887010982514 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-en) en-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 86.87447904613737 type value cos_sim_spearman 87.06554974065622 type value euclidean_pearson 76.82669047851108 type value euclidean_spearman 75.45711985511991 type value manhattan_pearson 77.46644556452847 type value manhattan_spearman 76.0249120007112 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-tr) en-tr test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 17.784495723497468 type value cos_sim_spearman 11.79629537128697 type value euclidean_pearson -4.354328445994008 type value euclidean_spearman -6.984566116230058 type value manhattan_pearson -4.166751901507852 type value manhattan_spearman -6.984143198323786 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (es-en) es-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 76.9009642643449 type value cos_sim_spearman 78.21764726338341 type value euclidean_pearson 50.578959144342925 type value euclidean_spearman 51.664379260719606 type value manhattan_pearson 53.95690880393329 type value manhattan_spearman 54.910058464050785 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (es-es) es-es test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 86.41638022270219 type value cos_sim_spearman 86.00477030366811 type value euclidean_pearson 79.7224037788285 type value euclidean_spearman 79.21417626867616 type value manhattan_pearson 80.29412412756984 type value manhattan_spearman 79.49460867616206 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (fr-en) fr-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 79.90432664091082 type value cos_sim_spearman 80.46007940700204 type value euclidean_pearson 49.25348015214428 type value euclidean_spearman 47.13113020475859 type value manhattan_pearson 54.57291204043908 type value manhattan_spearman 51.98559736896087 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (it-en) it-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 52.55164822309034 type value cos_sim_spearman 51.57629192137736 type value euclidean_pearson 16.63360593235354 type value euclidean_spearman 14.479679923782912 type value manhattan_pearson 18.524867185117472 type value manhattan_spearman 16.65940056664755 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (nl-en) nl-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 46.83690919715875 type value cos_sim_spearman 45.84993650002922 type value euclidean_pearson 6.173128686815117 type value euclidean_spearman 6.260781946306191 type value manhattan_pearson 7.328440452367316 type value manhattan_spearman 7.370842306497447 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (en) en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 64.97916914277232 type value cos_sim_spearman 66.13392188807865 type value euclidean_pearson 65.3921146908468 type value euclidean_spearman 65.8381588635056 type value manhattan_pearson 65.8866165769975 type value manhattan_spearman 66.27774050472219 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de) de test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 25.605130445111545 type value cos_sim_spearman 30.054844562369254 type value euclidean_pearson 23.890611005408196 type value euclidean_spearman 29.07902600726761 type value manhattan_pearson 24.239478426621833 type value manhattan_spearman 29.48547576782375 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es) es test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 61.6665616159781 type value cos_sim_spearman 65.41310206289988 type value euclidean_pearson 68.38805493215008 type value euclidean_spearman 65.22777377603435 type value manhattan_pearson 69.37445390454346 type value manhattan_spearman 66.02437701858754 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (pl) pl test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 15.302891825626372 type value cos_sim_spearman 31.134517255070097 type value euclidean_pearson 12.672592658843143 type value euclidean_spearman 29.14881036784207 type value manhattan_pearson 13.528545327757735 type value manhattan_spearman 29.56217928148797 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (tr) tr test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 28.79299114515319 type value cos_sim_spearman 47.135864983626206 type value euclidean_pearson 40.66410787594309 type value euclidean_spearman 45.09585593138228 type value manhattan_pearson 42.02561630700308 type value manhattan_spearman 45.43979983670554 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (ar) ar test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 46.00096625052943 type value cos_sim_spearman 58.67147426715496 type value euclidean_pearson 54.7154367422438 type value euclidean_spearman 59.003235142442634 type value manhattan_pearson 56.3116235357115 type value manhattan_spearman 60.12956331404423 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (ru) ru test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 29.3396354650316 type value cos_sim_spearman 43.3632935734809 type value euclidean_pearson 31.18506539466593 type value euclidean_spearman 37.531745324803815 type value manhattan_pearson 32.829038232529015 type value manhattan_spearman 38.04574361589953 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh) zh test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 62.9596148375188 type value cos_sim_spearman 66.77653412402461 type value euclidean_pearson 64.53156585980886 type value euclidean_spearman 66.2884373036083 type value manhattan_pearson 65.2831035495143 type value manhattan_spearman 66.83641945244322 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (fr) fr test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 79.9138821493919 type value cos_sim_spearman 80.38097535004677 type value euclidean_pearson 76.2401499094322 type value euclidean_spearman 77.00897050735907 type value manhattan_pearson 76.69531453728563 type value manhattan_spearman 77.83189696428695 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-en) de-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 51.27009640779202 type value cos_sim_spearman 51.16120562029285 type value euclidean_pearson 52.20594985566323 type value euclidean_spearman 52.75331049709882 type value manhattan_pearson 52.2725118792549 type value manhattan_spearman 53.614847968995115 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es-en) es-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 70.46044814118835 type value cos_sim_spearman 75.05760236668672 type value euclidean_pearson 72.80128921879461 type value euclidean_spearman 73.81164755219257 type value manhattan_pearson 72.7863795809044 type value manhattan_spearman 73.65932033818906 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (it) it test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 61.89276840435938 type value cos_sim_spearman 65.65042955732055 type value euclidean_pearson 61.22969491863841 type value euclidean_spearman 63.451215637904724 type value manhattan_pearson 61.16138956945465 type value manhattan_spearman 63.34966179331079 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (pl-en) pl-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 56.377577221753626 type value cos_sim_spearman 53.31223653270353 type value euclidean_pearson 26.488793041564307 type value euclidean_spearman 19.524551741701472 type value manhattan_pearson 24.322868054606474 type value manhattan_spearman 19.50371443994939 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh-en) zh-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 69.3634693673425 type value cos_sim_spearman 68.45051245419702 type value euclidean_pearson 56.1417414374769 type value euclidean_spearman 55.89891749631458 type value manhattan_pearson 57.266417430882925 type value manhattan_spearman 56.57927102744128 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es-it) es-it test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 60.04169437653179 type value cos_sim_spearman 65.49531007553446 type value euclidean_pearson 58.583860732586324 type value euclidean_spearman 58.80034792537441 type value manhattan_pearson 59.02513161664622 type value manhattan_spearman 58.42942047904558 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-fr) de-fr test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 48.81035211493999 type value cos_sim_spearman 53.27599246786967 type value euclidean_pearson 52.25710699032889 type value euclidean_spearman 55.22995695529873 type value manhattan_pearson 51.894901893217884 type value manhattan_spearman 54.95919975149795 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-pl) de-pl test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 36.75993101477816 type value cos_sim_spearman 43.050156692479355 type value euclidean_pearson 51.49021084746248 type value euclidean_spearman 49.54771253090078 type value manhattan_pearson 54.68410760796417 type value manhattan_spearman 48.19277197691717 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (fr-pl) fr-pl test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 48.553763306386486 type value cos_sim_spearman 28.17180849095055 type value euclidean_pearson 17.50739087826514 type value euclidean_spearman 16.903085094570333 type value manhattan_pearson 20.750046512534112 type value manhattan_spearman 5.634361698190111 task dataset metrics type STS type name config split revision mteb/stsbenchmark-sts MTEB STSBenchmark default test 8913289635987208e6e7c72789e4be2fe94b6abd type value cos_sim_pearson 82.17107190594417 type value cos_sim_spearman 80.89611873505183 type value euclidean_pearson 71.82491561814403 type value euclidean_spearman 70.33608835403274 type value manhattan_pearson 71.89538332420133 type value manhattan_spearman 70.36082395775944 task dataset metrics type Reranking type name config split revision mteb/scidocs-reranking MTEB SciDocsRR default test 56a6d0140cf6356659e2a7c1413286a774468d44 type value map 79.77047154974562 type value mrr 94.25887021475256 task dataset metrics type Retrieval type name config split revision scifact MTEB SciFact default test a75ae049398addde9b70f6b268875f5cbce99089 type value map_at_1 56.328 type value map_at_10 67.167 type value map_at_100 67.721 type value map_at_1000 67.735 type value map_at_3 64.20400000000001 type value map_at_5 65.904 type value mrr_at_1 59.667 type value mrr_at_10 68.553 type value mrr_at_100 68.992 type value mrr_at_1000 69.004 type value mrr_at_3 66.22200000000001 type value mrr_at_5 67.739 type value ndcg_at_1 59.667 type value ndcg_at_10 72.111 type value ndcg_at_100 74.441 type value ndcg_at_1000 74.90599999999999 type value ndcg_at_3 67.11399999999999 type value ndcg_at_5 69.687 type value precision_at_1 59.667 type value precision_at_10 9.733 type value precision_at_100 1.09 type value precision_at_1000 0.11299999999999999 type value precision_at_3 26.444000000000003 type value precision_at_5 17.599999999999998 type value recall_at_1 56.328 type value recall_at_10 85.8 type value recall_at_100 96.167 type value recall_at_1000 100.0 type value recall_at_3 72.433 type value recall_at_5 78.972 task dataset metrics type PairClassification type name config split revision mteb/sprintduplicatequestions-pairclassification MTEB SprintDuplicateQuestions default test 5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea type value cos_sim_accuracy 99.8019801980198 type value cos_sim_ap 94.92527097094644 type value cos_sim_f1 89.91935483870968 type value cos_sim_precision 90.65040650406505 type value cos_sim_recall 89.2 type value dot_accuracy 99.51782178217822 type value dot_ap 81.30756869559929 type value dot_f1 75.88235294117648 type value dot_precision 74.42307692307692 type value dot_recall 77.4 type value euclidean_accuracy 99.73069306930694 type value euclidean_ap 91.05040371796932 type value euclidean_f1 85.7889237199582 type value euclidean_precision 89.82494529540482 type value euclidean_recall 82.1 type value manhattan_accuracy 99.73762376237623 type value manhattan_ap 91.4823412839869 type value manhattan_f1 86.39836984207845 type value manhattan_precision 88.05815160955348 type value manhattan_recall 84.8 type value max_accuracy 99.8019801980198 type value max_ap 94.92527097094644 type value max_f1 89.91935483870968 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering MTEB StackExchangeClustering default test 70a89468f6dccacc6aa2b12a6eac54e74328f235 type value v_measure 55.13046832022158 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering-p2p MTEB StackExchangeClusteringP2P default test d88009ab563dd0b16cfaf4436abaf97fa3550cf0 type value v_measure 34.31252463546675 task dataset metrics type Reranking type name config split revision mteb/stackoverflowdupquestions-reranking MTEB StackOverflowDupQuestions default test ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9 type value map 51.06639688231414 type value mrr 51.80205415499534 task dataset metrics type Summarization type name config split revision mteb/summeval MTEB SummEval default test 8753c2788d36c01fc6f05d03fe3f7268d63f9122 type value cos_sim_pearson 31.963331462886957 type value cos_sim_spearman 33.59510652629926 type value dot_pearson 29.033733540882123 type value dot_spearman 31.550290638315504 task dataset metrics type Retrieval type name config split revision trec-covid MTEB TRECCOVID default test 2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217 type value map_at_1 0.23600000000000002 type value map_at_10 2.09 type value map_at_100 12.466000000000001 type value map_at_1000 29.852 type value map_at_3 0.6859999999999999 type value map_at_5 1.099 type value mrr_at_1 88.0 type value mrr_at_10 94.0 type value mrr_at_100 94.0 type value mrr_at_1000 94.0 type value mrr_at_3 94.0 type value mrr_at_5 94.0 type value ndcg_at_1 86.0 type value ndcg_at_10 81.368 type value ndcg_at_100 61.879 type value ndcg_at_1000 55.282 type value ndcg_at_3 84.816 type value ndcg_at_5 82.503 type value precision_at_1 88.0 type value precision_at_10 85.6 type value precision_at_100 63.85999999999999 type value precision_at_1000 24.682000000000002 type value precision_at_3 88.667 type value precision_at_5 86.0 type value recall_at_1 0.23600000000000002 type value recall_at_10 2.25 type value recall_at_100 15.488 type value recall_at_1000 52.196 type value recall_at_3 0.721 type value recall_at_5 1.159 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (sqi-eng) sqi-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 12.7 type value f1 10.384182044950325 type value precision 9.805277385275312 type value recall 12.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fry-eng) fry-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 30.63583815028902 type value f1 24.623726947426373 type value precision 22.987809919828013 type value recall 30.63583815028902 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kur-eng) kur-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 10.487804878048781 type value f1 8.255945048627975 type value precision 7.649047253615001 type value recall 10.487804878048781 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tur-eng) tur-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 8.5 type value f1 6.154428783776609 type value precision 5.680727638128585 type value recall 8.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (deu-eng) deu-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 73.0 type value f1 70.10046605876393 type value precision 69.0018253968254 type value recall 73.0 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nld-eng) nld-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 32.7 type value f1 29.7428583868239 type value precision 28.81671359506905 type value recall 32.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ron-eng) ron-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 31.5 type value f1 27.228675552174003 type value precision 25.950062299847747 type value recall 31.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ang-eng) ang-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 35.82089552238806 type value f1 28.75836980510979 type value precision 26.971643613434658 type value recall 35.82089552238806 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ido-eng) ido-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 49.8 type value f1 43.909237401451776 type value precision 41.944763440988936 type value recall 49.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (jav-eng) jav-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 18.536585365853657 type value f1 15.020182570246751 type value precision 14.231108073213337 type value recall 18.536585365853657 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (isl-eng) isl-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 8.7 type value f1 6.2934784902885355 type value precision 5.685926293425392 type value recall 8.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (slv-eng) slv-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 12.879708383961116 type value f1 10.136118341751114 type value precision 9.571444036679436 type value recall 12.879708383961116 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cym-eng) cym-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 9.217391304347826 type value f1 6.965003297761793 type value precision 6.476093529199119 type value recall 9.217391304347826 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kaz-eng) kaz-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 4.3478260869565215 type value f1 3.3186971707677397 type value precision 3.198658632552104 type value recall 4.3478260869565215 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (est-eng) est-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 6.9 type value f1 4.760708297894056 type value precision 4.28409511756074 type value recall 6.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (heb-eng) heb-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 2.1999999999999997 type value f1 1.6862703878117107 type value precision 1.6048118233915603 type value recall 2.1999999999999997 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gla-eng) gla-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 3.0156815440289506 type value f1 2.0913257250659134 type value precision 1.9072775486461648 type value recall 3.0156815440289506 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mar-eng) mar-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 49.0 type value f1 45.5254456536713 type value precision 44.134609250398725 type value recall 49.0 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lat-eng) lat-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 33.5 type value f1 28.759893973182564 type value precision 27.401259116024836 type value recall 33.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bel-eng) bel-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 10.2 type value f1 8.030039981676275 type value precision 7.548748077210127 type value recall 10.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pms-eng) pms-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 38.095238095238095 type value f1 31.944999250262406 type value precision 30.04452690166976 type value recall 38.095238095238095 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gle-eng) gle-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 4.8 type value f1 3.2638960786708067 type value precision 3.0495382950729644 type value recall 4.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pes-eng) pes-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 15.8 type value f1 12.131087470371275 type value precision 11.141304011547815 type value recall 15.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nob-eng) nob-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 23.3 type value f1 21.073044636921384 type value precision 20.374220568287285 type value recall 23.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bul-eng) bul-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 24.9 type value f1 20.091060685364987 type value precision 18.899700591081224 type value recall 24.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cbk-eng) cbk-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 70.1 type value f1 64.62940836940835 type value precision 62.46559523809524 type value recall 70.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hun-eng) hun-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 7.199999999999999 type value f1 5.06613460576115 type value precision 4.625224463391809 type value recall 7.199999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (uig-eng) uig-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 1.7999999999999998 type value f1 1.2716249514772895 type value precision 1.2107445914723798 type value recall 1.7999999999999998 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (rus-eng) rus-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 65.5 type value f1 59.84399711399712 type value precision 57.86349567099567 type value recall 65.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (spa-eng) spa-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 95.7 type value f1 94.48333333333333 type value precision 93.89999999999999 type value recall 95.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hye-eng) hye-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 0.8086253369272237 type value f1 0.4962046191492002 type value precision 0.47272438578554393 type value recall 0.8086253369272237 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tel-eng) tel-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 69.23076923076923 type value f1 64.6227941099736 type value precision 63.03795877325289 type value recall 69.23076923076923 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (afr-eng) afr-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 20.599999999999998 type value f1 16.62410040660465 type value precision 15.598352437967069 type value recall 20.599999999999998 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mon-eng) mon-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 4.318181818181818 type value f1 2.846721192535661 type value precision 2.6787861417537147 type value recall 4.318181818181818 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (arz-eng) arz-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 74.84276729559748 type value f1 70.6638714185884 type value precision 68.86792452830188 type value recall 74.84276729559748 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hrv-eng) hrv-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 15.9 type value f1 12.793698974586706 type value precision 12.088118017657736 type value recall 15.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nov-eng) nov-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 59.92217898832685 type value f1 52.23086900129701 type value precision 49.25853869433636 type value recall 59.92217898832685 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (gsw-eng) gsw-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 27.350427350427353 type value f1 21.033781033781032 type value precision 19.337955491801644 type value recall 27.350427350427353 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nds-eng) nds-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 29.299999999999997 type value f1 23.91597452425777 type value precision 22.36696598364942 type value recall 29.299999999999997 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ukr-eng) ukr-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 27.3 type value f1 22.059393517688886 type value precision 20.503235534170887 type value recall 27.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (uzb-eng) uzb-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 8.177570093457943 type value f1 4.714367017906037 type value precision 4.163882933965758 type value recall 8.177570093457943 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lit-eng) lit-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 5.800000000000001 type value f1 4.4859357432293825 type value precision 4.247814465614043 type value recall 5.800000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ina-eng) ina-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 78.4 type value f1 73.67166666666667 type value precision 71.83285714285714 type value recall 78.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lfn-eng) lfn-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 50.3 type value f1 44.85221545883311 type value precision 43.04913026243909 type value recall 50.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (zsm-eng) zsm-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 83.5 type value f1 79.95151515151515 type value precision 78.53611111111111 type value recall 83.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ita-eng) ita-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 69.89999999999999 type value f1 65.03756269256269 type value precision 63.233519536019536 type value recall 69.89999999999999 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cmn-eng) cmn-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 93.2 type value f1 91.44666666666666 type value precision 90.63333333333333 type value recall 93.2 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (lvs-eng) lvs-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 8.3 type value f1 6.553388144729963 type value precision 6.313497782829976 type value recall 8.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (glg-eng) glg-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 83.6 type value f1 79.86243107769424 type value precision 78.32555555555555 type value recall 83.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ceb-eng) ceb-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 9.166666666666666 type value f1 6.637753604420271 type value precision 6.10568253585495 type value recall 9.166666666666666 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bre-eng) bre-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 7.3999999999999995 type value f1 4.6729483612322165 type value precision 4.103844520292658 type value recall 7.3999999999999995 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ben-eng) ben-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 80.30000000000001 type value f1 75.97666666666667 type value precision 74.16 type value recall 80.30000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swg-eng) swg-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 23.214285714285715 type value f1 16.88988095238095 type value precision 15.364937641723353 type value recall 23.214285714285715 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (arq-eng) arq-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 33.15038419319429 type value f1 27.747873024072415 type value precision 25.99320572578704 type value recall 33.15038419319429 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kab-eng) kab-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 2.6 type value f1 1.687059048752127 type value precision 1.5384884521299 type value recall 2.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fra-eng) fra-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 93.30000000000001 type value f1 91.44000000000001 type value precision 90.59166666666667 type value recall 93.30000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (por-eng) por-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 94.1 type value f1 92.61666666666667 type value precision 91.88333333333333 type value recall 94.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tat-eng) tat-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 5.0 type value f1 3.589591971281927 type value precision 3.3046491614532854 type value recall 5.0 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (oci-eng) oci-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 45.9 type value f1 40.171969141969136 type value precision 38.30764368870302 type value recall 45.9 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pol-eng) pol-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 16.900000000000002 type value f1 14.094365204207351 type value precision 13.276519841269844 type value recall 16.900000000000002 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (war-eng) war-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 12.8 type value f1 10.376574912567156 type value precision 9.758423963284509 type value recall 12.8 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (aze-eng) aze-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 8.1 type value f1 6.319455355175778 type value precision 5.849948830628881 type value recall 8.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (vie-eng) vie-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 95.5 type value f1 94.19666666666667 type value precision 93.60000000000001 type value recall 95.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (nno-eng) nno-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 19.1 type value f1 16.280080686081906 type value precision 15.451573089395668 type value recall 19.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cha-eng) cha-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 30.656934306569344 type value f1 23.2568647897115 type value precision 21.260309034031664 type value recall 30.656934306569344 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mhr-eng) mhr-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 2.1999999999999997 type value f1 1.556861047295521 type value precision 1.4555993437238521 type value recall 2.1999999999999997 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dan-eng) dan-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 27.500000000000004 type value f1 23.521682636223492 type value precision 22.345341306967683 type value recall 27.500000000000004 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ell-eng) ell-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 7.3999999999999995 type value f1 5.344253880846173 type value precision 4.999794279068863 type value recall 7.3999999999999995 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (amh-eng) amh-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 0.5952380952380952 type value f1 0.026455026455026457 type value precision 0.013528138528138528 type value recall 0.5952380952380952 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (pam-eng) pam-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 7.3 type value f1 5.853140211779251 type value precision 5.505563080945322 type value recall 7.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hsb-eng) hsb-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 13.250517598343686 type value f1 9.676349506190704 type value precision 8.930392053553216 type value recall 13.250517598343686 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (srp-eng) srp-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 14.499999999999998 type value f1 11.68912588067557 type value precision 11.024716513105519 type value recall 14.499999999999998 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (epo-eng) epo-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 30.099999999999998 type value f1 26.196880936315146 type value precision 25.271714086169478 type value recall 30.099999999999998 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kzj-eng) kzj-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 6.4 type value f1 5.1749445942023335 type value precision 4.975338142029625 type value recall 6.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (awa-eng) awa-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 39.39393939393939 type value f1 35.005707393767096 type value precision 33.64342032053631 type value recall 39.39393939393939 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fao-eng) fao-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 18.3206106870229 type value f1 12.610893447220345 type value precision 11.079228765297467 type value recall 18.3206106870229 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mal-eng) mal-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 85.58951965065502 type value f1 83.30363944928548 type value precision 82.40026591554977 type value recall 85.58951965065502 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ile-eng) ile-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 65.7 type value f1 59.589642857142856 type value precision 57.392826797385624 type value recall 65.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (bos-eng) bos-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 18.07909604519774 type value f1 13.65194306689995 type value precision 12.567953943826327 type value recall 18.07909604519774 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cor-eng) cor-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 4.6 type value f1 2.8335386392505013 type value precision 2.558444143575722 type value recall 4.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (cat-eng) cat-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 90.7 type value f1 88.30666666666666 type value precision 87.195 type value recall 90.7 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (eus-eng) eus-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 57.699999999999996 type value f1 53.38433067253876 type value precision 51.815451335350346 type value recall 57.699999999999996 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (yue-eng) yue-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 80.60000000000001 type value f1 77.0290354090354 type value precision 75.61685897435898 type value recall 80.60000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swe-eng) swe-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 24.6 type value f1 19.52814960069739 type value precision 18.169084599880502 type value recall 24.6 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dtp-eng) dtp-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 5.0 type value f1 3.4078491753102376 type value precision 3.1757682319102387 type value recall 5.0 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kat-eng) kat-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 1.2064343163538873 type value f1 0.4224313053283095 type value precision 0.3360484946842894 type value recall 1.2064343163538873 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (jpn-eng) jpn-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 76.1 type value f1 71.36246031746032 type value precision 69.5086544011544 type value recall 76.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (csb-eng) csb-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 14.229249011857709 type value f1 10.026578603653704 type value precision 9.09171178352764 type value recall 14.229249011857709 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (xho-eng) xho-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 8.450704225352112 type value f1 5.51214407186151 type value precision 4.928281812084629 type value recall 8.450704225352112 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (orv-eng) orv-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 7.664670658682635 type value f1 5.786190079917295 type value precision 5.3643643579244 type value recall 7.664670658682635 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ind-eng) ind-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 90.5 type value f1 88.03999999999999 type value precision 86.94833333333334 type value recall 90.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tuk-eng) tuk-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 7.389162561576355 type value f1 5.482366349556517 type value precision 5.156814449917898 type value recall 7.389162561576355 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (max-eng) max-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 41.54929577464789 type value f1 36.13520282534367 type value precision 34.818226488560995 type value recall 41.54929577464789 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (swh-eng) swh-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 20.76923076923077 type value f1 16.742497560177643 type value precision 15.965759712090138 type value recall 20.76923076923077 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (hin-eng) hin-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 88.1 type value f1 85.23176470588236 type value precision 84.04458333333334 type value recall 88.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (dsb-eng) dsb-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 11.899791231732777 type value f1 8.776706659565102 type value precision 8.167815946521582 type value recall 11.899791231732777 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ber-eng) ber-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 6.1 type value f1 4.916589537178435 type value precision 4.72523017415345 type value recall 6.1 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tam-eng) tam-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 76.54723127035831 type value f1 72.75787187839306 type value precision 71.43338442869005 type value recall 76.54723127035831 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (slk-eng) slk-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 11.700000000000001 type value f1 9.975679190026007 type value precision 9.569927715653522 type value recall 11.700000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tgl-eng) tgl-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 13.100000000000001 type value f1 10.697335850115408 type value precision 10.113816082086341 type value recall 13.100000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ast-eng) ast-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 76.37795275590551 type value f1 71.12860892388451 type value precision 68.89763779527559 type value recall 76.37795275590551 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (mkd-eng) mkd-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 13.700000000000001 type value f1 10.471861684067568 type value precision 9.602902567641697 type value recall 13.700000000000001 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (khm-eng) khm-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 0.554016620498615 type value f1 0.37034084643642423 type value precision 0.34676040281208437 type value recall 0.554016620498615 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ces-eng) ces-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 12.4 type value f1 9.552607451092534 type value precision 8.985175505050504 type value recall 12.4 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tzl-eng) tzl-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 33.65384615384615 type value f1 27.820512820512818 type value precision 26.09432234432234 type value recall 33.65384615384615 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (urd-eng) urd-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 74.5 type value f1 70.09686507936507 type value precision 68.3117857142857 type value recall 74.5 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (ara-eng) ara-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 88.3 type value f1 85.37333333333333 type value precision 84.05833333333334 type value recall 88.3 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (kor-eng) kor-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 25.0 type value f1 22.393124632031995 type value precision 21.58347686592367 type value recall 25.0 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (yid-eng) yid-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 0.589622641509434 type value f1 0.15804980033762941 type value precision 0.1393275384872965 type value recall 0.589622641509434 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (fin-eng) fin-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 4.1000000000000005 type value f1 3.4069011332551775 type value precision 3.1784507042253516 type value recall 4.1000000000000005 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (tha-eng) tha-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 3.102189781021898 type value f1 2.223851811694751 type value precision 2.103465682299194 type value recall 3.102189781021898 task dataset metrics type BitextMining type name config split revision mteb/tatoeba-bitext-mining MTEB Tatoeba (wuu-eng) wuu-eng test ed9e4a974f867fd9736efcf222fc3a26487387a5 type value accuracy 83.1 type value f1 79.58255835667599 type value precision 78.09708333333333 type value recall 83.1 task dataset metrics type Retrieval type name config split revision webis-touche2020 MTEB Touche2020 default test 527b7d77e16e343303e68cb6af11d6e18b9f7b3b type value map_at_1 2.322 type value map_at_10 8.959999999999999 type value map_at_100 15.136 type value map_at_1000 16.694 type value map_at_3 4.837000000000001 type value map_at_5 6.196 type value mrr_at_1 28.571 type value mrr_at_10 47.589999999999996 type value mrr_at_100 48.166 type value mrr_at_1000 48.169000000000004 type value mrr_at_3 43.197 type value mrr_at_5 45.646 type value ndcg_at_1 26.531 type value ndcg_at_10 23.982 type value ndcg_at_100 35.519 type value ndcg_at_1000 46.878 type value ndcg_at_3 26.801000000000002 type value ndcg_at_5 24.879 type value precision_at_1 28.571 type value precision_at_10 22.041 type value precision_at_100 7.4079999999999995 type value precision_at_1000 1.492 type value precision_at_3 28.571 type value precision_at_5 25.306 type value recall_at_1 2.322 type value recall_at_10 15.443999999999999 type value recall_at_100 45.918 type value recall_at_1000 79.952 type value recall_at_3 6.143 type value recall_at_5 8.737 task dataset metrics type Classification type name config split revision mteb/toxic_conversations_50k MTEB ToxicConversationsClassification default test edfaf9da55d3dd50d43143d90c1ac476895ae6de type value accuracy 66.5452 type value ap 12.99191723223892 type value f1 51.667665096195734 task dataset metrics type Classification type name config split revision mteb/tweet_sentiment_extraction MTEB TweetSentimentExtractionClassification default test 62146448f05be9e52a36b8ee9936447ea787eede type value accuracy 55.854555744199196 type value f1 56.131766302254185 task dataset metrics type Clustering type name config split revision mteb/twentynewsgroups-clustering MTEB TwentyNewsgroupsClustering default test 091a54f9a36281ce7d6590ec8c75dd485e7e01d4 type value v_measure 37.27891385518074 task dataset metrics type PairClassification type name config split revision mteb/twittersemeval2015-pairclassification MTEB TwitterSemEval2015 default test 70970daeab8776df92f5ea462b6173c0b46fd2d1 type value cos_sim_accuracy 83.53102461703523 type value cos_sim_ap 65.30753664579191 type value cos_sim_f1 61.739943872778305 type value cos_sim_precision 55.438891222175556 type value cos_sim_recall 69.65699208443272 type value dot_accuracy 80.38981939560112 type value dot_ap 53.52081118421347 type value dot_f1 54.232957844617346 type value dot_precision 48.43393486828459 type value dot_recall 61.60949868073878 type value euclidean_accuracy 82.23758717291531 type value euclidean_ap 60.361102792772535 type value euclidean_f1 57.50518791791561 type value euclidean_precision 51.06470106470107 type value euclidean_recall 65.8047493403694 type value manhattan_accuracy 82.14221851344102 type value manhattan_ap 60.341937223793366 type value manhattan_f1 57.53803596127247 type value manhattan_precision 51.08473188702415 type value manhattan_recall 65.85751978891821 type value max_accuracy 83.53102461703523 type value max_ap 65.30753664579191 type value max_f1 61.739943872778305 task dataset metrics type PairClassification type name config split revision mteb/twitterurlcorpus-pairclassification MTEB TwitterURLCorpus default test 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf type value cos_sim_accuracy 88.75305623471883 type value cos_sim_ap 85.46387153880272 type value cos_sim_f1 77.91527673159008 type value cos_sim_precision 72.93667315828353 type value cos_sim_recall 83.62334462580844 type value dot_accuracy 85.08169363915086 type value dot_ap 74.96808060965559 type value dot_f1 71.39685033990366 type value dot_precision 64.16948111759288 type value dot_recall 80.45888512473051 type value euclidean_accuracy 85.84235650250321 type value euclidean_ap 78.42045145247211 type value euclidean_f1 70.32669630775179 type value euclidean_precision 70.6298050788227 type value euclidean_recall 70.02617801047121 type value manhattan_accuracy 85.86176116738464 type value manhattan_ap 78.54012451558276 type value manhattan_f1 70.56508080693389 type value manhattan_precision 69.39626293456413 type value manhattan_recall 71.77394518016631 type value max_accuracy 88.75305623471883 type value max_ap 85.46387153880272 type value max_f1 77.91527673159008

Usage

For usage instructions, refer to: https://github.com/Muennighoff/sgpt#asymmetric-semantic-search-be

The model was trained with the command

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch examples/training/ms_marco/train_bi-encoder_mnrl.py --model_name bigscience/bloom-7b1 --train_batch_size 32 --eval_batch_size 16 --freezenonbias --specb --lr 4e-4 --wandb --wandbwatchlog gradients --pooling weightedmean --gradcache --chunksize 8

Evaluation Results

{"ndcgs": {"sgpt-bloom-7b1-msmarco": {"scifact": {"NDCG@10": 0.71824}, "nfcorpus": {"NDCG@10": 0.35748}, "arguana": {"NDCG@10": 0.47281}, "scidocs": {"NDCG@10": 0.18435}, "fiqa": {"NDCG@10": 0.35736}, "cqadupstack": {"NDCG@10": 0.3708525}, "quora": {"NDCG@10": 0.74655}, "trec-covid": {"NDCG@10": 0.82731}, "webis-touche2020": {"NDCG@10": 0.2365}}}

See the evaluation folder or MTEB for more results.

Training

The model was trained with the parameters:

DataLoader:

torch.utils.data.dataloader.DataLoader of length 15600 with parameters:

{'batch_size': 32, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}

The model uses BitFit, weighted-mean pooling & GradCache, for details see: https://arxiv.org/abs/2202.08904

Loss:

sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache

Parameters of the fit()-Method:

{
    "epochs": 10,
    "evaluation_steps": 0,
    "evaluator": "NoneType",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 0.0004
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 1000,
    "weight_decay": 0.01
}

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 300, 'do_lower_case': False}) with Transformer model: BloomModel 
  (1): Pooling({'word_embedding_dimension': 4096, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
)

Citing & Authors

@article{muennighoff2022sgpt,
  title={SGPT: GPT Sentence Embeddings for Semantic Search},
  author={Muennighoff, Niklas},
  journal={arXiv preprint arXiv:2202.08904},
  year={2022}
}