SGPT-125M-weightedmean-msmarco-specb-bitfit/README.md

pipeline_tag, tags, model-index

pipeline_tag	tags	model-index
sentence-similarity	sentence-transformers feature-extraction sentence-similarity mteb	name results SGPT-125M-weightedmean-msmarco-specb-bitfit task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en) en test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 61.23880597014926 type value ap 25.854431650388644 type value f1 55.751862762818604 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (de) de test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 56.88436830835117 type value ap 72.67279104379772 type value f1 54.449840243786404 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en-ext) en-ext test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 58.27586206896551 type value ap 14.067357642500387 type value f1 48.172318518691334 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (ja) ja test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 54.64668094218415 type value ap 11.776694555054965 type value f1 44.526622834078765 task dataset metrics type Classification type name config split revision mteb/amazon_polarity MTEB AmazonPolarityClassification default test 80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1 type value accuracy 65.401225 type value ap 60.22809958678552 type value f1 65.0251824898292 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (en) en test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 31.165999999999993 type value f1 30.908870050167437 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (de) de test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 24.79 type value f1 24.5833598854121 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (es) es test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 26.643999999999995 type value f1 26.39012792213563 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (fr) fr test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 26.386000000000003 type value f1 26.276867791454873 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (ja) ja test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 22.078000000000003 type value f1 21.797960290226843 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (zh) zh test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 24.274 type value f1 23.887054434822627 task dataset metrics type Retrieval type name config split revision arguana MTEB ArguAna default test 5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3 type value map_at_1 22.404 type value map_at_10 36.845 type value map_at_100 37.945 type value map_at_1000 37.966 type value map_at_3 31.78 type value map_at_5 34.608 type value mrr_at_1 22.902 type value mrr_at_10 37.034 type value mrr_at_100 38.134 type value mrr_at_1000 38.155 type value mrr_at_3 31.935000000000002 type value mrr_at_5 34.812 type value ndcg_at_1 22.404 type value ndcg_at_10 45.425 type value ndcg_at_100 50.354 type value ndcg_at_1000 50.873999999999995 type value ndcg_at_3 34.97 type value ndcg_at_5 40.081 type value precision_at_1 22.404 type value precision_at_10 7.303999999999999 type value precision_at_100 0.951 type value precision_at_1000 0.099 type value precision_at_3 14.746 type value precision_at_5 11.337 type value recall_at_1 22.404 type value recall_at_10 73.044 type value recall_at_100 95.092 type value recall_at_1000 99.075 type value recall_at_3 44.239 type value recall_at_5 56.686 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-p2p MTEB ArxivClusteringP2P default test 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8 type value v_measure 39.70858340673288 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-s2s MTEB ArxivClusteringS2S default test b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3 type value v_measure 28.242847713721048 task dataset metrics type Reranking type name config split revision mteb/askubuntudupquestions-reranking MTEB AskUbuntuDupQuestions default test 4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c type value map 55.83700395192393 type value mrr 70.3891307215407 task dataset metrics type STS type name config split revision mteb/biosses-sts MTEB BIOSSES default test 9ee918f184421b6bd48b78f6c714d86546106103 type value cos_sim_pearson 79.25366801756223 type value cos_sim_spearman 75.20954502580506 type value euclidean_pearson 78.79900722991617 type value euclidean_spearman 77.79996549607588 type value manhattan_pearson 78.18408109480399 type value manhattan_spearman 76.85958262303106 task dataset metrics type Classification type name config split revision mteb/banking77 MTEB Banking77Classification default test 44fa15921b4c889113cc5df03dd4901b49161ab7 type value accuracy 77.70454545454545 type value f1 77.6929000113803 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-p2p MTEB BiorxivClusteringP2P default test 11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55 type value v_measure 33.63260395543984 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-s2s MTEB BiorxivClusteringS2S default test c0fab014e1bcb8d3a5e31b2088972a1e01547dc1 type value v_measure 27.038042665369925 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackAndroidRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 22.139 type value map_at_10 28.839 type value map_at_100 30.023 type value map_at_1000 30.153000000000002 type value map_at_3 26.521 type value map_at_5 27.775 type value mrr_at_1 26.466 type value mrr_at_10 33.495000000000005 type value mrr_at_100 34.416999999999994 type value mrr_at_1000 34.485 type value mrr_at_3 31.402 type value mrr_at_5 32.496 type value ndcg_at_1 26.466 type value ndcg_at_10 33.372 type value ndcg_at_100 38.7 type value ndcg_at_1000 41.696 type value ndcg_at_3 29.443 type value ndcg_at_5 31.121 type value precision_at_1 26.466 type value precision_at_10 6.037 type value precision_at_100 1.0670000000000002 type value precision_at_1000 0.16199999999999998 type value precision_at_3 13.782 type value precision_at_5 9.757 type value recall_at_1 22.139 type value recall_at_10 42.39 type value recall_at_100 65.427 type value recall_at_1000 86.04899999999999 type value recall_at_3 31.127 type value recall_at_5 35.717999999999996 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackEnglishRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 20.652 type value map_at_10 27.558 type value map_at_100 28.473 type value map_at_1000 28.577 type value map_at_3 25.402 type value map_at_5 26.68 type value mrr_at_1 25.223000000000003 type value mrr_at_10 31.966 type value mrr_at_100 32.664 type value mrr_at_1000 32.724 type value mrr_at_3 30.074 type value mrr_at_5 31.249 type value ndcg_at_1 25.223000000000003 type value ndcg_at_10 31.694 type value ndcg_at_100 35.662 type value ndcg_at_1000 38.092 type value ndcg_at_3 28.294000000000004 type value ndcg_at_5 30.049 type value precision_at_1 25.223000000000003 type value precision_at_10 5.777 type value precision_at_100 0.9730000000000001 type value precision_at_1000 0.13999999999999999 type value precision_at_3 13.397 type value precision_at_5 9.605 type value recall_at_1 20.652 type value recall_at_10 39.367999999999995 type value recall_at_100 56.485 type value recall_at_1000 73.292 type value recall_at_3 29.830000000000002 type value recall_at_5 34.43 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGamingRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 25.180000000000003 type value map_at_10 34.579 type value map_at_100 35.589999999999996 type value map_at_1000 35.68 type value map_at_3 31.735999999999997 type value map_at_5 33.479 type value mrr_at_1 29.467 type value mrr_at_10 37.967 type value mrr_at_100 38.800000000000004 type value mrr_at_1000 38.858 type value mrr_at_3 35.465 type value mrr_at_5 37.057 type value ndcg_at_1 29.467 type value ndcg_at_10 39.796 type value ndcg_at_100 44.531 type value ndcg_at_1000 46.666000000000004 type value ndcg_at_3 34.676 type value ndcg_at_5 37.468 type value precision_at_1 29.467 type value precision_at_10 6.601999999999999 type value precision_at_100 0.9900000000000001 type value precision_at_1000 0.124 type value precision_at_3 15.568999999999999 type value precision_at_5 11.172 type value recall_at_1 25.180000000000003 type value recall_at_10 52.269 type value recall_at_100 73.574 type value recall_at_1000 89.141 type value recall_at_3 38.522 type value recall_at_5 45.323 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGisRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 16.303 type value map_at_10 21.629 type value map_at_100 22.387999999999998 type value map_at_1000 22.489 type value map_at_3 19.608 type value map_at_5 20.774 type value mrr_at_1 17.740000000000002 type value mrr_at_10 23.214000000000002 type value mrr_at_100 23.97 type value mrr_at_1000 24.054000000000002 type value mrr_at_3 21.243000000000002 type value mrr_at_5 22.322 type value ndcg_at_1 17.740000000000002 type value ndcg_at_10 25.113000000000003 type value ndcg_at_100 29.287999999999997 type value ndcg_at_1000 32.204 type value ndcg_at_3 21.111 type value ndcg_at_5 23.061999999999998 type value precision_at_1 17.740000000000002 type value precision_at_10 3.955 type value precision_at_100 0.644 type value precision_at_1000 0.093 type value precision_at_3 8.851 type value precision_at_5 6.418 type value recall_at_1 16.303 type value recall_at_10 34.487 type value recall_at_100 54.413999999999994 type value recall_at_1000 77.158 type value recall_at_3 23.733 type value recall_at_5 28.381 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackMathematicaRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 10.133000000000001 type value map_at_10 15.665999999999999 type value map_at_100 16.592000000000002 type value map_at_1000 16.733999999999998 type value map_at_3 13.625000000000002 type value map_at_5 14.721 type value mrr_at_1 12.562000000000001 type value mrr_at_10 18.487000000000002 type value mrr_at_100 19.391 type value mrr_at_1000 19.487 type value mrr_at_3 16.418 type value mrr_at_5 17.599999999999998 type value ndcg_at_1 12.562000000000001 type value ndcg_at_10 19.43 type value ndcg_at_100 24.546 type value ndcg_at_1000 28.193 type value ndcg_at_3 15.509999999999998 type value ndcg_at_5 17.322000000000003 type value precision_at_1 12.562000000000001 type value precision_at_10 3.794 type value precision_at_100 0.74 type value precision_at_1000 0.122 type value precision_at_3 7.546 type value precision_at_5 5.721 type value recall_at_1 10.133000000000001 type value recall_at_10 28.261999999999997 type value recall_at_100 51.742999999999995 type value recall_at_1000 78.075 type value recall_at_3 17.634 type value recall_at_5 22.128999999999998 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackPhysicsRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 19.991999999999997 type value map_at_10 27.346999999999998 type value map_at_100 28.582 type value map_at_1000 28.716 type value map_at_3 24.907 type value map_at_5 26.1 type value mrr_at_1 23.773 type value mrr_at_10 31.647 type value mrr_at_100 32.639 type value mrr_at_1000 32.706 type value mrr_at_3 29.195 type value mrr_at_5 30.484 type value ndcg_at_1 23.773 type value ndcg_at_10 32.322 type value ndcg_at_100 37.996 type value ndcg_at_1000 40.819 type value ndcg_at_3 27.876 type value ndcg_at_5 29.664 type value precision_at_1 23.773 type value precision_at_10 5.976999999999999 type value precision_at_100 1.055 type value precision_at_1000 0.15 type value precision_at_3 13.122 type value precision_at_5 9.451 type value recall_at_1 19.991999999999997 type value recall_at_10 43.106 type value recall_at_100 67.264 type value recall_at_1000 86.386 type value recall_at_3 30.392000000000003 type value recall_at_5 34.910999999999994 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackProgrammersRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 17.896 type value map_at_10 24.644 type value map_at_100 25.790000000000003 type value map_at_1000 25.913999999999998 type value map_at_3 22.694 type value map_at_5 23.69 type value mrr_at_1 21.346999999999998 type value mrr_at_10 28.594 type value mrr_at_100 29.543999999999997 type value mrr_at_1000 29.621 type value mrr_at_3 26.807 type value mrr_at_5 27.669 type value ndcg_at_1 21.346999999999998 type value ndcg_at_10 28.833 type value ndcg_at_100 34.272000000000006 type value ndcg_at_1000 37.355 type value ndcg_at_3 25.373 type value ndcg_at_5 26.756 type value precision_at_1 21.346999999999998 type value precision_at_10 5.2170000000000005 type value precision_at_100 0.954 type value precision_at_1000 0.13899999999999998 type value precision_at_3 11.948 type value precision_at_5 8.425 type value recall_at_1 17.896 type value recall_at_10 37.291000000000004 type value recall_at_100 61.138000000000005 type value recall_at_1000 83.212 type value recall_at_3 27.705999999999996 type value recall_at_5 31.234 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 17.195166666666665 type value map_at_10 23.329083333333333 type value map_at_100 24.30308333333333 type value map_at_1000 24.422416666666667 type value map_at_3 21.327416666666664 type value map_at_5 22.419999999999998 type value mrr_at_1 19.999916666666667 type value mrr_at_10 26.390166666666666 type value mrr_at_100 27.230999999999998 type value mrr_at_1000 27.308333333333334 type value mrr_at_3 24.4675 type value mrr_at_5 25.541083333333336 type value ndcg_at_1 19.999916666666667 type value ndcg_at_10 27.248666666666665 type value ndcg_at_100 32.00258333333334 type value ndcg_at_1000 34.9465 type value ndcg_at_3 23.58566666666667 type value ndcg_at_5 25.26341666666666 type value precision_at_1 19.999916666666667 type value precision_at_10 4.772166666666666 type value precision_at_100 0.847 type value precision_at_1000 0.12741666666666668 type value precision_at_3 10.756166666666669 type value precision_at_5 7.725416666666667 type value recall_at_1 17.195166666666665 type value recall_at_10 35.99083333333334 type value recall_at_100 57.467999999999996 type value recall_at_1000 78.82366666666667 type value recall_at_3 25.898499999999995 type value recall_at_5 30.084333333333333 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackStatsRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 16.779 type value map_at_10 21.557000000000002 type value map_at_100 22.338 type value map_at_1000 22.421 type value map_at_3 19.939 type value map_at_5 20.903 type value mrr_at_1 18.404999999999998 type value mrr_at_10 23.435 type value mrr_at_100 24.179000000000002 type value mrr_at_1000 24.25 type value mrr_at_3 21.907 type value mrr_at_5 22.781000000000002 type value ndcg_at_1 18.404999999999998 type value ndcg_at_10 24.515 type value ndcg_at_100 28.721000000000004 type value ndcg_at_1000 31.259999999999998 type value ndcg_at_3 21.508 type value ndcg_at_5 23.01 type value precision_at_1 18.404999999999998 type value precision_at_10 3.834 type value precision_at_100 0.641 type value precision_at_1000 0.093 type value precision_at_3 9.151 type value precision_at_5 6.503 type value recall_at_1 16.779 type value recall_at_10 31.730000000000004 type value recall_at_100 51.673 type value recall_at_1000 71.17599999999999 type value recall_at_3 23.518 type value recall_at_5 27.230999999999998 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackTexRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 9.279 type value map_at_10 13.822000000000001 type value map_at_100 14.533 type value map_at_1000 14.649999999999999 type value map_at_3 12.396 type value map_at_5 13.214 type value mrr_at_1 11.149000000000001 type value mrr_at_10 16.139 type value mrr_at_100 16.872 type value mrr_at_1000 16.964000000000002 type value mrr_at_3 14.613000000000001 type value mrr_at_5 15.486 type value ndcg_at_1 11.149000000000001 type value ndcg_at_10 16.82 type value ndcg_at_100 20.73 type value ndcg_at_1000 23.894000000000002 type value ndcg_at_3 14.11 type value ndcg_at_5 15.404000000000002 type value precision_at_1 11.149000000000001 type value precision_at_10 3.063 type value precision_at_100 0.587 type value precision_at_1000 0.1 type value precision_at_3 6.699 type value precision_at_5 4.928 type value recall_at_1 9.279 type value recall_at_10 23.745 type value recall_at_100 41.873 type value recall_at_1000 64.982 type value recall_at_3 16.152 type value recall_at_5 19.409000000000002 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackUnixRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 16.36 type value map_at_10 21.927 type value map_at_100 22.889 type value map_at_1000 22.994 type value map_at_3 20.433 type value map_at_5 21.337 type value mrr_at_1 18.75 type value mrr_at_10 24.859 type value mrr_at_100 25.746999999999996 type value mrr_at_1000 25.829 type value mrr_at_3 23.383000000000003 type value mrr_at_5 24.297 type value ndcg_at_1 18.75 type value ndcg_at_10 25.372 type value ndcg_at_100 30.342999999999996 type value ndcg_at_1000 33.286 type value ndcg_at_3 22.627 type value ndcg_at_5 24.04 type value precision_at_1 18.75 type value precision_at_10 4.1419999999999995 type value precision_at_100 0.738 type value precision_at_1000 0.11100000000000002 type value precision_at_3 10.261000000000001 type value precision_at_5 7.164 type value recall_at_1 16.36 type value recall_at_10 32.949 type value recall_at_100 55.552 type value recall_at_1000 77.09899999999999 type value recall_at_3 25.538 type value recall_at_5 29.008 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWebmastersRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 17.39 type value map_at_10 23.058 type value map_at_100 24.445 type value map_at_1000 24.637999999999998 type value map_at_3 21.037 type value map_at_5 21.966 type value mrr_at_1 19.96 type value mrr_at_10 26.301000000000002 type value mrr_at_100 27.297 type value mrr_at_1000 27.375 type value mrr_at_3 24.340999999999998 type value mrr_at_5 25.339 type value ndcg_at_1 19.96 type value ndcg_at_10 27.249000000000002 type value ndcg_at_100 32.997 type value ndcg_at_1000 36.359 type value ndcg_at_3 23.519000000000002 type value ndcg_at_5 24.915000000000003 type value precision_at_1 19.96 type value precision_at_10 5.356000000000001 type value precision_at_100 1.198 type value precision_at_1000 0.20400000000000001 type value precision_at_3 10.738 type value precision_at_5 7.904999999999999 type value recall_at_1 17.39 type value recall_at_10 35.254999999999995 type value recall_at_100 61.351 type value recall_at_1000 84.395 type value recall_at_3 25.194 type value recall_at_5 28.546 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWordpressRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 14.238999999999999 type value map_at_10 19.323 type value map_at_100 19.994 type value map_at_1000 20.102999999999998 type value map_at_3 17.631 type value map_at_5 18.401 type value mrr_at_1 15.157000000000002 type value mrr_at_10 20.578 type value mrr_at_100 21.252 type value mrr_at_1000 21.346999999999998 type value mrr_at_3 18.762 type value mrr_at_5 19.713 type value ndcg_at_1 15.157000000000002 type value ndcg_at_10 22.468 type value ndcg_at_100 26.245 type value ndcg_at_1000 29.534 type value ndcg_at_3 18.981 type value ndcg_at_5 20.349999999999998 type value precision_at_1 15.157000000000002 type value precision_at_10 3.512 type value precision_at_100 0.577 type value precision_at_1000 0.091 type value precision_at_3 8.01 type value precision_at_5 5.656 type value recall_at_1 14.238999999999999 type value recall_at_10 31.038 type value recall_at_100 49.122 type value recall_at_1000 74.919 type value recall_at_3 21.436 type value recall_at_5 24.692 task dataset metrics type Retrieval type name config split revision climate-fever MTEB ClimateFEVER default test 392b78eb68c07badcd7c2cd8f39af108375dfcce type value map_at_1 8.828 type value map_at_10 14.982000000000001 type value map_at_100 16.495 type value map_at_1000 16.658 type value map_at_3 12.366000000000001 type value map_at_5 13.655000000000001 type value mrr_at_1 19.088 type value mrr_at_10 29.29 type value mrr_at_100 30.291 type value mrr_at_1000 30.342000000000002 type value mrr_at_3 25.907000000000004 type value mrr_at_5 27.840999999999998 type value ndcg_at_1 19.088 type value ndcg_at_10 21.858 type value ndcg_at_100 28.323999999999998 type value ndcg_at_1000 31.561 type value ndcg_at_3 17.175 type value ndcg_at_5 18.869 type value precision_at_1 19.088 type value precision_at_10 6.9190000000000005 type value precision_at_100 1.376 type value precision_at_1000 0.197 type value precision_at_3 12.703999999999999 type value precision_at_5 9.993 type value recall_at_1 8.828 type value recall_at_10 27.381 type value recall_at_100 50.0 type value recall_at_1000 68.355 type value recall_at_3 16.118 type value recall_at_5 20.587 task dataset metrics type Retrieval type name config split revision dbpedia-entity MTEB DBPedia default test f097057d03ed98220bc7309ddb10b71a54d667d6 type value map_at_1 5.586 type value map_at_10 10.040000000000001 type value map_at_100 12.55 type value map_at_1000 13.123999999999999 type value map_at_3 7.75 type value map_at_5 8.835999999999999 type value mrr_at_1 42.25 type value mrr_at_10 51.205999999999996 type value mrr_at_100 51.818 type value mrr_at_1000 51.855 type value mrr_at_3 48.875 type value mrr_at_5 50.488 type value ndcg_at_1 32.25 type value ndcg_at_10 22.718 type value ndcg_at_100 24.359 type value ndcg_at_1000 29.232000000000003 type value ndcg_at_3 25.974000000000004 type value ndcg_at_5 24.291999999999998 type value precision_at_1 42.25 type value precision_at_10 17.75 type value precision_at_100 5.032 type value precision_at_1000 1.117 type value precision_at_3 28.833 type value precision_at_5 24.25 type value recall_at_1 5.586 type value recall_at_10 14.16 type value recall_at_100 28.051 type value recall_at_1000 45.157000000000004 type value recall_at_3 8.758000000000001 type value recall_at_5 10.975999999999999 task dataset metrics type Classification type name config split revision mteb/emotion MTEB EmotionClassification default test 829147f8f75a25f005913200eb5ed41fae320aa1 type value accuracy 39.075 type value f1 35.01420354708222 task dataset metrics type Retrieval type name config split revision fever MTEB FEVER default test 1429cf27e393599b8b359b9b72c666f96b2525f9 type value map_at_1 43.519999999999996 type value map_at_10 54.368 type value map_at_100 54.918 type value map_at_1000 54.942 type value map_at_3 51.712 type value map_at_5 53.33599999999999 type value mrr_at_1 46.955000000000005 type value mrr_at_10 58.219 type value mrr_at_100 58.73500000000001 type value mrr_at_1000 58.753 type value mrr_at_3 55.518 type value mrr_at_5 57.191 type value ndcg_at_1 46.955000000000005 type value ndcg_at_10 60.45 type value ndcg_at_100 63.047 type value ndcg_at_1000 63.712999999999994 type value ndcg_at_3 55.233 type value ndcg_at_5 58.072 type value precision_at_1 46.955000000000005 type value precision_at_10 8.267 type value precision_at_100 0.962 type value precision_at_1000 0.10300000000000001 type value precision_at_3 22.326999999999998 type value precision_at_5 14.940999999999999 type value recall_at_1 43.519999999999996 type value recall_at_10 75.632 type value recall_at_100 87.41600000000001 type value recall_at_1000 92.557 type value recall_at_3 61.597 type value recall_at_5 68.518 task dataset metrics type Retrieval type name config split revision fiqa MTEB FiQA2018 default test 41b686a7f28c59bcaaa5791efd47c67c8ebe28be type value map_at_1 9.549000000000001 type value map_at_10 15.762 type value map_at_100 17.142 type value map_at_1000 17.329 type value map_at_3 13.575000000000001 type value map_at_5 14.754000000000001 type value mrr_at_1 19.753 type value mrr_at_10 26.568 type value mrr_at_100 27.606 type value mrr_at_1000 27.68 type value mrr_at_3 24.203 type value mrr_at_5 25.668999999999997 type value ndcg_at_1 19.753 type value ndcg_at_10 21.118000000000002 type value ndcg_at_100 27.308 type value ndcg_at_1000 31.304 type value ndcg_at_3 18.319 type value ndcg_at_5 19.414 type value precision_at_1 19.753 type value precision_at_10 6.08 type value precision_at_100 1.204 type value precision_at_1000 0.192 type value precision_at_3 12.191 type value precision_at_5 9.383 type value recall_at_1 9.549000000000001 type value recall_at_10 26.131 type value recall_at_100 50.544999999999995 type value recall_at_1000 74.968 type value recall_at_3 16.951 type value recall_at_5 20.95 task dataset metrics type Retrieval type name config split revision hotpotqa MTEB HotpotQA default test 766870b35a1b9ca65e67a0d1913899973551fc6c type value map_at_1 25.544 type value map_at_10 32.62 type value map_at_100 33.275 type value map_at_1000 33.344 type value map_at_3 30.851 type value map_at_5 31.868999999999996 type value mrr_at_1 51.087 type value mrr_at_10 57.704 type value mrr_at_100 58.175 type value mrr_at_1000 58.207 type value mrr_at_3 56.106 type value mrr_at_5 57.074000000000005 type value ndcg_at_1 51.087 type value ndcg_at_10 40.876000000000005 type value ndcg_at_100 43.762 type value ndcg_at_1000 45.423 type value ndcg_at_3 37.65 type value ndcg_at_5 39.305 type value precision_at_1 51.087 type value precision_at_10 8.304 type value precision_at_100 1.059 type value precision_at_1000 0.128 type value precision_at_3 22.875999999999998 type value precision_at_5 15.033 type value recall_at_1 25.544 type value recall_at_10 41.519 type value recall_at_100 52.957 type value recall_at_1000 64.132 type value recall_at_3 34.315 type value recall_at_5 37.583 task dataset metrics type Classification type name config split revision mteb/imdb MTEB ImdbClassification default test 8d743909f834c38949e8323a8a6ce8721ea6c7f4 type value accuracy 58.6696 type value ap 55.3644880984279 type value f1 58.07942097405652 task dataset metrics type Retrieval type name config split revision msmarco MTEB MSMARCO default validation e6838a846e2408f22cf5cc337ebc83e0bcf77849 type value map_at_1 14.442 type value map_at_10 22.932 type value map_at_100 24.132 type value map_at_1000 24.213 type value map_at_3 20.002 type value map_at_5 21.636 type value mrr_at_1 14.841999999999999 type value mrr_at_10 23.416 type value mrr_at_100 24.593999999999998 type value mrr_at_1000 24.669 type value mrr_at_3 20.494 type value mrr_at_5 22.14 type value ndcg_at_1 14.841999999999999 type value ndcg_at_10 27.975 type value ndcg_at_100 34.143 type value ndcg_at_1000 36.370000000000005 type value ndcg_at_3 21.944 type value ndcg_at_5 24.881 type value precision_at_1 14.841999999999999 type value precision_at_10 4.537 type value precision_at_100 0.767 type value precision_at_1000 0.096 type value precision_at_3 9.322 type value precision_at_5 7.074 type value recall_at_1 14.442 type value recall_at_10 43.557 type value recall_at_100 72.904 type value recall_at_1000 90.40700000000001 type value recall_at_3 27.088 type value recall_at_5 34.144000000000005 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (en) en test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 86.95622435020519 type value f1 86.58363130708494 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (de) de test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 62.73034657650043 type value f1 60.78623915840713 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (es) es test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 67.54503002001334 type value f1 65.34879794116112 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (fr) fr test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 65.35233322893829 type value f1 62.994001882446646 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (hi) hi test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 45.37110075295806 type value f1 44.26285860740745 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (th) th test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 55.276672694394215 type value f1 53.28388179869587 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (en) en test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 62.25262197902417 type value f1 43.44084037148853 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (de) de test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 49.56043956043956 type value f1 32.86333673498598 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (es) es test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 49.93995997331555 type value f1 34.726671876888126 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (fr) fr test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 46.32947071719386 type value f1 32.325273615982795 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (hi) hi test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 32.208676945141626 type value f1 21.32185122815139 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (th) th test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 43.627486437613015 type value f1 27.04872922347508 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (af) af test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 40.548083389374575 type value f1 39.490307545239716 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (am) am test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 24.18291862811029 type value f1 23.437620034727473 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ar) ar test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 30.134498991257562 type value f1 28.787175191531283 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (az) az test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 35.88433086751849 type value f1 36.264500398782126 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (bn) bn test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 29.17283120376597 type value f1 27.8101616531901 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (cy) cy test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 41.788836583725626 type value f1 39.71413181054801 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (da) da test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 44.176193678547406 type value f1 42.192499826552286 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (de) de test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 42.07464694014795 type value f1 39.44188259183162 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (el) el test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 36.254203093476804 type value f1 34.46592715936761 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (en) en test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 61.40887693342301 type value f1 59.79854802683996 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (es) es test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 42.679892400807 type value f1 42.04801248338172 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fa) fa test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 35.59179556153329 type value f1 34.045862930486166 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fi) fi test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 40.036987222595826 type value f1 38.117703439362785 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fr) fr test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 43.43981170141224 type value f1 42.7084388987865 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (he) he test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 31.593813046402154 type value f1 29.98550522450782 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hi) hi test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 27.044384667114997 type value f1 27.313059184832667 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hu) hu test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.453261600538 type value f1 37.309189326110435 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hy) hy test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 27.979152656355076 type value f1 27.430939684346445 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (id) id test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 43.97108271687963 type value f1 43.40585705688761 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (is) is test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 40.302622730329524 type value f1 39.108052180520744 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (it) it test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 45.474108944182916 type value f1 45.85950328241134 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ja) ja test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 45.60860793544048 type value f1 43.94920708216737 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (jv) jv test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.668459986550104 type value f1 37.6990034018859 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ka) ka test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 25.6523201075992 type value f1 25.279084273189582 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (km) km test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 28.295225285810353 type value f1 26.645825638771548 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (kn) kn test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 23.480161398789505 type value f1 22.275241866506732 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ko) ko test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 36.55682582380632 type value f1 36.004753171063605 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (lv) lv test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 41.84936112979153 type value f1 41.38932672359119 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ml) ml test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 24.90921318090114 type value f1 23.968687483768807 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (mn) mn test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 29.86213853396099 type value f1 29.977152075255407 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ms) ms test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 42.42098184263618 type value f1 41.50877432664628 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (my) my test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 25.131136516476126 type value f1 23.938932214086776 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nb) nb test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 39.81506388702084 type value f1 38.809586587791664 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nl) nl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 43.62138533960995 type value f1 42.01386842914633 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pl) pl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 42.19569603227976 type value f1 40.00556559825827 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pt) pt test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 45.20847343644923 type value f1 44.24115005029051 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ro) ro test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 41.80901143241426 type value f1 40.474074848670085 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ru) ru test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 35.96839273705447 type value f1 35.095456843621 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sl) sl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 40.60524546065905 type value f1 39.302383051500136 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sq) sq test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 42.75722932078009 type value f1 41.53763931497389 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sv) sv test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 42.347007397444514 type value f1 41.04366017948627 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sw) sw test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 41.12306657700067 type value f1 39.712940473289024 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ta) ta test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 24.603227975790183 type value f1 23.969236788828606 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (te) te test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 25.03698722259583 type value f1 24.37196123281459 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (th) th test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 35.40013449899126 type value f1 35.063600413688036 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tl) tl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 41.19031607262945 type value f1 40.240432304273014 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tr) tr test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 36.405514458641555 type value f1 36.03844992856558 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ur) ur test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 25.934767989240076 type value f1 25.2074457023531 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (vi) vi test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.79959650302622 type value f1 37.160233794673125 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-CN) zh-CN test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 46.244115669132476 type value f1 44.367480561291906 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-TW) zh-TW test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 42.30665770006724 type value f1 41.9642223283514 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (af) af test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 43.2481506388702 type value f1 40.924230769590785 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (am) am test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 25.30262273032952 type value f1 24.937105830264066 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ar) ar test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 32.07128446536651 type value f1 31.80245816594883 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (az) az test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 36.681237390719566 type value f1 36.37219042508338 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (bn) bn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 29.56624075319435 type value f1 28.386042056362758 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (cy) cy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 42.1049092131809 type value f1 38.926150886991294 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (da) da test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 45.44384667114997 type value f1 42.578252395460005 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (de) de test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 43.211163416274374 type value f1 41.04465858304789 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (el) el test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 36.503026227303295 type value f1 34.49785095312759 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (en) en test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 69.73772696704773 type value f1 69.21759502909043 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (es) es test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 44.078681909885674 type value f1 43.05914426901129 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fa) fa test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 32.61264290517821 type value f1 32.02463177462754 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fi) fi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 40.35642232683255 type value f1 38.13642481807678 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fr) fr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 45.06724949562878 type value f1 43.19827608343738 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (he) he test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 32.178883658372555 type value f1 29.979761884698775 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hi) hi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 26.903160726294555 type value f1 25.833010434083363 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hu) hu test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 40.379959650302624 type value f1 37.93134355292882 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hy) hy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 28.375924680564896 type value f1 26.96255693013172 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (id) id test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 44.361129791526565 type value f1 43.54445012295126 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (is) is test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 39.290517821116346 type value f1 37.26982052174147 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (it) it test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 46.4694014794889 type value f1 44.060986162841566 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ja) ja test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 46.25756556825824 type value f1 45.625139456758816 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (jv) jv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 41.12642905178212 type value f1 39.54392378396527 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ka) ka test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 24.72763954270343 type value f1 23.337743140804484 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (km) km test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 29.741089441829182 type value f1 27.570876190083748 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (kn) kn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 23.850033624747816 type value f1 22.86733484540032 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ko) ko test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 36.56691324815064 type value f1 35.504081677134565 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (lv) lv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 40.928043039677206 type value f1 39.108589131211254 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ml) ml test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 25.527908540685946 type value f1 25.333391622280477 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (mn) mn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 29.105581708137183 type value f1 28.478235012692814 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ms) ms test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 43.78614660390047 type value f1 41.9640143926267 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (my) my test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 27.269670477471415 type value f1 26.228386764141852 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nb) nb test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 39.018157363819775 type value f1 37.641949339321854 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nl) nl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 45.35978480161399 type value f1 42.6851176096831 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pl) pl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 41.89307330195023 type value f1 40.888710642615024 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pt) pt test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 45.901143241425686 type value f1 44.496942353920545 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ro) ro test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 44.11566913248151 type value f1 41.953945105870616 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ru) ru test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 32.76395427034297 type value f1 31.436372571600934 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sl) sl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 40.504371217215876 type value f1 39.322752749628165 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sq) sq test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 42.51849361129792 type value f1 41.4139297118463 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sv) sv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 42.293207800941495 type value f1 40.50409536806683 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sw) sw test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 42.9993275050437 type value f1 41.045416224973266 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ta) ta test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 28.32548755884331 type value f1 27.276841995561867 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (te) te test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 26.593813046402154 type value f1 25.483878616197586 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (th) th test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 36.788836583725626 type value f1 34.603932909177686 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tl) tl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 42.5689307330195 type value f1 40.924469309079825 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tr) tr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 37.09482178883658 type value f1 37.949628822857164 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ur) ur test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 28.836583725622063 type value f1 27.806558655512344 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (vi) vi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 37.357094821788834 type value f1 37.507918961038165 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-CN) zh-CN test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 49.37794216543375 type value f1 47.20421153697707 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-TW) zh-TW test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 44.42165433759248 type value f1 44.34741861198931 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-p2p MTEB MedrxivClusteringP2P default test dcefc037ef84348e49b0d29109e891c01067226b type value v_measure 31.374938993074252 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-s2s MTEB MedrxivClusteringS2S default test 3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc type value v_measure 26.871455379644093 task dataset metrics type Reranking type name config split revision mteb/mind_small MTEB MindSmallReranking default test 3bdac13927fdc888b903db93b2ffdbd90b295a69 type value map 30.402396942935333 type value mrr 31.42600938803256 task dataset metrics type Retrieval type name config split revision nfcorpus MTEB NFCorpus default test 7eb63cc0c1eb59324d709ebed25fcab851fa7610 type value map_at_1 3.7740000000000005 type value map_at_10 7.614999999999999 type value map_at_100 9.574 type value map_at_1000 10.711 type value map_at_3 5.7540000000000004 type value map_at_5 6.6659999999999995 type value mrr_at_1 33.127 type value mrr_at_10 40.351 type value mrr_at_100 41.144 type value mrr_at_1000 41.202 type value mrr_at_3 38.029 type value mrr_at_5 39.190000000000005 type value ndcg_at_1 31.579 type value ndcg_at_10 22.792 type value ndcg_at_100 21.698999999999998 type value ndcg_at_1000 30.892999999999997 type value ndcg_at_3 26.828999999999997 type value ndcg_at_5 25.119000000000003 type value precision_at_1 33.127 type value precision_at_10 16.718 type value precision_at_100 5.7090000000000005 type value precision_at_1000 1.836 type value precision_at_3 24.768 type value precision_at_5 21.3 type value recall_at_1 3.7740000000000005 type value recall_at_10 10.302999999999999 type value recall_at_100 23.013 type value recall_at_1000 54.864999999999995 type value recall_at_3 6.554 type value recall_at_5 8.087 task dataset metrics type Retrieval type name config split revision nq MTEB NQ default test 6062aefc120bfe8ece5897809fb2e53bfe0d128c type value map_at_1 15.620999999999999 type value map_at_10 24.519 type value map_at_100 25.586 type value map_at_1000 25.662000000000003 type value map_at_3 21.619 type value map_at_5 23.232 type value mrr_at_1 17.497 type value mrr_at_10 26.301000000000002 type value mrr_at_100 27.235 type value mrr_at_1000 27.297 type value mrr_at_3 23.561 type value mrr_at_5 25.111 type value ndcg_at_1 17.497 type value ndcg_at_10 29.725 type value ndcg_at_100 34.824 type value ndcg_at_1000 36.907000000000004 type value ndcg_at_3 23.946 type value ndcg_at_5 26.739 type value precision_at_1 17.497 type value precision_at_10 5.2170000000000005 type value precision_at_100 0.8099999999999999 type value precision_at_1000 0.101 type value precision_at_3 11.114 type value precision_at_5 8.285 type value recall_at_1 15.620999999999999 type value recall_at_10 43.999 type value recall_at_100 67.183 type value recall_at_1000 83.174 type value recall_at_3 28.720000000000002 type value recall_at_5 35.154 task dataset metrics type Retrieval type name config split revision quora MTEB QuoraRetrieval default test 6205996560df11e3a3da9ab4f926788fc30a7db4 type value map_at_1 54.717000000000006 type value map_at_10 67.514 type value map_at_100 68.484 type value map_at_1000 68.523 type value map_at_3 64.169 type value map_at_5 66.054 type value mrr_at_1 62.46000000000001 type value mrr_at_10 71.503 type value mrr_at_100 71.91499999999999 type value mrr_at_1000 71.923 type value mrr_at_3 69.46799999999999 type value mrr_at_5 70.677 type value ndcg_at_1 62.480000000000004 type value ndcg_at_10 72.98 type value ndcg_at_100 76.023 type value ndcg_at_1000 76.512 type value ndcg_at_3 68.138 type value ndcg_at_5 70.458 type value precision_at_1 62.480000000000004 type value precision_at_10 11.373 type value precision_at_100 1.437 type value precision_at_1000 0.154 type value precision_at_3 29.622999999999998 type value precision_at_5 19.918 type value recall_at_1 54.717000000000006 type value recall_at_10 84.745 type value recall_at_100 96.528 type value recall_at_1000 99.39 type value recall_at_3 71.60600000000001 type value recall_at_5 77.511 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering MTEB RedditClustering default test b2805658ae38990172679479369a78b86de8c390 type value v_measure 40.23390747226228 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering-p2p MTEB RedditClusteringP2P default test 385e3cb46b4cfa89021f56c4380204149d0efe33 type value v_measure 49.090518272935626 task dataset metrics type Retrieval type name config split revision scidocs MTEB SCIDOCS default test 5c59ef3e437a0a9651c8fe6fde943e7dce59fba5 type value map_at_1 3.028 type value map_at_10 6.968000000000001 type value map_at_100 8.200000000000001 type value map_at_1000 8.432 type value map_at_3 5.3069999999999995 type value map_at_5 6.099 type value mrr_at_1 14.799999999999999 type value mrr_at_10 22.425 type value mrr_at_100 23.577 type value mrr_at_1000 23.669999999999998 type value mrr_at_3 20.233 type value mrr_at_5 21.318 type value ndcg_at_1 14.799999999999999 type value ndcg_at_10 12.206 type value ndcg_at_100 17.799 type value ndcg_at_1000 22.891000000000002 type value ndcg_at_3 12.128 type value ndcg_at_5 10.212 type value precision_at_1 14.799999999999999 type value precision_at_10 6.17 type value precision_at_100 1.428 type value precision_at_1000 0.266 type value precision_at_3 11.333 type value precision_at_5 8.74 type value recall_at_1 3.028 type value recall_at_10 12.522 type value recall_at_100 28.975 type value recall_at_1000 54.038 type value recall_at_3 6.912999999999999 type value recall_at_5 8.883000000000001 task dataset metrics type STS type name config split revision mteb/sickr-sts MTEB SICK-R default test 20a6d6f312dd54037fe07a32d58e5e168867909d type value cos_sim_pearson 76.62983928119752 type value cos_sim_spearman 65.92910683118656 type value euclidean_pearson 71.10290039690963 type value euclidean_spearman 64.80076622426652 type value manhattan_pearson 70.8944726230188 type value manhattan_spearman 64.75082576033986 task dataset metrics type STS type name config split revision mteb/sts12-sts MTEB STS12 default test fdf84275bb8ce4b49c971d02e84dd1abc677a50f type value cos_sim_pearson 74.42679147085553 type value cos_sim_spearman 66.52980061546658 type value euclidean_pearson 74.87039477408763 type value euclidean_spearman 70.63397666902786 type value manhattan_pearson 74.97015137513088 type value manhattan_spearman 70.75951355434326 task dataset metrics type STS type name config split revision mteb/sts13-sts MTEB STS13 default test 1591bfcbe8c69d4bf7fe2a16e2451017832cafb9 type value cos_sim_pearson 75.62472426599543 type value cos_sim_spearman 76.1662886374236 type value euclidean_pearson 76.3297128081315 type value euclidean_spearman 77.19385151966563 type value manhattan_pearson 76.50363291423257 type value manhattan_spearman 77.37081896355399 task dataset metrics type STS type name config split revision mteb/sts14-sts MTEB STS14 default test e2125984e7df8b7871f6ae9949cf6b6795e7c54b type value cos_sim_pearson 74.48227705407035 type value cos_sim_spearman 69.04572664009687 type value euclidean_pearson 71.76138185714849 type value euclidean_spearman 68.93415452043307 type value manhattan_pearson 71.68010915543306 type value manhattan_spearman 68.99176321262806 task dataset metrics type STS type name config split revision mteb/sts15-sts MTEB STS15 default test 1cd7298cac12a96a373b6a2f18738bb3e739a9b6 type value cos_sim_pearson 78.1566527175902 type value cos_sim_spearman 79.23677712825851 type value euclidean_pearson 76.29138438696417 type value euclidean_spearman 77.20108266215374 type value manhattan_pearson 76.27464935799118 type value manhattan_spearman 77.15286174478099 task dataset metrics type STS type name config split revision mteb/sts16-sts MTEB STS16 default test 360a0b2dff98700d09e634a01e1cc1624d3e42cd type value cos_sim_pearson 75.068454465977 type value cos_sim_spearman 76.06792422441929 type value euclidean_pearson 70.64605440627699 type value euclidean_spearman 70.21776051117844 type value manhattan_pearson 70.32479295054918 type value manhattan_spearman 69.89782458638528 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (ko-ko) ko-ko test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 39.43327289939437 type value cos_sim_spearman 52.386010275505654 type value euclidean_pearson 46.40999904885745 type value euclidean_spearman 51.00333465175934 type value manhattan_pearson 46.55753533133655 type value manhattan_spearman 51.07550440519388 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (ar-ar) ar-ar test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 55.54431928210687 type value cos_sim_spearman 55.61674586076298 type value euclidean_pearson 58.07442713714088 type value euclidean_spearman 55.74066216931719 type value manhattan_pearson 57.84021675638542 type value manhattan_spearman 55.20365812536853 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-ar) en-ar test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 11.378463868809098 type value cos_sim_spearman 8.209569244801065 type value euclidean_pearson 1.07041700730406 type value euclidean_spearman 2.2052197108931892 type value manhattan_pearson 0.7671300251104268 type value manhattan_spearman 3.430645020535567 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-de) en-de test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 32.71403560929013 type value cos_sim_spearman 30.18181775929109 type value euclidean_pearson 25.57368595910298 type value euclidean_spearman 23.316649115731376 type value manhattan_pearson 24.144200325329614 type value manhattan_spearman 21.64621546338457 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-en) en-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 83.36340470799158 type value cos_sim_spearman 84.95398260629699 type value euclidean_pearson 80.69876969911644 type value euclidean_spearman 80.97451731130427 type value manhattan_pearson 80.65869354146945 type value manhattan_spearman 80.8540858718528 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-tr) en-tr test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 1.9200044163754912 type value cos_sim_spearman 1.0393399782021342 type value euclidean_pearson 1.1376003191297994 type value euclidean_spearman 1.8947106671763914 type value manhattan_pearson 3.8362564474484335 type value manhattan_spearman 4.242750882792888 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (es-en) es-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 26.561262451099577 type value cos_sim_spearman 28.776666666659906 type value euclidean_pearson 14.640410196999088 type value euclidean_spearman 16.10557011701786 type value manhattan_pearson 15.019405495911272 type value manhattan_spearman 15.37192083104197 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (es-es) es-es test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 69.7544202001433 type value cos_sim_spearman 71.88444295144646 type value euclidean_pearson 73.84934185952773 type value euclidean_spearman 73.26911108021089 type value manhattan_pearson 74.04354196954574 type value manhattan_spearman 73.37650787943872 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (fr-en) fr-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 27.70511842301491 type value cos_sim_spearman 26.339466714066447 type value euclidean_pearson 9.323158236506385 type value euclidean_spearman 7.32083231520273 type value manhattan_pearson 7.807399527573071 type value manhattan_spearman 5.525546663067113 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (it-en) it-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 24.226521799447692 type value cos_sim_spearman 20.72992940458968 type value euclidean_pearson 6.753378617205011 type value euclidean_spearman 6.281654679029505 type value manhattan_pearson 7.087180250449323 type value manhattan_spearman 6.41611659259516 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (nl-en) nl-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 29.131412364061234 type value cos_sim_spearman 25.053429612793547 type value euclidean_pearson 10.657141303962 type value euclidean_spearman 9.712124819778452 type value manhattan_pearson 12.481782693315688 type value manhattan_spearman 11.287958480905973 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (en) en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 64.04750650962879 type value cos_sim_spearman 65.66183708171826 type value euclidean_pearson 66.90887604405887 type value euclidean_spearman 66.89814072484552 type value manhattan_pearson 67.31627110509089 type value manhattan_spearman 67.01048176165322 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de) de test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 19.26519187000913 type value cos_sim_spearman 21.987647321429005 type value euclidean_pearson 17.850618752342946 type value euclidean_spearman 22.86669392885474 type value manhattan_pearson 18.16183594260708 type value manhattan_spearman 23.637510352837907 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es) es test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 34.221261828226936 type value cos_sim_spearman 49.811823238907664 type value euclidean_pearson 44.50394399762147 type value euclidean_spearman 50.959184495072876 type value manhattan_pearson 45.83191034038624 type value manhattan_spearman 50.190409866117946 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (pl) pl test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 3.620381732096531 type value cos_sim_spearman 23.30843951799194 type value euclidean_pearson 0.965453312113125 type value euclidean_spearman 24.235967620790316 type value manhattan_pearson 1.4408922275701606 type value manhattan_spearman 25.161920137046096 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (tr) tr test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 16.69489628726267 type value cos_sim_spearman 34.66348380997687 type value euclidean_pearson 29.415825529188606 type value euclidean_spearman 38.33011033170646 type value manhattan_pearson 31.23273195263394 type value manhattan_spearman 39.10055785755795 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (ar) ar test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 9.134927430889528 type value cos_sim_spearman 28.18922448944151 type value euclidean_pearson 19.86814169549051 type value euclidean_spearman 27.519588644948627 type value manhattan_pearson 21.80949221238945 type value manhattan_spearman 28.25217200494078 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (ru) ru test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 3.6386482942352085 type value cos_sim_spearman 9.068119621940966 type value euclidean_pearson 0.8123129118737714 type value euclidean_spearman 9.173672890166147 type value manhattan_pearson 0.754518899822658 type value manhattan_spearman 8.431719541986524 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh) zh test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 2.972091574908432 type value cos_sim_spearman 25.48511383289232 type value euclidean_pearson 12.751569670148918 type value euclidean_spearman 24.940721642439286 type value manhattan_pearson 14.310238482989826 type value manhattan_spearman 24.69821216148647 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (fr) fr test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 54.4745185734135 type value cos_sim_spearman 67.66493409568727 type value euclidean_pearson 60.13580336797049 type value euclidean_spearman 66.12319300814538 type value manhattan_pearson 60.816210368708155 type value manhattan_spearman 65.70010026716766 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-en) de-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 49.37865412588201 type value cos_sim_spearman 53.07135629778897 type value euclidean_pearson 49.29201416711091 type value euclidean_spearman 50.54523702399645 type value manhattan_pearson 51.265764141268534 type value manhattan_spearman 51.979086403193605 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es-en) es-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 44.925652392562135 type value cos_sim_spearman 49.51253904767726 type value euclidean_pearson 48.79346518897415 type value euclidean_spearman 51.47957870101565 type value manhattan_pearson 49.51314553898044 type value manhattan_spearman 51.895207893189166 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (it) it test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 45.241690321111875 type value cos_sim_spearman 48.24795739512037 type value euclidean_pearson 49.22719494399897 type value euclidean_spearman 49.64102442042809 type value manhattan_pearson 49.497887732970256 type value manhattan_spearman 49.940515338096304 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (pl-en) pl-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 36.42138324083909 type value cos_sim_spearman 36.79867489417801 type value euclidean_pearson 27.760612942610084 type value euclidean_spearman 29.140966500287625 type value manhattan_pearson 28.456674031350115 type value manhattan_spearman 27.46356370924497 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh-en) zh-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 26.55350664089358 type value cos_sim_spearman 28.681707196975008 type value euclidean_pearson 12.613577889195138 type value euclidean_spearman 13.589493311702933 type value manhattan_pearson 11.640157427420958 type value manhattan_spearman 10.345223941212415 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es-it) es-it test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 38.54682179114309 type value cos_sim_spearman 45.782560880405704 type value euclidean_pearson 46.496857002368486 type value euclidean_spearman 48.21270426410012 type value manhattan_pearson 46.871839119374044 type value manhattan_spearman 47.556987773851525 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-fr) de-fr test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 35.12956772546032 type value cos_sim_spearman 32.96920218281008 type value euclidean_pearson 34.23140384382136 type value euclidean_spearman 32.19303153191447 type value manhattan_pearson 34.189468276600635 type value manhattan_spearman 34.887065709732376 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-pl) de-pl test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 30.507667380509634 type value cos_sim_spearman 20.447284723752716 type value euclidean_pearson 29.662041381794474 type value euclidean_spearman 20.939990379746757 type value manhattan_pearson 32.5112080506328 type value manhattan_spearman 23.773047901712495 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (fr-pl) fr-pl test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 71.10820459712156 type value cos_sim_spearman 61.97797868009122 type value euclidean_pearson 60.30910689156633 type value euclidean_spearman 61.97797868009122 type value manhattan_pearson 66.3405176964038 type value manhattan_spearman 61.97797868009122 task dataset metrics type STS type name config split revision mteb/stsbenchmark-sts MTEB STSBenchmark default test 8913289635987208e6e7c72789e4be2fe94b6abd type value cos_sim_pearson 76.53032504460737 type value cos_sim_spearman 75.33716094627373 type value euclidean_pearson 69.64662673290599 type value euclidean_spearman 67.30188896368857 type value manhattan_pearson 69.45096082050807 type value manhattan_spearman 67.0718727259371 task dataset metrics type Reranking type name config split revision mteb/scidocs-reranking MTEB SciDocsRR default test 56a6d0140cf6356659e2a7c1413286a774468d44 type value map 71.33941904192648 type value mrr 89.73766429648782 task dataset metrics type Retrieval type name config split revision scifact MTEB SciFact default test a75ae049398addde9b70f6b268875f5cbce99089 type value map_at_1 43.333 type value map_at_10 52.364 type value map_at_100 53.184 type value map_at_1000 53.234 type value map_at_3 49.832 type value map_at_5 51.244 type value mrr_at_1 45.333 type value mrr_at_10 53.455 type value mrr_at_100 54.191 type value mrr_at_1000 54.235 type value mrr_at_3 51.556000000000004 type value mrr_at_5 52.622 type value ndcg_at_1 45.333 type value ndcg_at_10 56.899 type value ndcg_at_100 60.702 type value ndcg_at_1000 62.046 type value ndcg_at_3 52.451 type value ndcg_at_5 54.534000000000006 type value precision_at_1 45.333 type value precision_at_10 7.8 type value precision_at_100 0.987 type value precision_at_1000 0.11 type value precision_at_3 20.778 type value precision_at_5 13.866999999999999 type value recall_at_1 43.333 type value recall_at_10 69.69999999999999 type value recall_at_100 86.9 type value recall_at_1000 97.6 type value recall_at_3 57.81699999999999 type value recall_at_5 62.827999999999996 task dataset metrics type PairClassification type name config split revision mteb/sprintduplicatequestions-pairclassification MTEB SprintDuplicateQuestions default test 5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea type value cos_sim_accuracy 99.7 type value cos_sim_ap 89.88577913120001 type value cos_sim_f1 84.62694041061593 type value cos_sim_precision 84.7542627883651 type value cos_sim_recall 84.5 type value dot_accuracy 99.24752475247524 type value dot_ap 56.81855467290009 type value dot_f1 56.084126189283936 type value dot_precision 56.16850551654965 type value dot_recall 56.00000000000001 type value euclidean_accuracy 99.7059405940594 type value euclidean_ap 90.12451226491524 type value euclidean_f1 84.44211629125196 type value euclidean_precision 88.66886688668868 type value euclidean_recall 80.60000000000001 type value manhattan_accuracy 99.7128712871287 type value manhattan_ap 90.67590584183216 type value manhattan_f1 84.85436893203884 type value manhattan_precision 82.45283018867924 type value manhattan_recall 87.4 type value max_accuracy 99.7128712871287 type value max_ap 90.67590584183216 type value max_f1 84.85436893203884 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering MTEB StackExchangeClustering default test 70a89468f6dccacc6aa2b12a6eac54e74328f235 type value v_measure 52.74481093815175 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering-p2p MTEB StackExchangeClusteringP2P default test d88009ab563dd0b16cfaf4436abaf97fa3550cf0 type value v_measure 32.65999453562101 task dataset metrics type Reranking type name config split revision mteb/stackoverflowdupquestions-reranking MTEB StackOverflowDupQuestions default test ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9 type value map 44.74498464555465 type value mrr 45.333879764026825 task dataset metrics type Summarization type name config split revision mteb/summeval MTEB SummEval default test 8753c2788d36c01fc6f05d03fe3f7268d63f9122 type value cos_sim_pearson 29,603788751645216 type value cos_sim_spearman 29.705103354786033 type value dot_pearson 28.07425338095399 type value dot_spearman 26.841406359135367 task dataset metrics type Retrieval type name config split revision trec-covid MTEB TRECCOVID default test 2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217 type value map_at_1 0.241 type value map_at_10 1.672 type value map_at_100 7.858999999999999 type value map_at_1000 17.616 type value map_at_3 0.631 type value map_at_5 0.968 type value mrr_at_1 90.0 type value mrr_at_10 92.952 type value mrr_at_100 93.036 type value mrr_at_1000 93.036 type value mrr_at_3 92.667 type value mrr_at_5 92.667 type value ndcg_at_1 83.0 type value ndcg_at_10 70.30199999999999 type value ndcg_at_100 48.149 type value ndcg_at_1000 40.709 type value ndcg_at_3 79.173 type value ndcg_at_5 75.347 type value precision_at_1 90.0 type value precision_at_10 72.6 type value precision_at_100 48.46 type value precision_at_1000 18.093999999999998 type value precision_at_3 84.0 type value precision_at_5 78.8 type value recall_at_1 0.241 type value recall_at_10 1.814 type value recall_at_100 11.141 type value recall_at_1000 37.708999999999996 type value recall_at_3 0.647 type value recall_at_5 1.015 task dataset metrics type Retrieval type name config split revision webis-touche2020 MTEB Touche2020 default test 527b7d77e16e343303e68cb6af11d6e18b9f7b3b type value map_at_1 2.782 type value map_at_10 9.06 type value map_at_100 14.571000000000002 type value map_at_1000 16.006999999999998 type value map_at_3 5.037 type value map_at_5 6.63 type value mrr_at_1 34.694 type value mrr_at_10 48.243 type value mrr_at_100 49.065 type value mrr_at_1000 49.065 type value mrr_at_3 44.897999999999996 type value mrr_at_5 46.428999999999995 type value ndcg_at_1 31.633 type value ndcg_at_10 22.972 type value ndcg_at_100 34.777 type value ndcg_at_1000 45.639 type value ndcg_at_3 26.398 type value ndcg_at_5 24.418 type value precision_at_1 34.694 type value precision_at_10 19.796 type value precision_at_100 7.224 type value precision_at_1000 1.4449999999999998 type value precision_at_3 26.531 type value precision_at_5 23.265 type value recall_at_1 2.782 type value recall_at_10 14.841 type value recall_at_100 44.86 type value recall_at_1000 78.227 type value recall_at_3 5.959 type value recall_at_5 8.969000000000001 task dataset metrics type Classification type name config split revision mteb/toxic_conversations_50k MTEB ToxicConversationsClassification default test edfaf9da55d3dd50d43143d90c1ac476895ae6de type value accuracy 62.657999999999994 type value ap 10.96353161716344 type value f1 48.294226423442645 task dataset metrics type Classification type name config split revision mteb/tweet_sentiment_extraction MTEB TweetSentimentExtractionClassification default test 62146448f05be9e52a36b8ee9936447ea787eede type value accuracy 52.40803621958121 type value f1 52.61009636022186 task dataset metrics type Clustering type name config split revision mteb/twentynewsgroups-clustering MTEB TwentyNewsgroupsClustering default test 091a54f9a36281ce7d6590ec8c75dd485e7e01d4 type value v_measure 32.12697126747911 task dataset metrics type PairClassification type name config split revision mteb/twittersemeval2015-pairclassification MTEB TwitterSemEval2015 default test 70970daeab8776df92f5ea462b6173c0b46fd2d1 type value cos_sim_accuracy 80.69976753889253 type value cos_sim_ap 54.74680676121268 type value cos_sim_f1 53.18923998590391 type value cos_sim_precision 47.93563413084904 type value cos_sim_recall 59.73614775725594 type value dot_accuracy 79.3348036001669 type value dot_ap 48.46902128933627 type value dot_f1 50.480109739369006 type value dot_precision 42.06084051345173 type value dot_recall 63.113456464379944 type value euclidean_accuracy 79.78780473266973 type value euclidean_ap 50.258327255164815 type value euclidean_f1 49.655838666827684 type value euclidean_precision 45.78044978846582 type value euclidean_recall 54.24802110817942 type value manhattan_accuracy 79.76992310901831 type value manhattan_ap 49.89892485714363 type value manhattan_f1 49.330433787341185 type value manhattan_precision 43.56175459874672 type value manhattan_recall 56.86015831134564 type value max_accuracy 80.69976753889253 type value max_ap 54.74680676121268 type value max_f1 53.18923998590391 task dataset metrics type PairClassification type name config split revision mteb/twitterurlcorpus-pairclassification MTEB TwitterURLCorpus default test 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf type value cos_sim_accuracy 86.90573213800597 type value cos_sim_ap 81.05760818661524 type value cos_sim_f1 73.64688856729379 type value cos_sim_precision 69.46491946491946 type value cos_sim_recall 78.3646442870342 type value dot_accuracy 83.80680715644041 type value dot_ap 72.49774005947461 type value dot_f1 68.68460650173216 type value dot_precision 62.954647507858105 type value dot_recall 75.56205728364644 type value euclidean_accuracy 85.97430822369697 type value euclidean_ap 78.86101740829326 type value euclidean_f1 71.07960824663695 type value euclidean_precision 70.36897306270279 type value euclidean_recall 71.8047428395442 type value manhattan_accuracy 85.94132029339853 type value manhattan_ap 78.77876711171923 type value manhattan_f1 71.07869075515912 type value manhattan_precision 69.80697847067557 type value manhattan_recall 72.39759778256852 type value max_accuracy 86.90573213800597 type value max_ap 81.05760818661524 type value max_f1 73.64688856729379

SGPT-125M-weightedmean-msmarco-specb-bitfit

Usage

For usage instructions, refer to our codebase: https://github.com/Muennighoff/sgpt

Evaluation Results

For eval results, refer to the eval folder or our paper: https://arxiv.org/abs/2202.08904

Training

The model was trained with the parameters:

DataLoader:

torch.utils.data.dataloader.DataLoader of length 15600 with parameters:

{'batch_size': 32, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}

Loss:

sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss with parameters:

{'scale': 20.0, 'similarity_fct': 'cos_sim'}

Parameters of the fit()-Method:

{
    "epochs": 10,
    "evaluation_steps": 0,
    "evaluator": "NoneType",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 0.0002
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 1000,
    "weight_decay": 0.01
}

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 300, 'do_lower_case': False}) with Transformer model: GPTNeoModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
)

Citing & Authors

@article{muennighoff2022sgpt,
  title={SGPT: GPT Sentence Embeddings for Semantic Search},
  author={Muennighoff, Niklas},
  journal={arXiv preprint arXiv:2202.08904},
  year={2022}
}