SGPT-125M-weightedmean-nli-bitfit/README.md

pipeline_tag, tags, model-index

pipeline_tag	tags	model-index
sentence-similarity	sentence-transformers feature-extraction sentence-similarity mteb	name results SGPT-125M-weightedmean-nli-bitfit task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en) en test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 65.88059701492537 type value ap 28.685493163579785 type value f1 59.79951005816335 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (de) de test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 59.07922912205568 type value ap 73.91887421019034 type value f1 56.6316368658711 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en-ext) en-ext test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 64.91754122938531 type value ap 16.360681214864226 type value f1 53.126592061523766 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (ja) ja test 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 type value accuracy 56.423982869378996 type value ap 12.143003571907899 type value f1 45.76363777987471 task dataset metrics type Classification type name config split revision mteb/amazon_polarity MTEB AmazonPolarityClassification default test 80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1 type value accuracy 74.938225 type value ap 69.58187110320567 type value f1 74.72744058439321 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (en) en test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 35.098 type value f1 34.73265651435726 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (de) de test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 24.516 type value f1 24.21748200448397 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (es) es test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 29.097999999999995 type value f1 28.620040162757093 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (fr) fr test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 27.395999999999997 type value f1 27.146888644986284 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (ja) ja test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 21.724 type value f1 21.37230564276654 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (zh) zh test c379a6705fec24a2493fa68e011692605f44e119 type value accuracy 23.976 type value f1 23.741137981755482 task dataset metrics type Retrieval type name config split revision arguana MTEB ArguAna default test 5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3 type value map_at_1 13.442000000000002 type value map_at_10 24.275 type value map_at_100 25.588 type value map_at_1000 25.659 type value map_at_3 20.092 type value map_at_5 22.439999999999998 type value ndcg_at_1 13.442000000000002 type value ndcg_at_10 31.04 type value ndcg_at_100 37.529 type value ndcg_at_1000 39.348 type value ndcg_at_3 22.342000000000002 type value ndcg_at_5 26.595999999999997 type value precision_at_1 13.442000000000002 type value precision_at_10 5.299 type value precision_at_100 0.836 type value precision_at_1000 0.098 type value precision_at_3 9.625 type value precision_at_5 7.852 type value recall_at_1 13.442000000000002 type value recall_at_10 52.986999999999995 type value recall_at_100 83.64200000000001 type value recall_at_1000 97.795 type value recall_at_3 28.876 type value recall_at_5 39.26 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-p2p MTEB ArxivClusteringP2P default test 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8 type value v_measure 34.742482477870766 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-s2s MTEB ArxivClusteringS2S default test b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3 type value v_measure 24.67870651472156 task dataset metrics type Clustering type name config split revision slvnwhrl/blurbs-clustering-s2s MTEB BlurbsClusteringS2S default test 9bfff9a7f8f6dc6ffc9da71c48dd48b68696471d type value v_measure 8.00311862863495 task dataset metrics type Reranking type name config split revision mteb/askubuntudupquestions-reranking MTEB AskUbuntuDupQuestions default test 4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c type value map 52.63439984994702 type value mrr 65.75704612408214 task dataset metrics type STS type name config split revision mteb/biosses-sts MTEB BIOSSES default test 9ee918f184421b6bd48b78f6c714d86546106103 type value cos_sim_pearson 72.78000135012542 type value cos_sim_spearman 70.92812216947605 type value euclidean_pearson 77.1169214949292 type value euclidean_spearman 77.10175681583313 type value manhattan_pearson 76.84527031837595 type value manhattan_spearman 77.0704308008438 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (de-en) de-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 1.0960334029227559 type value f1 1.0925539318023658 type value precision 1.0908141962421711 type value recall 1.0960334029227559 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (fr-en) fr-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 0.02201188641866608 type value f1 0.02201188641866608 type value precision 0.02201188641866608 type value recall 0.02201188641866608 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (ru-en) ru-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 0.0 type value f1 0.0 type value precision 0.0 type value recall 0.0 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (zh-en) zh-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 0.0 type value f1 0.0 type value precision 0.0 type value recall 0.0 task dataset metrics type Classification type name config split revision mteb/banking77 MTEB Banking77Classification default test 44fa15921b4c889113cc5df03dd4901b49161ab7 type value accuracy 74.67857142857142 type value f1 74.61743413995573 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-p2p MTEB BiorxivClusteringP2P default test 11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55 type value v_measure 28.93427045246491 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-s2s MTEB BiorxivClusteringS2S default test c0fab014e1bcb8d3a5e31b2088972a1e01547dc1 type value v_measure 23.080939123955474 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackAndroidRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 18.221999999999998 type value map_at_10 24.506 type value map_at_100 25.611 type value map_at_1000 25.758 type value map_at_3 22.264999999999997 type value map_at_5 23.698 type value ndcg_at_1 23.033 type value ndcg_at_10 28.719 type value ndcg_at_100 33.748 type value ndcg_at_1000 37.056 type value ndcg_at_3 25.240000000000002 type value ndcg_at_5 27.12 type value precision_at_1 23.033 type value precision_at_10 5.408 type value precision_at_100 1.004 type value precision_at_1000 0.158 type value precision_at_3 11.874 type value precision_at_5 8.927 type value recall_at_1 18.221999999999998 type value recall_at_10 36.355 type value recall_at_100 58.724 type value recall_at_1000 81.33500000000001 type value recall_at_3 26.334000000000003 type value recall_at_5 31.4 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackEnglishRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 12.058 type value map_at_10 16.051000000000002 type value map_at_100 16.772000000000002 type value map_at_1000 16.871 type value map_at_3 14.78 type value map_at_5 15.5 type value ndcg_at_1 15.35 type value ndcg_at_10 18.804000000000002 type value ndcg_at_100 22.346 type value ndcg_at_1000 25.007 type value ndcg_at_3 16.768 type value ndcg_at_5 17.692 type value precision_at_1 15.35 type value precision_at_10 3.51 type value precision_at_100 0.664 type value precision_at_1000 0.11100000000000002 type value precision_at_3 7.983 type value precision_at_5 5.656 type value recall_at_1 12.058 type value recall_at_10 23.644000000000002 type value recall_at_100 39.76 type value recall_at_1000 58.56 type value recall_at_3 17.541999999999998 type value recall_at_5 20.232 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGamingRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 21.183 type value map_at_10 28.9 type value map_at_100 29.858 type value map_at_1000 29.953999999999997 type value map_at_3 26.58 type value map_at_5 27.912 type value ndcg_at_1 24.765 type value ndcg_at_10 33.339999999999996 type value ndcg_at_100 37.997 type value ndcg_at_1000 40.416000000000004 type value ndcg_at_3 29.044999999999998 type value ndcg_at_5 31.121 type value precision_at_1 24.765 type value precision_at_10 5.599 type value precision_at_100 0.8699999999999999 type value precision_at_1000 0.11499999999999999 type value precision_at_3 13.270999999999999 type value precision_at_5 9.367 type value recall_at_1 21.183 type value recall_at_10 43.875 type value recall_at_100 65.005 type value recall_at_1000 83.017 type value recall_at_3 32.232 type value recall_at_5 37.308 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGisRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 11.350999999999999 type value map_at_10 14.953 type value map_at_100 15.623000000000001 type value map_at_1000 15.716 type value map_at_3 13.603000000000002 type value map_at_5 14.343 type value ndcg_at_1 12.429 type value ndcg_at_10 17.319000000000003 type value ndcg_at_100 20.990000000000002 type value ndcg_at_1000 23.899 type value ndcg_at_3 14.605 type value ndcg_at_5 15.89 type value precision_at_1 12.429 type value precision_at_10 2.701 type value precision_at_100 0.48700000000000004 type value precision_at_1000 0.078 type value precision_at_3 6.026 type value precision_at_5 4.3839999999999995 type value recall_at_1 11.350999999999999 type value recall_at_10 23.536 type value recall_at_100 40.942 type value recall_at_1000 64.05 type value recall_at_3 16.195 type value recall_at_5 19.264 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackMathematicaRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 8.08 type value map_at_10 11.691 type value map_at_100 12.312 type value map_at_1000 12.439 type value map_at_3 10.344000000000001 type value map_at_5 10.996 type value ndcg_at_1 10.697 type value ndcg_at_10 14.48 type value ndcg_at_100 18.160999999999998 type value ndcg_at_1000 21.886 type value ndcg_at_3 11.872 type value ndcg_at_5 12.834000000000001 type value precision_at_1 10.697 type value precision_at_10 2.811 type value precision_at_100 0.551 type value precision_at_1000 0.10200000000000001 type value precision_at_3 5.804 type value precision_at_5 4.154 type value recall_at_1 8.08 type value recall_at_10 20.235 type value recall_at_100 37.525999999999996 type value recall_at_1000 65.106 type value recall_at_3 12.803999999999998 type value recall_at_5 15.498999999999999 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackPhysicsRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 13.908999999999999 type value map_at_10 19.256 type value map_at_100 20.286 type value map_at_1000 20.429 type value map_at_3 17.399 type value map_at_5 18.398999999999997 type value ndcg_at_1 17.421 type value ndcg_at_10 23.105999999999998 type value ndcg_at_100 28.128999999999998 type value ndcg_at_1000 31.480999999999998 type value ndcg_at_3 19.789 type value ndcg_at_5 21.237000000000002 type value precision_at_1 17.421 type value precision_at_10 4.331 type value precision_at_100 0.839 type value precision_at_1000 0.131 type value precision_at_3 9.4 type value precision_at_5 6.776 type value recall_at_1 13.908999999999999 type value recall_at_10 31.086999999999996 type value recall_at_100 52.946000000000005 type value recall_at_1000 76.546 type value recall_at_3 21.351 type value recall_at_5 25.264999999999997 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackProgrammersRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 12.598 type value map_at_10 17.304 type value map_at_100 18.209 type value map_at_1000 18.328 type value map_at_3 15.784 type value map_at_5 16.669999999999998 type value ndcg_at_1 15.867999999999999 type value ndcg_at_10 20.623 type value ndcg_at_100 25.093 type value ndcg_at_1000 28.498 type value ndcg_at_3 17.912 type value ndcg_at_5 19.198 type value precision_at_1 15.867999999999999 type value precision_at_10 3.7670000000000003 type value precision_at_100 0.716 type value precision_at_1000 0.11800000000000001 type value precision_at_3 8.638 type value precision_at_5 6.21 type value recall_at_1 12.598 type value recall_at_10 27.144000000000002 type value recall_at_100 46.817 type value recall_at_1000 71.86099999999999 type value recall_at_3 19.231 type value recall_at_5 22.716 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 12.738416666666666 type value map_at_10 17.235916666666668 type value map_at_100 18.063333333333333 type value map_at_1000 18.18433333333333 type value map_at_3 15.74775 type value map_at_5 16.57825 type value ndcg_at_1 15.487416666666665 type value ndcg_at_10 20.290166666666668 type value ndcg_at_100 24.41291666666666 type value ndcg_at_1000 27.586333333333336 type value ndcg_at_3 17.622083333333332 type value ndcg_at_5 18.859916666666667 type value precision_at_1 15.487416666666665 type value precision_at_10 3.6226666666666665 type value precision_at_100 0.6820833333333334 type value precision_at_1000 0.11216666666666666 type value precision_at_3 8.163749999999999 type value precision_at_5 5.865416666666667 type value recall_at_1 12.738416666666666 type value recall_at_10 26.599416666666663 type value recall_at_100 45.41258333333334 type value recall_at_1000 68.7565 type value recall_at_3 19.008166666666668 type value recall_at_5 22.24991666666667 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackStatsRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 12.307 type value map_at_10 15.440000000000001 type value map_at_100 16.033 type value map_at_1000 16.14 type value map_at_3 14.393 type value map_at_5 14.856 type value ndcg_at_1 14.571000000000002 type value ndcg_at_10 17.685000000000002 type value ndcg_at_100 20.882 type value ndcg_at_1000 23.888 type value ndcg_at_3 15.739 type value ndcg_at_5 16.391 type value precision_at_1 14.571000000000002 type value precision_at_10 2.883 type value precision_at_100 0.49100000000000005 type value precision_at_1000 0.08 type value precision_at_3 7.0040000000000004 type value precision_at_5 4.693 type value recall_at_1 12.307 type value recall_at_10 22.566 type value recall_at_100 37.469 type value recall_at_1000 60.550000000000004 type value recall_at_3 16.742 type value recall_at_5 18.634 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackTexRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 6.496 type value map_at_10 9.243 type value map_at_100 9.841 type value map_at_1000 9.946000000000002 type value map_at_3 8.395 type value map_at_5 8.872 type value ndcg_at_1 8.224 type value ndcg_at_10 11.24 type value ndcg_at_100 14.524999999999999 type value ndcg_at_1000 17.686 type value ndcg_at_3 9.617 type value ndcg_at_5 10.37 type value precision_at_1 8.224 type value precision_at_10 2.0820000000000003 type value precision_at_100 0.443 type value precision_at_1000 0.08499999999999999 type value precision_at_3 4.623 type value precision_at_5 3.331 type value recall_at_1 6.496 type value recall_at_10 15.310000000000002 type value recall_at_100 30.680000000000003 type value recall_at_1000 54.335 type value recall_at_3 10.691 type value recall_at_5 12.687999999999999 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackUnixRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 13.843 type value map_at_10 17.496000000000002 type value map_at_100 18.304000000000002 type value map_at_1000 18.426000000000002 type value map_at_3 16.225 type value map_at_5 16.830000000000002 type value ndcg_at_1 16.698 type value ndcg_at_10 20.301 type value ndcg_at_100 24.523 type value ndcg_at_1000 27.784 type value ndcg_at_3 17.822 type value ndcg_at_5 18.794 type value precision_at_1 16.698 type value precision_at_10 3.3579999999999997 type value precision_at_100 0.618 type value precision_at_1000 0.101 type value precision_at_3 7.898 type value precision_at_5 5.428999999999999 type value recall_at_1 13.843 type value recall_at_10 25.887999999999998 type value recall_at_100 45.028 type value recall_at_1000 68.991 type value recall_at_3 18.851000000000003 type value recall_at_5 21.462 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWebmastersRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 13.757 type value map_at_10 19.27 type value map_at_100 20.461 type value map_at_1000 20.641000000000002 type value map_at_3 17.865000000000002 type value map_at_5 18.618000000000002 type value ndcg_at_1 16.996 type value ndcg_at_10 22.774 type value ndcg_at_100 27.675 type value ndcg_at_1000 31.145 type value ndcg_at_3 20.691000000000003 type value ndcg_at_5 21.741 type value precision_at_1 16.996 type value precision_at_10 4.545 type value precision_at_100 1.036 type value precision_at_1000 0.185 type value precision_at_3 10.145 type value precision_at_5 7.391 type value recall_at_1 13.757 type value recall_at_10 28.233999999999998 type value recall_at_100 51.05499999999999 type value recall_at_1000 75.35300000000001 type value recall_at_3 21.794 type value recall_at_5 24.614 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWordpressRetrieval default test 2b9f5791698b5be7bc5e10535c8690f20043c3db type value map_at_1 9.057 type value map_at_10 12.720999999999998 type value map_at_100 13.450000000000001 type value map_at_1000 13.564000000000002 type value map_at_3 11.34 type value map_at_5 12.245000000000001 type value ndcg_at_1 9.797 type value ndcg_at_10 15.091 type value ndcg_at_100 18.886 type value ndcg_at_1000 22.29 type value ndcg_at_3 12.365 type value ndcg_at_5 13.931 type value precision_at_1 9.797 type value precision_at_10 2.477 type value precision_at_100 0.466 type value precision_at_1000 0.082 type value precision_at_3 5.299 type value precision_at_5 4.067 type value recall_at_1 9.057 type value recall_at_10 21.319 type value recall_at_100 38.999 type value recall_at_1000 65.374 type value recall_at_3 14.331 type value recall_at_5 17.916999999999998 task dataset metrics type Retrieval type name config split revision climate-fever MTEB ClimateFEVER default test 392b78eb68c07badcd7c2cd8f39af108375dfcce type value map_at_1 3.714 type value map_at_10 6.926 type value map_at_100 7.879 type value map_at_1000 8.032 type value map_at_3 5.504 type value map_at_5 6.357 type value ndcg_at_1 8.86 type value ndcg_at_10 11.007 type value ndcg_at_100 16.154 type value ndcg_at_1000 19.668 type value ndcg_at_3 8.103 type value ndcg_at_5 9.456000000000001 type value precision_at_1 8.86 type value precision_at_10 3.7199999999999998 type value precision_at_100 0.9169999999999999 type value precision_at_1000 0.156 type value precision_at_3 6.254 type value precision_at_5 5.380999999999999 type value recall_at_1 3.714 type value recall_at_10 14.382 type value recall_at_100 33.166000000000004 type value recall_at_1000 53.444 type value recall_at_3 7.523000000000001 type value recall_at_5 10.91 task dataset metrics type Retrieval type name config split revision dbpedia-entity MTEB DBPedia default test f097057d03ed98220bc7309ddb10b71a54d667d6 type value map_at_1 1.764 type value map_at_10 3.8600000000000003 type value map_at_100 5.457 type value map_at_1000 5.938000000000001 type value map_at_3 2.667 type value map_at_5 3.2199999999999998 type value ndcg_at_1 14.000000000000002 type value ndcg_at_10 10.868 type value ndcg_at_100 12.866 type value ndcg_at_1000 17.43 type value ndcg_at_3 11.943 type value ndcg_at_5 11.66 type value precision_at_1 19.25 type value precision_at_10 10.274999999999999 type value precision_at_100 3.527 type value precision_at_1000 0.9119999999999999 type value precision_at_3 14.917 type value precision_at_5 13.5 type value recall_at_1 1.764 type value recall_at_10 6.609 type value recall_at_100 17.616 type value recall_at_1000 33.085 type value recall_at_3 3.115 type value recall_at_5 4.605 task dataset metrics type Classification type name config split revision mteb/emotion MTEB EmotionClassification default test 829147f8f75a25f005913200eb5ed41fae320aa1 type value accuracy 42.225 type value f1 37.563516542112104 task dataset metrics type Retrieval type name config split revision fever MTEB FEVER default test 1429cf27e393599b8b359b9b72c666f96b2525f9 type value map_at_1 11.497 type value map_at_10 15.744 type value map_at_100 16.3 type value map_at_1000 16.365 type value map_at_3 14.44 type value map_at_5 15.18 type value ndcg_at_1 12.346 type value ndcg_at_10 18.398999999999997 type value ndcg_at_100 21.399 type value ndcg_at_1000 23.442 type value ndcg_at_3 15.695 type value ndcg_at_5 17.027 type value precision_at_1 12.346 type value precision_at_10 2.798 type value precision_at_100 0.445 type value precision_at_1000 0.063 type value precision_at_3 6.586 type value precision_at_5 4.665 type value recall_at_1 11.497 type value recall_at_10 25.636 type value recall_at_100 39.894 type value recall_at_1000 56.181000000000004 type value recall_at_3 18.273 type value recall_at_5 21.474 task dataset metrics type Retrieval type name config split revision fiqa MTEB FiQA2018 default test 41b686a7f28c59bcaaa5791efd47c67c8ebe28be type value map_at_1 3.637 type value map_at_10 6.084 type value map_at_100 6.9190000000000005 type value map_at_1000 7.1080000000000005 type value map_at_3 5.071 type value map_at_5 5.5649999999999995 type value ndcg_at_1 7.407 type value ndcg_at_10 8.94 type value ndcg_at_100 13.594999999999999 type value ndcg_at_1000 18.29 type value ndcg_at_3 7.393 type value ndcg_at_5 7.854 type value precision_at_1 7.407 type value precision_at_10 2.778 type value precision_at_100 0.75 type value precision_at_1000 0.154 type value precision_at_3 5.144 type value precision_at_5 3.981 type value recall_at_1 3.637 type value recall_at_10 11.821 type value recall_at_100 30.18 type value recall_at_1000 60.207 type value recall_at_3 6.839 type value recall_at_5 8.649 task dataset metrics type Retrieval type name config split revision hotpotqa MTEB HotpotQA default test 766870b35a1b9ca65e67a0d1913899973551fc6c type value map_at_1 9.676 type value map_at_10 13.350999999999999 type value map_at_100 13.919 type value map_at_1000 14.01 type value map_at_3 12.223 type value map_at_5 12.812000000000001 type value ndcg_at_1 19.352 type value ndcg_at_10 17.727 type value ndcg_at_100 20.837 type value ndcg_at_1000 23.412 type value ndcg_at_3 15.317 type value ndcg_at_5 16.436 type value precision_at_1 19.352 type value precision_at_10 3.993 type value precision_at_100 0.651 type value precision_at_1000 0.1 type value precision_at_3 9.669 type value precision_at_5 6.69 type value recall_at_1 9.676 type value recall_at_10 19.966 type value recall_at_100 32.573 type value recall_at_1000 49.905 type value recall_at_3 14.504 type value recall_at_5 16.725 task dataset metrics type Classification type name config split revision mteb/imdb MTEB ImdbClassification default test 8d743909f834c38949e8323a8a6ce8721ea6c7f4 type value accuracy 62.895999999999994 type value ap 58.47769349850157 type value f1 62.67885149592086 task dataset metrics type Retrieval type name config split revision msmarco MTEB MSMARCO default validation e6838a846e2408f22cf5cc337ebc83e0bcf77849 type value map_at_1 2.88 type value map_at_10 4.914000000000001 type value map_at_100 5.459 type value map_at_1000 5.538 type value map_at_3 4.087 type value map_at_5 4.518 type value ndcg_at_1 2.937 type value ndcg_at_10 6.273 type value ndcg_at_100 9.426 type value ndcg_at_1000 12.033000000000001 type value ndcg_at_3 4.513 type value ndcg_at_5 5.292 type value precision_at_1 2.937 type value precision_at_10 1.089 type value precision_at_100 0.27699999999999997 type value precision_at_1000 0.051000000000000004 type value precision_at_3 1.9290000000000003 type value precision_at_5 1.547 type value recall_at_1 2.88 type value recall_at_10 10.578 type value recall_at_100 26.267000000000003 type value recall_at_1000 47.589999999999996 type value recall_at_3 5.673 type value recall_at_5 7.545 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (en) en test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 81.51846785225717 type value f1 81.648869152345 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (de) de test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 60.37475345167653 type value f1 58.452649375517026 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (es) es test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 67.36824549699799 type value f1 65.35927434998516 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (fr) fr test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 63.12871907297212 type value f1 61.37620329272278 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (hi) hi test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 47.04553603442094 type value f1 46.20389912644561 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (th) th test a7e2a951126a26fc8c6a69f835f33a346ba259e3 type value accuracy 52.282097649186255 type value f1 50.75489206473579 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (en) en test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 58.2421340629275 type value f1 40.11696046622642 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (de) de test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 45.069033530571986 type value f1 30.468468273374967 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (es) es test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 48.80920613742495 type value f1 32.65985375400447 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (fr) fr test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 44.337613529595984 type value f1 29.302047435606436 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (hi) hi test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 34.198637504481894 type value f1 22.063706032248408 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (th) th test 6299947a7777084cc2d4b64235bf7190381ce755 type value accuracy 43.11030741410488 type value f1 26.92408933648504 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (af) af test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 37.79421654337593 type value f1 36.81580701507746 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (am) am test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 23.722259583053127 type value f1 23.235269695764273 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ar) ar test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 29.64021519838601 type value f1 28.273175327650137 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (az) az test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 39.4754539340955 type value f1 39.25997361415121 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (bn) bn test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 26.550100874243444 type value f1 25.607924873522975 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (cy) cy test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.78278412911904 type value f1 37.64180582626517 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (da) da test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 43.557498318762605 type value f1 41.35305173800667 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (de) de test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 40.39340954942838 type value f1 38.33393219528934 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (el) el test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 37.28648285137861 type value f1 36.64005906680284 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (en) en test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 58.080026899798256 type value f1 56.49243881660991 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (es) es test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 41.176866173503704 type value f1 40.66779962225799 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fa) fa test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 36.422326832548755 type value f1 34.6441738042885 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fi) fi test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.75588433086752 type value f1 37.26725894668694 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (fr) fr test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 43.67182246133153 type value f1 42.351846624566605 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (he) he test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 31.980497646267658 type value f1 30.557928872809008 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hi) hi test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 28.039677202420982 type value f1 28.428418145508306 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hu) hu test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.13718897108272 type value f1 37.057406988196874 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (hy) hy test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 26.05245460659045 type value f1 25.25483953344816 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (id) id test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 41.156691324815064 type value f1 40.83715033247605 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (is) is test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.62811028917284 type value f1 37.67691901246032 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (it) it test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 44.0383322125084 type value f1 43.77259010877456 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ja) ja test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 46.20712844653666 type value f1 44.66632875940824 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (jv) jv test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 37.60591795561533 type value f1 36.581071742378015 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ka) ka test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 24.47209145931405 type value f1 24.238209697895606 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (km) km test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 26.23739071956961 type value f1 25.378783150845052 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (kn) kn test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 17.831203765971754 type value f1 17.275078420466343 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ko) ko test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 37.266308002689975 type value f1 36.92473791708214 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (lv) lv test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 40.93140551445864 type value f1 40.825227889641965 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ml) ml test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 17.88500336247478 type value f1 17.621569082971817 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (mn) mn test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 32.975790181573636 type value f1 33.402014633349665 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ms) ms test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 40.91123066577001 type value f1 40.09538559124075 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (my) my test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 17.834566240753194 type value f1 17.006381849454314 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nb) nb test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 39.47881640887693 type value f1 37.819934317839305 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (nl) nl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 41.76193678547412 type value f1 40.281991759509694 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pl) pl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 42.61936785474109 type value f1 40.83673914649905 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (pt) pt test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 44.54270342972427 type value f1 43.45243164278448 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ro) ro test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 39.96973772696705 type value f1 38.74209466530094 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ru) ru test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 37.461331540013454 type value f1 36.91132021821187 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sl) sl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.28850033624748 type value f1 37.37259394049676 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sq) sq test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 40.95494283792872 type value f1 39.767707902869084 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sv) sv test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 41.85272360457296 type value f1 40.42848260365438 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (sw) sw test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.328850033624754 type value f1 36.90334596675622 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ta) ta test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 19.031607262945528 type value f1 18.66510306325761 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (te) te test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 19.38466711499664 type value f1 19.186399376652535 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (th) th test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 34.088769334229994 type value f1 34.20383086009429 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tl) tl test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 40.285810356422324 type value f1 39.361500249640414 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (tr) tr test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.860121049092136 type value f1 37.81916859627235 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (ur) ur test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 27.834566240753194 type value f1 26.898389386106487 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (vi) vi test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 38.70544720914593 type value f1 38.280026442024415 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-CN) zh-CN test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 45.78009414929387 type value f1 44.21526778674136 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-TW) zh-TW test 072a486a144adf7f4479a4a0dddb2152e161e1ea type value accuracy 42.32010759919301 type value f1 42.25772977490916 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (af) af test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 40.24546065904506 type value f1 38.79924050989544 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (am) am test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 25.68930733019502 type value f1 25.488166279162712 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ar) ar test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 32.39744451916611 type value f1 31.863029579075775 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (az) az test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 40.53127101546738 type value f1 39.707079033948936 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (bn) bn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 27.23268325487559 type value f1 26.443653281858793 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (cy) cy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 38.69872225958305 type value f1 36.55930387892567 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (da) da test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 44.75453934095494 type value f1 42.87356484024154 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (de) de test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 41.355077336919976 type value f1 39.82365179458047 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (el) el test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 38.43981170141224 type value f1 37.02538368296387 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (en) en test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 66.33826496301278 type value f1 65.89634765029932 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (es) es test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 44.17955615332885 type value f1 43.10228811620319 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fa) fa test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 34.82851378614661 type value f1 33.95952441502803 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fi) fi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 40.561533288500335 type value f1 38.04939011733627 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fr) fr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 45.917955615332886 type value f1 44.65741971572902 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (he) he test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 32.08473436449227 type value f1 29.53932929808133 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hi) hi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 28.369199731002016 type value f1 27.52902837981212 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hu) hu test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 39.49226630800269 type value f1 37.3272340470504 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hy) hy test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 25.904505716207133 type value f1 24.547396574853444 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (id) id test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 40.95830531271016 type value f1 40.177843177422226 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (is) is test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 38.564223268325485 type value f1 37.35307758495248 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (it) it test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 46.58708809683928 type value f1 44.103900526804985 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ja) ja test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 46.24747814391393 type value f1 45.4107101796664 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (jv) jv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 39.6570275722932 type value f1 38.82737576832412 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ka) ka test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 25.279085406859448 type value f1 23.662661686788493 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (km) km test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 28.97108271687962 type value f1 27.195758324189246 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (kn) kn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 19.27370544720915 type value f1 18.694271924323637 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ko) ko test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 35.729657027572294 type value f1 34.38287006177308 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (lv) lv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 39.57296570275723 type value f1 38.074945140886925 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ml) ml test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 19.895763281775388 type value f1 20.00931364846829 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (mn) mn test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 32.431069266980494 type value f1 31.395958664782576 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ms) ms test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 42.32347007397445 type value f1 40.81374026314701 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (my) my test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 20.864156018829856 type value f1 20.409870408935436 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nb) nb test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 40.47074646940148 type value f1 39.19044149415904 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nl) nl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 43.591123066577 type value f1 41.43420363064241 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pl) pl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 41.876260928043045 type value f1 41.192117676667614 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pt) pt test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 46.30800268997983 type value f1 45.25536730126799 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ro) ro test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 42.525218560860786 type value f1 41.02418109296485 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ru) ru test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 35.94821788836584 type value f1 35.08598314806566 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sl) sl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 38.69199731002017 type value f1 37.68119408674127 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sq) sq test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 40.474108944182916 type value f1 39.480530387013594 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sv) sv test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 41.523201075991935 type value f1 40.20097996024383 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sw) sw test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 39.54942837928716 type value f1 38.185561243338064 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ta) ta test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 22.8782784129119 type value f1 22.239467186721456 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (te) te test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 20.51445864156019 type value f1 19.999047885530217 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (th) th test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 34.92602555480834 type value f1 33.24016717215723 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tl) tl test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 40.74983187626093 type value f1 39.30274328728882 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tr) tr test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 39.06859448554136 type value f1 39.21542039662971 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ur) ur test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 29.747814391392062 type value f1 28.261836892220447 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (vi) vi test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 38.02286482851379 type value f1 37.8742438608697 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-CN) zh-CN test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 48.550773369199725 type value f1 46.7399625882649 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-TW) zh-TW test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 45.17821116341628 type value f1 44.84809741811729 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-p2p MTEB MedrxivClusteringP2P default test dcefc037ef84348e49b0d29109e891c01067226b type value v_measure 28.301902023313875 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-s2s MTEB MedrxivClusteringS2S default test 3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc type value v_measure 24.932123582259287 task dataset metrics type Reranking type name config split revision mteb/mind_small MTEB MindSmallReranking default test 3bdac13927fdc888b903db93b2ffdbd90b295a69 type value map 29.269341041468326 type value mrr 30.132140876875717 task dataset metrics type Retrieval type name config split revision nfcorpus MTEB NFCorpus default test 7eb63cc0c1eb59324d709ebed25fcab851fa7610 type value map_at_1 1.2269999999999999 type value map_at_10 3.081 type value map_at_100 4.104 type value map_at_1000 4.989 type value map_at_3 2.221 type value map_at_5 2.535 type value ndcg_at_1 15.015 type value ndcg_at_10 11.805 type value ndcg_at_100 12.452 type value ndcg_at_1000 22.284000000000002 type value ndcg_at_3 13.257 type value ndcg_at_5 12.199 type value precision_at_1 16.409000000000002 type value precision_at_10 9.102 type value precision_at_100 3.678 type value precision_at_1000 1.609 type value precision_at_3 12.797 type value precision_at_5 10.464 type value recall_at_1 1.2269999999999999 type value recall_at_10 5.838 type value recall_at_100 15.716 type value recall_at_1000 48.837 type value recall_at_3 2.828 type value recall_at_5 3.697 task dataset metrics type Retrieval type name config split revision nq MTEB NQ default test 6062aefc120bfe8ece5897809fb2e53bfe0d128c type value map_at_1 3.515 type value map_at_10 5.884 type value map_at_100 6.510000000000001 type value map_at_1000 6.598999999999999 type value map_at_3 4.8919999999999995 type value map_at_5 5.391 type value ndcg_at_1 4.056 type value ndcg_at_10 7.6259999999999994 type value ndcg_at_100 11.08 type value ndcg_at_1000 13.793 type value ndcg_at_3 5.537 type value ndcg_at_5 6.45 type value precision_at_1 4.056 type value precision_at_10 1.4569999999999999 type value precision_at_100 0.347 type value precision_at_1000 0.061 type value precision_at_3 2.6069999999999998 type value precision_at_5 2.086 type value recall_at_1 3.515 type value recall_at_10 12.312 type value recall_at_100 28.713 type value recall_at_1000 50.027 type value recall_at_3 6.701 type value recall_at_5 8.816 task dataset metrics type Retrieval type name config split revision quora MTEB QuoraRetrieval default test 6205996560df11e3a3da9ab4f926788fc30a7db4 type value map_at_1 61.697 type value map_at_10 74.20400000000001 type value map_at_100 75.023 type value map_at_1000 75.059 type value map_at_3 71.265 type value map_at_5 73.001 type value ndcg_at_1 70.95 type value ndcg_at_10 78.96 type value ndcg_at_100 81.26 type value ndcg_at_1000 81.679 type value ndcg_at_3 75.246 type value ndcg_at_5 77.092 type value precision_at_1 70.95 type value precision_at_10 11.998000000000001 type value precision_at_100 1.451 type value precision_at_1000 0.154 type value precision_at_3 32.629999999999995 type value precision_at_5 21.573999999999998 type value recall_at_1 61.697 type value recall_at_10 88.23299999999999 type value recall_at_100 96.961 type value recall_at_1000 99.401 type value recall_at_3 77.689 type value recall_at_5 82.745 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering MTEB RedditClustering default test b2805658ae38990172679479369a78b86de8c390 type value v_measure 33.75741018380938 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering-p2p MTEB RedditClusteringP2P default test 385e3cb46b4cfa89021f56c4380204149d0efe33 type value v_measure 41.00799910099266 task dataset metrics type Retrieval type name config split revision scidocs MTEB SCIDOCS default test 5c59ef3e437a0a9651c8fe6fde943e7dce59fba5 type value map_at_1 1.72 type value map_at_10 3.8240000000000003 type value map_at_100 4.727 type value map_at_1000 4.932 type value map_at_3 2.867 type value map_at_5 3.3230000000000004 type value ndcg_at_1 8.5 type value ndcg_at_10 7.133000000000001 type value ndcg_at_100 11.911 type value ndcg_at_1000 16.962 type value ndcg_at_3 6.763 type value ndcg_at_5 5.832 type value precision_at_1 8.5 type value precision_at_10 3.6799999999999997 type value precision_at_100 1.0670000000000002 type value precision_at_1000 0.22999999999999998 type value precision_at_3 6.2330000000000005 type value precision_at_5 5.0200000000000005 type value recall_at_1 1.72 type value recall_at_10 7.487000000000001 type value recall_at_100 21.683 type value recall_at_1000 46.688 type value recall_at_3 3.798 type value recall_at_5 5.113 task dataset metrics type STS type name config split revision mteb/sickr-sts MTEB SICK-R default test 20a6d6f312dd54037fe07a32d58e5e168867909d type value cos_sim_pearson 80.96286245858941 type value cos_sim_spearman 74.57093488947429 type value euclidean_pearson 75.50377970259402 type value euclidean_spearman 71.7498004622999 type value manhattan_pearson 75.3256836091382 type value manhattan_spearman 71.80676733410375 task dataset metrics type STS type name config split revision mteb/sts12-sts MTEB STS12 default test fdf84275bb8ce4b49c971d02e84dd1abc677a50f type value cos_sim_pearson 80.20938796088339 type value cos_sim_spearman 69.16914010333394 type value euclidean_pearson 79.33415250097545 type value euclidean_spearman 71.46707320292745 type value manhattan_pearson 79.73669837981976 type value manhattan_spearman 71.87919511134902 task dataset metrics type STS type name config split revision mteb/sts13-sts MTEB STS13 default test 1591bfcbe8c69d4bf7fe2a16e2451017832cafb9 type value cos_sim_pearson 76.401935081936 type value cos_sim_spearman 77.23446219694267 type value euclidean_pearson 74.61017160439877 type value euclidean_spearman 75.85871531365609 type value manhattan_pearson 74.83034779539724 type value manhattan_spearman 75.95948993588429 task dataset metrics type STS type name config split revision mteb/sts14-sts MTEB STS14 default test e2125984e7df8b7871f6ae9949cf6b6795e7c54b type value cos_sim_pearson 75.35551963935667 type value cos_sim_spearman 70.98892671568665 type value euclidean_pearson 73.24467338564628 type value euclidean_spearman 71.97533151639425 type value manhattan_pearson 73.2776559359938 type value manhattan_spearman 72.2221421456084 task dataset metrics type STS type name config split revision mteb/sts15-sts MTEB STS15 default test 1cd7298cac12a96a373b6a2f18738bb3e739a9b6 type value cos_sim_pearson 79.05293131911803 type value cos_sim_spearman 79.7379478259805 type value euclidean_pearson 78.17016171851057 type value euclidean_spearman 78.76038607583105 type value manhattan_pearson 78.4994607532332 type value manhattan_spearman 79.13026720132872 task dataset metrics type STS type name config split revision mteb/sts16-sts MTEB STS16 default test 360a0b2dff98700d09e634a01e1cc1624d3e42cd type value cos_sim_pearson 76.04750373932828 type value cos_sim_spearman 77.93230986462234 type value euclidean_pearson 75.8320302521164 type value euclidean_spearman 76.83154481579385 type value manhattan_pearson 75.98713517720608 type value manhattan_spearman 76.95479705521507 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (ko-ko) ko-ko test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 43.0464619152799 type value cos_sim_spearman 45.65606588928089 type value euclidean_pearson 45.69437788355499 type value euclidean_spearman 45.08552742346606 type value manhattan_pearson 45.87166698903681 type value manhattan_spearman 45.155963016434164 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (ar-ar) ar-ar test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 53.27469278912148 type value cos_sim_spearman 54.16113207623789 type value euclidean_pearson 55.97026429327157 type value euclidean_spearman 54.71320909074608 type value manhattan_pearson 56.12511774278802 type value manhattan_spearman 55.22875659158676 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-ar) en-ar test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 1.5482997790039945 type value cos_sim_spearman 1.7208386347363582 type value euclidean_pearson 6.727915670345885 type value euclidean_spearman 6.112826908474543 type value manhattan_pearson 4.94386093060865 type value manhattan_spearman 5.018174110623732 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-de) en-de test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 27.5420218362265 type value cos_sim_spearman 25.483838431031007 type value euclidean_pearson 6.268684143856358 type value euclidean_spearman 5.877961421091679 type value manhattan_pearson 2.667237739227861 type value manhattan_spearman 2.5683839956554775 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-en) en-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 85.32029757646663 type value cos_sim_spearman 87.32720847297225 type value euclidean_pearson 81.12594485791254 type value euclidean_spearman 81.1531079489332 type value manhattan_pearson 81.32899414704019 type value manhattan_spearman 81.3897040261192 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-tr) en-tr test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 4.37162299241808 type value cos_sim_spearman 2.0879072561774543 type value euclidean_pearson 3.0725243785454595 type value euclidean_spearman 5.3721339279483535 type value manhattan_pearson 4.867795293367359 type value manhattan_spearman 7.9397069840018775 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (es-en) es-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 20.306030448858603 type value cos_sim_spearman 21.93220782551375 type value euclidean_pearson 3.878631934602361 type value euclidean_spearman 5.171796902725965 type value manhattan_pearson 7.13020644036815 type value manhattan_spearman 7.707315591498748 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (es-es) es-es test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 66.81873207478459 type value cos_sim_spearman 67.80273445636502 type value euclidean_pearson 70.60654682977268 type value euclidean_spearman 69.4566208379486 type value manhattan_pearson 70.9548461896642 type value manhattan_spearman 69.78323323058773 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (fr-en) fr-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 21.366487281202602 type value cos_sim_spearman 18.90627528698481 type value euclidean_pearson 2.3390998579461995 type value euclidean_spearman 4.151213674012541 type value manhattan_pearson 2.234831868844863 type value manhattan_spearman 4.555291328501442 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (it-en) it-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 20.73153177251085 type value cos_sim_spearman 16.3855949033176 type value euclidean_pearson 8.734648741714238 type value euclidean_spearman 10.75672244732182 type value manhattan_pearson 7.536654126608877 type value manhattan_spearman 8.330065460047296 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (nl-en) nl-en test 9fc37e8c632af1c87a3d23e685d49552a02582a0 type value cos_sim_pearson 26.618435024084253 type value cos_sim_spearman 23.488974089577816 type value euclidean_pearson 3.1310350304707866 type value euclidean_spearman 3.1242598481634665 type value manhattan_pearson 1.1096752982707008 type value manhattan_spearman 1.4591693078765848 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (en) en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 59.17638344661753 type value cos_sim_spearman 59.636760071130865 type value euclidean_pearson 56.68753290255448 type value euclidean_spearman 57.613280258574484 type value manhattan_pearson 56.92312052723706 type value manhattan_spearman 57.76774918418505 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de) de test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 10.322254716987457 type value cos_sim_spearman 11.0033092996862 type value euclidean_pearson 6.006926471684402 type value euclidean_spearman 10.972140246688376 type value manhattan_pearson 5.933298751861177 type value manhattan_spearman 11.030111585680233 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es) es test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 43.38031880545056 type value cos_sim_spearman 43.05358201410913 type value euclidean_pearson 42.72327196362553 type value euclidean_spearman 42.55163899944477 type value manhattan_pearson 44.01557499780587 type value manhattan_spearman 43.12473221615855 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (pl) pl test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 4.291290504363136 type value cos_sim_spearman 14.912727487893479 type value euclidean_pearson 3.2855132112394485 type value euclidean_spearman 16.575204463951025 type value manhattan_pearson 3.2398776723465814 type value manhattan_spearman 16.841985772913855 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (tr) tr test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 4.102739498555817 type value cos_sim_spearman 3.818238576547375 type value euclidean_pearson 2.3181033496453556 type value euclidean_spearman 5.1826811802703565 type value manhattan_pearson 4.8006179265256455 type value manhattan_spearman 6.738401400306252 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (ar) ar test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 2.38765395226737 type value cos_sim_spearman 5.173899391162327 type value euclidean_pearson 3.0710263954769825 type value euclidean_spearman 5.04922290903982 type value manhattan_pearson 3.7826314109861703 type value manhattan_spearman 5.042238232170212 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (ru) ru test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 7.6735490672676345 type value cos_sim_spearman 3.3631215256878892 type value euclidean_pearson 4.64331702652217 type value euclidean_spearman 3.6129205171334324 type value manhattan_pearson 4.011231736076196 type value manhattan_spearman 3.233959766173701 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh) zh test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 0.06167614416104335 type value cos_sim_spearman 6.521685391703255 type value euclidean_pearson 4.884572579069032 type value euclidean_spearman 5.59058032900239 type value manhattan_pearson 6.139838096573897 type value manhattan_spearman 5.0060884837066215 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (fr) fr test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 53.19490347682836 type value cos_sim_spearman 54.56055727079527 type value euclidean_pearson 52.55574442039842 type value euclidean_spearman 52.94640154371587 type value manhattan_pearson 53.275993040454196 type value manhattan_spearman 53.174561503510155 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-en) de-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 51.151158530122146 type value cos_sim_spearman 53.926925081736655 type value euclidean_pearson 44.55629287737235 type value euclidean_spearman 46.222372143731384 type value manhattan_pearson 42.831322151459005 type value manhattan_spearman 45.70991764985799 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es-en) es-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 30.36194885126792 type value cos_sim_spearman 32.739632941633836 type value euclidean_pearson 29.83135800843496 type value euclidean_spearman 31.114406001326923 type value manhattan_pearson 31.264502938148286 type value manhattan_spearman 33.3112040753475 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (it) it test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 35.23883630335275 type value cos_sim_spearman 33.67797082086704 type value euclidean_pearson 34.878640693874544 type value euclidean_spearman 33.525189235133496 type value manhattan_pearson 34.22761246389947 type value manhattan_spearman 32.713218497609176 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (pl-en) pl-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 19.809302548119547 type value cos_sim_spearman 20.540370202115497 type value euclidean_pearson 23.006803962133016 type value euclidean_spearman 22.96270653079511 type value manhattan_pearson 25.40168317585851 type value manhattan_spearman 25.421508137540865 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh-en) zh-en test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 20.393500955410488 type value cos_sim_spearman 26.705713693011603 type value euclidean_pearson 18.168376767724585 type value euclidean_spearman 19.260826601517245 type value manhattan_pearson 18.302619990671527 type value manhattan_spearman 19.4691037846159 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (es-it) es-it test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 36.58919983075148 type value cos_sim_spearman 35.989722099974045 type value euclidean_pearson 41.045112547574206 type value euclidean_spearman 39.322301680629835 type value manhattan_pearson 41.36802503205308 type value manhattan_spearman 40.76270030293609 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-fr) de-fr test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 26.350936227950083 type value cos_sim_spearman 25.108218032460343 type value euclidean_pearson 28.61681094744849 type value euclidean_spearman 27.350990203943592 type value manhattan_pearson 30.527977072984513 type value manhattan_spearman 26.403339990640813 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (de-pl) de-pl test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 20.056269198600322 type value cos_sim_spearman 20.939990379746757 type value euclidean_pearson 18.942765438962198 type value euclidean_spearman 21.709842967237446 type value manhattan_pearson 23.643909798655123 type value manhattan_spearman 23.58828328071473 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (fr-pl) fr-pl test 2de6ce8c1921b71a755b262c6b57fef195dd7906 type value cos_sim_pearson 19.563740271419395 type value cos_sim_spearman 5.634361698190111 type value euclidean_pearson 16.833522619239474 type value euclidean_spearman 16.903085094570333 type value manhattan_pearson 5.805392712660814 type value manhattan_spearman 16.903085094570333 task dataset metrics type STS type name config split revision mteb/stsbenchmark-sts MTEB STSBenchmark default test 8913289635987208e6e7c72789e4be2fe94b6abd type value cos_sim_pearson 80.00905671833966 type value cos_sim_spearman 79.54269211027272 type value euclidean_pearson 79.51954544247441 type value euclidean_spearman 78.93670303434288 type value manhattan_pearson 79.47610653340678 type value manhattan_spearman 79.07344156719613 task dataset metrics type Reranking type name config split revision mteb/scidocs-reranking MTEB SciDocsRR default test 56a6d0140cf6356659e2a7c1413286a774468d44 type value map 68.35710819755543 type value mrr 88.05442832403617 task dataset metrics type Retrieval type name config split revision scifact MTEB SciFact default test a75ae049398addde9b70f6b268875f5cbce99089 type value map_at_1 21.556 type value map_at_10 27.982000000000003 type value map_at_100 28.937 type value map_at_1000 29.058 type value map_at_3 25.644 type value map_at_5 26.996 type value ndcg_at_1 23.333000000000002 type value ndcg_at_10 31.787 type value ndcg_at_100 36.647999999999996 type value ndcg_at_1000 39.936 type value ndcg_at_3 27.299 type value ndcg_at_5 29.659000000000002 type value precision_at_1 23.333000000000002 type value precision_at_10 4.867 type value precision_at_100 0.743 type value precision_at_1000 0.10200000000000001 type value precision_at_3 11.333 type value precision_at_5 8.133 type value recall_at_1 21.556 type value recall_at_10 42.333 type value recall_at_100 65.706 type value recall_at_1000 91.489 type value recall_at_3 30.361 type value recall_at_5 36.222 task dataset metrics type PairClassification type name config split revision mteb/sprintduplicatequestions-pairclassification MTEB SprintDuplicateQuestions default test 5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea type value cos_sim_accuracy 99.49306930693069 type value cos_sim_ap 77.7308550291728 type value cos_sim_f1 71.78978681209718 type value cos_sim_precision 71.1897738446411 type value cos_sim_recall 72.39999999999999 type value dot_accuracy 99.08118811881188 type value dot_ap 30.267748833368234 type value dot_f1 34.335201222618444 type value dot_precision 34.994807892004154 type value dot_recall 33.7 type value euclidean_accuracy 99.51683168316832 type value euclidean_ap 78.64498778235628 type value euclidean_f1 73.09149972929075 type value euclidean_precision 79.69303423848878 type value euclidean_recall 67.5 type value manhattan_accuracy 99.53168316831683 type value manhattan_ap 79.45274878693958 type value manhattan_f1 74.19863373620599 type value manhattan_precision 78.18383167220377 type value manhattan_recall 70.6 type value max_accuracy 99.53168316831683 type value max_ap 79.45274878693958 type value max_f1 74.19863373620599 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering MTEB StackExchangeClustering default test 70a89468f6dccacc6aa2b12a6eac54e74328f235 type value v_measure 44.59127540530939 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering-p2p MTEB StackExchangeClusteringP2P default test d88009ab563dd0b16cfaf4436abaf97fa3550cf0 type value v_measure 28.230204578753636 task dataset metrics type Reranking type name config split revision mteb/stackoverflowdupquestions-reranking MTEB StackOverflowDupQuestions default test ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9 type value map 39.96520488022785 type value mrr 40.189248047703934 task dataset metrics type Summarization type name config split revision mteb/summeval MTEB SummEval default test 8753c2788d36c01fc6f05d03fe3f7268d63f9122 type value cos_sim_pearson 30.56303767714449 type value cos_sim_spearman 30.256847004390487 type value dot_pearson 29.453520030995005 type value dot_spearman 29.561732550926777 task dataset metrics type Retrieval type name config split revision trec-covid MTEB TRECCOVID default test 2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217 type value map_at_1 0.11299999999999999 type value map_at_10 0.733 type value map_at_100 3.313 type value map_at_1000 7.355 type value map_at_3 0.28200000000000003 type value map_at_5 0.414 type value ndcg_at_1 42.0 type value ndcg_at_10 39.31 type value ndcg_at_100 26.904 type value ndcg_at_1000 23.778 type value ndcg_at_3 42.775999999999996 type value ndcg_at_5 41.554 type value precision_at_1 48.0 type value precision_at_10 43.0 type value precision_at_100 27.08 type value precision_at_1000 11.014 type value precision_at_3 48.0 type value precision_at_5 45.6 type value recall_at_1 0.11299999999999999 type value recall_at_10 0.976 type value recall_at_100 5.888 type value recall_at_1000 22.634999999999998 type value recall_at_3 0.329 type value recall_at_5 0.518 task dataset metrics type Retrieval type name config split revision webis-touche2020 MTEB Touche2020 default test 527b7d77e16e343303e68cb6af11d6e18b9f7b3b type value map_at_1 0.645 type value map_at_10 4.1160000000000005 type value map_at_100 7.527 type value map_at_1000 8.677999999999999 type value map_at_3 1.6019999999999999 type value map_at_5 2.6 type value ndcg_at_1 10.204 type value ndcg_at_10 12.27 type value ndcg_at_100 22.461000000000002 type value ndcg_at_1000 33.543 type value ndcg_at_3 9.982000000000001 type value ndcg_at_5 11.498 type value precision_at_1 10.204 type value precision_at_10 12.245000000000001 type value precision_at_100 5.286 type value precision_at_1000 1.2630000000000001 type value precision_at_3 10.884 type value precision_at_5 13.061 type value recall_at_1 0.645 type value recall_at_10 8.996 type value recall_at_100 33.666000000000004 type value recall_at_1000 67.704 type value recall_at_3 2.504 type value recall_at_5 4.95 task dataset metrics type Classification type name config split revision mteb/toxic_conversations_50k MTEB ToxicConversationsClassification default test edfaf9da55d3dd50d43143d90c1ac476895ae6de type value accuracy 62.7862 type value ap 10.958454618347831 type value f1 48.37243417046763 task dataset metrics type Classification type name config split revision mteb/tweet_sentiment_extraction MTEB TweetSentimentExtractionClassification default test 62146448f05be9e52a36b8ee9936447ea787eede type value accuracy 54.821731748726656 type value f1 55.14729314789282 task dataset metrics type Clustering type name config split revision mteb/twentynewsgroups-clustering MTEB TwentyNewsgroupsClustering default test 091a54f9a36281ce7d6590ec8c75dd485e7e01d4 type value v_measure 28.24295128553035 task dataset metrics type PairClassification type name config split revision mteb/twittersemeval2015-pairclassification MTEB TwitterSemEval2015 default test 70970daeab8776df92f5ea462b6173c0b46fd2d1 type value cos_sim_accuracy 81.5640460153782 type value cos_sim_ap 57.094095366921536 type value cos_sim_f1 55.29607083563918 type value cos_sim_precision 47.62631077216397 type value cos_sim_recall 65.91029023746702 type value dot_accuracy 78.81623651427549 type value dot_ap 47.42989400382077 type value dot_f1 51.25944584382871 type value dot_precision 42.55838271174625 type value dot_recall 64.43271767810026 type value euclidean_accuracy 80.29445073612685 type value euclidean_ap 53.42012231336148 type value euclidean_f1 51.867783563504645 type value euclidean_precision 45.4203013481364 type value euclidean_recall 60.4485488126649 type value manhattan_accuracy 80.2884901949097 type value manhattan_ap 53.43205271323232 type value manhattan_f1 52.014165559982295 type value manhattan_precision 44.796035074342356 type value manhattan_recall 62.00527704485488 type value max_accuracy 81.5640460153782 type value max_ap 57.094095366921536 type value max_f1 55.29607083563918 task dataset metrics type PairClassification type name config split revision mteb/twitterurlcorpus-pairclassification MTEB TwitterURLCorpus default test 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf type value cos_sim_accuracy 86.63018589668955 type value cos_sim_ap 80.51063771262909 type value cos_sim_f1 72.70810586950793 type value cos_sim_precision 71.14123627790467 type value cos_sim_recall 74.3455497382199 type value dot_accuracy 82.41743315092948 type value dot_ap 69.2393381283664 type value dot_f1 65.61346624814597 type value dot_precision 59.43260638630257 type value dot_recall 73.22913458577148 type value euclidean_accuracy 86.49435324251951 type value euclidean_ap 80.28100477250926 type value euclidean_f1 72.58242344489099 type value euclidean_precision 67.44662568576906 type value euclidean_recall 78.56482907299045 type value manhattan_accuracy 86.59525749990297 type value manhattan_ap 80.37850832566262 type value manhattan_f1 72.59435321233073 type value manhattan_precision 68.19350473612991 type value manhattan_recall 77.60240221743148 type value max_accuracy 86.63018589668955 type value max_ap 80.51063771262909 type value max_f1 72.70810586950793

SGPT-125M-weightedmean-nli-bitfit

Usage

For usage instructions, refer to our codebase: https://github.com/Muennighoff/sgpt

Evaluation Results

For eval results, refer to the eval folder or our paper: https://arxiv.org/abs/2202.08904

Training

The model was trained with the parameters:

DataLoader:

sentence_transformers.datasets.NoDuplicatesDataLoader.NoDuplicatesDataLoader of length 8807 with parameters:

{'batch_size': 64}

Loss:

sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss with parameters:

{'scale': 20.0, 'similarity_fct': 'cos_sim'}

Parameters of the fit()-Method:

{
    "epochs": 1,
    "evaluation_steps": 880,
    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 0.0002
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 881,
    "weight_decay": 0.01
}

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 75, 'do_lower_case': False}) with Transformer model: GPTNeoModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
)

Citing & Authors

@article{muennighoff2022sgpt,
  title={SGPT: GPT Sentence Embeddings for Semantic Search},
  author={Muennighoff, Niklas},
  journal={arXiv preprint arXiv:2202.08904},
  year={2022}
}