bge-small-en-v1.5/README.md

tags, model-index, license, language

tags	model-index	license	language
sentence-transformers feature-extraction sentence-similarity transformers mteb	name results bge-small-en-v1.5 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en) en test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 73.79104477611939 type value ap 37.21923821573361 type value f1 68.0914945617093 task dataset metrics type Classification type name config split revision mteb/amazon_polarity MTEB AmazonPolarityClassification default test e2d317d38cd51312af73b3d32a06d1a08b442046 type value accuracy 92.75377499999999 type value ap 89.46766124546022 type value f1 92.73884001331487 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (en) en test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 46.986 type value f1 46.55936786727896 task dataset metrics type Retrieval type name config split revision arguana MTEB ArguAna default test None type value map_at_1 35.846000000000004 type value map_at_10 51.388 type value map_at_100 52.132999999999996 type value map_at_1000 52.141000000000005 type value map_at_3 47.037 type value map_at_5 49.579 type value mrr_at_1 36.558 type value mrr_at_10 51.658 type value mrr_at_100 52.402 type value mrr_at_1000 52.410000000000004 type value mrr_at_3 47.345 type value mrr_at_5 49.797999999999995 type value ndcg_at_1 35.846000000000004 type value ndcg_at_10 59.550000000000004 type value ndcg_at_100 62.596 type value ndcg_at_1000 62.759 type value ndcg_at_3 50.666999999999994 type value ndcg_at_5 55.228 type value precision_at_1 35.846000000000004 type value precision_at_10 8.542 type value precision_at_100 0.984 type value precision_at_1000 0.1 type value precision_at_3 20.389 type value precision_at_5 14.438 type value recall_at_1 35.846000000000004 type value recall_at_10 85.42 type value recall_at_100 98.43499999999999 type value recall_at_1000 99.644 type value recall_at_3 61.166 type value recall_at_5 72.191 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-p2p MTEB ArxivClusteringP2P default test a122ad7f3f0291bf49cc6f4d32aa80929df69d5d type value v_measure 47.402770198163594 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-s2s MTEB ArxivClusteringS2S default test f910caf1a6075f7329cdf8c1a6135696f37dbd53 type value v_measure 40.01545436974177 task dataset metrics type Reranking type name config split revision mteb/askubuntudupquestions-reranking MTEB AskUbuntuDupQuestions default test 2000358ca161889fa9c082cb41daa8dcfb161a54 type value map 62.586465273207196 type value mrr 74.42169019038825 task dataset metrics type STS type name config split revision mteb/biosses-sts MTEB BIOSSES default test d3fb88f8f02e40887cd149695127462bbcf29b4a type value cos_sim_pearson 85.1891186537969 type value cos_sim_spearman 83.75492046087288 type value euclidean_pearson 84.11766204805357 type value euclidean_spearman 84.01456493126516 type value manhattan_pearson 84.2132950502772 type value manhattan_spearman 83.89227298813377 task dataset metrics type Classification type name config split revision mteb/banking77 MTEB Banking77Classification default test 0fd18e25b25c072e09e0d92ab615fda904d66300 type value accuracy 85.74025974025975 type value f1 85.71493566466381 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-p2p MTEB BiorxivClusteringP2P default test 65b79d1d13f80053f67aca9498d9402c2d9f1f40 type value v_measure 38.467181385006434 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-s2s MTEB BiorxivClusteringS2S default test 258694dd0231531bc1fd9de6ceb52a0853c6d908 type value v_measure 34.719496037339056 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackAndroidRetrieval default test None type value map_at_1 29.587000000000003 type value map_at_10 41.114 type value map_at_100 42.532 type value map_at_1000 42.661 type value map_at_3 37.483 type value map_at_5 39.652 type value mrr_at_1 36.338 type value mrr_at_10 46.763 type value mrr_at_100 47.393 type value mrr_at_1000 47.445 type value mrr_at_3 43.538 type value mrr_at_5 45.556000000000004 type value ndcg_at_1 36.338 type value ndcg_at_10 47.658 type value ndcg_at_100 52.824000000000005 type value ndcg_at_1000 54.913999999999994 type value ndcg_at_3 41.989 type value ndcg_at_5 44.944 type value precision_at_1 36.338 type value precision_at_10 9.156 type value precision_at_100 1.4789999999999999 type value precision_at_1000 0.196 type value precision_at_3 20.076 type value precision_at_5 14.85 type value recall_at_1 29.587000000000003 type value recall_at_10 60.746 type value recall_at_100 82.157 type value recall_at_1000 95.645 type value recall_at_3 44.821 type value recall_at_5 52.819 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackEnglishRetrieval default test None type value map_at_1 30.239 type value map_at_10 39.989000000000004 type value map_at_100 41.196 type value map_at_1000 41.325 type value map_at_3 37.261 type value map_at_5 38.833 type value mrr_at_1 37.516 type value mrr_at_10 46.177 type value mrr_at_100 46.806 type value mrr_at_1000 46.849000000000004 type value mrr_at_3 44.002 type value mrr_at_5 45.34 type value ndcg_at_1 37.516 type value ndcg_at_10 45.586 type value ndcg_at_100 49.897000000000006 type value ndcg_at_1000 51.955 type value ndcg_at_3 41.684 type value ndcg_at_5 43.617 type value precision_at_1 37.516 type value precision_at_10 8.522 type value precision_at_100 1.374 type value precision_at_1000 0.184 type value precision_at_3 20.105999999999998 type value precision_at_5 14.152999999999999 type value recall_at_1 30.239 type value recall_at_10 55.03 type value recall_at_100 73.375 type value recall_at_1000 86.29599999999999 type value recall_at_3 43.269000000000005 type value recall_at_5 48.878 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGamingRetrieval default test None type value map_at_1 38.338 type value map_at_10 50.468999999999994 type value map_at_100 51.553000000000004 type value map_at_1000 51.608 type value map_at_3 47.107 type value map_at_5 49.101 type value mrr_at_1 44.201 type value mrr_at_10 54.057 type value mrr_at_100 54.764 type value mrr_at_1000 54.791000000000004 type value mrr_at_3 51.56699999999999 type value mrr_at_5 53.05 type value ndcg_at_1 44.201 type value ndcg_at_10 56.379000000000005 type value ndcg_at_100 60.645 type value ndcg_at_1000 61.73499999999999 type value ndcg_at_3 50.726000000000006 type value ndcg_at_5 53.58500000000001 type value precision_at_1 44.201 type value precision_at_10 9.141 type value precision_at_100 1.216 type value precision_at_1000 0.135 type value precision_at_3 22.654 type value precision_at_5 15.723999999999998 type value recall_at_1 38.338 type value recall_at_10 70.30499999999999 type value recall_at_100 88.77199999999999 type value recall_at_1000 96.49799999999999 type value recall_at_3 55.218 type value recall_at_5 62.104000000000006 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGisRetrieval default test None type value map_at_1 25.682 type value map_at_10 33.498 type value map_at_100 34.461000000000006 type value map_at_1000 34.544000000000004 type value map_at_3 30.503999999999998 type value map_at_5 32.216 type value mrr_at_1 27.683999999999997 type value mrr_at_10 35.467999999999996 type value mrr_at_100 36.32 type value mrr_at_1000 36.386 type value mrr_at_3 32.618 type value mrr_at_5 34.262 type value ndcg_at_1 27.683999999999997 type value ndcg_at_10 38.378 type value ndcg_at_100 43.288 type value ndcg_at_1000 45.413 type value ndcg_at_3 32.586 type value ndcg_at_5 35.499 type value precision_at_1 27.683999999999997 type value precision_at_10 5.864 type value precision_at_100 0.882 type value precision_at_1000 0.11 type value precision_at_3 13.446 type value precision_at_5 9.718 type value recall_at_1 25.682 type value recall_at_10 51.712 type value recall_at_100 74.446 type value recall_at_1000 90.472 type value recall_at_3 36.236000000000004 type value recall_at_5 43.234 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackMathematicaRetrieval default test None type value map_at_1 16.073999999999998 type value map_at_10 24.352999999999998 type value map_at_100 25.438 type value map_at_1000 25.545 type value map_at_3 21.614 type value map_at_5 23.104 type value mrr_at_1 19.776 type value mrr_at_10 28.837000000000003 type value mrr_at_100 29.755 type value mrr_at_1000 29.817 type value mrr_at_3 26.201999999999998 type value mrr_at_5 27.714 type value ndcg_at_1 19.776 type value ndcg_at_10 29.701 type value ndcg_at_100 35.307 type value ndcg_at_1000 37.942 type value ndcg_at_3 24.764 type value ndcg_at_5 27.025 type value precision_at_1 19.776 type value precision_at_10 5.659 type value precision_at_100 0.971 type value precision_at_1000 0.133 type value precision_at_3 12.065 type value precision_at_5 8.905000000000001 type value recall_at_1 16.073999999999998 type value recall_at_10 41.647 type value recall_at_100 66.884 type value recall_at_1000 85.91499999999999 type value recall_at_3 27.916 type value recall_at_5 33.729 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackPhysicsRetrieval default test None type value map_at_1 28.444999999999997 type value map_at_10 38.218999999999994 type value map_at_100 39.595 type value map_at_1000 39.709 type value map_at_3 35.586 type value map_at_5 36.895 type value mrr_at_1 34.841 type value mrr_at_10 44.106 type value mrr_at_100 44.98 type value mrr_at_1000 45.03 type value mrr_at_3 41.979 type value mrr_at_5 43.047999999999995 type value ndcg_at_1 34.841 type value ndcg_at_10 43.922 type value ndcg_at_100 49.504999999999995 type value ndcg_at_1000 51.675000000000004 type value ndcg_at_3 39.858 type value ndcg_at_5 41.408 type value precision_at_1 34.841 type value precision_at_10 7.872999999999999 type value precision_at_100 1.2449999999999999 type value precision_at_1000 0.161 type value precision_at_3 18.993 type value precision_at_5 13.032 type value recall_at_1 28.444999999999997 type value recall_at_10 54.984 type value recall_at_100 78.342 type value recall_at_1000 92.77 type value recall_at_3 42.842999999999996 type value recall_at_5 47.247 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackProgrammersRetrieval default test None type value map_at_1 23.072 type value map_at_10 32.354 type value map_at_100 33.800000000000004 type value map_at_1000 33.908 type value map_at_3 29.232000000000003 type value map_at_5 31.049 type value mrr_at_1 29.110000000000003 type value mrr_at_10 38.03 type value mrr_at_100 39.032 type value mrr_at_1000 39.086999999999996 type value mrr_at_3 35.407 type value mrr_at_5 36.76 type value ndcg_at_1 29.110000000000003 type value ndcg_at_10 38.231 type value ndcg_at_100 44.425 type value ndcg_at_1000 46.771 type value ndcg_at_3 33.095 type value ndcg_at_5 35.459 type value precision_at_1 29.110000000000003 type value precision_at_10 7.215000000000001 type value precision_at_100 1.2109999999999999 type value precision_at_1000 0.157 type value precision_at_3 16.058 type value precision_at_5 11.644 type value recall_at_1 23.072 type value recall_at_10 50.285999999999994 type value recall_at_100 76.596 type value recall_at_1000 92.861 type value recall_at_3 35.702 type value recall_at_5 42.152 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackRetrieval default test None type value map_at_1 24.937916666666666 type value map_at_10 33.755250000000004 type value map_at_100 34.955999999999996 type value map_at_1000 35.070499999999996 type value map_at_3 30.98708333333333 type value map_at_5 32.51491666666666 type value mrr_at_1 29.48708333333333 type value mrr_at_10 37.92183333333334 type value mrr_at_100 38.76583333333333 type value mrr_at_1000 38.82466666666667 type value mrr_at_3 35.45125 type value mrr_at_5 36.827000000000005 type value ndcg_at_1 29.48708333333333 type value ndcg_at_10 39.05225 type value ndcg_at_100 44.25983333333334 type value ndcg_at_1000 46.568333333333335 type value ndcg_at_3 34.271583333333325 type value ndcg_at_5 36.483916666666666 type value precision_at_1 29.48708333333333 type value precision_at_10 6.865749999999999 type value precision_at_100 1.1195833333333332 type value precision_at_1000 0.15058333333333335 type value precision_at_3 15.742083333333333 type value precision_at_5 11.221916666666667 type value recall_at_1 24.937916666666666 type value recall_at_10 50.650416666666665 type value recall_at_100 73.55383333333334 type value recall_at_1000 89.61691666666667 type value recall_at_3 37.27808333333334 type value recall_at_5 42.99475 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackStatsRetrieval default test None type value map_at_1 23.947 type value map_at_10 30.575000000000003 type value map_at_100 31.465 type value map_at_1000 31.558000000000003 type value map_at_3 28.814 type value map_at_5 29.738999999999997 type value mrr_at_1 26.994 type value mrr_at_10 33.415 type value mrr_at_100 34.18 type value mrr_at_1000 34.245 type value mrr_at_3 31.621 type value mrr_at_5 32.549 type value ndcg_at_1 26.994 type value ndcg_at_10 34.482 type value ndcg_at_100 38.915 type value ndcg_at_1000 41.355 type value ndcg_at_3 31.139 type value ndcg_at_5 32.589 type value precision_at_1 26.994 type value precision_at_10 5.322 type value precision_at_100 0.8160000000000001 type value precision_at_1000 0.11100000000000002 type value precision_at_3 13.344000000000001 type value precision_at_5 8.988 type value recall_at_1 23.947 type value recall_at_10 43.647999999999996 type value recall_at_100 63.851 type value recall_at_1000 82.0 type value recall_at_3 34.288000000000004 type value recall_at_5 38.117000000000004 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackTexRetrieval default test None type value map_at_1 16.197 type value map_at_10 22.968 type value map_at_100 24.095 type value map_at_1000 24.217 type value map_at_3 20.771 type value map_at_5 21.995 type value mrr_at_1 19.511 type value mrr_at_10 26.55 type value mrr_at_100 27.500999999999998 type value mrr_at_1000 27.578999999999997 type value mrr_at_3 24.421 type value mrr_at_5 25.604 type value ndcg_at_1 19.511 type value ndcg_at_10 27.386 type value ndcg_at_100 32.828 type value ndcg_at_1000 35.739 type value ndcg_at_3 23.405 type value ndcg_at_5 25.255 type value precision_at_1 19.511 type value precision_at_10 5.017 type value precision_at_100 0.91 type value precision_at_1000 0.133 type value precision_at_3 11.023 type value precision_at_5 8.025 type value recall_at_1 16.197 type value recall_at_10 37.09 type value recall_at_100 61.778 type value recall_at_1000 82.56599999999999 type value recall_at_3 26.034000000000002 type value recall_at_5 30.762 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackUnixRetrieval default test None type value map_at_1 25.41 type value map_at_10 33.655 type value map_at_100 34.892 type value map_at_1000 34.995 type value map_at_3 30.94 type value map_at_5 32.303 type value mrr_at_1 29.477999999999998 type value mrr_at_10 37.443 type value mrr_at_100 38.383 type value mrr_at_1000 38.440000000000005 type value mrr_at_3 34.949999999999996 type value mrr_at_5 36.228 type value ndcg_at_1 29.477999999999998 type value ndcg_at_10 38.769 type value ndcg_at_100 44.245000000000005 type value ndcg_at_1000 46.593 type value ndcg_at_3 33.623 type value ndcg_at_5 35.766 type value precision_at_1 29.477999999999998 type value precision_at_10 6.455 type value precision_at_100 1.032 type value precision_at_1000 0.135 type value precision_at_3 14.893999999999998 type value precision_at_5 10.485 type value recall_at_1 25.41 type value recall_at_10 50.669 type value recall_at_100 74.084 type value recall_at_1000 90.435 type value recall_at_3 36.679 type value recall_at_5 41.94 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWebmastersRetrieval default test None type value map_at_1 23.339 type value map_at_10 31.852000000000004 type value map_at_100 33.411 type value map_at_1000 33.62 type value map_at_3 28.929 type value map_at_5 30.542 type value mrr_at_1 28.063 type value mrr_at_10 36.301 type value mrr_at_100 37.288 type value mrr_at_1000 37.349 type value mrr_at_3 33.663 type value mrr_at_5 35.165 type value ndcg_at_1 28.063 type value ndcg_at_10 37.462 type value ndcg_at_100 43.620999999999995 type value ndcg_at_1000 46.211 type value ndcg_at_3 32.68 type value ndcg_at_5 34.981 type value precision_at_1 28.063 type value precision_at_10 7.1739999999999995 type value precision_at_100 1.486 type value precision_at_1000 0.23500000000000001 type value precision_at_3 15.217 type value precision_at_5 11.265 type value recall_at_1 23.339 type value recall_at_10 48.376999999999995 type value recall_at_100 76.053 type value recall_at_1000 92.455 type value recall_at_3 34.735 type value recall_at_5 40.71 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWordpressRetrieval default test None type value map_at_1 18.925 type value map_at_10 26.017000000000003 type value map_at_100 27.034000000000002 type value map_at_1000 27.156000000000002 type value map_at_3 23.604 type value map_at_5 24.75 type value mrr_at_1 20.333000000000002 type value mrr_at_10 27.915 type value mrr_at_100 28.788000000000004 type value mrr_at_1000 28.877999999999997 type value mrr_at_3 25.446999999999996 type value mrr_at_5 26.648 type value ndcg_at_1 20.333000000000002 type value ndcg_at_10 30.673000000000002 type value ndcg_at_100 35.618 type value ndcg_at_1000 38.517 type value ndcg_at_3 25.71 type value ndcg_at_5 27.679 type value precision_at_1 20.333000000000002 type value precision_at_10 4.9910000000000005 type value precision_at_100 0.8130000000000001 type value precision_at_1000 0.117 type value precision_at_3 11.029 type value precision_at_5 7.8740000000000006 type value recall_at_1 18.925 type value recall_at_10 43.311 type value recall_at_100 66.308 type value recall_at_1000 87.49 type value recall_at_3 29.596 type value recall_at_5 34.245 task dataset metrics type Retrieval type name config split revision climate-fever MTEB ClimateFEVER default test None type value map_at_1 13.714 type value map_at_10 23.194 type value map_at_100 24.976000000000003 type value map_at_1000 25.166 type value map_at_3 19.709 type value map_at_5 21.523999999999997 type value mrr_at_1 30.619000000000003 type value mrr_at_10 42.563 type value mrr_at_100 43.386 type value mrr_at_1000 43.423 type value mrr_at_3 39.555 type value mrr_at_5 41.268 type value ndcg_at_1 30.619000000000003 type value ndcg_at_10 31.836 type value ndcg_at_100 38.652 type value ndcg_at_1000 42.088 type value ndcg_at_3 26.733 type value ndcg_at_5 28.435 type value precision_at_1 30.619000000000003 type value precision_at_10 9.751999999999999 type value precision_at_100 1.71 type value precision_at_1000 0.23500000000000001 type value precision_at_3 19.935 type value precision_at_5 14.984 type value recall_at_1 13.714 type value recall_at_10 37.26 type value recall_at_100 60.546 type value recall_at_1000 79.899 type value recall_at_3 24.325 type value recall_at_5 29.725 task dataset metrics type Retrieval type name config split revision dbpedia-entity MTEB DBPedia default test None type value map_at_1 8.462 type value map_at_10 18.637 type value map_at_100 26.131999999999998 type value map_at_1000 27.607 type value map_at_3 13.333 type value map_at_5 15.654000000000002 type value mrr_at_1 66.25 type value mrr_at_10 74.32600000000001 type value mrr_at_100 74.60900000000001 type value mrr_at_1000 74.62 type value mrr_at_3 72.667 type value mrr_at_5 73.817 type value ndcg_at_1 53.87499999999999 type value ndcg_at_10 40.028999999999996 type value ndcg_at_100 44.199 type value ndcg_at_1000 51.629999999999995 type value ndcg_at_3 44.113 type value ndcg_at_5 41.731 type value precision_at_1 66.25 type value precision_at_10 31.900000000000002 type value precision_at_100 10.043000000000001 type value precision_at_1000 1.926 type value precision_at_3 47.417 type value precision_at_5 40.65 type value recall_at_1 8.462 type value recall_at_10 24.293 type value recall_at_100 50.146 type value recall_at_1000 74.034 type value recall_at_3 14.967 type value recall_at_5 18.682000000000002 task dataset metrics type Classification type name config split revision mteb/emotion MTEB EmotionClassification default test 4f58c6b202a23cf9a4da393831edf4f9183cad37 type value accuracy 47.84499999999999 type value f1 42.48106691979349 task dataset metrics type Retrieval type name config split revision fever MTEB FEVER default test None type value map_at_1 74.034 type value map_at_10 82.76 type value map_at_100 82.968 type value map_at_1000 82.98299999999999 type value map_at_3 81.768 type value map_at_5 82.418 type value mrr_at_1 80.048 type value mrr_at_10 87.64999999999999 type value mrr_at_100 87.712 type value mrr_at_1000 87.713 type value mrr_at_3 87.01100000000001 type value mrr_at_5 87.466 type value ndcg_at_1 80.048 type value ndcg_at_10 86.643 type value ndcg_at_100 87.361 type value ndcg_at_1000 87.606 type value ndcg_at_3 85.137 type value ndcg_at_5 86.016 type value precision_at_1 80.048 type value precision_at_10 10.372 type value precision_at_100 1.093 type value precision_at_1000 0.11299999999999999 type value precision_at_3 32.638 type value precision_at_5 20.177 type value recall_at_1 74.034 type value recall_at_10 93.769 type value recall_at_100 96.569 type value recall_at_1000 98.039 type value recall_at_3 89.581 type value recall_at_5 91.906 task dataset metrics type Retrieval type name config split revision fiqa MTEB FiQA2018 default test None type value map_at_1 20.5 type value map_at_10 32.857 type value map_at_100 34.589 type value map_at_1000 34.778 type value map_at_3 29.160999999999998 type value map_at_5 31.033 type value mrr_at_1 40.123 type value mrr_at_10 48.776 type value mrr_at_100 49.495 type value mrr_at_1000 49.539 type value mrr_at_3 46.605000000000004 type value mrr_at_5 47.654 type value ndcg_at_1 40.123 type value ndcg_at_10 40.343 type value ndcg_at_100 46.56 type value ndcg_at_1000 49.777 type value ndcg_at_3 37.322 type value ndcg_at_5 37.791000000000004 type value precision_at_1 40.123 type value precision_at_10 11.08 type value precision_at_100 1.752 type value precision_at_1000 0.232 type value precision_at_3 24.897 type value precision_at_5 17.809 type value recall_at_1 20.5 type value recall_at_10 46.388 type value recall_at_100 69.552 type value recall_at_1000 89.011 type value recall_at_3 33.617999999999995 type value recall_at_5 38.211 task dataset metrics type Retrieval type name config split revision hotpotqa MTEB HotpotQA default test None type value map_at_1 39.135999999999996 type value map_at_10 61.673 type value map_at_100 62.562 type value map_at_1000 62.62 type value map_at_3 58.467999999999996 type value map_at_5 60.463 type value mrr_at_1 78.271 type value mrr_at_10 84.119 type value mrr_at_100 84.29299999999999 type value mrr_at_1000 84.299 type value mrr_at_3 83.18900000000001 type value mrr_at_5 83.786 type value ndcg_at_1 78.271 type value ndcg_at_10 69.935 type value ndcg_at_100 73.01299999999999 type value ndcg_at_1000 74.126 type value ndcg_at_3 65.388 type value ndcg_at_5 67.906 type value precision_at_1 78.271 type value precision_at_10 14.562 type value precision_at_100 1.6969999999999998 type value precision_at_1000 0.184 type value precision_at_3 41.841 type value precision_at_5 27.087 type value recall_at_1 39.135999999999996 type value recall_at_10 72.809 type value recall_at_100 84.86200000000001 type value recall_at_1000 92.208 type value recall_at_3 62.76199999999999 type value recall_at_5 67.718 task dataset metrics type Classification type name config split revision mteb/imdb MTEB ImdbClassification default test 3d86128a09e091d6018b6d26cad27f2739fc2db7 type value accuracy 90.60600000000001 type value ap 86.6579587804335 type value f1 90.5938853929307 task dataset metrics type Retrieval type name config split revision msmarco MTEB MSMARCO default dev None type value map_at_1 21.852 type value map_at_10 33.982 type value map_at_100 35.116 type value map_at_1000 35.167 type value map_at_3 30.134 type value map_at_5 32.340999999999994 type value mrr_at_1 22.479 type value mrr_at_10 34.594 type value mrr_at_100 35.672 type value mrr_at_1000 35.716 type value mrr_at_3 30.84 type value mrr_at_5 32.998 type value ndcg_at_1 22.493 type value ndcg_at_10 40.833000000000006 type value ndcg_at_100 46.357 type value ndcg_at_1000 47.637 type value ndcg_at_3 32.995999999999995 type value ndcg_at_5 36.919000000000004 type value precision_at_1 22.493 type value precision_at_10 6.465999999999999 type value precision_at_100 0.9249999999999999 type value precision_at_1000 0.104 type value precision_at_3 14.030999999999999 type value precision_at_5 10.413 type value recall_at_1 21.852 type value recall_at_10 61.934999999999995 type value recall_at_100 87.611 type value recall_at_1000 97.441 type value recall_at_3 40.583999999999996 type value recall_at_5 49.992999999999995 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (en) en test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 93.36069311445507 type value f1 93.16456330371453 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (en) en test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 74.74692202462381 type value f1 58.17903579421599 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (en) en test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 74.80833893745796 type value f1 72.70786592684664 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (en) en test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 78.69872225958305 type value f1 78.61626934504731 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-p2p MTEB MedrxivClusteringP2P default test e7a26af6f3ae46b30dde8737f02c07b1505bcc73 type value v_measure 33.058658628717694 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-s2s MTEB MedrxivClusteringS2S default test 35191c8c0dca72d8ff3efcd72aa802307d469663 type value v_measure 30.85561739360599 task dataset metrics type Reranking type name config split revision mteb/mind_small MTEB MindSmallReranking default test 3bdac13927fdc888b903db93b2ffdbd90b295a69 type value map 31.290259910144385 type value mrr 32.44223046102856 task dataset metrics type Retrieval type name config split revision nfcorpus MTEB NFCorpus default test None type value map_at_1 5.288 type value map_at_10 12.267999999999999 type value map_at_100 15.557000000000002 type value map_at_1000 16.98 type value map_at_3 8.866 type value map_at_5 10.418 type value mrr_at_1 43.653 type value mrr_at_10 52.681 type value mrr_at_100 53.315999999999995 type value mrr_at_1000 53.357 type value mrr_at_3 51.393 type value mrr_at_5 51.903999999999996 type value ndcg_at_1 42.415000000000006 type value ndcg_at_10 34.305 type value ndcg_at_100 30.825999999999997 type value ndcg_at_1000 39.393 type value ndcg_at_3 39.931 type value ndcg_at_5 37.519999999999996 type value precision_at_1 43.653 type value precision_at_10 25.728 type value precision_at_100 7.932 type value precision_at_1000 2.07 type value precision_at_3 38.184000000000005 type value precision_at_5 32.879000000000005 type value recall_at_1 5.288 type value recall_at_10 16.195 type value recall_at_100 31.135 type value recall_at_1000 61.531000000000006 type value recall_at_3 10.313 type value recall_at_5 12.754999999999999 task dataset metrics type Retrieval type name config split revision nq MTEB NQ default test None type value map_at_1 28.216 type value map_at_10 42.588 type value map_at_100 43.702999999999996 type value map_at_1000 43.739 type value map_at_3 38.177 type value map_at_5 40.754000000000005 type value mrr_at_1 31.866 type value mrr_at_10 45.189 type value mrr_at_100 46.056000000000004 type value mrr_at_1000 46.081 type value mrr_at_3 41.526999999999994 type value mrr_at_5 43.704 type value ndcg_at_1 31.837 type value ndcg_at_10 50.178 type value ndcg_at_100 54.98800000000001 type value ndcg_at_1000 55.812 type value ndcg_at_3 41.853 type value ndcg_at_5 46.153 type value precision_at_1 31.837 type value precision_at_10 8.43 type value precision_at_100 1.1119999999999999 type value precision_at_1000 0.11900000000000001 type value precision_at_3 19.023 type value precision_at_5 13.911000000000001 type value recall_at_1 28.216 type value recall_at_10 70.8 type value recall_at_100 91.857 type value recall_at_1000 97.941 type value recall_at_3 49.196 type value recall_at_5 59.072 task dataset metrics type Retrieval type name config split revision quora MTEB QuoraRetrieval default test None type value map_at_1 71.22800000000001 type value map_at_10 85.115 type value map_at_100 85.72 type value map_at_1000 85.737 type value map_at_3 82.149 type value map_at_5 84.029 type value mrr_at_1 81.96 type value mrr_at_10 88.00200000000001 type value mrr_at_100 88.088 type value mrr_at_1000 88.089 type value mrr_at_3 87.055 type value mrr_at_5 87.715 type value ndcg_at_1 82.01 type value ndcg_at_10 88.78 type value ndcg_at_100 89.91 type value ndcg_at_1000 90.013 type value ndcg_at_3 85.957 type value ndcg_at_5 87.56 type value precision_at_1 82.01 type value precision_at_10 13.462 type value precision_at_100 1.528 type value precision_at_1000 0.157 type value precision_at_3 37.553 type value precision_at_5 24.732000000000003 type value recall_at_1 71.22800000000001 type value recall_at_10 95.69 type value recall_at_100 99.531 type value recall_at_1000 99.98 type value recall_at_3 87.632 type value recall_at_5 92.117 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering MTEB RedditClustering default test 24640382cdbf8abc73003fb0fa6d111a705499eb type value v_measure 52.31768034366916 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering-p2p MTEB RedditClusteringP2P default test 282350215ef01743dc01b456c7f5241fa8937f16 type value v_measure 60.640266772723606 task dataset metrics type Retrieval type name config split revision scidocs MTEB SCIDOCS default test None type value map_at_1 4.7780000000000005 type value map_at_10 12.299 type value map_at_100 14.363000000000001 type value map_at_1000 14.71 type value map_at_3 8.738999999999999 type value map_at_5 10.397 type value mrr_at_1 23.599999999999998 type value mrr_at_10 34.845 type value mrr_at_100 35.916 type value mrr_at_1000 35.973 type value mrr_at_3 31.7 type value mrr_at_5 33.535 type value ndcg_at_1 23.599999999999998 type value ndcg_at_10 20.522000000000002 type value ndcg_at_100 28.737000000000002 type value ndcg_at_1000 34.596 type value ndcg_at_3 19.542 type value ndcg_at_5 16.958000000000002 type value precision_at_1 23.599999999999998 type value precision_at_10 10.67 type value precision_at_100 2.259 type value precision_at_1000 0.367 type value precision_at_3 18.333 type value precision_at_5 14.879999999999999 type value recall_at_1 4.7780000000000005 type value recall_at_10 21.617 type value recall_at_100 45.905 type value recall_at_1000 74.42 type value recall_at_3 11.148 type value recall_at_5 15.082999999999998 task dataset metrics type STS type name config split revision mteb/sickr-sts MTEB SICK-R default test a6ea5a8cab320b040a23452cc28066d9beae2cee type value cos_sim_pearson 83.22372750297885 type value cos_sim_spearman 79.40972617119405 type value euclidean_pearson 80.6101072020434 type value euclidean_spearman 79.53844217225202 type value manhattan_pearson 80.57265975286111 type value manhattan_spearman 79.46335611792958 task dataset metrics type STS type name config split revision mteb/sts12-sts MTEB STS12 default test a0d554a64d88156834ff5ae9920b964011b16384 type value cos_sim_pearson 85.43713315520749 type value cos_sim_spearman 77.44128693329532 type value euclidean_pearson 81.63869928101123 type value euclidean_spearman 77.29512977961515 type value manhattan_pearson 81.63704185566183 type value manhattan_spearman 77.29909412738657 task dataset metrics type STS type name config split revision mteb/sts13-sts MTEB STS13 default test 7e90230a92c190f1bf69ae9002b8cea547a64cca type value cos_sim_pearson 81.59451537860527 type value cos_sim_spearman 82.97994638856723 type value euclidean_pearson 82.89478688288412 type value euclidean_spearman 83.58740751053104 type value manhattan_pearson 82.69140840941608 type value manhattan_spearman 83.33665956040555 task dataset metrics type STS type name config split revision mteb/sts14-sts MTEB STS14 default test 6031580fec1f6af667f0bd2da0a551cf4f0b2375 type value cos_sim_pearson 82.00756527711764 type value cos_sim_spearman 81.83560996841379 type value euclidean_pearson 82.07684151976518 type value euclidean_spearman 82.00913052060511 type value manhattan_pearson 82.05690778488794 type value manhattan_spearman 82.02260252019525 task dataset metrics type STS type name config split revision mteb/sts15-sts MTEB STS15 default test ae752c7c21bf194d8b67fd573edf7ae58183cbe3 type value cos_sim_pearson 86.13710262895447 type value cos_sim_spearman 87.26412811156248 type value euclidean_pearson 86.94151453230228 type value euclidean_spearman 87.5363796699571 type value manhattan_pearson 86.86989424083748 type value manhattan_spearman 87.47315940781353 task dataset metrics type STS type name config split revision mteb/sts16-sts MTEB STS16 default test 4d8694f8f0e0100860b497b999b3dbed754a0513 type value cos_sim_pearson 83.0230597603627 type value cos_sim_spearman 84.93344499318864 type value euclidean_pearson 84.23754743431141 type value euclidean_spearman 85.09707376597099 type value manhattan_pearson 84.04325160987763 type value manhattan_spearman 84.89353071339909 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-en) en-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 86.75620824563921 type value cos_sim_spearman 87.15065513706398 type value euclidean_pearson 88.26281533633521 type value euclidean_spearman 87.51963738643983 type value manhattan_pearson 88.25599267618065 type value manhattan_spearman 87.58048736047483 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (en) en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 64.74645319195137 type value cos_sim_spearman 65.29996325037214 type value euclidean_pearson 67.04297794086443 type value euclidean_spearman 65.43841726694343 type value manhattan_pearson 67.39459955690904 type value manhattan_spearman 65.92864704413651 task dataset metrics type STS type name config split revision mteb/stsbenchmark-sts MTEB STSBenchmark default test b0fddb56ed78048fa8b90373c8a3cfc37b684831 type value cos_sim_pearson 84.31291020270801 type value cos_sim_spearman 85.86473738688068 type value euclidean_pearson 85.65537275064152 type value euclidean_spearman 86.13087454209642 type value manhattan_pearson 85.43946955047609 type value manhattan_spearman 85.91568175344916 task dataset metrics type Reranking type name config split revision mteb/scidocs-reranking MTEB SciDocsRR default test d3c5e1fc0b855ab6097bf1cda04dd73947d7caab type value map 85.93798118350695 type value mrr 95.93536274908824 task dataset metrics type Retrieval type name config split revision scifact MTEB SciFact default test None type value map_at_1 57.594 type value map_at_10 66.81899999999999 type value map_at_100 67.368 type value map_at_1000 67.4 type value map_at_3 64.061 type value map_at_5 65.47 type value mrr_at_1 60.667 type value mrr_at_10 68.219 type value mrr_at_100 68.655 type value mrr_at_1000 68.684 type value mrr_at_3 66.22200000000001 type value mrr_at_5 67.289 type value ndcg_at_1 60.667 type value ndcg_at_10 71.275 type value ndcg_at_100 73.642 type value ndcg_at_1000 74.373 type value ndcg_at_3 66.521 type value ndcg_at_5 68.581 type value precision_at_1 60.667 type value precision_at_10 9.433 type value precision_at_100 1.0699999999999998 type value precision_at_1000 0.11299999999999999 type value precision_at_3 25.556 type value precision_at_5 16.8 type value recall_at_1 57.594 type value recall_at_10 83.622 type value recall_at_100 94.167 type value recall_at_1000 99.667 type value recall_at_3 70.64399999999999 type value recall_at_5 75.983 task dataset metrics type PairClassification type name config split revision mteb/sprintduplicatequestions-pairclassification MTEB SprintDuplicateQuestions default test d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 type value cos_sim_accuracy 99.85841584158416 type value cos_sim_ap 96.66996142314342 type value cos_sim_f1 92.83208020050125 type value cos_sim_precision 93.06532663316584 type value cos_sim_recall 92.60000000000001 type value dot_accuracy 99.85841584158416 type value dot_ap 96.6775307676576 type value dot_f1 92.69289729177312 type value dot_precision 94.77533960292581 type value dot_recall 90.7 type value euclidean_accuracy 99.86138613861387 type value euclidean_ap 96.6338454403108 type value euclidean_f1 92.92214357937311 type value euclidean_precision 93.96728016359918 type value euclidean_recall 91.9 type value manhattan_accuracy 99.86237623762376 type value manhattan_ap 96.60370449645053 type value manhattan_f1 92.91177970423253 type value manhattan_precision 94.7970863683663 type value manhattan_recall 91.10000000000001 type value max_accuracy 99.86237623762376 type value max_ap 96.6775307676576 type value max_f1 92.92214357937311 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering MTEB StackExchangeClustering default test 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 type value v_measure 60.77977058695198 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering-p2p MTEB StackExchangeClusteringP2P default test 815ca46b2622cec33ccafc3735d572c266efdb44 type value v_measure 35.2725272535638 task dataset metrics type Reranking type name config split revision mteb/stackoverflowdupquestions-reranking MTEB StackOverflowDupQuestions default test e185fbe320c72810689fc5848eb6114e1ef5ec69 type value map 53.64052466362125 type value mrr 54.533067014684654 task dataset metrics type Summarization type name config split revision mteb/summeval MTEB SummEval default test cda12ad7615edc362dbf25a00fdd61d3b1eaf93c type value cos_sim_pearson 30.677624219206578 type value cos_sim_spearman 30.121368518123447 type value dot_pearson 30.69870088041608 type value dot_spearman 29.61284927093751 task dataset metrics type Retrieval type name config split revision trec-covid MTEB TRECCOVID default test None type value map_at_1 0.22 type value map_at_10 1.855 type value map_at_100 9.885 type value map_at_1000 23.416999999999998 type value map_at_3 0.637 type value map_at_5 1.024 type value mrr_at_1 88.0 type value mrr_at_10 93.067 type value mrr_at_100 93.067 type value mrr_at_1000 93.067 type value mrr_at_3 92.667 type value mrr_at_5 93.067 type value ndcg_at_1 82.0 type value ndcg_at_10 75.899 type value ndcg_at_100 55.115 type value ndcg_at_1000 48.368 type value ndcg_at_3 79.704 type value ndcg_at_5 78.39699999999999 type value precision_at_1 88.0 type value precision_at_10 79.60000000000001 type value precision_at_100 56.06 type value precision_at_1000 21.206 type value precision_at_3 84.667 type value precision_at_5 83.2 type value recall_at_1 0.22 type value recall_at_10 2.078 type value recall_at_100 13.297 type value recall_at_1000 44.979 type value recall_at_3 0.6689999999999999 type value recall_at_5 1.106 task dataset metrics type Retrieval type name config split revision webis-touche2020 MTEB Touche2020 default test None type value map_at_1 2.258 type value map_at_10 10.439 type value map_at_100 16.89 type value map_at_1000 18.407999999999998 type value map_at_3 5.668 type value map_at_5 7.718 type value mrr_at_1 32.653 type value mrr_at_10 51.159 type value mrr_at_100 51.714000000000006 type value mrr_at_1000 51.714000000000006 type value mrr_at_3 47.959 type value mrr_at_5 50.407999999999994 type value ndcg_at_1 29.592000000000002 type value ndcg_at_10 26.037 type value ndcg_at_100 37.924 type value ndcg_at_1000 49.126999999999995 type value ndcg_at_3 30.631999999999998 type value ndcg_at_5 28.571 type value precision_at_1 32.653 type value precision_at_10 22.857 type value precision_at_100 7.754999999999999 type value precision_at_1000 1.529 type value precision_at_3 34.014 type value precision_at_5 29.796 type value recall_at_1 2.258 type value recall_at_10 16.554 type value recall_at_100 48.439 type value recall_at_1000 82.80499999999999 type value recall_at_3 7.283 type value recall_at_5 10.732 task dataset metrics type Classification type name config split revision mteb/toxic_conversations_50k MTEB ToxicConversationsClassification default test d7c0de2777da35d6aae2200a62c6e0e5af397c4c type value accuracy 69.8858 type value ap 13.835684144362109 type value f1 53.803351693244586 task dataset metrics type Classification type name config split revision mteb/tweet_sentiment_extraction MTEB TweetSentimentExtractionClassification default test d604517c81ca91fe16a244d1248fc021f9ecee7a type value accuracy 60.50650820599886 type value f1 60.84357825979259 task dataset metrics type Clustering type name config split revision mteb/twentynewsgroups-clustering MTEB TwentyNewsgroupsClustering default test 6125ec4e24fa026cec8a478383ee943acfbd5449 type value v_measure 48.52131044852134 task dataset metrics type PairClassification type name config split revision mteb/twittersemeval2015-pairclassification MTEB TwitterSemEval2015 default test 70970daeab8776df92f5ea462b6173c0b46fd2d1 type value cos_sim_accuracy 85.59337187816654 type value cos_sim_ap 73.23925826533437 type value cos_sim_f1 67.34693877551021 type value cos_sim_precision 62.40432237730752 type value cos_sim_recall 73.13984168865434 type value dot_accuracy 85.31322644096085 type value dot_ap 72.30723963807422 type value dot_f1 66.47051612112296 type value dot_precision 62.0792305930845 type value dot_recall 71.53034300791556 type value euclidean_accuracy 85.61125350181797 type value euclidean_ap 73.32843720487845 type value euclidean_f1 67.36549633745895 type value euclidean_precision 64.60755813953489 type value euclidean_recall 70.36939313984169 type value manhattan_accuracy 85.63509566668654 type value manhattan_ap 73.16658488311325 type value manhattan_f1 67.20597386434349 type value manhattan_precision 63.60424028268551 type value manhattan_recall 71.2401055408971 type value max_accuracy 85.63509566668654 type value max_ap 73.32843720487845 type value max_f1 67.36549633745895 task dataset metrics type PairClassification type name config split revision mteb/twitterurlcorpus-pairclassification MTEB TwitterURLCorpus default test 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf type value cos_sim_accuracy 88.33779640625606 type value cos_sim_ap 84.83868375898157 type value cos_sim_f1 77.16506154017773 type value cos_sim_precision 74.62064005753327 type value cos_sim_recall 79.88912842623961 type value dot_accuracy 88.02732176815307 type value dot_ap 83.95089283763002 type value dot_f1 76.29635101196631 type value dot_precision 73.31771720613288 type value dot_recall 79.52725592854944 type value euclidean_accuracy 88.44452206310397 type value euclidean_ap 84.98384576824827 type value euclidean_f1 77.29311047696697 type value euclidean_precision 74.51232583065381 type value euclidean_recall 80.28949799815214 type value manhattan_accuracy 88.47362906042613 type value manhattan_ap 84.91421462218432 type value manhattan_f1 77.05107637204792 type value manhattan_precision 74.74484256243214 type value manhattan_recall 79.50415768401602 type value max_accuracy 88.47362906042613 type value max_ap 84.98384576824827 type value max_f1 77.29311047696697	mit	en

FlagEmbedding

Model List | FAQ | Usage | Evaluation | Train | Contact | Citation | License

More details please refer to our Github: FlagEmbedding.

If you are looking for a model that supports more languages, longer texts, and other retrieval methods, you can try using bge-m3.

English | 中文

FlagEmbedding focuses on retrieval-augmented LLMs, consisting of the following projects currently:

• Long-Context LLM: Activation Beacon
• Fine-tuning of LM : LM-Cocktail
• Dense Retrieval: BGE-M3, LLM Embedder, BGE Embedding
• Reranker Model: BGE Reranker
• Benchmark: C-MTEB

News

• 1/30/2024: Release BGE-M3, a new member to BGE model series! M3 stands for Multi-linguality (100+ languages), Multi-granularities (input length up to 8192), Multi-Functionality (unification of dense, lexical, multi-vec/colbert retrieval). It is the first embedding model which supports all three retrieval methods, achieving new SOTA on multi-lingual (MIRACL) and cross-lingual (MKQA) benchmarks. Technical Report and Code. 🔥
• 1/9/2024: Release Activation-Beacon, an effective, efficient, compatible, and low-cost (training) method to extend the context length of LLM. Technical Report 🔥
• 12/24/2023: Release LLaRA, a LLaMA-7B based dense retriever, leading to state-of-the-art performances on MS MARCO and BEIR. Model and code will be open-sourced. Please stay tuned. Technical Report 🔥
• 11/23/2023: Release LM-Cocktail, a method to maintain general capabilities during fine-tuning by merging multiple language models. Technical Report 🔥
• 10/12/2023: Release LLM-Embedder, a unified embedding model to support diverse retrieval augmentation needs for LLMs. Technical Report
• 09/15/2023: The technical report of BGE has been released
• 09/15/2023: The massive training data of BGE has been released
• 09/12/2023: New models:
  • New reranker model: release cross-encoder models BAAI/bge-reranker-base and BAAI/bge-reranker-large, which are more powerful than embedding model. We recommend to use/fine-tune them to re-rank top-k documents returned by embedding models.
  • update embedding model: release bge-*-v1.5 embedding model to alleviate the issue of the similarity distribution, and enhance its retrieval ability without instruction.

More

• 09/07/2023: Update fine-tune code: Add script to mine hard negatives and support adding instruction during fine-tuning.
• 08/09/2023: BGE Models are integrated into Langchain, you can use it like this; C-MTEB leaderboard is available.
• 08/05/2023: Release base-scale and small-scale models, best performance among the models of the same size 🤗
• 08/02/2023: Release bge-large-*(short for BAAI General Embedding) Models, rank 1st on MTEB and C-MTEB benchmark! 🎉 🎉
• 08/01/2023: We release the Chinese Massive Text Embedding Benchmark (C-MTEB), consisting of 31 test dataset.

Model List

bge is short for BAAI general embedding.

Model	Language		Description	query instruction for retrieval [1]
BAAI/bge-m3	Multilingual	Inference Fine-tune	Multi-Functionality(dense retrieval, sparse retrieval, multi-vector(colbert)), Multi-Linguality, and Multi-Granularity(8192 tokens)
BAAI/llm-embedder	English	Inference Fine-tune	a unified embedding model to support diverse retrieval augmentation needs for LLMs	See README
BAAI/bge-reranker-large	Chinese and English	Inference Fine-tune	a cross-encoder model which is more accurate but less efficient [2]
BAAI/bge-reranker-base	Chinese and English	Inference Fine-tune	a cross-encoder model which is more accurate but less efficient [2]
BAAI/bge-large-en-v1.5	English	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	Represent this sentence for searching relevant passages:
BAAI/bge-base-en-v1.5	English	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	Represent this sentence for searching relevant passages:
BAAI/bge-small-en-v1.5	English	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	Represent this sentence for searching relevant passages:
BAAI/bge-large-zh-v1.5	Chinese	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	为这个句子生成表示以用于检索相关文章：
BAAI/bge-base-zh-v1.5	Chinese	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	为这个句子生成表示以用于检索相关文章：
BAAI/bge-small-zh-v1.5	Chinese	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	为这个句子生成表示以用于检索相关文章：
BAAI/bge-large-en	English	Inference Fine-tune	🏆 rank 1st in MTEB leaderboard	Represent this sentence for searching relevant passages:
BAAI/bge-base-en	English	Inference Fine-tune	a base-scale model but with similar ability to bge-large-en	Represent this sentence for searching relevant passages:
BAAI/bge-small-en	English	Inference Fine-tune	a small-scale model but with competitive performance	Represent this sentence for searching relevant passages:
BAAI/bge-large-zh	Chinese	Inference Fine-tune	🏆 rank 1st in C-MTEB benchmark	为这个句子生成表示以用于检索相关文章：
BAAI/bge-base-zh	Chinese	Inference Fine-tune	a base-scale model but with similar ability to bge-large-zh	为这个句子生成表示以用于检索相关文章：
BAAI/bge-small-zh	Chinese	Inference Fine-tune	a small-scale model but with competitive performance	为这个句子生成表示以用于检索相关文章：

[1]: If you need to search the relevant passages to a query, we suggest to add the instruction to the query; in other cases, no instruction is needed, just use the original query directly. In all cases, no instruction needs to be added to passages.

[2]: Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding. To balance the accuracy and time cost, cross-encoder is widely used to re-rank top-k documents retrieved by other simple models. For examples, use bge embedding model to retrieve top 100 relevant documents, and then use bge reranker to re-rank the top 100 document to get the final top-3 results.

All models have been uploaded to Huggingface Hub, and you can see them at https://huggingface.co/BAAI. If you cannot open the Huggingface Hub, you also can download the models at https://model.baai.ac.cn/models .

Frequently asked questions

1. How to fine-tune bge embedding model?

Following this example to prepare data and fine-tune your model. Some suggestions:

• Mine hard negatives following this example, which can improve the retrieval performance.
• If you pre-train bge on your data, the pre-trained model cannot be directly used to calculate similarity, and it must be fine-tuned with contrastive learning before computing similarity.
• If the accuracy of the fine-tuned model is still not high, it is recommended to use/fine-tune the cross-encoder model (bge-reranker) to re-rank top-k results. Hard negatives also are needed to fine-tune reranker.

2. The similarity score between two dissimilar sentences is higher than 0.5

Suggest to use bge v1.5, which alleviates the issue of the similarity distribution.

Since we finetune the models by contrastive learning with a temperature of 0.01, the similarity distribution of the current BGE model is about in the interval [0.6, 1]. So a similarity score greater than 0.5 does not indicate that the two sentences are similar.

For downstream tasks, such as passage retrieval or semantic similarity, what matters is the relative order of the scores, not the absolute value. If you need to filter similar sentences based on a similarity threshold, please select an appropriate similarity threshold based on the similarity distribution on your data (such as 0.8, 0.85, or even 0.9).

3. When does the query instruction need to be used

For the bge-*-v1.5, we improve its retrieval ability when not using instruction. No instruction only has a slight degradation in retrieval performance compared with using instruction. So you can generate embedding without instruction in all cases for convenience.

For a retrieval task that uses short queries to find long related documents, it is recommended to add instructions for these short queries. The best method to decide whether to add instructions for queries is choosing the setting that achieves better performance on your task. In all cases, the documents/passages do not need to add the instruction.

Usage

Usage for Embedding Model

Here are some examples for using bge models with FlagEmbedding, Sentence-Transformers, Langchain, or Huggingface Transformers.

Using FlagEmbedding

pip install -U FlagEmbedding

If it doesn't work for you, you can see FlagEmbedding for more methods to install FlagEmbedding.

from FlagEmbedding import FlagModel
sentences_1 = ["样例数据-1", "样例数据-2"]
sentences_2 = ["样例数据-3", "样例数据-4"]
model = FlagModel('BAAI/bge-large-zh-v1.5', 
                  query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章：",
                  use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
embeddings_1 = model.encode(sentences_1)
embeddings_2 = model.encode(sentences_2)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)

# for s2p(short query to long passage) retrieval task, suggest to use encode_queries() which will automatically add the instruction to each query
# corpus in retrieval task can still use encode() or encode_corpus(), since they don't need instruction
queries = ['query_1', 'query_2']
passages = ["样例文档-1", "样例文档-2"]
q_embeddings = model.encode_queries(queries)
p_embeddings = model.encode(passages)
scores = q_embeddings @ p_embeddings.T

For the value of the argument query_instruction_for_retrieval, see Model List.

By default, FlagModel will use all available GPUs when encoding. Please set os.environ["CUDA_VISIBLE_DEVICES"] to select specific GPUs. You also can set os.environ["CUDA_VISIBLE_DEVICES"]="" to make all GPUs unavailable.

Using Sentence-Transformers

You can also use the bge models with sentence-transformers:

pip install -U sentence-transformers

from sentence_transformers import SentenceTransformer
sentences_1 = ["样例数据-1", "样例数据-2"]
sentences_2 = ["样例数据-3", "样例数据-4"]
model = SentenceTransformer('BAAI/bge-large-zh-v1.5')
embeddings_1 = model.encode(sentences_1, normalize_embeddings=True)
embeddings_2 = model.encode(sentences_2, normalize_embeddings=True)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)

For s2p(short query to long passage) retrieval task, each short query should start with an instruction (instructions see Model List). But the instruction is not needed for passages.

from sentence_transformers import SentenceTransformer
queries = ['query_1', 'query_2']
passages = ["样例文档-1", "样例文档-2"]
instruction = "为这个句子生成表示以用于检索相关文章："

model = SentenceTransformer('BAAI/bge-large-zh-v1.5')
q_embeddings = model.encode([instruction+q for q in queries], normalize_embeddings=True)
p_embeddings = model.encode(passages, normalize_embeddings=True)
scores = q_embeddings @ p_embeddings.T

Using Langchain

You can use bge in langchain like this:

from langchain.embeddings import HuggingFaceBgeEmbeddings
model_name = "BAAI/bge-large-en-v1.5"
model_kwargs = {'device': 'cuda'}
encode_kwargs = {'normalize_embeddings': True} # set True to compute cosine similarity
model = HuggingFaceBgeEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs,
    query_instruction="为这个句子生成表示以用于检索相关文章："
)
model.query_instruction = "为这个句子生成表示以用于检索相关文章："

Using HuggingFace Transformers

With the transformers package, you can use the model like this: First, you pass your input through the transformer model, then you select the last hidden state of the first token (i.e., [CLS]) as the sentence embedding.

from transformers import AutoTokenizer, AutoModel
import torch
# Sentences we want sentence embeddings for
sentences = ["样例数据-1", "样例数据-2"]

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-large-zh-v1.5')
model = AutoModel.from_pretrained('BAAI/bge-large-zh-v1.5')
model.eval()

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
# for s2p(short query to long passage) retrieval task, add an instruction to query (not add instruction for passages)
# encoded_input = tokenizer([instruction + q for q in queries], padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)
    # Perform pooling. In this case, cls pooling.
    sentence_embeddings = model_output[0][:, 0]
# normalize embeddings
sentence_embeddings = torch.nn.functional.normalize(sentence_embeddings, p=2, dim=1)
print("Sentence embeddings:", sentence_embeddings)

Usage for Reranker

Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding. You can get a relevance score by inputting query and passage to the reranker. The reranker is optimized based cross-entropy loss, so the relevance score is not bounded to a specific range.

Using FlagEmbedding

pip install -U FlagEmbedding

Get relevance scores (higher scores indicate more relevance):

from FlagEmbedding import FlagReranker
reranker = FlagReranker('BAAI/bge-reranker-large', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation

score = reranker.compute_score(['query', 'passage'])
print(score)

scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']])
print(scores)

Using Huggingface transformers

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-large')
model = AutoModelForSequenceClassification.from_pretrained('BAAI/bge-reranker-large')
model.eval()

pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
with torch.no_grad():
    inputs = tokenizer(pairs, padding=True, truncation=True, return_tensors='pt', max_length=512)
    scores = model(**inputs, return_dict=True).logits.view(-1, ).float()
    print(scores)

Usage of the ONNX files

from optimum.onnxruntime import ORTModelForFeatureExtraction  # type: ignore

import torch
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-small-en-v1.5')
model = AutoModel.from_pretrained('BAAI/bge-small-en-v1.5')
model_ort = ORTModelForFeatureExtraction.from_pretrained('BAAI/bge-small-en-v1.5', file_name="onnx/model.onnx")

# Sentences we want sentence embeddings for
sentences = ["样例数据-1", "样例数据-2"]

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
# for s2p(short query to long passage) retrieval task, add an instruction to query (not add instruction for passages)
# encoded_input = tokenizer([instruction + q for q in queries], padding=True, truncation=True, return_tensors='pt')

model_output_ort = model_ort(**encoded_input)
# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# model_output and model_output_ort are identical

Usage via infinity

Its also possible to deploy the onnx files with the infinity_emb pip package. Recommended is device="cuda", engine="torch" with flash attention on gpu, and device="cpu", engine="optimum" for onnx inference.

import asyncio
from infinity_emb import AsyncEmbeddingEngine, EngineArgs

sentences = ["Embed this is sentence via Infinity.", "Paris is in France."]
engine = AsyncEmbeddingEngine.from_args(
    EngineArgs(model_name_or_path = "BAAI/bge-small-en-v1.5", device="cpu", engine="optimum" # or engine="torch"
))

async def main(): 
    async with engine:
        embeddings, usage = await engine.embed(sentences=sentences)
asyncio.run(main())

Evaluation

baai-general-embedding models achieve state-of-the-art performance on both MTEB and C-MTEB leaderboard! For more details and evaluation tools see our scripts.

• MTEB:

Model Name	Dimension	Sequence Length	Average (56)	Retrieval (15)	Clustering (11)	Pair Classification (3)	Reranking (4)	STS (10)	Summarization (1)	Classification (12)
BAAI/bge-large-en-v1.5	1024	512	64.23	54.29	46.08	87.12	60.03	83.11	31.61	75.97
BAAI/bge-base-en-v1.5	768	512	63.55	53.25	45.77	86.55	58.86	82.4	31.07	75.53
BAAI/bge-small-en-v1.5	384	512	62.17	51.68	43.82	84.92	58.36	81.59	30.12	74.14
bge-large-en	1024	512	63.98	53.9	46.98	85.8	59.48	81.56	32.06	76.21
bge-base-en	768	512	63.36	53.0	46.32	85.86	58.7	81.84	29.27	75.27
gte-large	1024	512	63.13	52.22	46.84	85.00	59.13	83.35	31.66	73.33
gte-base	768	512	62.39	51.14	46.2	84.57	58.61	82.3	31.17	73.01
e5-large-v2	1024	512	62.25	50.56	44.49	86.03	56.61	82.05	30.19	75.24
bge-small-en	384	512	62.11	51.82	44.31	83.78	57.97	80.72	30.53	74.37
instructor-xl	768	512	61.79	49.26	44.74	86.62	57.29	83.06	32.32	61.79
e5-base-v2	768	512	61.5	50.29	43.80	85.73	55.91	81.05	30.28	73.84
gte-small	384	512	61.36	49.46	44.89	83.54	57.7	82.07	30.42	72.31
text-embedding-ada-002	1536	8192	60.99	49.25	45.9	84.89	56.32	80.97	30.8	70.93
e5-small-v2	384	512	59.93	49.04	39.92	84.67	54.32	80.39	31.16	72.94
sentence-t5-xxl	768	512	59.51	42.24	43.72	85.06	56.42	82.63	30.08	73.42
all-mpnet-base-v2	768	514	57.78	43.81	43.69	83.04	59.36	80.28	27.49	65.07
sgpt-bloom-7b1-msmarco	4096	2048	57.59	48.22	38.93	81.9	55.65	77.74	33.6	66.19

• C-MTEB:
  We create the benchmark C-MTEB for Chinese text embedding which consists of 31 datasets from 6 tasks. Please refer to C_MTEB for a detailed introduction.

Model	Embedding dimension	Avg	Retrieval	STS	PairClassification	Classification	Reranking	Clustering
BAAI/bge-large-zh-v1.5	1024	64.53	70.46	56.25	81.6	69.13	65.84	48.99
BAAI/bge-base-zh-v1.5	768	63.13	69.49	53.72	79.75	68.07	65.39	47.53
BAAI/bge-small-zh-v1.5	512	57.82	61.77	49.11	70.41	63.96	60.92	44.18
BAAI/bge-large-zh	1024	64.20	71.53	54.98	78.94	68.32	65.11	48.39
bge-large-zh-noinstruct	1024	63.53	70.55	53	76.77	68.58	64.91	50.01
BAAI/bge-base-zh	768	62.96	69.53	54.12	77.5	67.07	64.91	47.63
multilingual-e5-large	1024	58.79	63.66	48.44	69.89	67.34	56.00	48.23
BAAI/bge-small-zh	512	58.27	63.07	49.45	70.35	63.64	61.48	45.09
m3e-base	768	57.10	56.91	50.47	63.99	67.52	59.34	47.68
m3e-large	1024	57.05	54.75	50.42	64.3	68.2	59.66	48.88
multilingual-e5-base	768	55.48	61.63	46.49	67.07	65.35	54.35	40.68
multilingual-e5-small	384	55.38	59.95	45.27	66.45	65.85	53.86	45.26
text-embedding-ada-002(OpenAI)	1536	53.02	52.0	43.35	69.56	64.31	54.28	45.68
luotuo	1024	49.37	44.4	42.78	66.62	61	49.25	44.39
text2vec-base	768	47.63	38.79	43.41	67.41	62.19	49.45	37.66
text2vec-large	1024	47.36	41.94	44.97	70.86	60.66	49.16	30.02

• Reranking: See C_MTEB for evaluation script.

Model	T2Reranking	T2RerankingZh2En*	T2RerankingEn2Zh*	MMarcoReranking	CMedQAv1	CMedQAv2	Avg
text2vec-base-multilingual	64.66	62.94	62.51	14.37	48.46	48.6	50.26
multilingual-e5-small	65.62	60.94	56.41	29.91	67.26	66.54	57.78
multilingual-e5-large	64.55	61.61	54.28	28.6	67.42	67.92	57.4
multilingual-e5-base	64.21	62.13	54.68	29.5	66.23	66.98	57.29
m3e-base	66.03	62.74	56.07	17.51	77.05	76.76	59.36
m3e-large	66.13	62.72	56.1	16.46	77.76	78.27	59.57
bge-base-zh-v1.5	66.49	63.25	57.02	29.74	80.47	84.88	63.64
bge-large-zh-v1.5	65.74	63.39	57.03	28.74	83.45	85.44	63.97
BAAI/bge-reranker-base	67.28	63.95	60.45	35.46	81.26	84.1	65.42
BAAI/bge-reranker-large	67.6	64.03	61.44	37.16	82.15	84.18	66.09

* : T2RerankingZh2En and T2RerankingEn2Zh are cross-language retrieval tasks

Train

BAAI Embedding

We pre-train the models using retromae and train them on large-scale pairs data using contrastive learning. You can fine-tune the embedding model on your data following our examples. We also provide a pre-train example. Note that the goal of pre-training is to reconstruct the text, and the pre-trained model cannot be used for similarity calculation directly, it needs to be fine-tuned. More training details for bge see baai_general_embedding.

BGE Reranker

Cross-encoder will perform full-attention over the input pair, which is more accurate than embedding model (i.e., bi-encoder) but more time-consuming than embedding model. Therefore, it can be used to re-rank the top-k documents returned by embedding model. We train the cross-encoder on a multilingual pair data, The data format is the same as embedding model, so you can fine-tune it easily following our example. More details please refer to ./FlagEmbedding/reranker/README.md

Contact

If you have any question or suggestion related to this project, feel free to open an issue or pull request. You also can email Shitao Xiao(stxiao@baai.ac.cn) and Zheng Liu(liuzheng@baai.ac.cn).

Citation

If you find this repository useful, please consider giving a star ⭐ and citation

@misc{bge_embedding,
      title={C-Pack: Packaged Resources To Advance General Chinese Embedding}, 
      author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff},
      year={2023},
      eprint={2309.07597},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

License

FlagEmbedding is licensed under the MIT License. The released models can be used for commercial purposes free of charge.