bge-large-en-v1.5/README.md

tags, model-index, license, language

tags	model-index	license	language
sentence-transformers feature-extraction sentence-similarity transformers mteb	name results bge-large-en-v1.5 task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en) en test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 75.8507462686567 type value ap 38.566457320228245 type value f1 69.69386648043475 task dataset metrics type Classification type name config split revision mteb/amazon_polarity MTEB AmazonPolarityClassification default test e2d317d38cd51312af73b3d32a06d1a08b442046 type value accuracy 92.416675 type value ap 89.1928861155922 type value f1 92.39477019574215 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (en) en test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 48.175999999999995 type value f1 47.80712792870253 task dataset metrics type Retrieval type name config split revision arguana MTEB ArguAna default test None type value map_at_1 40.184999999999995 type value map_at_10 55.654 type value map_at_100 56.25 type value map_at_1000 56.255 type value map_at_3 51.742999999999995 type value map_at_5 54.129000000000005 type value mrr_at_1 40.967 type value mrr_at_10 55.96 type value mrr_at_100 56.54900000000001 type value mrr_at_1000 56.554 type value mrr_at_3 51.980000000000004 type value mrr_at_5 54.44 type value ndcg_at_1 40.184999999999995 type value ndcg_at_10 63.542 type value ndcg_at_100 65.96499999999999 type value ndcg_at_1000 66.08699999999999 type value ndcg_at_3 55.582 type value ndcg_at_5 59.855000000000004 type value precision_at_1 40.184999999999995 type value precision_at_10 8.841000000000001 type value precision_at_100 0.987 type value precision_at_1000 0.1 type value precision_at_3 22.238 type value precision_at_5 15.405 type value recall_at_1 40.184999999999995 type value recall_at_10 88.407 type value recall_at_100 98.72 type value recall_at_1000 99.644 type value recall_at_3 66.714 type value recall_at_5 77.027 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-p2p MTEB ArxivClusteringP2P default test a122ad7f3f0291bf49cc6f4d32aa80929df69d5d type value v_measure 48.567077926750066 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-s2s MTEB ArxivClusteringS2S default test f910caf1a6075f7329cdf8c1a6135696f37dbd53 type value v_measure 43.19453389182364 task dataset metrics type Reranking type name config split revision mteb/askubuntudupquestions-reranking MTEB AskUbuntuDupQuestions default test 2000358ca161889fa9c082cb41daa8dcfb161a54 type value map 64.46555939623092 type value mrr 77.82361605768807 task dataset metrics type STS type name config split revision mteb/biosses-sts MTEB BIOSSES default test d3fb88f8f02e40887cd149695127462bbcf29b4a type value cos_sim_pearson 84.9554128814735 type value cos_sim_spearman 84.65373612172036 type value euclidean_pearson 83.2905059954138 type value euclidean_spearman 84.52240782811128 type value manhattan_pearson 82.99533802997436 type value manhattan_spearman 84.20673798475734 task dataset metrics type Classification type name config split revision mteb/banking77 MTEB Banking77Classification default test 0fd18e25b25c072e09e0d92ab615fda904d66300 type value accuracy 87.78896103896103 type value f1 87.77189310964883 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-p2p MTEB BiorxivClusteringP2P default test 65b79d1d13f80053f67aca9498d9402c2d9f1f40 type value v_measure 39.714538337650495 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-s2s MTEB BiorxivClusteringS2S default test 258694dd0231531bc1fd9de6ceb52a0853c6d908 type value v_measure 36.90108349284447 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackAndroidRetrieval default test None type value map_at_1 32.795 type value map_at_10 43.669000000000004 type value map_at_100 45.151 type value map_at_1000 45.278 type value map_at_3 40.006 type value map_at_5 42.059999999999995 type value mrr_at_1 39.771 type value mrr_at_10 49.826 type value mrr_at_100 50.504000000000005 type value mrr_at_1000 50.549 type value mrr_at_3 47.115 type value mrr_at_5 48.832 type value ndcg_at_1 39.771 type value ndcg_at_10 50.217999999999996 type value ndcg_at_100 55.454 type value ndcg_at_1000 57.37 type value ndcg_at_3 44.885000000000005 type value ndcg_at_5 47.419 type value precision_at_1 39.771 type value precision_at_10 9.642000000000001 type value precision_at_100 1.538 type value precision_at_1000 0.198 type value precision_at_3 21.268 type value precision_at_5 15.536 type value recall_at_1 32.795 type value recall_at_10 62.580999999999996 type value recall_at_100 84.438 type value recall_at_1000 96.492 type value recall_at_3 47.071000000000005 type value recall_at_5 54.079 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackEnglishRetrieval default test None type value map_at_1 32.671 type value map_at_10 43.334 type value map_at_100 44.566 type value map_at_1000 44.702999999999996 type value map_at_3 40.343 type value map_at_5 41.983 type value mrr_at_1 40.764 type value mrr_at_10 49.382 type value mrr_at_100 49.988 type value mrr_at_1000 50.03300000000001 type value mrr_at_3 47.293 type value mrr_at_5 48.51 type value ndcg_at_1 40.764 type value ndcg_at_10 49.039 type value ndcg_at_100 53.259 type value ndcg_at_1000 55.253 type value ndcg_at_3 45.091 type value ndcg_at_5 46.839999999999996 type value precision_at_1 40.764 type value precision_at_10 9.191 type value precision_at_100 1.476 type value precision_at_1000 0.19499999999999998 type value precision_at_3 21.72 type value precision_at_5 15.299 type value recall_at_1 32.671 type value recall_at_10 58.816 type value recall_at_100 76.654 type value recall_at_1000 89.05999999999999 type value recall_at_3 46.743 type value recall_at_5 51.783 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGamingRetrieval default test None type value map_at_1 40.328 type value map_at_10 53.32599999999999 type value map_at_100 54.37499999999999 type value map_at_1000 54.429 type value map_at_3 49.902 type value map_at_5 52.002 type value mrr_at_1 46.332 type value mrr_at_10 56.858 type value mrr_at_100 57.522 type value mrr_at_1000 57.54899999999999 type value mrr_at_3 54.472 type value mrr_at_5 55.996 type value ndcg_at_1 46.332 type value ndcg_at_10 59.313 type value ndcg_at_100 63.266999999999996 type value ndcg_at_1000 64.36 type value ndcg_at_3 53.815000000000005 type value ndcg_at_5 56.814 type value precision_at_1 46.332 type value precision_at_10 9.53 type value precision_at_100 1.238 type value precision_at_1000 0.13699999999999998 type value precision_at_3 24.054000000000002 type value precision_at_5 16.589000000000002 type value recall_at_1 40.328 type value recall_at_10 73.421 type value recall_at_100 90.059 type value recall_at_1000 97.81 type value recall_at_3 59.009 type value recall_at_5 66.352 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGisRetrieval default test None type value map_at_1 27.424 type value map_at_10 36.332 type value map_at_100 37.347 type value map_at_1000 37.422 type value map_at_3 33.743 type value map_at_5 35.176 type value mrr_at_1 29.153000000000002 type value mrr_at_10 38.233 type value mrr_at_100 39.109 type value mrr_at_1000 39.164 type value mrr_at_3 35.876000000000005 type value mrr_at_5 37.169000000000004 type value ndcg_at_1 29.153000000000002 type value ndcg_at_10 41.439 type value ndcg_at_100 46.42 type value ndcg_at_1000 48.242000000000004 type value ndcg_at_3 36.362 type value ndcg_at_5 38.743 type value precision_at_1 29.153000000000002 type value precision_at_10 6.315999999999999 type value precision_at_100 0.927 type value precision_at_1000 0.11199999999999999 type value precision_at_3 15.443000000000001 type value precision_at_5 10.644 type value recall_at_1 27.424 type value recall_at_10 55.364000000000004 type value recall_at_100 78.211 type value recall_at_1000 91.74600000000001 type value recall_at_3 41.379 type value recall_at_5 47.14 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackMathematicaRetrieval default test None type value map_at_1 19.601 type value map_at_10 27.826 type value map_at_100 29.017 type value map_at_1000 29.137 type value map_at_3 25.125999999999998 type value map_at_5 26.765 type value mrr_at_1 24.005000000000003 type value mrr_at_10 32.716 type value mrr_at_100 33.631 type value mrr_at_1000 33.694 type value mrr_at_3 29.934 type value mrr_at_5 31.630999999999997 type value ndcg_at_1 24.005000000000003 type value ndcg_at_10 33.158 type value ndcg_at_100 38.739000000000004 type value ndcg_at_1000 41.495 type value ndcg_at_3 28.185 type value ndcg_at_5 30.796 type value precision_at_1 24.005000000000003 type value precision_at_10 5.908 type value precision_at_100 1.005 type value precision_at_1000 0.13899999999999998 type value precision_at_3 13.391 type value precision_at_5 9.876 type value recall_at_1 19.601 type value recall_at_10 44.746 type value recall_at_100 68.82300000000001 type value recall_at_1000 88.215 type value recall_at_3 31.239 type value recall_at_5 37.695 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackPhysicsRetrieval default test None type value map_at_1 30.130000000000003 type value map_at_10 40.96 type value map_at_100 42.282 type value map_at_1000 42.392 type value map_at_3 37.889 type value map_at_5 39.661 type value mrr_at_1 36.958999999999996 type value mrr_at_10 46.835 type value mrr_at_100 47.644 type value mrr_at_1000 47.688 type value mrr_at_3 44.562000000000005 type value mrr_at_5 45.938 type value ndcg_at_1 36.958999999999996 type value ndcg_at_10 47.06 type value ndcg_at_100 52.345 type value ndcg_at_1000 54.35 type value ndcg_at_3 42.301 type value ndcg_at_5 44.635999999999996 type value precision_at_1 36.958999999999996 type value precision_at_10 8.479000000000001 type value precision_at_100 1.284 type value precision_at_1000 0.163 type value precision_at_3 20.244 type value precision_at_5 14.224999999999998 type value recall_at_1 30.130000000000003 type value recall_at_10 59.27 type value recall_at_100 81.195 type value recall_at_1000 94.21199999999999 type value recall_at_3 45.885 type value recall_at_5 52.016 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackProgrammersRetrieval default test None type value map_at_1 26.169999999999998 type value map_at_10 36.451 type value map_at_100 37.791000000000004 type value map_at_1000 37.897 type value map_at_3 33.109 type value map_at_5 34.937000000000005 type value mrr_at_1 32.877 type value mrr_at_10 42.368 type value mrr_at_100 43.201 type value mrr_at_1000 43.259 type value mrr_at_3 39.763999999999996 type value mrr_at_5 41.260000000000005 type value ndcg_at_1 32.877 type value ndcg_at_10 42.659000000000006 type value ndcg_at_100 48.161 type value ndcg_at_1000 50.345 type value ndcg_at_3 37.302 type value ndcg_at_5 39.722 type value precision_at_1 32.877 type value precision_at_10 7.9 type value precision_at_100 1.236 type value precision_at_1000 0.158 type value precision_at_3 17.846 type value precision_at_5 12.9 type value recall_at_1 26.169999999999998 type value recall_at_10 55.35 type value recall_at_100 78.755 type value recall_at_1000 93.518 type value recall_at_3 40.176 type value recall_at_5 46.589000000000006 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackRetrieval default test None type value map_at_1 27.15516666666667 type value map_at_10 36.65741666666667 type value map_at_100 37.84991666666666 type value map_at_1000 37.96316666666667 type value map_at_3 33.74974999999999 type value map_at_5 35.3765 type value mrr_at_1 32.08233333333334 type value mrr_at_10 41.033833333333334 type value mrr_at_100 41.84524999999999 type value mrr_at_1000 41.89983333333333 type value mrr_at_3 38.62008333333333 type value mrr_at_5 40.03441666666666 type value ndcg_at_1 32.08233333333334 type value ndcg_at_10 42.229 type value ndcg_at_100 47.26716666666667 type value ndcg_at_1000 49.43466666666667 type value ndcg_at_3 37.36408333333333 type value ndcg_at_5 39.6715 type value precision_at_1 32.08233333333334 type value precision_at_10 7.382583333333334 type value precision_at_100 1.16625 type value precision_at_1000 0.15408333333333332 type value precision_at_3 17.218 type value precision_at_5 12.21875 type value recall_at_1 27.15516666666667 type value recall_at_10 54.36683333333333 type value recall_at_100 76.37183333333333 type value recall_at_1000 91.26183333333333 type value recall_at_3 40.769916666666674 type value recall_at_5 46.702333333333335 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackStatsRetrieval default test None type value map_at_1 25.749 type value map_at_10 33.001999999999995 type value map_at_100 33.891 type value map_at_1000 33.993 type value map_at_3 30.703999999999997 type value map_at_5 31.959 type value mrr_at_1 28.834 type value mrr_at_10 35.955 type value mrr_at_100 36.709 type value mrr_at_1000 36.779 type value mrr_at_3 33.947 type value mrr_at_5 35.089 type value ndcg_at_1 28.834 type value ndcg_at_10 37.329 type value ndcg_at_100 41.79 type value ndcg_at_1000 44.169000000000004 type value ndcg_at_3 33.184999999999995 type value ndcg_at_5 35.107 type value precision_at_1 28.834 type value precision_at_10 5.7669999999999995 type value precision_at_100 0.876 type value precision_at_1000 0.11399999999999999 type value precision_at_3 14.213000000000001 type value precision_at_5 9.754999999999999 type value recall_at_1 25.749 type value recall_at_10 47.791 type value recall_at_100 68.255 type value recall_at_1000 85.749 type value recall_at_3 36.199 type value recall_at_5 41.071999999999996 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackTexRetrieval default test None type value map_at_1 17.777 type value map_at_10 25.201 type value map_at_100 26.423999999999996 type value map_at_1000 26.544 type value map_at_3 22.869 type value map_at_5 24.023 type value mrr_at_1 21.473 type value mrr_at_10 29.12 type value mrr_at_100 30.144 type value mrr_at_1000 30.215999999999998 type value mrr_at_3 26.933 type value mrr_at_5 28.051 type value ndcg_at_1 21.473 type value ndcg_at_10 30.003 type value ndcg_at_100 35.766 type value ndcg_at_1000 38.501000000000005 type value ndcg_at_3 25.773000000000003 type value ndcg_at_5 27.462999999999997 type value precision_at_1 21.473 type value precision_at_10 5.482 type value precision_at_100 0.975 type value precision_at_1000 0.13799999999999998 type value precision_at_3 12.205 type value precision_at_5 8.692 type value recall_at_1 17.777 type value recall_at_10 40.582 type value recall_at_100 66.305 type value recall_at_1000 85.636 type value recall_at_3 28.687 type value recall_at_5 33.089 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackUnixRetrieval default test None type value map_at_1 26.677 type value map_at_10 36.309000000000005 type value map_at_100 37.403999999999996 type value map_at_1000 37.496 type value map_at_3 33.382 type value map_at_5 34.98 type value mrr_at_1 31.343 type value mrr_at_10 40.549 type value mrr_at_100 41.342 type value mrr_at_1000 41.397 type value mrr_at_3 38.029 type value mrr_at_5 39.451 type value ndcg_at_1 31.343 type value ndcg_at_10 42.1 type value ndcg_at_100 47.089999999999996 type value ndcg_at_1000 49.222 type value ndcg_at_3 36.836999999999996 type value ndcg_at_5 39.21 type value precision_at_1 31.343 type value precision_at_10 7.164 type value precision_at_100 1.0959999999999999 type value precision_at_1000 0.13899999999999998 type value precision_at_3 16.915 type value precision_at_5 11.940000000000001 type value recall_at_1 26.677 type value recall_at_10 55.54599999999999 type value recall_at_100 77.094 type value recall_at_1000 92.01 type value recall_at_3 41.191 type value recall_at_5 47.006 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWebmastersRetrieval default test None type value map_at_1 24.501 type value map_at_10 33.102 type value map_at_100 34.676 type value map_at_1000 34.888000000000005 type value map_at_3 29.944 type value map_at_5 31.613999999999997 type value mrr_at_1 29.447000000000003 type value mrr_at_10 37.996 type value mrr_at_100 38.946 type value mrr_at_1000 38.995000000000005 type value mrr_at_3 35.079 type value mrr_at_5 36.69 type value ndcg_at_1 29.447000000000003 type value ndcg_at_10 39.232 type value ndcg_at_100 45.247 type value ndcg_at_1000 47.613 type value ndcg_at_3 33.922999999999995 type value ndcg_at_5 36.284 type value precision_at_1 29.447000000000003 type value precision_at_10 7.648000000000001 type value precision_at_100 1.516 type value precision_at_1000 0.23900000000000002 type value precision_at_3 16.008 type value precision_at_5 11.779 type value recall_at_1 24.501 type value recall_at_10 51.18899999999999 type value recall_at_100 78.437 type value recall_at_1000 92.842 type value recall_at_3 35.808 type value recall_at_5 42.197 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWordpressRetrieval default test None type value map_at_1 22.039 type value map_at_10 30.377 type value map_at_100 31.275 type value map_at_1000 31.379 type value map_at_3 27.98 type value map_at_5 29.358 type value mrr_at_1 24.03 type value mrr_at_10 32.568000000000005 type value mrr_at_100 33.403 type value mrr_at_1000 33.475 type value mrr_at_3 30.436999999999998 type value mrr_at_5 31.796000000000003 type value ndcg_at_1 24.03 type value ndcg_at_10 35.198 type value ndcg_at_100 39.668 type value ndcg_at_1000 42.296 type value ndcg_at_3 30.709999999999997 type value ndcg_at_5 33.024 type value precision_at_1 24.03 type value precision_at_10 5.564 type value precision_at_100 0.828 type value precision_at_1000 0.117 type value precision_at_3 13.309000000000001 type value precision_at_5 9.39 type value recall_at_1 22.039 type value recall_at_10 47.746 type value recall_at_100 68.23599999999999 type value recall_at_1000 87.852 type value recall_at_3 35.852000000000004 type value recall_at_5 41.410000000000004 task dataset metrics type Retrieval type name config split revision climate-fever MTEB ClimateFEVER default test None type value map_at_1 15.692999999999998 type value map_at_10 26.903 type value map_at_100 28.987000000000002 type value map_at_1000 29.176999999999996 type value map_at_3 22.137 type value map_at_5 24.758 type value mrr_at_1 35.57 type value mrr_at_10 47.821999999999996 type value mrr_at_100 48.608000000000004 type value mrr_at_1000 48.638999999999996 type value mrr_at_3 44.452000000000005 type value mrr_at_5 46.546 type value ndcg_at_1 35.57 type value ndcg_at_10 36.567 type value ndcg_at_100 44.085 type value ndcg_at_1000 47.24 type value ndcg_at_3 29.964000000000002 type value ndcg_at_5 32.511 type value precision_at_1 35.57 type value precision_at_10 11.485 type value precision_at_100 1.9619999999999997 type value precision_at_1000 0.256 type value precision_at_3 22.237000000000002 type value precision_at_5 17.471999999999998 type value recall_at_1 15.692999999999998 type value recall_at_10 43.056 type value recall_at_100 68.628 type value recall_at_1000 86.075 type value recall_at_3 26.918999999999997 type value recall_at_5 34.14 task dataset metrics type Retrieval type name config split revision dbpedia-entity MTEB DBPedia default test None type value map_at_1 9.53 type value map_at_10 20.951 type value map_at_100 30.136000000000003 type value map_at_1000 31.801000000000002 type value map_at_3 15.021 type value map_at_5 17.471999999999998 type value mrr_at_1 71.0 type value mrr_at_10 79.176 type value mrr_at_100 79.418 type value mrr_at_1000 79.426 type value mrr_at_3 78.125 type value mrr_at_5 78.61200000000001 type value ndcg_at_1 58.5 type value ndcg_at_10 44.106 type value ndcg_at_100 49.268 type value ndcg_at_1000 56.711999999999996 type value ndcg_at_3 48.934 type value ndcg_at_5 45.826 type value precision_at_1 71.0 type value precision_at_10 35.0 type value precision_at_100 11.360000000000001 type value precision_at_1000 2.046 type value precision_at_3 52.833 type value precision_at_5 44.15 type value recall_at_1 9.53 type value recall_at_10 26.811 type value recall_at_100 55.916999999999994 type value recall_at_1000 79.973 type value recall_at_3 16.413 type value recall_at_5 19.980999999999998 task dataset metrics type Classification type name config split revision mteb/emotion MTEB EmotionClassification default test 4f58c6b202a23cf9a4da393831edf4f9183cad37 type value accuracy 51.519999999999996 type value f1 46.36601294761231 task dataset metrics type Retrieval type name config split revision fever MTEB FEVER default test None type value map_at_1 74.413 type value map_at_10 83.414 type value map_at_100 83.621 type value map_at_1000 83.635 type value map_at_3 82.337 type value map_at_5 83.039 type value mrr_at_1 80.19800000000001 type value mrr_at_10 87.715 type value mrr_at_100 87.778 type value mrr_at_1000 87.779 type value mrr_at_3 87.106 type value mrr_at_5 87.555 type value ndcg_at_1 80.19800000000001 type value ndcg_at_10 87.182 type value ndcg_at_100 87.90299999999999 type value ndcg_at_1000 88.143 type value ndcg_at_3 85.60600000000001 type value ndcg_at_5 86.541 type value precision_at_1 80.19800000000001 type value precision_at_10 10.531 type value precision_at_100 1.113 type value precision_at_1000 0.11499999999999999 type value precision_at_3 32.933 type value precision_at_5 20.429 type value recall_at_1 74.413 type value recall_at_10 94.363 type value recall_at_100 97.165 type value recall_at_1000 98.668 type value recall_at_3 90.108 type value recall_at_5 92.52 task dataset metrics type Retrieval type name config split revision fiqa MTEB FiQA2018 default test None type value map_at_1 22.701 type value map_at_10 37.122 type value map_at_100 39.178000000000004 type value map_at_1000 39.326 type value map_at_3 32.971000000000004 type value map_at_5 35.332 type value mrr_at_1 44.753 type value mrr_at_10 53.452 type value mrr_at_100 54.198 type value mrr_at_1000 54.225 type value mrr_at_3 50.952 type value mrr_at_5 52.464 type value ndcg_at_1 44.753 type value ndcg_at_10 45.021 type value ndcg_at_100 52.028 type value ndcg_at_1000 54.596000000000004 type value ndcg_at_3 41.622 type value ndcg_at_5 42.736000000000004 type value precision_at_1 44.753 type value precision_at_10 12.284 type value precision_at_100 1.955 type value precision_at_1000 0.243 type value precision_at_3 27.828999999999997 type value precision_at_5 20.061999999999998 type value recall_at_1 22.701 type value recall_at_10 51.432 type value recall_at_100 77.009 type value recall_at_1000 92.511 type value recall_at_3 37.919000000000004 type value recall_at_5 44.131 task dataset metrics type Retrieval type name config split revision hotpotqa MTEB HotpotQA default test None type value map_at_1 40.189 type value map_at_10 66.24600000000001 type value map_at_100 67.098 type value map_at_1000 67.149 type value map_at_3 62.684 type value map_at_5 64.974 type value mrr_at_1 80.378 type value mrr_at_10 86.127 type value mrr_at_100 86.29299999999999 type value mrr_at_1000 86.297 type value mrr_at_3 85.31400000000001 type value mrr_at_5 85.858 type value ndcg_at_1 80.378 type value ndcg_at_10 74.101 type value ndcg_at_100 76.993 type value ndcg_at_1000 77.948 type value ndcg_at_3 69.232 type value ndcg_at_5 72.04599999999999 type value precision_at_1 80.378 type value precision_at_10 15.595999999999998 type value precision_at_100 1.7840000000000003 type value precision_at_1000 0.191 type value precision_at_3 44.884 type value precision_at_5 29.145 type value recall_at_1 40.189 type value recall_at_10 77.981 type value recall_at_100 89.21 type value recall_at_1000 95.48299999999999 type value recall_at_3 67.326 type value recall_at_5 72.863 task dataset metrics type Classification type name config split revision mteb/imdb MTEB ImdbClassification default test 3d86128a09e091d6018b6d26cad27f2739fc2db7 type value accuracy 92.84599999999999 type value ap 89.4710787567357 type value f1 92.83752676932258 task dataset metrics type Retrieval type name config split revision msmarco MTEB MSMARCO default dev None type value map_at_1 23.132 type value map_at_10 35.543 type value map_at_100 36.702 type value map_at_1000 36.748999999999995 type value map_at_3 31.737 type value map_at_5 33.927 type value mrr_at_1 23.782 type value mrr_at_10 36.204 type value mrr_at_100 37.29 type value mrr_at_1000 37.330999999999996 type value mrr_at_3 32.458999999999996 type value mrr_at_5 34.631 type value ndcg_at_1 23.782 type value ndcg_at_10 42.492999999999995 type value ndcg_at_100 47.985 type value ndcg_at_1000 49.141 type value ndcg_at_3 34.748000000000005 type value ndcg_at_5 38.651 type value precision_at_1 23.782 type value precision_at_10 6.665 type value precision_at_100 0.941 type value precision_at_1000 0.104 type value precision_at_3 14.776 type value precision_at_5 10.84 type value recall_at_1 23.132 type value recall_at_10 63.794 type value recall_at_100 89.027 type value recall_at_1000 97.807 type value recall_at_3 42.765 type value recall_at_5 52.11 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (en) en test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 94.59188326493388 type value f1 94.3842594786827 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (en) en test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 79.49384404924761 type value f1 59.7580539534629 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (en) en test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 77.56220578345663 type value f1 75.27228165561478 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (en) en test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 80.53463349024884 type value f1 80.4893958236536 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-p2p MTEB MedrxivClusteringP2P default test e7a26af6f3ae46b30dde8737f02c07b1505bcc73 type value v_measure 32.56100273484962 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-s2s MTEB MedrxivClusteringS2S default test 35191c8c0dca72d8ff3efcd72aa802307d469663 type value v_measure 31.470380028839607 task dataset metrics type Reranking type name config split revision mteb/mind_small MTEB MindSmallReranking default test 3bdac13927fdc888b903db93b2ffdbd90b295a69 type value map 32.06102792457849 type value mrr 33.30709199672238 task dataset metrics type Retrieval type name config split revision nfcorpus MTEB NFCorpus default test None type value map_at_1 6.776999999999999 type value map_at_10 14.924000000000001 type value map_at_100 18.955 type value map_at_1000 20.538999999999998 type value map_at_3 10.982 type value map_at_5 12.679000000000002 type value mrr_at_1 47.988 type value mrr_at_10 57.232000000000006 type value mrr_at_100 57.818999999999996 type value mrr_at_1000 57.847 type value mrr_at_3 54.901999999999994 type value mrr_at_5 56.481 type value ndcg_at_1 46.594 type value ndcg_at_10 38.129000000000005 type value ndcg_at_100 35.54 type value ndcg_at_1000 44.172 type value ndcg_at_3 43.025999999999996 type value ndcg_at_5 41.052 type value precision_at_1 47.988 type value precision_at_10 28.111000000000004 type value precision_at_100 8.929 type value precision_at_1000 2.185 type value precision_at_3 40.144000000000005 type value precision_at_5 35.232 type value recall_at_1 6.776999999999999 type value recall_at_10 19.289 type value recall_at_100 36.359 type value recall_at_1000 67.54 type value recall_at_3 11.869 type value recall_at_5 14.999 task dataset metrics type Retrieval type name config split revision nq MTEB NQ default test None type value map_at_1 31.108000000000004 type value map_at_10 47.126000000000005 type value map_at_100 48.171 type value map_at_1000 48.199 type value map_at_3 42.734 type value map_at_5 45.362 type value mrr_at_1 34.936 type value mrr_at_10 49.571 type value mrr_at_100 50.345 type value mrr_at_1000 50.363 type value mrr_at_3 45.959 type value mrr_at_5 48.165 type value ndcg_at_1 34.936 type value ndcg_at_10 55.028999999999996 type value ndcg_at_100 59.244 type value ndcg_at_1000 59.861 type value ndcg_at_3 46.872 type value ndcg_at_5 51.217999999999996 type value precision_at_1 34.936 type value precision_at_10 9.099 type value precision_at_100 1.145 type value precision_at_1000 0.12 type value precision_at_3 21.456 type value precision_at_5 15.411 type value recall_at_1 31.108000000000004 type value recall_at_10 76.53999999999999 type value recall_at_100 94.39 type value recall_at_1000 98.947 type value recall_at_3 55.572 type value recall_at_5 65.525 task dataset metrics type Retrieval type name config split revision quora MTEB QuoraRetrieval default test None type value map_at_1 71.56400000000001 type value map_at_10 85.482 type value map_at_100 86.114 type value map_at_1000 86.13 type value map_at_3 82.607 type value map_at_5 84.405 type value mrr_at_1 82.42 type value mrr_at_10 88.304 type value mrr_at_100 88.399 type value mrr_at_1000 88.399 type value mrr_at_3 87.37 type value mrr_at_5 88.024 type value ndcg_at_1 82.45 type value ndcg_at_10 89.06500000000001 type value ndcg_at_100 90.232 type value ndcg_at_1000 90.305 type value ndcg_at_3 86.375 type value ndcg_at_5 87.85300000000001 type value precision_at_1 82.45 type value precision_at_10 13.486999999999998 type value precision_at_100 1.534 type value precision_at_1000 0.157 type value precision_at_3 37.813 type value precision_at_5 24.773999999999997 type value recall_at_1 71.56400000000001 type value recall_at_10 95.812 type value recall_at_100 99.7 type value recall_at_1000 99.979 type value recall_at_3 87.966 type value recall_at_5 92.268 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering MTEB RedditClustering default test 24640382cdbf8abc73003fb0fa6d111a705499eb type value v_measure 57.241876648614145 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering-p2p MTEB RedditClusteringP2P default test 282350215ef01743dc01b456c7f5241fa8937f16 type value v_measure 64.66212576446223 task dataset metrics type Retrieval type name config split revision scidocs MTEB SCIDOCS default test None type value map_at_1 5.308 type value map_at_10 13.803 type value map_at_100 16.176 type value map_at_1000 16.561 type value map_at_3 9.761000000000001 type value map_at_5 11.802 type value mrr_at_1 26.200000000000003 type value mrr_at_10 37.621 type value mrr_at_100 38.767 type value mrr_at_1000 38.815 type value mrr_at_3 34.117 type value mrr_at_5 36.107 type value ndcg_at_1 26.200000000000003 type value ndcg_at_10 22.64 type value ndcg_at_100 31.567 type value ndcg_at_1000 37.623 type value ndcg_at_3 21.435000000000002 type value ndcg_at_5 18.87 type value precision_at_1 26.200000000000003 type value precision_at_10 11.74 type value precision_at_100 2.465 type value precision_at_1000 0.391 type value precision_at_3 20.033 type value precision_at_5 16.64 type value recall_at_1 5.308 type value recall_at_10 23.794999999999998 type value recall_at_100 50.015 type value recall_at_1000 79.283 type value recall_at_3 12.178 type value recall_at_5 16.882 task dataset metrics type STS type name config split revision mteb/sickr-sts MTEB SICK-R default test a6ea5a8cab320b040a23452cc28066d9beae2cee type value cos_sim_pearson 84.93231134675553 type value cos_sim_spearman 81.68319292603205 type value euclidean_pearson 81.8396814380367 type value euclidean_spearman 81.24641903349945 type value manhattan_pearson 81.84698799204274 type value manhattan_spearman 81.24269997904105 task dataset metrics type STS type name config split revision mteb/sts12-sts MTEB STS12 default test a0d554a64d88156834ff5ae9920b964011b16384 type value cos_sim_pearson 86.73241671587446 type value cos_sim_spearman 79.05091082971826 type value euclidean_pearson 83.91146869578044 type value euclidean_spearman 79.87978465370936 type value manhattan_pearson 83.90888338917678 type value manhattan_spearman 79.87482848584241 task dataset metrics type STS type name config split revision mteb/sts13-sts MTEB STS13 default test 7e90230a92c190f1bf69ae9002b8cea547a64cca type value cos_sim_pearson 85.14970731146177 type value cos_sim_spearman 86.37363490084627 type value euclidean_pearson 83.02154218530433 type value euclidean_spearman 83.80258761957367 type value manhattan_pearson 83.01664495119347 type value manhattan_spearman 83.77567458007952 task dataset metrics type STS type name config split revision mteb/sts14-sts MTEB STS14 default test 6031580fec1f6af667f0bd2da0a551cf4f0b2375 type value cos_sim_pearson 83.40474139886784 type value cos_sim_spearman 82.77768789165984 type value euclidean_pearson 80.7065877443695 type value euclidean_spearman 81.375940662505 type value manhattan_pearson 80.6507552270278 type value manhattan_spearman 81.32782179098741 task dataset metrics type STS type name config split revision mteb/sts15-sts MTEB STS15 default test ae752c7c21bf194d8b67fd573edf7ae58183cbe3 type value cos_sim_pearson 87.08585968722274 type value cos_sim_spearman 88.03110031451399 type value euclidean_pearson 85.74012019602384 type value euclidean_spearman 86.13592849438209 type value manhattan_pearson 85.74404842369206 type value manhattan_spearman 86.14492318960154 task dataset metrics type STS type name config split revision mteb/sts16-sts MTEB STS16 default test 4d8694f8f0e0100860b497b999b3dbed754a0513 type value cos_sim_pearson 84.95069052788875 type value cos_sim_spearman 86.4867991595147 type value euclidean_pearson 84.31013325754635 type value euclidean_spearman 85.01529258006482 type value manhattan_pearson 84.26995570085374 type value manhattan_spearman 84.96982104986162 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-en) en-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 87.54617647971897 type value cos_sim_spearman 87.49834181751034 type value euclidean_pearson 86.01015322577122 type value euclidean_spearman 84.63362652063199 type value manhattan_pearson 86.13807574475706 type value manhattan_spearman 84.7772370721132 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (en) en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 67.20047755786615 type value cos_sim_spearman 67.05324077987636 type value euclidean_pearson 66.91930642976601 type value euclidean_spearman 65.21491856099105 type value manhattan_pearson 66.78756851976624 type value manhattan_spearman 65.12356257740728 task dataset metrics type STS type name config split revision mteb/stsbenchmark-sts MTEB STSBenchmark default test b0fddb56ed78048fa8b90373c8a3cfc37b684831 type value cos_sim_pearson 86.19852871539686 type value cos_sim_spearman 87.5161895296395 type value euclidean_pearson 84.59848645207485 type value euclidean_spearman 85.26427328757919 type value manhattan_pearson 84.59747366996524 type value manhattan_spearman 85.24045855146915 task dataset metrics type Reranking type name config split revision mteb/scidocs-reranking MTEB SciDocsRR default test d3c5e1fc0b855ab6097bf1cda04dd73947d7caab type value map 87.63320317811032 type value mrr 96.26242947321379 task dataset metrics type Retrieval type name config split revision scifact MTEB SciFact default test None type value map_at_1 60.928000000000004 type value map_at_10 70.112 type value map_at_100 70.59299999999999 type value map_at_1000 70.623 type value map_at_3 66.846 type value map_at_5 68.447 type value mrr_at_1 64.0 type value mrr_at_10 71.212 type value mrr_at_100 71.616 type value mrr_at_1000 71.64500000000001 type value mrr_at_3 68.77799999999999 type value mrr_at_5 70.094 type value ndcg_at_1 64.0 type value ndcg_at_10 74.607 type value ndcg_at_100 76.416 type value ndcg_at_1000 77.102 type value ndcg_at_3 69.126 type value ndcg_at_5 71.41300000000001 type value precision_at_1 64.0 type value precision_at_10 9.933 type value precision_at_100 1.077 type value precision_at_1000 0.11299999999999999 type value precision_at_3 26.556 type value precision_at_5 17.467 type value recall_at_1 60.928000000000004 type value recall_at_10 87.322 type value recall_at_100 94.833 type value recall_at_1000 100.0 type value recall_at_3 72.628 type value recall_at_5 78.428 task dataset metrics type PairClassification type name config split revision mteb/sprintduplicatequestions-pairclassification MTEB SprintDuplicateQuestions default test d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 type value cos_sim_accuracy 99.86237623762376 type value cos_sim_ap 96.72586477206649 type value cos_sim_f1 93.01858362631845 type value cos_sim_precision 93.4409687184662 type value cos_sim_recall 92.60000000000001 type value dot_accuracy 99.78019801980199 type value dot_ap 93.72748205246228 type value dot_f1 89.04109589041096 type value dot_precision 87.16475095785441 type value dot_recall 91.0 type value euclidean_accuracy 99.85445544554456 type value euclidean_ap 96.6661459876145 type value euclidean_f1 92.58337481333997 type value euclidean_precision 92.17046580773042 type value euclidean_recall 93.0 type value manhattan_accuracy 99.85445544554456 type value manhattan_ap 96.6883549244056 type value manhattan_f1 92.57598405580468 type value manhattan_precision 92.25422045680239 type value manhattan_recall 92.9 type value max_accuracy 99.86237623762376 type value max_ap 96.72586477206649 type value max_f1 93.01858362631845 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering MTEB StackExchangeClustering default test 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 type value v_measure 66.39930057069995 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering-p2p MTEB StackExchangeClusteringP2P default test 815ca46b2622cec33ccafc3735d572c266efdb44 type value v_measure 34.96398659903402 task dataset metrics type Reranking type name config split revision mteb/stackoverflowdupquestions-reranking MTEB StackOverflowDupQuestions default test e185fbe320c72810689fc5848eb6114e1ef5ec69 type value map 55.946944700355395 type value mrr 56.97151398438164 task dataset metrics type Summarization type name config split revision mteb/summeval MTEB SummEval default test cda12ad7615edc362dbf25a00fdd61d3b1eaf93c type value cos_sim_pearson 31.541657650692905 type value cos_sim_spearman 31.605804192286303 type value dot_pearson 28.26905996736398 type value dot_spearman 27.864801765851187 task dataset metrics type Retrieval type name config split revision trec-covid MTEB TRECCOVID default test None type value map_at_1 0.22599999999999998 type value map_at_10 1.8870000000000002 type value map_at_100 9.78 type value map_at_1000 22.514 type value map_at_3 0.6669999999999999 type value map_at_5 1.077 type value mrr_at_1 82.0 type value mrr_at_10 89.86699999999999 type value mrr_at_100 89.86699999999999 type value mrr_at_1000 89.86699999999999 type value mrr_at_3 89.667 type value mrr_at_5 89.667 type value ndcg_at_1 79.0 type value ndcg_at_10 74.818 type value ndcg_at_100 53.715999999999994 type value ndcg_at_1000 47.082 type value ndcg_at_3 82.134 type value ndcg_at_5 79.81899999999999 type value precision_at_1 82.0 type value precision_at_10 78.0 type value precision_at_100 54.48 type value precision_at_1000 20.518 type value precision_at_3 87.333 type value precision_at_5 85.2 type value recall_at_1 0.22599999999999998 type value recall_at_10 2.072 type value recall_at_100 13.013 type value recall_at_1000 43.462 type value recall_at_3 0.695 type value recall_at_5 1.139 task dataset metrics type Retrieval type name config split revision webis-touche2020 MTEB Touche2020 default test None type value map_at_1 2.328 type value map_at_10 9.795 type value map_at_100 15.801000000000002 type value map_at_1000 17.23 type value map_at_3 4.734 type value map_at_5 6.644 type value mrr_at_1 30.612000000000002 type value mrr_at_10 46.902 type value mrr_at_100 47.495 type value mrr_at_1000 47.495 type value mrr_at_3 41.156 type value mrr_at_5 44.218 type value ndcg_at_1 28.571 type value ndcg_at_10 24.806 type value ndcg_at_100 36.419000000000004 type value ndcg_at_1000 47.272999999999996 type value ndcg_at_3 25.666 type value ndcg_at_5 25.448999999999998 type value precision_at_1 30.612000000000002 type value precision_at_10 23.061 type value precision_at_100 7.714 type value precision_at_1000 1.484 type value precision_at_3 26.531 type value precision_at_5 26.122 type value recall_at_1 2.328 type value recall_at_10 16.524 type value recall_at_100 47.179 type value recall_at_1000 81.22200000000001 type value recall_at_3 5.745 type value recall_at_5 9.339 task dataset metrics type Classification type name config split revision mteb/toxic_conversations_50k MTEB ToxicConversationsClassification default test d7c0de2777da35d6aae2200a62c6e0e5af397c4c type value accuracy 70.9142 type value ap 14.335574772555415 type value f1 54.62839595194111 task dataset metrics type Classification type name config split revision mteb/tweet_sentiment_extraction MTEB TweetSentimentExtractionClassification default test d604517c81ca91fe16a244d1248fc021f9ecee7a type value accuracy 59.94340690435768 type value f1 60.286487936731916 task dataset metrics type Clustering type name config split revision mteb/twentynewsgroups-clustering MTEB TwentyNewsgroupsClustering default test 6125ec4e24fa026cec8a478383ee943acfbd5449 type value v_measure 51.26597708987974 task dataset metrics type PairClassification type name config split revision mteb/twittersemeval2015-pairclassification MTEB TwitterSemEval2015 default test 70970daeab8776df92f5ea462b6173c0b46fd2d1 type value cos_sim_accuracy 87.48882398521786 type value cos_sim_ap 79.04326607602204 type value cos_sim_f1 71.64566826860633 type value cos_sim_precision 70.55512918905092 type value cos_sim_recall 72.77044854881267 type value dot_accuracy 84.19264469213805 type value dot_ap 67.96360043562528 type value dot_f1 64.06418393006827 type value dot_precision 58.64941898706424 type value dot_recall 70.58047493403694 type value euclidean_accuracy 87.45902127913214 type value euclidean_ap 78.9742237648272 type value euclidean_f1 71.5553235908142 type value euclidean_precision 70.77955601445535 type value euclidean_recall 72.34828496042216 type value manhattan_accuracy 87.41729749061214 type value manhattan_ap 78.90073137580596 type value manhattan_f1 71.3942611553533 type value manhattan_precision 68.52705653967483 type value manhattan_recall 74.51187335092348 type value max_accuracy 87.48882398521786 type value max_ap 79.04326607602204 type value max_f1 71.64566826860633 task dataset metrics type PairClassification type name config split revision mteb/twitterurlcorpus-pairclassification MTEB TwitterURLCorpus default test 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf type value cos_sim_accuracy 88.68125897465751 type value cos_sim_ap 85.6003454431979 type value cos_sim_f1 77.6957163958641 type value cos_sim_precision 73.0110366307807 type value cos_sim_recall 83.02279026793964 type value dot_accuracy 87.7672992587418 type value dot_ap 82.4971301112899 type value dot_f1 75.90528233151184 type value dot_precision 72.0370626469368 type value dot_recall 80.21250384970742 type value euclidean_accuracy 88.4503434625684 type value euclidean_ap 84.91949884748384 type value euclidean_f1 76.92365018444684 type value euclidean_precision 74.53245721712759 type value euclidean_recall 79.47336002463813 type value manhattan_accuracy 88.47556952691427 type value manhattan_ap 84.8963689101517 type value manhattan_f1 76.85901249256395 type value manhattan_precision 74.31693989071039 type value manhattan_recall 79.58115183246073 type value max_accuracy 88.68125897465751 type value max_ap 85.6003454431979 type value max_f1 77.6957163958641	mit	en

FlagEmbedding

Model List | FAQ | Usage | Evaluation | Train | Contact | Citation | License

For more details please refer to our Github: FlagEmbedding.

If you are looking for a model that supports more languages, longer texts, and other retrieval methods, you can try using bge-m3.

English | 中文

FlagEmbedding focuses on retrieval-augmented LLMs, consisting of the following projects currently:

• Long-Context LLM: Activation Beacon
• Fine-tuning of LM : LM-Cocktail
• Dense Retrieval: BGE-M3, LLM Embedder, BGE Embedding
• Reranker Model: BGE Reranker
• Benchmark: C-MTEB

News

• 1/30/2024: Release BGE-M3, a new member to BGE model series! M3 stands for Multi-linguality (100+ languages), Multi-granularities (input length up to 8192), Multi-Functionality (unification of dense, lexical, multi-vec/colbert retrieval). It is the first embedding model that supports all three retrieval methods, achieving new SOTA on multi-lingual (MIRACL) and cross-lingual (MKQA) benchmarks. Technical Report and Code. 🔥
• 1/9/2024: Release Activation-Beacon, an effective, efficient, compatible, and low-cost (training) method to extend the context length of LLM. Technical Report 🔥
• 12/24/2023: Release LLaRA, a LLaMA-7B based dense retriever, leading to state-of-the-art performances on MS MARCO and BEIR. Model and code will be open-sourced. Please stay tuned. Technical Report 🔥
• 11/23/2023: Release LM-Cocktail, a method to maintain general capabilities during fine-tuning by merging multiple language models. Technical Report 🔥
• 10/12/2023: Release LLM-Embedder, a unified embedding model to support diverse retrieval augmentation needs for LLMs. Technical Report
• 09/15/2023: The technical report and massive training data of BGE has been released
• 09/12/2023: New models:
  • New reranker model: release cross-encoder models BAAI/bge-reranker-base and BAAI/bge-reranker-large, which are more powerful than embedding model. We recommend to use/fine-tune them to re-rank top-k documents returned by embedding models.
  • update embedding model: release bge-*-v1.5 embedding model to alleviate the issue of the similarity distribution, and enhance its retrieval ability without instruction.

More

• 09/07/2023: Update fine-tune code: Add script to mine hard negatives and support adding instruction during fine-tuning.
• 08/09/2023: BGE Models are integrated into Langchain, you can use it like this; C-MTEB leaderboard is available.
• 08/05/2023: Release base-scale and small-scale models, best performance among the models of the same size 🤗
• 08/02/2023: Release bge-large-*(short for BAAI General Embedding) Models, rank 1st on MTEB and C-MTEB benchmark! 🎉 🎉
• 08/01/2023: We release the Chinese Massive Text Embedding Benchmark (C-MTEB), consisting of 31 test dataset.

Model List

bge is short for BAAI general embedding.

Model	Language		Description	query instruction for retrieval [1]
BAAI/bge-m3	Multilingual	Inference Fine-tune	Multi-Functionality(dense retrieval, sparse retrieval, multi-vector(colbert)), Multi-Linguality, and Multi-Granularity(8192 tokens)
BAAI/llm-embedder	English	Inference Fine-tune	a unified embedding model to support diverse retrieval augmentation needs for LLMs	See README
BAAI/bge-reranker-large	Chinese and English	Inference Fine-tune	a cross-encoder model which is more accurate but less efficient [2]
BAAI/bge-reranker-base	Chinese and English	Inference Fine-tune	a cross-encoder model which is more accurate but less efficient [2]
BAAI/bge-large-en-v1.5	English	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	Represent this sentence for searching relevant passages:
BAAI/bge-base-en-v1.5	English	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	Represent this sentence for searching relevant passages:
BAAI/bge-small-en-v1.5	English	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	Represent this sentence for searching relevant passages:
BAAI/bge-large-zh-v1.5	Chinese	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	为这个句子生成表示以用于检索相关文章：
BAAI/bge-base-zh-v1.5	Chinese	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	为这个句子生成表示以用于检索相关文章：
BAAI/bge-small-zh-v1.5	Chinese	Inference Fine-tune	version 1.5 with more reasonable similarity distribution	为这个句子生成表示以用于检索相关文章：
BAAI/bge-large-en	English	Inference Fine-tune	🏆 rank 1st in MTEB leaderboard	Represent this sentence for searching relevant passages:
BAAI/bge-base-en	English	Inference Fine-tune	a base-scale model but with similar ability to bge-large-en	Represent this sentence for searching relevant passages:
BAAI/bge-small-en	English	Inference Fine-tune	a small-scale model but with competitive performance	Represent this sentence for searching relevant passages:
BAAI/bge-large-zh	Chinese	Inference Fine-tune	🏆 rank 1st in C-MTEB benchmark	为这个句子生成表示以用于检索相关文章：
BAAI/bge-base-zh	Chinese	Inference Fine-tune	a base-scale model but with similar ability to bge-large-zh	为这个句子生成表示以用于检索相关文章：
BAAI/bge-small-zh	Chinese	Inference Fine-tune	a small-scale model but with competitive performance	为这个句子生成表示以用于检索相关文章：

[1]: If you need to search the relevant passages to a query, we suggest to add the instruction to the query; in other cases, no instruction is needed, just use the original query directly. In all cases, no instruction needs to be added to passages.

[2]: Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding. To balance the accuracy and time cost, cross-encoder is widely used to re-rank top-k documents retrieved by other simple models. For examples, use bge embedding model to retrieve top 100 relevant documents, and then use bge reranker to re-rank the top 100 document to get the final top-3 results.

All models have been uploaded to Huggingface Hub, and you can see them at https://huggingface.co/BAAI. If you cannot open the Huggingface Hub, you also can download the models at https://model.baai.ac.cn/models .

Frequently asked questions

1. How to fine-tune bge embedding model?

Following this example to prepare data and fine-tune your model. Some suggestions:

• Mine hard negatives following this example, which can improve the retrieval performance.
• If you pre-train bge on your data, the pre-trained model cannot be directly used to calculate similarity, and it must be fine-tuned with contrastive learning before computing similarity.
• If the accuracy of the fine-tuned model is still not high, it is recommended to use/fine-tune the cross-encoder model (bge-reranker) to re-rank top-k results. Hard negatives also are needed to fine-tune reranker.

2. The similarity score between two dissimilar sentences is higher than 0.5

Suggest to use bge v1.5, which alleviates the issue of the similarity distribution.

Since we finetune the models by contrastive learning with a temperature of 0.01, the similarity distribution of the current BGE model is about in the interval [0.6, 1]. So a similarity score greater than 0.5 does not indicate that the two sentences are similar.

For downstream tasks, such as passage retrieval or semantic similarity, what matters is the relative order of the scores, not the absolute value. If you need to filter similar sentences based on a similarity threshold, please select an appropriate similarity threshold based on the similarity distribution on your data (such as 0.8, 0.85, or even 0.9).

3. When does the query instruction need to be used

For the bge-*-v1.5, we improve its retrieval ability when not using instruction. No instruction only has a slight degradation in retrieval performance compared with using instruction. So you can generate embedding without instruction in all cases for convenience.

For a retrieval task that uses short queries to find long related documents, it is recommended to add instructions for these short queries. The best method to decide whether to add instructions for queries is choosing the setting that achieves better performance on your task. In all cases, the documents/passages do not need to add the instruction.

Usage

Usage for Embedding Model

Here are some examples for using bge models with FlagEmbedding, Sentence-Transformers, Langchain, or Huggingface Transformers.

Using FlagEmbedding

pip install -U FlagEmbedding

If it doesn't work for you, you can see FlagEmbedding for more methods to install FlagEmbedding.

from FlagEmbedding import FlagModel
sentences_1 = ["样例数据-1", "样例数据-2"]
sentences_2 = ["样例数据-3", "样例数据-4"]
model = FlagModel('BAAI/bge-large-zh-v1.5', 
                  query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章：",
                  use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
embeddings_1 = model.encode(sentences_1)
embeddings_2 = model.encode(sentences_2)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)

# for s2p(short query to long passage) retrieval task, suggest to use encode_queries() which will automatically add the instruction to each query
# corpus in retrieval task can still use encode() or encode_corpus(), since they don't need instruction
queries = ['query_1', 'query_2']
passages = ["样例文档-1", "样例文档-2"]
q_embeddings = model.encode_queries(queries)
p_embeddings = model.encode(passages)
scores = q_embeddings @ p_embeddings.T

For the value of the argument query_instruction_for_retrieval, see Model List.

By default, FlagModel will use all available GPUs when encoding. Please set os.environ["CUDA_VISIBLE_DEVICES"] to select specific GPUs. You also can set os.environ["CUDA_VISIBLE_DEVICES"]="" to make all GPUs unavailable.

Using Sentence-Transformers

You can also use the bge models with sentence-transformers:

pip install -U sentence-transformers

from sentence_transformers import SentenceTransformer
sentences_1 = ["样例数据-1", "样例数据-2"]
sentences_2 = ["样例数据-3", "样例数据-4"]
model = SentenceTransformer('BAAI/bge-large-zh-v1.5')
embeddings_1 = model.encode(sentences_1, normalize_embeddings=True)
embeddings_2 = model.encode(sentences_2, normalize_embeddings=True)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)

For s2p(short query to long passage) retrieval task, each short query should start with an instruction (instructions see Model List). But the instruction is not needed for passages.

from sentence_transformers import SentenceTransformer
queries = ['query_1', 'query_2']
passages = ["样例文档-1", "样例文档-2"]
instruction = "为这个句子生成表示以用于检索相关文章："

model = SentenceTransformer('BAAI/bge-large-zh-v1.5')
q_embeddings = model.encode([instruction+q for q in queries], normalize_embeddings=True)
p_embeddings = model.encode(passages, normalize_embeddings=True)
scores = q_embeddings @ p_embeddings.T

Using Langchain

You can use bge in langchain like this:

from langchain.embeddings import HuggingFaceBgeEmbeddings
model_name = "BAAI/bge-large-en-v1.5"
model_kwargs = {'device': 'cuda'}
encode_kwargs = {'normalize_embeddings': True} # set True to compute cosine similarity
model = HuggingFaceBgeEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs,
    query_instruction="为这个句子生成表示以用于检索相关文章："
)
model.query_instruction = "为这个句子生成表示以用于检索相关文章："

Using HuggingFace Transformers

With the transformers package, you can use the model like this: First, you pass your input through the transformer model, then you select the last hidden state of the first token (i.e., [CLS]) as the sentence embedding.

from transformers import AutoTokenizer, AutoModel
import torch
# Sentences we want sentence embeddings for
sentences = ["样例数据-1", "样例数据-2"]

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-large-zh-v1.5')
model = AutoModel.from_pretrained('BAAI/bge-large-zh-v1.5')
model.eval()

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
# for s2p(short query to long passage) retrieval task, add an instruction to query (not add instruction for passages)
# encoded_input = tokenizer([instruction + q for q in queries], padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)
    # Perform pooling. In this case, cls pooling.
    sentence_embeddings = model_output[0][:, 0]
# normalize embeddings
sentence_embeddings = torch.nn.functional.normalize(sentence_embeddings, p=2, dim=1)
print("Sentence embeddings:", sentence_embeddings)

Usage of the ONNX files

from optimum.onnxruntime import ORTModelForFeatureExtraction  # type: ignore

import torch
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-large-en-v1.5')
model = AutoModel.from_pretrained('BAAI/bge-large-en-v1.5', revision="refs/pr/13")
model_ort = ORTModelForFeatureExtraction.from_pretrained('BAAI/bge-large-en-v1.5', revision="refs/pr/13",file_name="onnx/model.onnx")

# Sentences we want sentence embeddings for
sentences = ["样例数据-1", "样例数据-2"]

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
# for s2p(short query to long passage) retrieval task, add an instruction to query (not add instruction for passages)
# encoded_input = tokenizer([instruction + q for q in queries], padding=True, truncation=True, return_tensors='pt')

model_output_ort = model_ort(**encoded_input)
# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# model_output and model_output_ort are identical

Its also possible to deploy the onnx files with the infinity_emb pip package.

import asyncio
from infinity_emb import AsyncEmbeddingEngine, EngineArgs

sentences = ["Embed this is sentence via Infinity.", "Paris is in France."]
engine = AsyncEmbeddingEngine.from_args(
    EngineArgs(model_name_or_path = "BAAI/bge-large-en-v1.5", device="cpu", engine="optimum" # or engine="torch"
))

async def main(): 
    async with engine:
        embeddings, usage = await engine.embed(sentences=sentences)
asyncio.run(main())

Usage for Reranker

Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding. You can get a relevance score by inputting query and passage to the reranker. The reranker is optimized based cross-entropy loss, so the relevance score is not bounded to a specific range.

Using FlagEmbedding

pip install -U FlagEmbedding

Get relevance scores (higher scores indicate more relevance):

from FlagEmbedding import FlagReranker
reranker = FlagReranker('BAAI/bge-reranker-large', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation

score = reranker.compute_score(['query', 'passage'])
print(score)

scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']])
print(scores)

Using Huggingface transformers

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-large')
model = AutoModelForSequenceClassification.from_pretrained('BAAI/bge-reranker-large')
model.eval()

pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
with torch.no_grad():
    inputs = tokenizer(pairs, padding=True, truncation=True, return_tensors='pt', max_length=512)
    scores = model(**inputs, return_dict=True).logits.view(-1, ).float()
    print(scores)

Evaluation

baai-general-embedding models achieve state-of-the-art performance on both MTEB and C-MTEB leaderboard! For more details and evaluation tools see our scripts.

• MTEB:

Model Name	Dimension	Sequence Length	Average (56)	Retrieval (15)	Clustering (11)	Pair Classification (3)	Reranking (4)	STS (10)	Summarization (1)	Classification (12)
BAAI/bge-large-en-v1.5	1024	512	64.23	54.29	46.08	87.12	60.03	83.11	31.61	75.97
BAAI/bge-base-en-v1.5	768	512	63.55	53.25	45.77	86.55	58.86	82.4	31.07	75.53
BAAI/bge-small-en-v1.5	384	512	62.17	51.68	43.82	84.92	58.36	81.59	30.12	74.14
bge-large-en	1024	512	63.98	53.9	46.98	85.8	59.48	81.56	32.06	76.21
bge-base-en	768	512	63.36	53.0	46.32	85.86	58.7	81.84	29.27	75.27
gte-large	1024	512	63.13	52.22	46.84	85.00	59.13	83.35	31.66	73.33
gte-base	768	512	62.39	51.14	46.2	84.57	58.61	82.3	31.17	73.01
e5-large-v2	1024	512	62.25	50.56	44.49	86.03	56.61	82.05	30.19	75.24
bge-small-en	384	512	62.11	51.82	44.31	83.78	57.97	80.72	30.53	74.37
instructor-xl	768	512	61.79	49.26	44.74	86.62	57.29	83.06	32.32	61.79
e5-base-v2	768	512	61.5	50.29	43.80	85.73	55.91	81.05	30.28	73.84
gte-small	384	512	61.36	49.46	44.89	83.54	57.7	82.07	30.42	72.31
text-embedding-ada-002	1536	8192	60.99	49.25	45.9	84.89	56.32	80.97	30.8	70.93
e5-small-v2	384	512	59.93	49.04	39.92	84.67	54.32	80.39	31.16	72.94
sentence-t5-xxl	768	512	59.51	42.24	43.72	85.06	56.42	82.63	30.08	73.42
all-mpnet-base-v2	768	514	57.78	43.81	43.69	83.04	59.36	80.28	27.49	65.07
sgpt-bloom-7b1-msmarco	4096	2048	57.59	48.22	38.93	81.9	55.65	77.74	33.6	66.19

• C-MTEB:
  We create the benchmark C-MTEB for Chinese text embedding which consists of 31 datasets from 6 tasks. Please refer to C_MTEB for a detailed introduction.

Model	Embedding dimension	Avg	Retrieval	STS	PairClassification	Classification	Reranking	Clustering
BAAI/bge-large-zh-v1.5	1024	64.53	70.46	56.25	81.6	69.13	65.84	48.99
BAAI/bge-base-zh-v1.5	768	63.13	69.49	53.72	79.75	68.07	65.39	47.53
BAAI/bge-small-zh-v1.5	512	57.82	61.77	49.11	70.41	63.96	60.92	44.18
BAAI/bge-large-zh	1024	64.20	71.53	54.98	78.94	68.32	65.11	48.39
bge-large-zh-noinstruct	1024	63.53	70.55	53	76.77	68.58	64.91	50.01
BAAI/bge-base-zh	768	62.96	69.53	54.12	77.5	67.07	64.91	47.63
multilingual-e5-large	1024	58.79	63.66	48.44	69.89	67.34	56.00	48.23
BAAI/bge-small-zh	512	58.27	63.07	49.45	70.35	63.64	61.48	45.09
m3e-base	768	57.10	56.91	50.47	63.99	67.52	59.34	47.68
m3e-large	1024	57.05	54.75	50.42	64.3	68.2	59.66	48.88
multilingual-e5-base	768	55.48	61.63	46.49	67.07	65.35	54.35	40.68
multilingual-e5-small	384	55.38	59.95	45.27	66.45	65.85	53.86	45.26
text-embedding-ada-002(OpenAI)	1536	53.02	52.0	43.35	69.56	64.31	54.28	45.68
luotuo	1024	49.37	44.4	42.78	66.62	61	49.25	44.39
text2vec-base	768	47.63	38.79	43.41	67.41	62.19	49.45	37.66
text2vec-large	1024	47.36	41.94	44.97	70.86	60.66	49.16	30.02

• Reranking: See C_MTEB for evaluation script.

Model	T2Reranking	T2RerankingZh2En*	T2RerankingEn2Zh*	MMarcoReranking	CMedQAv1	CMedQAv2	Avg
text2vec-base-multilingual	64.66	62.94	62.51	14.37	48.46	48.6	50.26
multilingual-e5-small	65.62	60.94	56.41	29.91	67.26	66.54	57.78
multilingual-e5-large	64.55	61.61	54.28	28.6	67.42	67.92	57.4
multilingual-e5-base	64.21	62.13	54.68	29.5	66.23	66.98	57.29
m3e-base	66.03	62.74	56.07	17.51	77.05	76.76	59.36
m3e-large	66.13	62.72	56.1	16.46	77.76	78.27	59.57
bge-base-zh-v1.5	66.49	63.25	57.02	29.74	80.47	84.88	63.64
bge-large-zh-v1.5	65.74	63.39	57.03	28.74	83.45	85.44	63.97
BAAI/bge-reranker-base	67.28	63.95	60.45	35.46	81.26	84.1	65.42
BAAI/bge-reranker-large	67.6	64.03	61.44	37.16	82.15	84.18	66.09

* : T2RerankingZh2En and T2RerankingEn2Zh are cross-language retrieval tasks

Train

BAAI Embedding

We pre-train the models using retromae and train them on large-scale pairs data using contrastive learning. You can fine-tune the embedding model on your data following our examples. We also provide a pre-train example. Note that the goal of pre-training is to reconstruct the text, and the pre-trained model cannot be used for similarity calculation directly, it needs to be fine-tuned. More training details for bge see baai_general_embedding.

BGE Reranker

Cross-encoder will perform full-attention over the input pair, which is more accurate than embedding model (i.e., bi-encoder) but more time-consuming than embedding model. Therefore, it can be used to re-rank the top-k documents returned by embedding model. We train the cross-encoder on a multilingual pair data, The data format is the same as embedding model, so you can fine-tune it easily following our example. More details please refer to ./FlagEmbedding/reranker/README.md

Contact

If you have any question or suggestion related to this project, feel free to open an issue or pull request. You also can email Shitao Xiao(stxiao@baai.ac.cn) and Zheng Liu(liuzheng@baai.ac.cn).

Citation

If you find this repository useful, please consider giving a star ⭐ and citation

@misc{bge_embedding,
      title={C-Pack: Packaged Resources To Advance General Chinese Embedding}, 
      author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff},
      year={2023},
      eprint={2309.07597},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

License

FlagEmbedding is licensed under the MIT License. The released models can be used for commercial purposes free of charge.