GritLM-7B/README.md

pipeline_tag, inference, license, datasets, tags, model-index, base_model

pipeline_tag	inference	license	datasets	tags	model-index	base_model
text-generation	true	apache-2.0	GritLM/tulu2	mteb	name results GritLM-7B task dataset metrics type Classification type name config split revision mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en) en test e8379541af4e31359cca9fbcf4b00f2671dba205 type value accuracy 81.17910447761194 type value ap 46.26260671758199 type value f1 75.44565719934167 task dataset metrics type Classification type name config split revision mteb/amazon_polarity MTEB AmazonPolarityClassification default test e2d317d38cd51312af73b3d32a06d1a08b442046 type value accuracy 96.5161 type value ap 94.79131981460425 type value f1 96.51506148413065 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (en) en test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 57.806000000000004 type value f1 56.78350156257903 task dataset metrics type Retrieval type name config split revision arguana MTEB ArguAna default test None type value map_at_1 38.478 type value map_at_10 54.955 type value map_at_100 54.955 type value map_at_1000 54.955 type value map_at_3 50.888999999999996 type value map_at_5 53.349999999999994 type value mrr_at_1 39.757999999999996 type value mrr_at_10 55.449000000000005 type value mrr_at_100 55.449000000000005 type value mrr_at_1000 55.449000000000005 type value mrr_at_3 51.37500000000001 type value mrr_at_5 53.822 type value ndcg_at_1 38.478 type value ndcg_at_10 63.239999999999995 type value ndcg_at_100 63.239999999999995 type value ndcg_at_1000 63.239999999999995 type value ndcg_at_3 54.935 type value ndcg_at_5 59.379000000000005 type value precision_at_1 38.478 type value precision_at_10 8.933 type value precision_at_100 0.893 type value precision_at_1000 0.089 type value precision_at_3 22.214 type value precision_at_5 15.491 type value recall_at_1 38.478 type value recall_at_10 89.331 type value recall_at_100 89.331 type value recall_at_1000 89.331 type value recall_at_3 66.643 type value recall_at_5 77.45400000000001 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-p2p MTEB ArxivClusteringP2P default test a122ad7f3f0291bf49cc6f4d32aa80929df69d5d type value v_measure 51.67144081472449 task dataset metrics type Clustering type name config split revision mteb/arxiv-clustering-s2s MTEB ArxivClusteringS2S default test f910caf1a6075f7329cdf8c1a6135696f37dbd53 type value v_measure 48.11256154264126 task dataset metrics type Reranking type name config split revision mteb/askubuntudupquestions-reranking MTEB AskUbuntuDupQuestions default test 2000358ca161889fa9c082cb41daa8dcfb161a54 type value map 67.33801955487878 type value mrr 80.71549487754474 task dataset metrics type STS type name config split revision mteb/biosses-sts MTEB BIOSSES default test d3fb88f8f02e40887cd149695127462bbcf29b4a type value cos_sim_pearson 88.1935203751726 type value cos_sim_spearman 86.35497970498659 type value euclidean_pearson 85.46910708503744 type value euclidean_spearman 85.13928935405485 type value manhattan_pearson 85.68373836333303 type value manhattan_spearman 85.40013867117746 task dataset metrics type Classification type name config split revision mteb/banking77 MTEB Banking77Classification default test 0fd18e25b25c072e09e0d92ab615fda904d66300 type value accuracy 88.46753246753248 type value f1 88.43006344981134 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-p2p MTEB BiorxivClusteringP2P default test 65b79d1d13f80053f67aca9498d9402c2d9f1f40 type value v_measure 40.86793640310432 task dataset metrics type Clustering type name config split revision mteb/biorxiv-clustering-s2s MTEB BiorxivClusteringS2S default test 258694dd0231531bc1fd9de6ceb52a0853c6d908 type value v_measure 39.80291334130727 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackAndroidRetrieval default test None type value map_at_1 38.421 type value map_at_10 52.349000000000004 type value map_at_100 52.349000000000004 type value map_at_1000 52.349000000000004 type value map_at_3 48.17 type value map_at_5 50.432 type value mrr_at_1 47.353 type value mrr_at_10 58.387 type value mrr_at_100 58.387 type value mrr_at_1000 58.387 type value mrr_at_3 56.199 type value mrr_at_5 57.487 type value ndcg_at_1 47.353 type value ndcg_at_10 59.202 type value ndcg_at_100 58.848 type value ndcg_at_1000 58.831999999999994 type value ndcg_at_3 54.112 type value ndcg_at_5 56.312 type value precision_at_1 47.353 type value precision_at_10 11.459 type value precision_at_100 1.146 type value precision_at_1000 0.11499999999999999 type value precision_at_3 26.133 type value precision_at_5 18.627 type value recall_at_1 38.421 type value recall_at_10 71.89 type value recall_at_100 71.89 type value recall_at_1000 71.89 type value recall_at_3 56.58 type value recall_at_5 63.125 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackEnglishRetrieval default test None type value map_at_1 38.025999999999996 type value map_at_10 50.590999999999994 type value map_at_100 51.99700000000001 type value map_at_1000 52.11599999999999 type value map_at_3 47.435 type value map_at_5 49.236000000000004 type value mrr_at_1 48.28 type value mrr_at_10 56.814 type value mrr_at_100 57.446 type value mrr_at_1000 57.476000000000006 type value mrr_at_3 54.958 type value mrr_at_5 56.084999999999994 type value ndcg_at_1 48.28 type value ndcg_at_10 56.442 type value ndcg_at_100 60.651999999999994 type value ndcg_at_1000 62.187000000000005 type value ndcg_at_3 52.866 type value ndcg_at_5 54.515 type value precision_at_1 48.28 type value precision_at_10 10.586 type value precision_at_100 1.6310000000000002 type value precision_at_1000 0.20600000000000002 type value precision_at_3 25.945 type value precision_at_5 18.076 type value recall_at_1 38.025999999999996 type value recall_at_10 66.11399999999999 type value recall_at_100 83.339 type value recall_at_1000 92.413 type value recall_at_3 54.493 type value recall_at_5 59.64699999999999 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGamingRetrieval default test None type value map_at_1 47.905 type value map_at_10 61.58 type value map_at_100 62.605 type value map_at_1000 62.637 type value map_at_3 58.074000000000005 type value map_at_5 60.260000000000005 type value mrr_at_1 54.42 type value mrr_at_10 64.847 type value mrr_at_100 65.403 type value mrr_at_1000 65.41900000000001 type value mrr_at_3 62.675000000000004 type value mrr_at_5 64.101 type value ndcg_at_1 54.42 type value ndcg_at_10 67.394 type value ndcg_at_100 70.846 type value ndcg_at_1000 71.403 type value ndcg_at_3 62.025 type value ndcg_at_5 65.032 type value precision_at_1 54.42 type value precision_at_10 10.646 type value precision_at_100 1.325 type value precision_at_1000 0.13999999999999999 type value precision_at_3 27.398 type value precision_at_5 18.796 type value recall_at_1 47.905 type value recall_at_10 80.84599999999999 type value recall_at_100 95.078 type value recall_at_1000 98.878 type value recall_at_3 67.05600000000001 type value recall_at_5 74.261 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackGisRetrieval default test None type value map_at_1 30.745 type value map_at_10 41.021 type value map_at_100 41.021 type value map_at_1000 41.021 type value map_at_3 37.714999999999996 type value map_at_5 39.766 type value mrr_at_1 33.559 type value mrr_at_10 43.537 type value mrr_at_100 43.537 type value mrr_at_1000 43.537 type value mrr_at_3 40.546 type value mrr_at_5 42.439 type value ndcg_at_1 33.559 type value ndcg_at_10 46.781 type value ndcg_at_100 46.781 type value ndcg_at_1000 46.781 type value ndcg_at_3 40.516000000000005 type value ndcg_at_5 43.957 type value precision_at_1 33.559 type value precision_at_10 7.198 type value precision_at_100 0.72 type value precision_at_1000 0.07200000000000001 type value precision_at_3 17.1 type value precision_at_5 12.316 type value recall_at_1 30.745 type value recall_at_10 62.038000000000004 type value recall_at_100 62.038000000000004 type value recall_at_1000 62.038000000000004 type value recall_at_3 45.378 type value recall_at_5 53.580000000000005 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackMathematicaRetrieval default test None type value map_at_1 19.637999999999998 type value map_at_10 31.05 type value map_at_100 31.05 type value map_at_1000 31.05 type value map_at_3 27.628000000000004 type value map_at_5 29.767 type value mrr_at_1 25 type value mrr_at_10 36.131 type value mrr_at_100 36.131 type value mrr_at_1000 36.131 type value mrr_at_3 33.333 type value mrr_at_5 35.143 type value ndcg_at_1 25 type value ndcg_at_10 37.478 type value ndcg_at_100 37.469 type value ndcg_at_1000 37.469 type value ndcg_at_3 31.757999999999996 type value ndcg_at_5 34.821999999999996 type value precision_at_1 25 type value precision_at_10 7.188999999999999 type value precision_at_100 0.719 type value precision_at_1000 0.07200000000000001 type value precision_at_3 15.837000000000002 type value precision_at_5 11.841 type value recall_at_1 19.637999999999998 type value recall_at_10 51.836000000000006 type value recall_at_100 51.836000000000006 type value recall_at_1000 51.836000000000006 type value recall_at_3 36.384 type value recall_at_5 43.964 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackPhysicsRetrieval default test None type value map_at_1 34.884 type value map_at_10 47.88 type value map_at_100 47.88 type value map_at_1000 47.88 type value map_at_3 43.85 type value map_at_5 46.414 type value mrr_at_1 43.022 type value mrr_at_10 53.569 type value mrr_at_100 53.569 type value mrr_at_1000 53.569 type value mrr_at_3 51.075 type value mrr_at_5 52.725 type value ndcg_at_1 43.022 type value ndcg_at_10 54.461000000000006 type value ndcg_at_100 54.388000000000005 type value ndcg_at_1000 54.388000000000005 type value ndcg_at_3 48.864999999999995 type value ndcg_at_5 52.032000000000004 type value precision_at_1 43.022 type value precision_at_10 9.885 type value precision_at_100 0.988 type value precision_at_1000 0.099 type value precision_at_3 23.612 type value precision_at_5 16.997 type value recall_at_1 34.884 type value recall_at_10 68.12899999999999 type value recall_at_100 68.12899999999999 type value recall_at_1000 68.12899999999999 type value recall_at_3 52.428 type value recall_at_5 60.662000000000006 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackProgrammersRetrieval default test None type value map_at_1 31.588 type value map_at_10 43.85 type value map_at_100 45.317 type value map_at_1000 45.408 type value map_at_3 39.73 type value map_at_5 42.122 type value mrr_at_1 38.927 type value mrr_at_10 49.582 type value mrr_at_100 50.39 type value mrr_at_1000 50.426 type value mrr_at_3 46.518 type value mrr_at_5 48.271 type value ndcg_at_1 38.927 type value ndcg_at_10 50.605999999999995 type value ndcg_at_100 56.22200000000001 type value ndcg_at_1000 57.724 type value ndcg_at_3 44.232 type value ndcg_at_5 47.233999999999995 type value precision_at_1 38.927 type value precision_at_10 9.429 type value precision_at_100 1.435 type value precision_at_1000 0.172 type value precision_at_3 21.271 type value precision_at_5 15.434000000000001 type value recall_at_1 31.588 type value recall_at_10 64.836 type value recall_at_100 88.066 type value recall_at_1000 97.748 type value recall_at_3 47.128 type value recall_at_5 54.954 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackRetrieval default test None type value map_at_1 31.956083333333336 type value map_at_10 43.33483333333333 type value map_at_100 44.64883333333333 type value map_at_1000 44.75 type value map_at_3 39.87741666666666 type value map_at_5 41.86766666666667 type value mrr_at_1 38.06341666666667 type value mrr_at_10 47.839666666666666 type value mrr_at_100 48.644000000000005 type value mrr_at_1000 48.68566666666667 type value mrr_at_3 45.26358333333334 type value mrr_at_5 46.790000000000006 type value ndcg_at_1 38.06341666666667 type value ndcg_at_10 49.419333333333334 type value ndcg_at_100 54.50166666666667 type value ndcg_at_1000 56.161166666666674 type value ndcg_at_3 43.982416666666666 type value ndcg_at_5 46.638083333333334 type value precision_at_1 38.06341666666667 type value precision_at_10 8.70858333333333 type value precision_at_100 1.327 type value precision_at_1000 0.165 type value precision_at_3 20.37816666666667 type value precision_at_5 14.516333333333334 type value recall_at_1 31.956083333333336 type value recall_at_10 62.69458333333334 type value recall_at_100 84.46433333333334 type value recall_at_1000 95.58449999999999 type value recall_at_3 47.52016666666666 type value recall_at_5 54.36066666666666 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackStatsRetrieval default test None type value map_at_1 28.912 type value map_at_10 38.291 type value map_at_100 39.44 type value map_at_1000 39.528 type value map_at_3 35.638 type value map_at_5 37.218 type value mrr_at_1 32.822 type value mrr_at_10 41.661 type value mrr_at_100 42.546 type value mrr_at_1000 42.603 type value mrr_at_3 39.238 type value mrr_at_5 40.726 type value ndcg_at_1 32.822 type value ndcg_at_10 43.373 type value ndcg_at_100 48.638 type value ndcg_at_1000 50.654999999999994 type value ndcg_at_3 38.643 type value ndcg_at_5 41.126000000000005 type value precision_at_1 32.822 type value precision_at_10 6.8709999999999996 type value precision_at_100 1.032 type value precision_at_1000 0.128 type value precision_at_3 16.82 type value precision_at_5 11.718 type value recall_at_1 28.912 type value recall_at_10 55.376999999999995 type value recall_at_100 79.066 type value recall_at_1000 93.664 type value recall_at_3 42.569 type value recall_at_5 48.719 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackTexRetrieval default test None type value map_at_1 22.181 type value map_at_10 31.462 type value map_at_100 32.73 type value map_at_1000 32.848 type value map_at_3 28.57 type value map_at_5 30.182 type value mrr_at_1 27.185 type value mrr_at_10 35.846000000000004 type value mrr_at_100 36.811 type value mrr_at_1000 36.873 type value mrr_at_3 33.437 type value mrr_at_5 34.813 type value ndcg_at_1 27.185 type value ndcg_at_10 36.858000000000004 type value ndcg_at_100 42.501 type value ndcg_at_1000 44.945 type value ndcg_at_3 32.066 type value ndcg_at_5 34.29 type value precision_at_1 27.185 type value precision_at_10 6.752 type value precision_at_100 1.111 type value precision_at_1000 0.151 type value precision_at_3 15.290000000000001 type value precision_at_5 11.004999999999999 type value recall_at_1 22.181 type value recall_at_10 48.513 type value recall_at_100 73.418 type value recall_at_1000 90.306 type value recall_at_3 35.003 type value recall_at_5 40.876000000000005 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackUnixRetrieval default test None type value map_at_1 33.934999999999995 type value map_at_10 44.727 type value map_at_100 44.727 type value map_at_1000 44.727 type value map_at_3 40.918 type value map_at_5 42.961 type value mrr_at_1 39.646 type value mrr_at_10 48.898 type value mrr_at_100 48.898 type value mrr_at_1000 48.898 type value mrr_at_3 45.896 type value mrr_at_5 47.514 type value ndcg_at_1 39.646 type value ndcg_at_10 50.817 type value ndcg_at_100 50.803 type value ndcg_at_1000 50.803 type value ndcg_at_3 44.507999999999996 type value ndcg_at_5 47.259 type value precision_at_1 39.646 type value precision_at_10 8.759 type value precision_at_100 0.876 type value precision_at_1000 0.08800000000000001 type value precision_at_3 20.274 type value precision_at_5 14.366000000000001 type value recall_at_1 33.934999999999995 type value recall_at_10 65.037 type value recall_at_100 65.037 type value recall_at_1000 65.037 type value recall_at_3 47.439 type value recall_at_5 54.567 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWebmastersRetrieval default test None type value map_at_1 32.058 type value map_at_10 43.137 type value map_at_100 43.137 type value map_at_1000 43.137 type value map_at_3 39.882 type value map_at_5 41.379 type value mrr_at_1 38.933 type value mrr_at_10 48.344 type value mrr_at_100 48.344 type value mrr_at_1000 48.344 type value mrr_at_3 45.652 type value mrr_at_5 46.877 type value ndcg_at_1 38.933 type value ndcg_at_10 49.964 type value ndcg_at_100 49.242000000000004 type value ndcg_at_1000 49.222 type value ndcg_at_3 44.605 type value ndcg_at_5 46.501999999999995 type value precision_at_1 38.933 type value precision_at_10 9.427000000000001 type value precision_at_100 0.943 type value precision_at_1000 0.094 type value precision_at_3 20.685000000000002 type value precision_at_5 14.585 type value recall_at_1 32.058 type value recall_at_10 63.074 type value recall_at_100 63.074 type value recall_at_1000 63.074 type value recall_at_3 47.509 type value recall_at_5 52.455 task dataset metrics type Retrieval type name config split revision BeIR/cqadupstack MTEB CQADupstackWordpressRetrieval default test None type value map_at_1 26.029000000000003 type value map_at_10 34.646 type value map_at_100 34.646 type value map_at_1000 34.646 type value map_at_3 31.456 type value map_at_5 33.138 type value mrr_at_1 28.281 type value mrr_at_10 36.905 type value mrr_at_100 36.905 type value mrr_at_1000 36.905 type value mrr_at_3 34.011 type value mrr_at_5 35.638 type value ndcg_at_1 28.281 type value ndcg_at_10 40.159 type value ndcg_at_100 40.159 type value ndcg_at_1000 40.159 type value ndcg_at_3 33.995 type value ndcg_at_5 36.836999999999996 type value precision_at_1 28.281 type value precision_at_10 6.358999999999999 type value precision_at_100 0.636 type value precision_at_1000 0.064 type value precision_at_3 14.233 type value precision_at_5 10.314 type value recall_at_1 26.029000000000003 type value recall_at_10 55.08 type value recall_at_100 55.08 type value recall_at_1000 55.08 type value recall_at_3 38.487 type value recall_at_5 45.308 task dataset metrics type Retrieval type name config split revision climate-fever MTEB ClimateFEVER default test None type value map_at_1 12.842999999999998 type value map_at_10 22.101000000000003 type value map_at_100 24.319 type value map_at_1000 24.51 type value map_at_3 18.372 type value map_at_5 20.323 type value mrr_at_1 27.948 type value mrr_at_10 40.321 type value mrr_at_100 41.262 type value mrr_at_1000 41.297 type value mrr_at_3 36.558 type value mrr_at_5 38.824999999999996 type value ndcg_at_1 27.948 type value ndcg_at_10 30.906 type value ndcg_at_100 38.986 type value ndcg_at_1000 42.136 type value ndcg_at_3 24.911 type value ndcg_at_5 27.168999999999997 type value precision_at_1 27.948 type value precision_at_10 9.798 type value precision_at_100 1.8399999999999999 type value precision_at_1000 0.243 type value precision_at_3 18.328 type value precision_at_5 14.502 type value recall_at_1 12.842999999999998 type value recall_at_10 37.245 type value recall_at_100 64.769 type value recall_at_1000 82.055 type value recall_at_3 23.159 type value recall_at_5 29.113 task dataset metrics type Retrieval type name config split revision dbpedia-entity MTEB DBPedia default test None type value map_at_1 8.934000000000001 type value map_at_10 21.915000000000003 type value map_at_100 21.915000000000003 type value map_at_1000 21.915000000000003 type value map_at_3 14.623 type value map_at_5 17.841 type value mrr_at_1 71.25 type value mrr_at_10 78.994 type value mrr_at_100 78.994 type value mrr_at_1000 78.994 type value mrr_at_3 77.208 type value mrr_at_5 78.55799999999999 type value ndcg_at_1 60.62499999999999 type value ndcg_at_10 46.604 type value ndcg_at_100 35.653 type value ndcg_at_1000 35.531 type value ndcg_at_3 50.605 type value ndcg_at_5 48.730000000000004 type value precision_at_1 71.25 type value precision_at_10 37.75 type value precision_at_100 3.775 type value precision_at_1000 0.377 type value precision_at_3 54.417 type value precision_at_5 48.15 type value recall_at_1 8.934000000000001 type value recall_at_10 28.471000000000004 type value recall_at_100 28.471000000000004 type value recall_at_1000 28.471000000000004 type value recall_at_3 16.019 type value recall_at_5 21.410999999999998 task dataset metrics type Classification type name config split revision mteb/emotion MTEB EmotionClassification default test 4f58c6b202a23cf9a4da393831edf4f9183cad37 type value accuracy 52.81 type value f1 47.987573380720114 task dataset metrics type Retrieval type name config split revision fever MTEB FEVER default test None type value map_at_1 66.81899999999999 type value map_at_10 78.034 type value map_at_100 78.034 type value map_at_1000 78.034 type value map_at_3 76.43100000000001 type value map_at_5 77.515 type value mrr_at_1 71.542 type value mrr_at_10 81.638 type value mrr_at_100 81.638 type value mrr_at_1000 81.638 type value mrr_at_3 80.403 type value mrr_at_5 81.256 type value ndcg_at_1 71.542 type value ndcg_at_10 82.742 type value ndcg_at_100 82.741 type value ndcg_at_1000 82.741 type value ndcg_at_3 80.039 type value ndcg_at_5 81.695 type value precision_at_1 71.542 type value precision_at_10 10.387 type value precision_at_100 1.039 type value precision_at_1000 0.104 type value precision_at_3 31.447999999999997 type value precision_at_5 19.91 type value recall_at_1 66.81899999999999 type value recall_at_10 93.372 type value recall_at_100 93.372 type value recall_at_1000 93.372 type value recall_at_3 86.33 type value recall_at_5 90.347 task dataset metrics type Retrieval type name config split revision fiqa MTEB FiQA2018 default test None type value map_at_1 31.158 type value map_at_10 52.017 type value map_at_100 54.259 type value map_at_1000 54.367 type value map_at_3 45.738 type value map_at_5 49.283 type value mrr_at_1 57.87 type value mrr_at_10 66.215 type value mrr_at_100 66.735 type value mrr_at_1000 66.75 type value mrr_at_3 64.043 type value mrr_at_5 65.116 type value ndcg_at_1 57.87 type value ndcg_at_10 59.946999999999996 type value ndcg_at_100 66.31099999999999 type value ndcg_at_1000 67.75999999999999 type value ndcg_at_3 55.483000000000004 type value ndcg_at_5 56.891000000000005 type value precision_at_1 57.87 type value precision_at_10 16.497 type value precision_at_100 2.321 type value precision_at_1000 0.258 type value precision_at_3 37.14 type value precision_at_5 27.067999999999998 type value recall_at_1 31.158 type value recall_at_10 67.381 type value recall_at_100 89.464 type value recall_at_1000 97.989 type value recall_at_3 50.553000000000004 type value recall_at_5 57.824 task dataset metrics type Retrieval type name config split revision hotpotqa MTEB HotpotQA default test None type value map_at_1 42.073 type value map_at_10 72.418 type value map_at_100 73.175 type value map_at_1000 73.215 type value map_at_3 68.791 type value map_at_5 71.19 type value mrr_at_1 84.146 type value mrr_at_10 88.994 type value mrr_at_100 89.116 type value mrr_at_1000 89.12 type value mrr_at_3 88.373 type value mrr_at_5 88.82 type value ndcg_at_1 84.146 type value ndcg_at_10 79.404 type value ndcg_at_100 81.83200000000001 type value ndcg_at_1000 82.524 type value ndcg_at_3 74.595 type value ndcg_at_5 77.474 type value precision_at_1 84.146 type value precision_at_10 16.753999999999998 type value precision_at_100 1.8599999999999999 type value precision_at_1000 0.19499999999999998 type value precision_at_3 48.854 type value precision_at_5 31.579 type value recall_at_1 42.073 type value recall_at_10 83.768 type value recall_at_100 93.018 type value recall_at_1000 97.481 type value recall_at_3 73.282 type value recall_at_5 78.947 task dataset metrics type Classification type name config split revision mteb/imdb MTEB ImdbClassification default test 3d86128a09e091d6018b6d26cad27f2739fc2db7 type value accuracy 94.9968 type value ap 92.93892195862824 type value f1 94.99327998213761 task dataset metrics type Retrieval type name config split revision msmarco MTEB MSMARCO default dev None type value map_at_1 21.698 type value map_at_10 34.585 type value map_at_100 35.782000000000004 type value map_at_1000 35.825 type value map_at_3 30.397999999999996 type value map_at_5 32.72 type value mrr_at_1 22.192 type value mrr_at_10 35.085 type value mrr_at_100 36.218 type value mrr_at_1000 36.256 type value mrr_at_3 30.986000000000004 type value mrr_at_5 33.268 type value ndcg_at_1 22.192 type value ndcg_at_10 41.957 type value ndcg_at_100 47.658 type value ndcg_at_1000 48.697 type value ndcg_at_3 33.433 type value ndcg_at_5 37.551 type value precision_at_1 22.192 type value precision_at_10 6.781 type value precision_at_100 0.963 type value precision_at_1000 0.105 type value precision_at_3 14.365 type value precision_at_5 10.713000000000001 type value recall_at_1 21.698 type value recall_at_10 64.79 type value recall_at_100 91.071 type value recall_at_1000 98.883 type value recall_at_3 41.611 type value recall_at_5 51.459999999999994 task dataset metrics type Classification type name config split revision mteb/mtop_domain MTEB MTOPDomainClassification (en) en test d80d48c1eb48d3562165c59d59d0034df9fff0bf type value accuracy 96.15823073415413 type value f1 96.00362034963248 task dataset metrics type Classification type name config split revision mteb/mtop_intent MTEB MTOPIntentClassification (en) en test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba type value accuracy 87.12722298221614 type value f1 70.46888967516227 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (en) en test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 80.77673167451245 type value f1 77.60202561132175 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (en) en test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 82.09145931405514 type value f1 81.7701921473406 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-p2p MTEB MedrxivClusteringP2P default test e7a26af6f3ae46b30dde8737f02c07b1505bcc73 type value v_measure 36.52153488185864 task dataset metrics type Clustering type name config split revision mteb/medrxiv-clustering-s2s MTEB MedrxivClusteringS2S default test 35191c8c0dca72d8ff3efcd72aa802307d469663 type value v_measure 36.80090398444147 task dataset metrics type Reranking type name config split revision mteb/mind_small MTEB MindSmallReranking default test 3bdac13927fdc888b903db93b2ffdbd90b295a69 type value map 31.807141746058605 type value mrr 32.85025611455029 task dataset metrics type Retrieval type name config split revision nfcorpus MTEB NFCorpus default test None type value map_at_1 6.920999999999999 type value map_at_10 16.049 type value map_at_100 16.049 type value map_at_1000 16.049 type value map_at_3 11.865 type value map_at_5 13.657 type value mrr_at_1 53.87 type value mrr_at_10 62.291 type value mrr_at_100 62.291 type value mrr_at_1000 62.291 type value mrr_at_3 60.681 type value mrr_at_5 61.61 type value ndcg_at_1 51.23799999999999 type value ndcg_at_10 40.892 type value ndcg_at_100 26.951999999999998 type value ndcg_at_1000 26.474999999999998 type value ndcg_at_3 46.821 type value ndcg_at_5 44.333 type value precision_at_1 53.251000000000005 type value precision_at_10 30.124000000000002 type value precision_at_100 3.012 type value precision_at_1000 0.301 type value precision_at_3 43.55 type value precision_at_5 38.266 type value recall_at_1 6.920999999999999 type value recall_at_10 20.852 type value recall_at_100 20.852 type value recall_at_1000 20.852 type value recall_at_3 13.628000000000002 type value recall_at_5 16.273 task dataset metrics type Retrieval type name config split revision nq MTEB NQ default test None type value map_at_1 46.827999999999996 type value map_at_10 63.434000000000005 type value map_at_100 63.434000000000005 type value map_at_1000 63.434000000000005 type value map_at_3 59.794000000000004 type value map_at_5 62.08 type value mrr_at_1 52.288999999999994 type value mrr_at_10 65.95 type value mrr_at_100 65.95 type value mrr_at_1000 65.95 type value mrr_at_3 63.413 type value mrr_at_5 65.08 type value ndcg_at_1 52.288999999999994 type value ndcg_at_10 70.301 type value ndcg_at_100 70.301 type value ndcg_at_1000 70.301 type value ndcg_at_3 63.979 type value ndcg_at_5 67.582 type value precision_at_1 52.288999999999994 type value precision_at_10 10.576 type value precision_at_100 1.058 type value precision_at_1000 0.106 type value precision_at_3 28.177000000000003 type value precision_at_5 19.073 type value recall_at_1 46.827999999999996 type value recall_at_10 88.236 type value recall_at_100 88.236 type value recall_at_1000 88.236 type value recall_at_3 72.371 type value recall_at_5 80.56 task dataset metrics type Retrieval type name config split revision quora MTEB QuoraRetrieval default test None type value map_at_1 71.652 type value map_at_10 85.953 type value map_at_100 85.953 type value map_at_1000 85.953 type value map_at_3 83.05399999999999 type value map_at_5 84.89 type value mrr_at_1 82.42 type value mrr_at_10 88.473 type value mrr_at_100 88.473 type value mrr_at_1000 88.473 type value mrr_at_3 87.592 type value mrr_at_5 88.211 type value ndcg_at_1 82.44 type value ndcg_at_10 89.467 type value ndcg_at_100 89.33 type value ndcg_at_1000 89.33 type value ndcg_at_3 86.822 type value ndcg_at_5 88.307 type value precision_at_1 82.44 type value precision_at_10 13.616 type value precision_at_100 1.362 type value precision_at_1000 0.136 type value precision_at_3 38.117000000000004 type value precision_at_5 25.05 type value recall_at_1 71.652 type value recall_at_10 96.224 type value recall_at_100 96.224 type value recall_at_1000 96.224 type value recall_at_3 88.571 type value recall_at_5 92.812 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering MTEB RedditClustering default test 24640382cdbf8abc73003fb0fa6d111a705499eb type value v_measure 61.295010338050474 task dataset metrics type Clustering type name config split revision mteb/reddit-clustering-p2p MTEB RedditClusteringP2P default test 282350215ef01743dc01b456c7f5241fa8937f16 type value v_measure 67.26380819328142 task dataset metrics type Retrieval type name config split revision scidocs MTEB SCIDOCS default test None type value map_at_1 5.683 type value map_at_10 14.924999999999999 type value map_at_100 17.532 type value map_at_1000 17.875 type value map_at_3 10.392 type value map_at_5 12.592 type value mrr_at_1 28.000000000000004 type value mrr_at_10 39.951 type value mrr_at_100 41.025 type value mrr_at_1000 41.056 type value mrr_at_3 36.317 type value mrr_at_5 38.412 type value ndcg_at_1 28.000000000000004 type value ndcg_at_10 24.410999999999998 type value ndcg_at_100 33.79 type value ndcg_at_1000 39.035 type value ndcg_at_3 22.845 type value ndcg_at_5 20.080000000000002 type value precision_at_1 28.000000000000004 type value precision_at_10 12.790000000000001 type value precision_at_100 2.633 type value precision_at_1000 0.388 type value precision_at_3 21.367 type value precision_at_5 17.7 type value recall_at_1 5.683 type value recall_at_10 25.91 type value recall_at_100 53.443 type value recall_at_1000 78.73 type value recall_at_3 13.003 type value recall_at_5 17.932000000000002 task dataset metrics type STS type name config split revision mteb/sickr-sts MTEB SICK-R default test a6ea5a8cab320b040a23452cc28066d9beae2cee type value cos_sim_pearson 84.677978681023 type value cos_sim_spearman 83.13093441058189 type value euclidean_pearson 83.35535759341572 type value euclidean_spearman 83.42583744219611 type value manhattan_pearson 83.2243124045889 type value manhattan_spearman 83.39801618652632 task dataset metrics type STS type name config split revision mteb/sts12-sts MTEB STS12 default test a0d554a64d88156834ff5ae9920b964011b16384 type value cos_sim_pearson 81.68960206569666 type value cos_sim_spearman 77.3368966488535 type value euclidean_pearson 77.62828980560303 type value euclidean_spearman 76.77951481444651 type value manhattan_pearson 77.88637240839041 type value manhattan_spearman 77.22157841466188 task dataset metrics type STS type name config split revision mteb/sts13-sts MTEB STS13 default test 7e90230a92c190f1bf69ae9002b8cea547a64cca type value cos_sim_pearson 84.18745821650724 type value cos_sim_spearman 85.04423285574542 type value euclidean_pearson 85.46604816931023 type value euclidean_spearman 85.5230593932974 type value manhattan_pearson 85.57912805986261 type value manhattan_spearman 85.65955905111873 task dataset metrics type STS type name config split revision mteb/sts14-sts MTEB STS14 default test 6031580fec1f6af667f0bd2da0a551cf4f0b2375 type value cos_sim_pearson 83.6715333300355 type value cos_sim_spearman 82.9058522514908 type value euclidean_pearson 83.9640357424214 type value euclidean_spearman 83.60415457472637 type value manhattan_pearson 84.05621005853469 type value manhattan_spearman 83.87077724707746 task dataset metrics type STS type name config split revision mteb/sts15-sts MTEB STS15 default test ae752c7c21bf194d8b67fd573edf7ae58183cbe3 type value cos_sim_pearson 87.82422928098886 type value cos_sim_spearman 88.12660311894628 type value euclidean_pearson 87.50974805056555 type value euclidean_spearman 87.91957275596677 type value manhattan_pearson 87.74119404878883 type value manhattan_spearman 88.2808922165719 task dataset metrics type STS type name config split revision mteb/sts16-sts MTEB STS16 default test 4d8694f8f0e0100860b497b999b3dbed754a0513 type value cos_sim_pearson 84.80605838552093 type value cos_sim_spearman 86.24123388765678 type value euclidean_pearson 85.32648347339814 type value euclidean_spearman 85.60046671950158 type value manhattan_pearson 85.53800168487811 type value manhattan_spearman 85.89542420480763 task dataset metrics type STS type name config split revision mteb/sts17-crosslingual-sts MTEB STS17 (en-en) en-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d type value cos_sim_pearson 89.87540978988132 type value cos_sim_spearman 90.12715295099461 type value euclidean_pearson 91.61085993525275 type value euclidean_spearman 91.31835942311758 type value manhattan_pearson 91.57500202032934 type value manhattan_spearman 91.1790925526635 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (en) en test eea2b4fe26a775864c896887d910b76a8098ad3f type value cos_sim_pearson 69.87136205329556 type value cos_sim_spearman 68.6253154635078 type value euclidean_pearson 68.91536015034222 type value euclidean_spearman 67.63744649352542 type value manhattan_pearson 69.2000713045275 type value manhattan_spearman 68.16002901587316 task dataset metrics type STS type name config split revision mteb/stsbenchmark-sts MTEB STSBenchmark default test b0fddb56ed78048fa8b90373c8a3cfc37b684831 type value cos_sim_pearson 85.21849551039082 type value cos_sim_spearman 85.6392959372461 type value euclidean_pearson 85.92050852609488 type value euclidean_spearman 85.97205649009734 type value manhattan_pearson 86.1031154802254 type value manhattan_spearman 86.26791155517466 task dataset metrics type Reranking type name config split revision mteb/scidocs-reranking MTEB SciDocsRR default test d3c5e1fc0b855ab6097bf1cda04dd73947d7caab type value map 86.83953958636627 type value mrr 96.71167612344082 task dataset metrics type Retrieval type name config split revision scifact MTEB SciFact default test None type value map_at_1 64.994 type value map_at_10 74.763 type value map_at_100 75.127 type value map_at_1000 75.143 type value map_at_3 71.824 type value map_at_5 73.71 type value mrr_at_1 68.333 type value mrr_at_10 75.749 type value mrr_at_100 75.922 type value mrr_at_1000 75.938 type value mrr_at_3 73.556 type value mrr_at_5 74.739 type value ndcg_at_1 68.333 type value ndcg_at_10 79.174 type value ndcg_at_100 80.41 type value ndcg_at_1000 80.804 type value ndcg_at_3 74.361 type value ndcg_at_5 76.861 type value precision_at_1 68.333 type value precision_at_10 10.333 type value precision_at_100 1.0999999999999999 type value precision_at_1000 0.11299999999999999 type value precision_at_3 28.778 type value precision_at_5 19.067 type value recall_at_1 64.994 type value recall_at_10 91.822 type value recall_at_100 97 type value recall_at_1000 100 type value recall_at_3 78.878 type value recall_at_5 85.172 task dataset metrics type PairClassification type name config split revision mteb/sprintduplicatequestions-pairclassification MTEB SprintDuplicateQuestions default test d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 type value cos_sim_accuracy 99.72079207920792 type value cos_sim_ap 93.00265215525152 type value cos_sim_f1 85.06596306068602 type value cos_sim_precision 90.05586592178771 type value cos_sim_recall 80.60000000000001 type value dot_accuracy 99.66039603960397 type value dot_ap 91.22371407479089 type value dot_f1 82.34693877551021 type value dot_precision 84.0625 type value dot_recall 80.7 type value euclidean_accuracy 99.71881188118812 type value euclidean_ap 92.88449963304728 type value euclidean_f1 85.19480519480518 type value euclidean_precision 88.64864864864866 type value euclidean_recall 82 type value manhattan_accuracy 99.73267326732673 type value manhattan_ap 93.23055393056883 type value manhattan_f1 85.88957055214725 type value manhattan_precision 87.86610878661088 type value manhattan_recall 84 type value max_accuracy 99.73267326732673 type value max_ap 93.23055393056883 type value max_f1 85.88957055214725 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering MTEB StackExchangeClustering default test 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 type value v_measure 77.3305735900358 task dataset metrics type Clustering type name config split revision mteb/stackexchange-clustering-p2p MTEB StackExchangeClusteringP2P default test 815ca46b2622cec33ccafc3735d572c266efdb44 type value v_measure 41.32967136540674 task dataset metrics type Reranking type name config split revision mteb/stackoverflowdupquestions-reranking MTEB StackOverflowDupQuestions default test e185fbe320c72810689fc5848eb6114e1ef5ec69 type value map 55.95514866379359 type value mrr 56.95423245055598 task dataset metrics type Summarization type name config split revision mteb/summeval MTEB SummEval default test cda12ad7615edc362dbf25a00fdd61d3b1eaf93c type value cos_sim_pearson 30.783007208997144 type value cos_sim_spearman 30.373444721540533 type value dot_pearson 29.210604111143905 type value dot_spearman 29.98809758085659 task dataset metrics type Retrieval type name config split revision trec-covid MTEB TRECCOVID default test None type value map_at_1 0.234 type value map_at_10 1.894 type value map_at_100 1.894 type value map_at_1000 1.894 type value map_at_3 0.636 type value map_at_5 1 type value mrr_at_1 88 type value mrr_at_10 93.667 type value mrr_at_100 93.667 type value mrr_at_1000 93.667 type value mrr_at_3 93.667 type value mrr_at_5 93.667 type value ndcg_at_1 85 type value ndcg_at_10 74.798 type value ndcg_at_100 16.462 type value ndcg_at_1000 7.0889999999999995 type value ndcg_at_3 80.754 type value ndcg_at_5 77.319 type value precision_at_1 88 type value precision_at_10 78 type value precision_at_100 7.8 type value precision_at_1000 0.7799999999999999 type value precision_at_3 83.333 type value precision_at_5 80.80000000000001 type value recall_at_1 0.234 type value recall_at_10 2.093 type value recall_at_100 2.093 type value recall_at_1000 2.093 type value recall_at_3 0.662 type value recall_at_5 1.0739999999999998 task dataset metrics type Retrieval type name config split revision webis-touche2020 MTEB Touche2020 default test None type value map_at_1 2.703 type value map_at_10 10.866000000000001 type value map_at_100 10.866000000000001 type value map_at_1000 10.866000000000001 type value map_at_3 5.909 type value map_at_5 7.35 type value mrr_at_1 36.735 type value mrr_at_10 53.583000000000006 type value mrr_at_100 53.583000000000006 type value mrr_at_1000 53.583000000000006 type value mrr_at_3 49.32 type value mrr_at_5 51.769 type value ndcg_at_1 34.694 type value ndcg_at_10 27.926000000000002 type value ndcg_at_100 22.701 type value ndcg_at_1000 22.701 type value ndcg_at_3 32.073 type value ndcg_at_5 28.327999999999996 type value precision_at_1 36.735 type value precision_at_10 24.694 type value precision_at_100 2.469 type value precision_at_1000 0.247 type value precision_at_3 31.973000000000003 type value precision_at_5 26.939 type value recall_at_1 2.703 type value recall_at_10 17.702 type value recall_at_100 17.702 type value recall_at_1000 17.702 type value recall_at_3 7.208 type value recall_at_5 9.748999999999999 task dataset metrics type Classification type name config split revision mteb/toxic_conversations_50k MTEB ToxicConversationsClassification default test d7c0de2777da35d6aae2200a62c6e0e5af397c4c type value accuracy 70.79960000000001 type value ap 15.467565415565815 type value f1 55.28639823443618 task dataset metrics type Classification type name config split revision mteb/tweet_sentiment_extraction MTEB TweetSentimentExtractionClassification default test d604517c81ca91fe16a244d1248fc021f9ecee7a type value accuracy 64.7792869269949 type value f1 65.08597154774318 task dataset metrics type Clustering type name config split revision mteb/twentynewsgroups-clustering MTEB TwentyNewsgroupsClustering default test 6125ec4e24fa026cec8a478383ee943acfbd5449 type value v_measure 55.70352297774293 task dataset metrics type PairClassification type name config split revision mteb/twittersemeval2015-pairclassification MTEB TwitterSemEval2015 default test 70970daeab8776df92f5ea462b6173c0b46fd2d1 type value cos_sim_accuracy 88.27561542588067 type value cos_sim_ap 81.08262141256193 type value cos_sim_f1 73.82341501361338 type value cos_sim_precision 72.5720112159062 type value cos_sim_recall 75.11873350923483 type value dot_accuracy 86.66030875603504 type value dot_ap 76.6052349228621 type value dot_f1 70.13897280966768 type value dot_precision 64.70457079152732 type value dot_recall 76.56992084432717 type value euclidean_accuracy 88.37098408535495 type value euclidean_ap 81.12515230092113 type value euclidean_f1 74.10338225909379 type value euclidean_precision 71.76761433868974 type value euclidean_recall 76.59630606860158 type value manhattan_accuracy 88.34118137926924 type value manhattan_ap 80.95751834536561 type value manhattan_f1 73.9119496855346 type value manhattan_precision 70.625 type value manhattan_recall 77.5197889182058 type value max_accuracy 88.37098408535495 type value max_ap 81.12515230092113 type value max_f1 74.10338225909379 task dataset metrics type PairClassification type name config split revision mteb/twitterurlcorpus-pairclassification MTEB TwitterURLCorpus default test 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf type value cos_sim_accuracy 89.79896767182831 type value cos_sim_ap 87.40071784061065 type value cos_sim_f1 79.87753144712087 type value cos_sim_precision 76.67304015296367 type value cos_sim_recall 83.3615645210964 type value dot_accuracy 88.95486474948578 type value dot_ap 86.00227979119943 type value dot_f1 78.54601474525914 type value dot_precision 75.00525394045535 type value dot_recall 82.43763473975977 type value euclidean_accuracy 89.7892653393876 type value euclidean_ap 87.42174706480819 type value euclidean_f1 80.07283321194465 type value euclidean_precision 75.96738529574351 type value euclidean_recall 84.6473668001232 type value manhattan_accuracy 89.8474793340319 type value manhattan_ap 87.47814292587448 type value manhattan_f1 80.15461150280949 type value manhattan_precision 74.88798234468 type value manhattan_recall 86.21804742839544 type value max_accuracy 89.8474793340319 type value max_ap 87.47814292587448 type value max_f1 80.15461150280949	mistralai/Mistral-7B-v0.1

Model Summary

GritLM is a generative representational instruction tuned language model. It unifies text representation (embedding) and text generation into a single model achieving state-of-the-art performance on both types of tasks.

• Repository: ContextualAI/gritlm
• Paper: https://arxiv.org/abs/2402.09906
• Logs: https://wandb.ai/muennighoff/gritlm/runs/0uui712t/overview
• Script: https://github.com/ContextualAI/gritlm/blob/main/scripts/training/train_gritlm_7b.sh

Model	Description
GritLM 7B	Mistral 7B finetuned using GRIT
GritLM 8x7B	Mixtral 8x7B finetuned using GRIT

Use

The model usage is documented here.

Citation

@misc{muennighoff2024generative,
      title={Generative Representational Instruction Tuning}, 
      author={Niklas Muennighoff and Hongjin Su and Liang Wang and Nan Yang and Furu Wei and Tao Yu and Amanpreet Singh and Douwe Kiela},
      year={2024},
      eprint={2402.09906},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}