tags, model-index, language, license
tags model-index language license
mteb
Sentence Transformers
sentence-similarity
sentence-transformers
name results
multilingual-e5-small
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en) en test e8379541af4e31359cca9fbcf4b00f2671dba205
type value
accuracy 73.79104477611939
type value
ap 36.9996434842022
type value
f1 67.95453679103099
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (de) de test e8379541af4e31359cca9fbcf4b00f2671dba205
type value
accuracy 71.64882226980728
type value
ap 82.11942130026586
type value
f1 69.87963421606715
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (en-ext) en-ext test e8379541af4e31359cca9fbcf4b00f2671dba205
type value
accuracy 75.8095952023988
type value
ap 24.46869495579561
type value
f1 63.00108480037597
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_counterfactual MTEB AmazonCounterfactualClassification (ja) ja test e8379541af4e31359cca9fbcf4b00f2671dba205
type value
accuracy 64.186295503212
type value
ap 15.496804690197042
type value
f1 52.07153895475031
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_polarity MTEB AmazonPolarityClassification default test e2d317d38cd51312af73b3d32a06d1a08b442046
type value
accuracy 88.699325
type value
ap 85.27039559917269
type value
f1 88.65556295032513
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (en) en test 1399c76144fd37290681b995c656ef9b2e06e26d
type value
accuracy 44.69799999999999
type value
f1 43.73187348654165
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (de) de test 1399c76144fd37290681b995c656ef9b2e06e26d
type value
accuracy 40.245999999999995
type value
f1 39.3863530637684
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (es) es test 1399c76144fd37290681b995c656ef9b2e06e26d
type value
accuracy 40.394
type value
f1 39.301223469483446
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (fr) fr test 1399c76144fd37290681b995c656ef9b2e06e26d
type value
accuracy 38.864
type value
f1 37.97974261868003
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (ja) ja test 1399c76144fd37290681b995c656ef9b2e06e26d
type value
accuracy 37.682
type value
f1 37.07399369768313
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (zh) zh test 1399c76144fd37290681b995c656ef9b2e06e26d
type value
accuracy 37.504
type value
f1 36.62317273874278
task dataset metrics
type
Retrieval
type name config split revision
arguana MTEB ArguAna default test None
type value
map_at_1 19.061
type value
map_at_10 31.703
type value
map_at_100 32.967
type value
map_at_1000 33.001000000000005
type value
map_at_3 27.466
type value
map_at_5 29.564
type value
mrr_at_1 19.559
type value
mrr_at_10 31.874999999999996
type value
mrr_at_100 33.146
type value
mrr_at_1000 33.18
type value
mrr_at_3 27.667
type value
mrr_at_5 29.74
type value
ndcg_at_1 19.061
type value
ndcg_at_10 39.062999999999995
type value
ndcg_at_100 45.184000000000005
type value
ndcg_at_1000 46.115
type value
ndcg_at_3 30.203000000000003
type value
ndcg_at_5 33.953
type value
precision_at_1 19.061
type value
precision_at_10 6.279999999999999
type value
precision_at_100 0.9129999999999999
type value
precision_at_1000 0.099
type value
precision_at_3 12.706999999999999
type value
precision_at_5 9.431000000000001
type value
recall_at_1 19.061
type value
recall_at_10 62.802
type value
recall_at_100 91.323
type value
recall_at_1000 98.72
type value
recall_at_3 38.122
type value
recall_at_5 47.155
task dataset metrics
type
Clustering
type name config split revision
mteb/arxiv-clustering-p2p MTEB ArxivClusteringP2P default test a122ad7f3f0291bf49cc6f4d32aa80929df69d5d
type value
v_measure 39.22266660528253
task dataset metrics
type
Clustering
type name config split revision
mteb/arxiv-clustering-s2s MTEB ArxivClusteringS2S default test f910caf1a6075f7329cdf8c1a6135696f37dbd53
type value
v_measure 30.79980849482483
task dataset metrics
type
Reranking
type name config split revision
mteb/askubuntudupquestions-reranking MTEB AskUbuntuDupQuestions default test 2000358ca161889fa9c082cb41daa8dcfb161a54
type value
map 57.8790068352054
type value
mrr 71.78791276436706
task dataset metrics
type
STS
type name config split revision
mteb/biosses-sts MTEB BIOSSES default test d3fb88f8f02e40887cd149695127462bbcf29b4a
type value
cos_sim_pearson 82.36328364043163
type value
cos_sim_spearman 82.26211536195868
type value
euclidean_pearson 80.3183865039173
type value
euclidean_spearman 79.88495276296132
type value
manhattan_pearson 80.14484480692127
type value
manhattan_spearman 80.39279565980743
task dataset metrics
type
BitextMining
type name config split revision
mteb/bucc-bitext-mining MTEB BUCC (de-en) de-en test d51519689f32196a32af33b075a01d0e7c51e252
type value
accuracy 98.0375782881002
type value
f1 97.86012526096033
type value
precision 97.77139874739039
type value
recall 98.0375782881002
task dataset metrics
type
BitextMining
type name config split revision
mteb/bucc-bitext-mining MTEB BUCC (fr-en) fr-en test d51519689f32196a32af33b075a01d0e7c51e252
type value
accuracy 93.35241030156286
type value
f1 92.66050333846944
type value
precision 92.3306919069631
type value
recall 93.35241030156286
task dataset metrics
type
BitextMining
type name config split revision
mteb/bucc-bitext-mining MTEB BUCC (ru-en) ru-en test d51519689f32196a32af33b075a01d0e7c51e252
type value
accuracy 94.0699688257707
type value
f1 93.50236693222492
type value
precision 93.22791825424315
type value
recall 94.0699688257707
task dataset metrics
type
BitextMining
type name config split revision
mteb/bucc-bitext-mining MTEB BUCC (zh-en) zh-en test d51519689f32196a32af33b075a01d0e7c51e252
type value
accuracy 89.25750394944708
type value
f1 88.79234684921889
type value
precision 88.57293312269616
type value
recall 89.25750394944708
task dataset metrics
type
Classification
type name config split revision
mteb/banking77 MTEB Banking77Classification default test 0fd18e25b25c072e09e0d92ab615fda904d66300
type value
accuracy 79.41558441558442
type value
f1 79.25886487487219
task dataset metrics
type
Clustering
type name config split revision
mteb/biorxiv-clustering-p2p MTEB BiorxivClusteringP2P default test 65b79d1d13f80053f67aca9498d9402c2d9f1f40
type value
v_measure 35.747820820329736
task dataset metrics
type
Clustering
type name config split revision
mteb/biorxiv-clustering-s2s MTEB BiorxivClusteringS2S default test 258694dd0231531bc1fd9de6ceb52a0853c6d908
type value
v_measure 27.045143830596146
task dataset metrics
type
Retrieval
type name config split revision
BeIR/cqadupstack MTEB CQADupstackRetrieval default test None
type value
map_at_1 24.252999999999997
type value
map_at_10 31.655916666666666
type value
map_at_100 32.680749999999996
type value
map_at_1000 32.79483333333334
type value
map_at_3 29.43691666666666
type value
map_at_5 30.717416666666665
type value
mrr_at_1 28.602750000000004
type value
mrr_at_10 35.56875
type value
mrr_at_100 36.3595
type value
mrr_at_1000 36.427749999999996
type value
mrr_at_3 33.586166666666664
type value
mrr_at_5 34.73641666666666
type value
ndcg_at_1 28.602750000000004
type value
ndcg_at_10 36.06933333333334
type value
ndcg_at_100 40.70141666666667
type value
ndcg_at_1000 43.24341666666667
type value
ndcg_at_3 32.307916666666664
type value
ndcg_at_5 34.129999999999995
type value
precision_at_1 28.602750000000004
type value
precision_at_10 6.097666666666667
type value
precision_at_100 0.9809166666666668
type value
precision_at_1000 0.13766666666666663
type value
precision_at_3 14.628166666666667
type value
precision_at_5 10.266916666666667
type value
recall_at_1 24.252999999999997
type value
recall_at_10 45.31916666666667
type value
recall_at_100 66.03575000000001
type value
recall_at_1000 83.94708333333334
type value
recall_at_3 34.71941666666666
type value
recall_at_5 39.46358333333333
task dataset metrics
type
Retrieval
type name config split revision
climate-fever MTEB ClimateFEVER default test None
type value
map_at_1 9.024000000000001
type value
map_at_10 15.644
type value
map_at_100 17.154
type value
map_at_1000 17.345
type value
map_at_3 13.028
type value
map_at_5 14.251
type value
mrr_at_1 19.674
type value
mrr_at_10 29.826999999999998
type value
mrr_at_100 30.935000000000002
type value
mrr_at_1000 30.987
type value
mrr_at_3 26.645000000000003
type value
mrr_at_5 28.29
type value
ndcg_at_1 19.674
type value
ndcg_at_10 22.545
type value
ndcg_at_100 29.207
type value
ndcg_at_1000 32.912
type value
ndcg_at_3 17.952
type value
ndcg_at_5 19.363
type value
precision_at_1 19.674
type value
precision_at_10 7.212000000000001
type value
precision_at_100 1.435
type value
precision_at_1000 0.212
type value
precision_at_3 13.507
type value
precision_at_5 10.397
type value
recall_at_1 9.024000000000001
type value
recall_at_10 28.077999999999996
type value
recall_at_100 51.403
type value
recall_at_1000 72.406
type value
recall_at_3 16.768
type value
recall_at_5 20.737
task dataset metrics
type
Retrieval
type name config split revision
dbpedia-entity MTEB DBPedia default test None
type value
map_at_1 8.012
type value
map_at_10 17.138
type value
map_at_100 24.146
type value
map_at_1000 25.622
type value
map_at_3 12.552
type value
map_at_5 14.435
type value
mrr_at_1 62.25000000000001
type value
mrr_at_10 71.186
type value
mrr_at_100 71.504
type value
mrr_at_1000 71.514
type value
mrr_at_3 69.333
type value
mrr_at_5 70.408
type value
ndcg_at_1 49.75
type value
ndcg_at_10 37.76
type value
ndcg_at_100 42.071
type value
ndcg_at_1000 49.309
type value
ndcg_at_3 41.644
type value
ndcg_at_5 39.812999999999995
type value
precision_at_1 62.25000000000001
type value
precision_at_10 30.15
type value
precision_at_100 9.753
type value
precision_at_1000 1.9189999999999998
type value
precision_at_3 45.667
type value
precision_at_5 39.15
type value
recall_at_1 8.012
type value
recall_at_10 22.599
type value
recall_at_100 48.068
type value
recall_at_1000 71.328
type value
recall_at_3 14.043
type value
recall_at_5 17.124
task dataset metrics
type
Classification
type name config split revision
mteb/emotion MTEB EmotionClassification default test 4f58c6b202a23cf9a4da393831edf4f9183cad37
type value
accuracy 42.455
type value
f1 37.59462649781862
task dataset metrics
type
Retrieval
type name config split revision
fever MTEB FEVER default test None
type value
map_at_1 58.092
type value
map_at_10 69.586
type value
map_at_100 69.968
type value
map_at_1000 69.982
type value
map_at_3 67.48100000000001
type value
map_at_5 68.915
type value
mrr_at_1 62.166
type value
mrr_at_10 73.588
type value
mrr_at_100 73.86399999999999
type value
mrr_at_1000 73.868
type value
mrr_at_3 71.6
type value
mrr_at_5 72.99
type value
ndcg_at_1 62.166
type value
ndcg_at_10 75.27199999999999
type value
ndcg_at_100 76.816
type value
ndcg_at_1000 77.09700000000001
type value
ndcg_at_3 71.36
type value
ndcg_at_5 73.785
type value
precision_at_1 62.166
type value
precision_at_10 9.716
type value
precision_at_100 1.065
type value
precision_at_1000 0.11
type value
precision_at_3 28.278
type value
precision_at_5 18.343999999999998
type value
recall_at_1 58.092
type value
recall_at_10 88.73400000000001
type value
recall_at_100 95.195
type value
recall_at_1000 97.04599999999999
type value
recall_at_3 78.45
type value
recall_at_5 84.316
task dataset metrics
type
Retrieval
type name config split revision
fiqa MTEB FiQA2018 default test None
type value
map_at_1 16.649
type value
map_at_10 26.457000000000004
type value
map_at_100 28.169
type value
map_at_1000 28.352
type value
map_at_3 23.305
type value
map_at_5 25.169000000000004
type value
mrr_at_1 32.407000000000004
type value
mrr_at_10 40.922
type value
mrr_at_100 41.931000000000004
type value
mrr_at_1000 41.983
type value
mrr_at_3 38.786
type value
mrr_at_5 40.205999999999996
type value
ndcg_at_1 32.407000000000004
type value
ndcg_at_10 33.314
type value
ndcg_at_100 40.312
type value
ndcg_at_1000 43.685
type value
ndcg_at_3 30.391000000000002
type value
ndcg_at_5 31.525
type value
precision_at_1 32.407000000000004
type value
precision_at_10 8.966000000000001
type value
precision_at_100 1.6019999999999999
type value
precision_at_1000 0.22200000000000003
type value
precision_at_3 20.165
type value
precision_at_5 14.722
type value
recall_at_1 16.649
type value
recall_at_10 39.117000000000004
type value
recall_at_100 65.726
type value
recall_at_1000 85.784
type value
recall_at_3 27.914
type value
recall_at_5 33.289
task dataset metrics
type
Retrieval
type name config split revision
hotpotqa MTEB HotpotQA default test None
type value
map_at_1 36.253
type value
map_at_10 56.16799999999999
type value
map_at_100 57.06099999999999
type value
map_at_1000 57.126
type value
map_at_3 52.644999999999996
type value
map_at_5 54.909
type value
mrr_at_1 72.505
type value
mrr_at_10 79.66
type value
mrr_at_100 79.869
type value
mrr_at_1000 79.88
type value
mrr_at_3 78.411
type value
mrr_at_5 79.19800000000001
type value
ndcg_at_1 72.505
type value
ndcg_at_10 65.094
type value
ndcg_at_100 68.219
type value
ndcg_at_1000 69.515
type value
ndcg_at_3 59.99
type value
ndcg_at_5 62.909000000000006
type value
precision_at_1 72.505
type value
precision_at_10 13.749
type value
precision_at_100 1.619
type value
precision_at_1000 0.179
type value
precision_at_3 38.357
type value
precision_at_5 25.313000000000002
type value
recall_at_1 36.253
type value
recall_at_10 68.744
type value
recall_at_100 80.925
type value
recall_at_1000 89.534
type value
recall_at_3 57.535000000000004
type value
recall_at_5 63.282000000000004
task dataset metrics
type
Classification
type name config split revision
mteb/imdb MTEB ImdbClassification default test 3d86128a09e091d6018b6d26cad27f2739fc2db7
type value
accuracy 80.82239999999999
type value
ap 75.65895781725314
type value
f1 80.75880969095746
task dataset metrics
type
Retrieval
type name config split revision
msmarco MTEB MSMARCO default dev None
type value
map_at_1 21.624
type value
map_at_10 34.075
type value
map_at_100 35.229
type value
map_at_1000 35.276999999999994
type value
map_at_3 30.245
type value
map_at_5 32.42
type value
mrr_at_1 22.264
type value
mrr_at_10 34.638000000000005
type value
mrr_at_100 35.744
type value
mrr_at_1000 35.787
type value
mrr_at_3 30.891000000000002
type value
mrr_at_5 33.042
type value
ndcg_at_1 22.264
type value
ndcg_at_10 40.991
type value
ndcg_at_100 46.563
type value
ndcg_at_1000 47.743
type value
ndcg_at_3 33.198
type value
ndcg_at_5 37.069
type value
precision_at_1 22.264
type value
precision_at_10 6.5089999999999995
type value
precision_at_100 0.9299999999999999
type value
precision_at_1000 0.10300000000000001
type value
precision_at_3 14.216999999999999
type value
precision_at_5 10.487
type value
recall_at_1 21.624
type value
recall_at_10 62.303
type value
recall_at_100 88.124
type value
recall_at_1000 97.08
type value
recall_at_3 41.099999999999994
type value
recall_at_5 50.381
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_domain MTEB MTOPDomainClassification (en) en test d80d48c1eb48d3562165c59d59d0034df9fff0bf
type value
accuracy 91.06703146374831
type value
f1 90.86867815863172
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_domain MTEB MTOPDomainClassification (de) de test d80d48c1eb48d3562165c59d59d0034df9fff0bf
type value
accuracy 87.46970977740209
type value
f1 86.36832872036588
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_domain MTEB MTOPDomainClassification (es) es test d80d48c1eb48d3562165c59d59d0034df9fff0bf
type value
accuracy 89.26951300867245
type value
f1 88.93561193959502
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_domain MTEB MTOPDomainClassification (fr) fr test d80d48c1eb48d3562165c59d59d0034df9fff0bf
type value
accuracy 84.22799874725963
type value
f1 84.30490069236556
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_domain MTEB MTOPDomainClassification (hi) hi test d80d48c1eb48d3562165c59d59d0034df9fff0bf
type value
accuracy 86.02007888131948
type value
f1 85.39376041027991
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_domain MTEB MTOPDomainClassification (th) th test d80d48c1eb48d3562165c59d59d0034df9fff0bf
type value
accuracy 85.34900542495481
type value
f1 85.39859673336713
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_intent MTEB MTOPIntentClassification (en) en test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
type value
accuracy 71.078431372549
type value
f1 53.45071102002276
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_intent MTEB MTOPIntentClassification (de) de test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
type value
accuracy 65.85798816568047
type value
f1 46.53112748993529
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_intent MTEB MTOPIntentClassification (es) es test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
type value
accuracy 67.96864576384256
type value
f1 45.966703022829506
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_intent MTEB MTOPIntentClassification (fr) fr test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
type value
accuracy 61.31537738803633
type value
f1 45.52601712835461
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_intent MTEB MTOPIntentClassification (hi) hi test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
type value
accuracy 66.29616349946218
type value
f1 47.24166485726613
task dataset metrics
type
Classification
type name config split revision
mteb/mtop_intent MTEB MTOPIntentClassification (th) th test ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
type value
accuracy 67.51537070524412
type value
f1 49.463476319014276
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (af) af test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 57.06792199058508
type value
f1 54.094921857502285
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (am) am test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 51.960322797579025
type value
f1 48.547371223370945
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (ar) ar test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 54.425016812373904
type value
f1 50.47069202054312
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (az) az test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 59.798251513113655
type value
f1 57.05013069086648
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (bn) bn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 59.37794216543376
type value
f1 56.3607992649805
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (cy) cy test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 46.56018829858777
type value
f1 43.87319715715134
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (da) da test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 62.9724277067922
type value
f1 59.36480066245562
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (de) de test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 62.72696704774715
type value
f1 59.143595966615855
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (el) el test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 61.5971755211836
type value
f1 59.169445724946726
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (en) en test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 70.29589778076665
type value
f1 67.7577001808977
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (es) es test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 66.31136516476126
type value
f1 64.52032955983242
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (fa) fa test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 65.54472091459314
type value
f1 61.47903120066317
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (fi) fi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 61.45595158036314
type value
f1 58.0891846024637
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (fr) fr test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 65.47074646940149
type value
f1 62.84830858877575
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (he) he test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 58.046402151983855
type value
f1 55.269074430533195
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (hi) hi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 64.06523201075991
type value
f1 61.35339643021369
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (hu) hu test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 60.954942837928726
type value
f1 57.07035922704846
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (hy) hy test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 57.404169468728995
type value
f1 53.94259011839138
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (id) id test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 64.16610625420309
type value
f1 61.337103431499365
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (is) is test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 52.262945527908535
type value
f1 49.7610691598921
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (it) it test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 65.54472091459314
type value
f1 63.469099018440154
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (ja) ja test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 68.22797579018157
type value
f1 64.89098471083001
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (jv) jv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 50.847343644922674
type value
f1 47.8536963168393
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (ka) ka test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 48.45326160053799
type value
f1 46.370078045805556
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (km) km test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 42.83120376597175
type value
f1 39.68948521599982
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (kn) kn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 57.5084061869536
type value
f1 53.961876160401545
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (ko) ko test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 63.7895090786819
type value
f1 61.134223684676
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (lv) lv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 54.98991257565569
type value
f1 52.579862862826296
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (ml) ml test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 61.90316072629456
type value
f1 58.203024538290336
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (mn) mn test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 57.09818426361802
type value
f1 54.22718458445455
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (ms) ms test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 58.991257565568255
type value
f1 55.84892781767421
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (my) my test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 55.901143241425686
type value
f1 52.25264332199797
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (nb) nb test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 61.96368527236047
type value
f1 58.927243876153454
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (nl) nl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 65.64223268325489
type value
f1 62.340453718379706
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (pl) pl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 64.52589105581708
type value
f1 61.661113187022174
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (pt) pt test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 66.84599865501009
type value
f1 64.59342572873005
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (ro) ro test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 60.81035642232684
type value
f1 57.5169089806797
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (ru) ru test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 65.75991930060525
type value
f1 62.89531115787938
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (sl) sl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 56.51647612642906
type value
f1 54.33154780100043
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (sq) sq test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 57.985877605917956
type value
f1 54.46187524463802
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (sv) sv test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 65.03026227303296
type value
f1 62.34377392877748
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (sw) sw test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 53.567585743106925
type value
f1 50.73770655983206
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (ta) ta test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 57.2595830531271
type value
f1 53.657327291708626
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (te) te test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 57.82784129119032
type value
f1 54.82518072665301
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (th) th test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 64.06859448554137
type value
f1 63.00185280500495
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (tl) tl test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 58.91055817081371
type value
f1 55.54116301224262
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (tr) tr test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 63.54404841963686
type value
f1 59.57650946030184
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (ur) ur test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 59.27706792199059
type value
f1 56.50010066083435
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (vi) vi test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 64.0719569603228
type value
f1 61.817075925647956
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-CN) zh-CN test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 68.23806321452591
type value
f1 65.24917026029749
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-TW) zh-TW test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
type value
accuracy 62.53530598520511
type value
f1 61.71131132295768
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (af) af test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 63.04303967720243
type value
f1 60.3950085685985
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (am) am test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 56.83591123066578
type value
f1 54.95059828830849
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ar) ar test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 59.62340282447881
type value
f1 59.525159996498225
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (az) az test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 60.85406859448555
type value
f1 59.129299095681276
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (bn) bn test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 62.76731674512441
type value
f1 61.159560612627715
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (cy) cy test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 50.181573638197705
type value
f1 46.98422176289957
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (da) da test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 68.92737054472092
type value
f1 67.69135611952979
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (de) de test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 69.18964357767318
type value
f1 68.46106138186214
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (el) el test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 67.0712844653665
type value
f1 66.75545422473901
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (en) en test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 74.4754539340955
type value
f1 74.38427146553252
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (es) es test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 69.82515131136518
type value
f1 69.63516462173847
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fa) fa test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 68.70880968392737
type value
f1 67.45420662567926
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fi) fi test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 65.95494283792871
type value
f1 65.06191009049222
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (fr) fr test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 68.75924680564896
type value
f1 68.30833379585945
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (he) he test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 63.806321452589096
type value
f1 63.273048243765054
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hi) hi test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 67.68997982515133
type value
f1 66.54703855381324
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hu) hu test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 66.46940147948891
type value
f1 65.91017343463396
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (hy) hy test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 59.49899125756556
type value
f1 57.90333469917769
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (id) id test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 67.9219905850706
type value
f1 67.23169403762938
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (is) is test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 56.486213853396094
type value
f1 54.85282355583758
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (it) it test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 69.04169468728985
type value
f1 68.83833333320462
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ja) ja test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 73.88702084734365
type value
f1 74.04474735232299
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (jv) jv test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 56.63416274377943
type value
f1 55.11332211687954
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ka) ka test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 52.23604572965702
type value
f1 50.86529813991055
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (km) km test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 46.62407531943511
type value
f1 43.63485467164535
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (kn) kn test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 59.15601882985878
type value
f1 57.522837510959924
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ko) ko test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 69.84532616005382
type value
f1 69.60021127179697
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (lv) lv test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 56.65770006724949
type value
f1 55.84219135523227
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ml) ml test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 66.53665097511768
type value
f1 65.09087787792639
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (mn) mn test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 59.31405514458642
type value
f1 58.06135303831491
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ms) ms test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 64.88231338264964
type value
f1 62.751099407787926
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (my) my test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 58.86012104909213
type value
f1 56.29118323058282
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nb) nb test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 67.37390719569602
type value
f1 66.27922244885102
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (nl) nl test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 70.8675184936113
type value
f1 70.22146529932019
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pl) pl test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 68.2212508406187
type value
f1 67.77454802056282
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (pt) pt test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 68.18090114324143
type value
f1 68.03737625431621
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ro) ro test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 64.65030262273034
type value
f1 63.792945486912856
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ru) ru test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 69.48217888365838
type value
f1 69.96028997292197
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sl) sl test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 60.17821116341627
type value
f1 59.3935969827171
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sq) sq test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 62.86146603900471
type value
f1 60.133692735032376
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sv) sv test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 70.89441829186282
type value
f1 70.03064076194089
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (sw) sw test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 58.15063887020847
type value
f1 56.23326278499678
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ta) ta test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 59.43846671149966
type value
f1 57.70440450281974
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (te) te test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 60.8507061197041
type value
f1 59.22916396061171
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (th) th test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 70.65568258238063
type value
f1 69.90736239440633
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tl) tl test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 60.8843308675185
type value
f1 59.30332663713599
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (tr) tr test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 68.05312710154674
type value
f1 67.44024062594775
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (ur) ur test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 62.111634162743776
type value
f1 60.89083013084519
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (vi) vi test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 67.44115669132482
type value
f1 67.92227541674552
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-CN) zh-CN test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 74.4687289845326
type value
f1 74.16376793486025
task dataset metrics
type
Classification
type name config split revision
mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-TW) zh-TW test 7d571f92784cd94a019292a1f45445077d0ef634
type value
accuracy 68.31876260928043
type value
f1 68.5246745215607
task dataset metrics
type
Clustering
type name config split revision
mteb/medrxiv-clustering-p2p MTEB MedrxivClusteringP2P default test e7a26af6f3ae46b30dde8737f02c07b1505bcc73
type value
v_measure 30.90431696479766
task dataset metrics
type
Clustering
type name config split revision
mteb/medrxiv-clustering-s2s MTEB MedrxivClusteringS2S default test 35191c8c0dca72d8ff3efcd72aa802307d469663
type value
v_measure 27.259158476693774
task dataset metrics
type
Reranking
type name config split revision
mteb/mind_small MTEB MindSmallReranking default test 3bdac13927fdc888b903db93b2ffdbd90b295a69
type value
map 30.28445330838555
type value
mrr 31.15758529581164
task dataset metrics
type
Retrieval
type name config split revision
nfcorpus MTEB NFCorpus default test None
type value
map_at_1 5.353
type value
map_at_10 11.565
type value
map_at_100 14.097000000000001
type value
map_at_1000 15.354999999999999
type value
map_at_3 8.749
type value
map_at_5 9.974
type value
mrr_at_1 42.105
type value
mrr_at_10 50.589
type value
mrr_at_100 51.187000000000005
type value
mrr_at_1000 51.233
type value
mrr_at_3 48.246
type value
mrr_at_5 49.546
type value
ndcg_at_1 40.402
type value
ndcg_at_10 31.009999999999998
type value
ndcg_at_100 28.026
type value
ndcg_at_1000 36.905
type value
ndcg_at_3 35.983
type value
ndcg_at_5 33.764
type value
precision_at_1 42.105
type value
precision_at_10 22.786
type value
precision_at_100 6.916
type value
precision_at_1000 1.981
type value
precision_at_3 33.333
type value
precision_at_5 28.731
type value
recall_at_1 5.353
type value
recall_at_10 15.039
type value
recall_at_100 27.348
type value
recall_at_1000 59.453
type value
recall_at_3 9.792
type value
recall_at_5 11.882
task dataset metrics
type
Retrieval
type name config split revision
nq MTEB NQ default test None
type value
map_at_1 33.852
type value
map_at_10 48.924
type value
map_at_100 49.854
type value
map_at_1000 49.886
type value
map_at_3 44.9
type value
map_at_5 47.387
type value
mrr_at_1 38.035999999999994
type value
mrr_at_10 51.644
type value
mrr_at_100 52.339
type value
mrr_at_1000 52.35999999999999
type value
mrr_at_3 48.421
type value
mrr_at_5 50.468999999999994
type value
ndcg_at_1 38.007000000000005
type value
ndcg_at_10 56.293000000000006
type value
ndcg_at_100 60.167
type value
ndcg_at_1000 60.916000000000004
type value
ndcg_at_3 48.903999999999996
type value
ndcg_at_5 52.978
type value
precision_at_1 38.007000000000005
type value
precision_at_10 9.041
type value
precision_at_100 1.1199999999999999
type value
precision_at_1000 0.11900000000000001
type value
precision_at_3 22.084
type value
precision_at_5 15.608
type value
recall_at_1 33.852
type value
recall_at_10 75.893
type value
recall_at_100 92.589
type value
recall_at_1000 98.153
type value
recall_at_3 56.969
type value
recall_at_5 66.283
task dataset metrics
type
Retrieval
type name config split revision
quora MTEB QuoraRetrieval default test None
type value
map_at_1 69.174
type value
map_at_10 82.891
type value
map_at_100 83.545
type value
map_at_1000 83.56700000000001
type value
map_at_3 79.944
type value
map_at_5 81.812
type value
mrr_at_1 79.67999999999999
type value
mrr_at_10 86.279
type value
mrr_at_100 86.39
type value
mrr_at_1000 86.392
type value
mrr_at_3 85.21
type value
mrr_at_5 85.92999999999999
type value
ndcg_at_1 79.69000000000001
type value
ndcg_at_10 86.929
type value
ndcg_at_100 88.266
type value
ndcg_at_1000 88.428
type value
ndcg_at_3 83.899
type value
ndcg_at_5 85.56700000000001
type value
precision_at_1 79.69000000000001
type value
precision_at_10 13.161000000000001
type value
precision_at_100 1.513
type value
precision_at_1000 0.156
type value
precision_at_3 36.603
type value
precision_at_5 24.138
type value
recall_at_1 69.174
type value
recall_at_10 94.529
type value
recall_at_100 99.15
type value
recall_at_1000 99.925
type value
recall_at_3 85.86200000000001
type value
recall_at_5 90.501
task dataset metrics
type
Clustering
type name config split revision
mteb/reddit-clustering MTEB RedditClustering default test 24640382cdbf8abc73003fb0fa6d111a705499eb
type value
v_measure 39.13064340585255
task dataset metrics
type
Clustering
type name config split revision
mteb/reddit-clustering-p2p MTEB RedditClusteringP2P default test 282350215ef01743dc01b456c7f5241fa8937f16
type value
v_measure 58.97884249325877
task dataset metrics
type
Retrieval
type name config split revision
scidocs MTEB SCIDOCS default test None
type value
map_at_1 3.4680000000000004
type value
map_at_10 7.865
type value
map_at_100 9.332
type value
map_at_1000 9.587
type value
map_at_3 5.800000000000001
type value
map_at_5 6.8790000000000004
type value
mrr_at_1 17.0
type value
mrr_at_10 25.629
type value
mrr_at_100 26.806
type value
mrr_at_1000 26.889000000000003
type value
mrr_at_3 22.8
type value
mrr_at_5 24.26
type value
ndcg_at_1 17.0
type value
ndcg_at_10 13.895
type value
ndcg_at_100 20.491999999999997
type value
ndcg_at_1000 25.759999999999998
type value
ndcg_at_3 13.347999999999999
type value
ndcg_at_5 11.61
type value
precision_at_1 17.0
type value
precision_at_10 7.090000000000001
type value
precision_at_100 1.669
type value
precision_at_1000 0.294
type value
precision_at_3 12.3
type value
precision_at_5 10.02
type value
recall_at_1 3.4680000000000004
type value
recall_at_10 14.363000000000001
type value
recall_at_100 33.875
type value
recall_at_1000 59.711999999999996
type value
recall_at_3 7.483
type value
recall_at_5 10.173
task dataset metrics
type
STS
type name config split revision
mteb/sickr-sts MTEB SICK-R default test a6ea5a8cab320b040a23452cc28066d9beae2cee
type value
cos_sim_pearson 83.04084311714061
type value
cos_sim_spearman 77.51342467443078
type value
euclidean_pearson 80.0321166028479
type value
euclidean_spearman 77.29249114733226
type value
manhattan_pearson 80.03105964262431
type value
manhattan_spearman 77.22373689514794
task dataset metrics
type
STS
type name config split revision
mteb/sts12-sts MTEB STS12 default test a0d554a64d88156834ff5ae9920b964011b16384
type value
cos_sim_pearson 84.1680158034387
type value
cos_sim_spearman 76.55983344071117
type value
euclidean_pearson 79.75266678300143
type value
euclidean_spearman 75.34516823467025
type value
manhattan_pearson 79.75959151517357
type value
manhattan_spearman 75.42330344141912
task dataset metrics
type
STS
type name config split revision
mteb/sts13-sts MTEB STS13 default test 7e90230a92c190f1bf69ae9002b8cea547a64cca
type value
cos_sim_pearson 76.48898993209346
type value
cos_sim_spearman 76.96954120323366
type value
euclidean_pearson 76.94139109279668
type value
euclidean_spearman 76.85860283201711
type value
manhattan_pearson 76.6944095091912
type value
manhattan_spearman 76.61096912972553
task dataset metrics
type
STS
type name config split revision
mteb/sts14-sts MTEB STS14 default test 6031580fec1f6af667f0bd2da0a551cf4f0b2375
type value
cos_sim_pearson 77.85082366246944
type value
cos_sim_spearman 75.52053350101731
type value
euclidean_pearson 77.1165845070926
type value
euclidean_spearman 75.31216065884388
type value
manhattan_pearson 77.06193941833494
type value
manhattan_spearman 75.31003701700112
task dataset metrics
type
STS
type name config split revision
mteb/sts15-sts MTEB STS15 default test ae752c7c21bf194d8b67fd573edf7ae58183cbe3
type value
cos_sim_pearson 86.36305246526497
type value
cos_sim_spearman 87.11704613927415
type value
euclidean_pearson 86.04199125810939
type value
euclidean_spearman 86.51117572414263
type value
manhattan_pearson 86.0805106816633
type value
manhattan_spearman 86.52798366512229
task dataset metrics
type
STS
type name config split revision
mteb/sts16-sts MTEB STS16 default test 4d8694f8f0e0100860b497b999b3dbed754a0513
type value
cos_sim_pearson 82.18536255599724
type value
cos_sim_spearman 83.63377151025418
type value
euclidean_pearson 83.24657467993141
type value
euclidean_spearman 84.02751481993825
type value
manhattan_pearson 83.11941806582371
type value
manhattan_spearman 83.84251281019304
task dataset metrics
type
STS
type name config split revision
mteb/sts17-crosslingual-sts MTEB STS17 (ko-ko) ko-ko test af5e6fb845001ecf41f4c1e033ce921939a2a68d
type value
cos_sim_pearson 78.95816528475514
type value
cos_sim_spearman 78.86607380120462
type value
euclidean_pearson 78.51268699230545
type value
euclidean_spearman 79.11649316502229
type value
manhattan_pearson 78.32367302808157
type value
manhattan_spearman 78.90277699624637
task dataset metrics
type
STS
type name config split revision
mteb/sts17-crosslingual-sts MTEB STS17 (ar-ar) ar-ar test af5e6fb845001ecf41f4c1e033ce921939a2a68d
type value
cos_sim_pearson 72.89126914997624
type value
cos_sim_spearman 73.0296921832678
type value
euclidean_pearson 71.50385903677738
type value
euclidean_spearman 73.13368899716289
type value
manhattan_pearson 71.47421463379519
type value
manhattan_spearman 73.03383242946575
task dataset metrics
type
STS
type name config split revision
mteb/sts17-crosslingual-sts MTEB STS17 (en-ar) en-ar test af5e6fb845001ecf41f4c1e033ce921939a2a68d
type value
cos_sim_pearson 59.22923684492637
type value
cos_sim_spearman 57.41013211368396
type value
euclidean_pearson 61.21107388080905
type value
euclidean_spearman 60.07620768697254
type value
manhattan_pearson 59.60157142786555
type value
manhattan_spearman 59.14069604103739
task dataset metrics
type
STS
type name config split revision
mteb/sts17-crosslingual-sts MTEB STS17 (en-de) en-de test af5e6fb845001ecf41f4c1e033ce921939a2a68d
type value
cos_sim_pearson 76.24345978774299
type value
cos_sim_spearman 77.24225743830719
type value
euclidean_pearson 76.66226095469165
type value
euclidean_spearman 77.60708820493146
type value
manhattan_pearson 76.05303324760429
type value
manhattan_spearman 76.96353149912348
task dataset metrics
type
STS
type name config split revision
mteb/sts17-crosslingual-sts MTEB STS17 (en-en) en-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d
type value
cos_sim_pearson 85.50879160160852
type value
cos_sim_spearman 86.43594662965224
type value
euclidean_pearson 86.06846012826577
type value
euclidean_spearman 86.02041395794136
type value
manhattan_pearson 86.10916255616904
type value
manhattan_spearman 86.07346068198953
task dataset metrics
type
STS
type name config split revision
mteb/sts17-crosslingual-sts MTEB STS17 (en-tr) en-tr test af5e6fb845001ecf41f4c1e033ce921939a2a68d
type value
cos_sim_pearson 58.39803698977196
type value
cos_sim_spearman 55.96910950423142
type value
euclidean_pearson 58.17941175613059
type value
euclidean_spearman 55.03019330522745
type value
manhattan_pearson 57.333358138183286
type value
manhattan_spearman 54.04614023149965
task dataset metrics
type
STS
type name config split revision
mteb/sts17-crosslingual-sts MTEB STS17 (es-en) es-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d
type value
cos_sim_pearson 70.98304089637197
type value
cos_sim_spearman 72.44071656215888
type value
euclidean_pearson 72.19224359033983
type value
euclidean_spearman 73.89871188913025
type value
manhattan_pearson 71.21098311547406
type value
manhattan_spearman 72.93405764824821
task dataset metrics
type
STS
type name config split revision
mteb/sts17-crosslingual-sts MTEB STS17 (es-es) es-es test af5e6fb845001ecf41f4c1e033ce921939a2a68d
type value
cos_sim_pearson 85.99792397466308
type value
cos_sim_spearman 84.83824377879495
type value
euclidean_pearson 85.70043288694438
type value
euclidean_spearman 84.70627558703686
type value
manhattan_pearson 85.89570850150801
type value
manhattan_spearman 84.95806105313007
task dataset metrics
type
STS
type name config split revision
mteb/sts17-crosslingual-sts MTEB STS17 (fr-en) fr-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d
type value
cos_sim_pearson 72.21850322994712
type value
cos_sim_spearman 72.28669398117248
type value
euclidean_pearson 73.40082510412948
type value
euclidean_spearman 73.0326539281865
type value
manhattan_pearson 71.8659633964841
type value
manhattan_spearman 71.57817425823303
task dataset metrics
type
STS
type name config split revision
mteb/sts17-crosslingual-sts MTEB STS17 (it-en) it-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d
type value
cos_sim_pearson 75.80921368595645
type value
cos_sim_spearman 77.33209091229315
type value
euclidean_pearson 76.53159540154829
type value
euclidean_spearman 78.17960842810093
type value
manhattan_pearson 76.13530186637601
type value
manhattan_spearman 78.00701437666875
task dataset metrics
type
STS
type name config split revision
mteb/sts17-crosslingual-sts MTEB STS17 (nl-en) nl-en test af5e6fb845001ecf41f4c1e033ce921939a2a68d
type value
cos_sim_pearson 74.74980608267349
type value
cos_sim_spearman 75.37597374318821
type value
euclidean_pearson 74.90506081911661
type value
euclidean_spearman 75.30151613124521
type value
manhattan_pearson 74.62642745918002
type value
manhattan_spearman 75.18619716592303
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (en) en test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 59.632662289205584
type value
cos_sim_spearman 60.938543391610914
type value
euclidean_pearson 62.113200529767056
type value
euclidean_spearman 61.410312633261164
type value
manhattan_pearson 61.75494698945686
type value
manhattan_spearman 60.92726195322362
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (de) de test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 45.283470551557244
type value
cos_sim_spearman 53.44833015864201
type value
euclidean_pearson 41.17892011120893
type value
euclidean_spearman 53.81441383126767
type value
manhattan_pearson 41.17482200420659
type value
manhattan_spearman 53.82180269276363
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (es) es test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 60.5069165306236
type value
cos_sim_spearman 66.87803259033826
type value
euclidean_pearson 63.5428979418236
type value
euclidean_spearman 66.9293576586897
type value
manhattan_pearson 63.59789526178922
type value
manhattan_spearman 66.86555009875066
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (pl) pl test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 28.23026196280264
type value
cos_sim_spearman 35.79397812652861
type value
euclidean_pearson 17.828102102767353
type value
euclidean_spearman 35.721501145568894
type value
manhattan_pearson 17.77134274219677
type value
manhattan_spearman 35.98107902846267
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (tr) tr test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 56.51946541393812
type value
cos_sim_spearman 63.714686006214485
type value
euclidean_pearson 58.32104651305898
type value
euclidean_spearman 62.237110895702216
type value
manhattan_pearson 58.579416468759185
type value
manhattan_spearman 62.459738981727
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (ar) ar test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 48.76009839569795
type value
cos_sim_spearman 56.65188431953149
type value
euclidean_pearson 50.997682160915595
type value
euclidean_spearman 55.99910008818135
type value
manhattan_pearson 50.76220659606342
type value
manhattan_spearman 55.517347595391456
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (ru) ru test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 51.232731157702425
type value
cos_sim_spearman 59.89531877658345
type value
euclidean_pearson 49.937914570348376
type value
euclidean_spearman 60.220905659334036
type value
manhattan_pearson 50.00987996844193
type value
manhattan_spearman 60.081341480977926
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (zh) zh test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 54.717524559088005
type value
cos_sim_spearman 66.83570886252286
type value
euclidean_pearson 58.41338625505467
type value
euclidean_spearman 66.68991427704938
type value
manhattan_pearson 58.78638572916807
type value
manhattan_spearman 66.58684161046335
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (fr) fr test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 73.2962042954962
type value
cos_sim_spearman 76.58255504852025
type value
euclidean_pearson 75.70983192778257
type value
euclidean_spearman 77.4547684870542
type value
manhattan_pearson 75.75565853870485
type value
manhattan_spearman 76.90208974949428
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (de-en) de-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 54.47396266924846
type value
cos_sim_spearman 56.492267162048606
type value
euclidean_pearson 55.998505203070195
type value
euclidean_spearman 56.46447012960222
type value
manhattan_pearson 54.873172394430995
type value
manhattan_spearman 56.58111534551218
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (es-en) es-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 69.87177267688686
type value
cos_sim_spearman 74.57160943395763
type value
euclidean_pearson 70.88330406826788
type value
euclidean_spearman 74.29767636038422
type value
manhattan_pearson 71.38245248369536
type value
manhattan_spearman 74.53102232732175
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (it) it test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 72.80225656959544
type value
cos_sim_spearman 76.52646173725735
type value
euclidean_pearson 73.95710720200799
type value
euclidean_spearman 76.54040031984111
type value
manhattan_pearson 73.89679971946774
type value
manhattan_spearman 76.60886958161574
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (pl-en) pl-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 70.70844249898789
type value
cos_sim_spearman 72.68571783670241
type value
euclidean_pearson 72.38800772441031
type value
euclidean_spearman 72.86804422703312
type value
manhattan_pearson 71.29840508203515
type value
manhattan_spearman 71.86264441749513
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (zh-en) zh-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 58.647478923935694
type value
cos_sim_spearman 63.74453623540931
type value
euclidean_pearson 59.60138032437505
type value
euclidean_spearman 63.947930832166065
type value
manhattan_pearson 58.59735509491861
type value
manhattan_spearman 62.082503844627404
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (es-it) es-it test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 65.8722516867162
type value
cos_sim_spearman 71.81208592523012
type value
euclidean_pearson 67.95315252165956
type value
euclidean_spearman 73.00749822046009
type value
manhattan_pearson 68.07884688638924
type value
manhattan_spearman 72.34210325803069
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (de-fr) de-fr test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 54.5405814240949
type value
cos_sim_spearman 60.56838649023775
type value
euclidean_pearson 53.011731611314104
type value
euclidean_spearman 58.533194841668426
type value
manhattan_pearson 53.623067729338494
type value
manhattan_spearman 58.018756154446926
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (de-pl) de-pl test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 13.611046866216112
type value
cos_sim_spearman 28.238192909158492
type value
euclidean_pearson 22.16189199885129
type value
euclidean_spearman 35.012895679076564
type value
manhattan_pearson 21.969771178698387
type value
manhattan_spearman 32.456985088607475
task dataset metrics
type
STS
type name config split revision
mteb/sts22-crosslingual-sts MTEB STS22 (fr-pl) fr-pl test 6d1ba47164174a496b7fa5d3569dae26a6813b80
type value
cos_sim_pearson 74.58077407011655
type value
cos_sim_spearman 84.51542547285167
type value
euclidean_pearson 74.64613843596234
type value
euclidean_spearman 84.51542547285167
type value
manhattan_pearson 75.15335973101396
type value
manhattan_spearman 84.51542547285167
task dataset metrics
type
STS
type name config split revision
mteb/stsbenchmark-sts MTEB STSBenchmark default test b0fddb56ed78048fa8b90373c8a3cfc37b684831
type value
cos_sim_pearson 82.0739825531578
type value
cos_sim_spearman 84.01057479311115
type value
euclidean_pearson 83.85453227433344
type value
euclidean_spearman 84.01630226898655
type value
manhattan_pearson 83.75323603028978
type value
manhattan_spearman 83.89677983727685
task dataset metrics
type
Reranking
type name config split revision
mteb/scidocs-reranking MTEB SciDocsRR default test d3c5e1fc0b855ab6097bf1cda04dd73947d7caab
type value
map 78.12945623123957
type value
mrr 93.87738713719106
task dataset metrics
type
Retrieval
type name config split revision
scifact MTEB SciFact default test None
type value
map_at_1 52.983000000000004
type value
map_at_10 62.946000000000005
type value
map_at_100 63.514
type value
map_at_1000 63.554
type value
map_at_3 60.183
type value
map_at_5 61.672000000000004
type value
mrr_at_1 55.667
type value
mrr_at_10 64.522
type value
mrr_at_100 64.957
type value
mrr_at_1000 64.995
type value
mrr_at_3 62.388999999999996
type value
mrr_at_5 63.639
type value
ndcg_at_1 55.667
type value
ndcg_at_10 67.704
type value
ndcg_at_100 70.299
type value
ndcg_at_1000 71.241
type value
ndcg_at_3 62.866
type value
ndcg_at_5 65.16999999999999
type value
precision_at_1 55.667
type value
precision_at_10 9.033
type value
precision_at_100 1.053
type value
precision_at_1000 0.11299999999999999
type value
precision_at_3 24.444
type value
precision_at_5 16.133
type value
recall_at_1 52.983000000000004
type value
recall_at_10 80.656
type value
recall_at_100 92.5
type value
recall_at_1000 99.667
type value
recall_at_3 67.744
type value
recall_at_5 73.433
task dataset metrics
type
PairClassification
type name config split revision
mteb/sprintduplicatequestions-pairclassification MTEB SprintDuplicateQuestions default test d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46
type value
cos_sim_accuracy 99.72772277227723
type value
cos_sim_ap 92.17845897992215
type value
cos_sim_f1 85.9746835443038
type value
cos_sim_precision 87.07692307692308
type value
cos_sim_recall 84.89999999999999
type value
dot_accuracy 99.3039603960396
type value
dot_ap 60.70244020124878
type value
dot_f1 59.92742353551063
type value
dot_precision 62.21743810548978
type value
dot_recall 57.8
type value
euclidean_accuracy 99.71683168316832
type value
euclidean_ap 91.53997039964659
type value
euclidean_f1 84.88372093023257
type value
euclidean_precision 90.02242152466367
type value
euclidean_recall 80.30000000000001
type value
manhattan_accuracy 99.72376237623763
type value
manhattan_ap 91.80756777790289
type value
manhattan_f1 85.48468106479157
type value
manhattan_precision 85.8728557013118
type value
manhattan_recall 85.1
type value
max_accuracy 99.72772277227723
type value
max_ap 92.17845897992215
type value
max_f1 85.9746835443038
task dataset metrics
type
Clustering
type name config split revision
mteb/stackexchange-clustering MTEB StackExchangeClustering default test 6cbc1f7b2bc0622f2e39d2c77fa502909748c259
type value
v_measure 53.52464042600003
task dataset metrics
type
Clustering
type name config split revision
mteb/stackexchange-clustering-p2p MTEB StackExchangeClusteringP2P default test 815ca46b2622cec33ccafc3735d572c266efdb44
type value
v_measure 32.071631948736
task dataset metrics
type
Reranking
type name config split revision
mteb/stackoverflowdupquestions-reranking MTEB StackOverflowDupQuestions default test e185fbe320c72810689fc5848eb6114e1ef5ec69
type value
map 49.19552407604654
type value
mrr 49.95269130379425
task dataset metrics
type
Summarization
type name config split revision
mteb/summeval MTEB SummEval default test cda12ad7615edc362dbf25a00fdd61d3b1eaf93c
type value
cos_sim_pearson 29.345293033095427
type value
cos_sim_spearman 29.976931423258403
type value
dot_pearson 27.047078008958408
type value
dot_spearman 27.75894368380218
task dataset metrics
type
Retrieval
type name config split revision
trec-covid MTEB TRECCOVID default test None
type value
map_at_1 0.22
type value
map_at_10 1.706
type value
map_at_100 9.634
type value
map_at_1000 23.665
type value
map_at_3 0.5950000000000001
type value
map_at_5 0.95
type value
mrr_at_1 86.0
type value
mrr_at_10 91.8
type value
mrr_at_100 91.8
type value
mrr_at_1000 91.8
type value
mrr_at_3 91.0
type value
mrr_at_5 91.8
type value
ndcg_at_1 80.0
type value
ndcg_at_10 72.573
type value
ndcg_at_100 53.954
type value
ndcg_at_1000 47.760999999999996
type value
ndcg_at_3 76.173
type value
ndcg_at_5 75.264
type value
precision_at_1 86.0
type value
precision_at_10 76.4
type value
precision_at_100 55.50000000000001
type value
precision_at_1000 21.802
type value
precision_at_3 81.333
type value
precision_at_5 80.4
type value
recall_at_1 0.22
type value
recall_at_10 1.925
type value
recall_at_100 12.762
type value
recall_at_1000 44.946000000000005
type value
recall_at_3 0.634
type value
recall_at_5 1.051
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (sqi-eng) sqi-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 91.0
type value
f1 88.55666666666666
type value
precision 87.46166666666667
type value
recall 91.0
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (fry-eng) fry-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 57.22543352601156
type value
f1 51.03220478943021
type value
precision 48.8150289017341
type value
recall 57.22543352601156
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (kur-eng) kur-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 46.58536585365854
type value
f1 39.66870798578116
type value
precision 37.416085946573745
type value
recall 46.58536585365854
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (tur-eng) tur-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 89.7
type value
f1 86.77999999999999
type value
precision 85.45333333333332
type value
recall 89.7
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (deu-eng) deu-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 97.39999999999999
type value
f1 96.58333333333331
type value
precision 96.2
type value
recall 97.39999999999999
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (nld-eng) nld-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 92.4
type value
f1 90.3
type value
precision 89.31666666666668
type value
recall 92.4
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ron-eng) ron-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 86.9
type value
f1 83.67190476190476
type value
precision 82.23333333333332
type value
recall 86.9
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ang-eng) ang-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 50.0
type value
f1 42.23229092632078
type value
precision 39.851634683724235
type value
recall 50.0
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ido-eng) ido-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 76.3
type value
f1 70.86190476190477
type value
precision 68.68777777777777
type value
recall 76.3
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (jav-eng) jav-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 57.073170731707314
type value
f1 50.658958927251604
type value
precision 48.26480836236933
type value
recall 57.073170731707314
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (isl-eng) isl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 68.2
type value
f1 62.156507936507936
type value
precision 59.84964285714286
type value
recall 68.2
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (slv-eng) slv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 77.52126366950182
type value
f1 72.8496210148701
type value
precision 70.92171498003819
type value
recall 77.52126366950182
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (cym-eng) cym-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 70.78260869565217
type value
f1 65.32422360248447
type value
precision 63.063067367415194
type value
recall 70.78260869565217
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (kaz-eng) kaz-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 78.43478260869566
type value
f1 73.02608695652172
type value
precision 70.63768115942028
type value
recall 78.43478260869566
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (est-eng) est-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 60.9
type value
f1 55.309753694581275
type value
precision 53.130476190476195
type value
recall 60.9
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (heb-eng) heb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 72.89999999999999
type value
f1 67.92023809523809
type value
precision 65.82595238095237
type value
recall 72.89999999999999
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (gla-eng) gla-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 46.80337756332931
type value
f1 39.42174900558496
type value
precision 36.97101116280851
type value
recall 46.80337756332931
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (mar-eng) mar-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 89.8
type value
f1 86.79
type value
precision 85.375
type value
recall 89.8
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (lat-eng) lat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 47.199999999999996
type value
f1 39.95484348984349
type value
precision 37.561071428571424
type value
recall 47.199999999999996
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (bel-eng) bel-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 87.8
type value
f1 84.68190476190475
type value
precision 83.275
type value
recall 87.8
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (pms-eng) pms-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 48.76190476190476
type value
f1 42.14965986394558
type value
precision 39.96743626743626
type value
recall 48.76190476190476
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (gle-eng) gle-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 66.10000000000001
type value
f1 59.58580086580086
type value
precision 57.150238095238095
type value
recall 66.10000000000001
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (pes-eng) pes-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 87.3
type value
f1 84.0
type value
precision 82.48666666666666
type value
recall 87.3
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (nob-eng) nob-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 90.4
type value
f1 87.79523809523809
type value
precision 86.6
type value
recall 90.4
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (bul-eng) bul-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 87.0
type value
f1 83.81
type value
precision 82.36666666666666
type value
recall 87.0
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (cbk-eng) cbk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 63.9
type value
f1 57.76533189033189
type value
precision 55.50595238095239
type value
recall 63.9
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (hun-eng) hun-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 76.1
type value
f1 71.83690476190478
type value
precision 70.04928571428573
type value
recall 76.1
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (uig-eng) uig-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 66.3
type value
f1 59.32626984126984
type value
precision 56.62535714285713
type value
recall 66.3
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (rus-eng) rus-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 90.60000000000001
type value
f1 87.96333333333334
type value
precision 86.73333333333333
type value
recall 90.60000000000001
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (spa-eng) spa-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 93.10000000000001
type value
f1 91.10000000000001
type value
precision 90.16666666666666
type value
recall 93.10000000000001
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (hye-eng) hye-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 85.71428571428571
type value
f1 82.29142600436403
type value
precision 80.8076626877166
type value
recall 85.71428571428571
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (tel-eng) tel-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 88.88888888888889
type value
f1 85.7834757834758
type value
precision 84.43732193732193
type value
recall 88.88888888888889
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (afr-eng) afr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 88.5
type value
f1 85.67190476190476
type value
precision 84.43333333333332
type value
recall 88.5
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (mon-eng) mon-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 82.72727272727273
type value
f1 78.21969696969695
type value
precision 76.18181818181819
type value
recall 82.72727272727273
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (arz-eng) arz-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 61.0062893081761
type value
f1 55.13976240391334
type value
precision 52.92112499659669
type value
recall 61.0062893081761
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (hrv-eng) hrv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 89.5
type value
f1 86.86666666666666
type value
precision 85.69166666666668
type value
recall 89.5
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (nov-eng) nov-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 73.54085603112841
type value
f1 68.56031128404669
type value
precision 66.53047989623866
type value
recall 73.54085603112841
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (gsw-eng) gsw-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 43.58974358974359
type value
f1 36.45299145299145
type value
precision 33.81155881155882
type value
recall 43.58974358974359
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (nds-eng) nds-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 59.599999999999994
type value
f1 53.264689754689755
type value
precision 50.869166666666665
type value
recall 59.599999999999994
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ukr-eng) ukr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 85.2
type value
f1 81.61666666666665
type value
precision 80.02833333333335
type value
recall 85.2
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (uzb-eng) uzb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 63.78504672897196
type value
f1 58.00029669188548
type value
precision 55.815809968847354
type value
recall 63.78504672897196
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (lit-eng) lit-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 66.5
type value
f1 61.518333333333345
type value
precision 59.622363699102834
type value
recall 66.5
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ina-eng) ina-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 88.6
type value
f1 85.60222222222221
type value
precision 84.27916666666665
type value
recall 88.6
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (lfn-eng) lfn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 58.699999999999996
type value
f1 52.732375957375965
type value
precision 50.63214035964035
type value
recall 58.699999999999996
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (zsm-eng) zsm-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 92.10000000000001
type value
f1 89.99666666666667
type value
precision 89.03333333333333
type value
recall 92.10000000000001
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ita-eng) ita-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 90.10000000000001
type value
f1 87.55666666666667
type value
precision 86.36166666666668
type value
recall 90.10000000000001
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (cmn-eng) cmn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 91.4
type value
f1 88.89000000000001
type value
precision 87.71166666666666
type value
recall 91.4
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (lvs-eng) lvs-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 65.7
type value
f1 60.67427750410509
type value
precision 58.71785714285714
type value
recall 65.7
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (glg-eng) glg-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 85.39999999999999
type value
f1 81.93190476190475
type value
precision 80.37833333333333
type value
recall 85.39999999999999
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ceb-eng) ceb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 47.833333333333336
type value
f1 42.006625781625786
type value
precision 40.077380952380956
type value
recall 47.833333333333336
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (bre-eng) bre-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 10.4
type value
f1 8.24465007215007
type value
precision 7.664597069597071
type value
recall 10.4
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ben-eng) ben-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 82.6
type value
f1 77.76333333333334
type value
precision 75.57833333333332
type value
recall 82.6
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (swg-eng) swg-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 52.67857142857143
type value
f1 44.302721088435376
type value
precision 41.49801587301587
type value
recall 52.67857142857143
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (arq-eng) arq-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 28.3205268935236
type value
f1 22.426666605171157
type value
precision 20.685900116470915
type value
recall 28.3205268935236
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (kab-eng) kab-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 22.7
type value
f1 17.833970473970474
type value
precision 16.407335164835164
type value
recall 22.7
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (fra-eng) fra-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 92.2
type value
f1 89.92999999999999
type value
precision 88.87
type value
recall 92.2
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (por-eng) por-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 91.4
type value
f1 89.25
type value
precision 88.21666666666667
type value
recall 91.4
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (tat-eng) tat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 69.19999999999999
type value
f1 63.38269841269841
type value
precision 61.14773809523809
type value
recall 69.19999999999999
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (oci-eng) oci-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 48.8
type value
f1 42.839915639915645
type value
precision 40.770287114845935
type value
recall 48.8
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (pol-eng) pol-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 88.8
type value
f1 85.90666666666668
type value
precision 84.54166666666666
type value
recall 88.8
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (war-eng) war-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 46.6
type value
f1 40.85892920804686
type value
precision 38.838223114604695
type value
recall 46.6
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (aze-eng) aze-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 84.0
type value
f1 80.14190476190475
type value
precision 78.45333333333333
type value
recall 84.0
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (vie-eng) vie-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 90.5
type value
f1 87.78333333333333
type value
precision 86.5
type value
recall 90.5
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (nno-eng) nno-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 74.5
type value
f1 69.48397546897547
type value
precision 67.51869047619049
type value
recall 74.5
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (cha-eng) cha-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 32.846715328467155
type value
f1 27.828177499710343
type value
precision 26.63451511991658
type value
recall 32.846715328467155
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (mhr-eng) mhr-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 8.0
type value
f1 6.07664116764988
type value
precision 5.544177607179943
type value
recall 8.0
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (dan-eng) dan-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 87.6
type value
f1 84.38555555555554
type value
precision 82.91583333333334
type value
recall 87.6
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ell-eng) ell-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 87.5
type value
f1 84.08333333333331
type value
precision 82.47333333333333
type value
recall 87.5
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (amh-eng) amh-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 80.95238095238095
type value
f1 76.13095238095238
type value
precision 74.05753968253967
type value
recall 80.95238095238095
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (pam-eng) pam-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 8.799999999999999
type value
f1 6.971422975172975
type value
precision 6.557814916172301
type value
recall 8.799999999999999
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (hsb-eng) hsb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 44.099378881987576
type value
f1 37.01649742022413
type value
precision 34.69420618488942
type value
recall 44.099378881987576
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (srp-eng) srp-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 84.3
type value
f1 80.32666666666667
type value
precision 78.60666666666665
type value
recall 84.3
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (epo-eng) epo-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 92.5
type value
f1 90.49666666666666
type value
precision 89.56666666666668
type value
recall 92.5
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (kzj-eng) kzj-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 10.0
type value
f1 8.268423529875141
type value
precision 7.878118605532398
type value
recall 10.0
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (awa-eng) awa-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 79.22077922077922
type value
f1 74.27128427128426
type value
precision 72.28715728715729
type value
recall 79.22077922077922
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (fao-eng) fao-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 65.64885496183206
type value
f1 58.87495456197747
type value
precision 55.992366412213734
type value
recall 65.64885496183206
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (mal-eng) mal-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 96.06986899563319
type value
f1 94.78408539543909
type value
precision 94.15332362930616
type value
recall 96.06986899563319
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ile-eng) ile-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 77.2
type value
f1 71.72571428571428
type value
precision 69.41000000000001
type value
recall 77.2
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (bos-eng) bos-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 86.4406779661017
type value
f1 83.2391713747646
type value
precision 81.74199623352166
type value
recall 86.4406779661017
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (cor-eng) cor-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 8.4
type value
f1 6.017828743398003
type value
precision 5.4829865484756795
type value
recall 8.4
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (cat-eng) cat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 83.5
type value
f1 79.74833333333333
type value
precision 78.04837662337664
type value
recall 83.5
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (eus-eng) eus-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 60.4
type value
f1 54.467301587301584
type value
precision 52.23242424242424
type value
recall 60.4
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (yue-eng) yue-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 74.9
type value
f1 69.68699134199134
type value
precision 67.59873015873016
type value
recall 74.9
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (swe-eng) swe-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 88.0
type value
f1 84.9652380952381
type value
precision 83.66166666666666
type value
recall 88.0
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (dtp-eng) dtp-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 9.1
type value
f1 7.681244588744588
type value
precision 7.370043290043291
type value
recall 9.1
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (kat-eng) kat-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 80.9651474530831
type value
f1 76.84220605132133
type value
precision 75.19606398962966
type value
recall 80.9651474530831
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (jpn-eng) jpn-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 86.9
type value
f1 83.705
type value
precision 82.3120634920635
type value
recall 86.9
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (csb-eng) csb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 29.64426877470356
type value
f1 23.98763072676116
type value
precision 22.506399397703746
type value
recall 29.64426877470356
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (xho-eng) xho-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 70.4225352112676
type value
f1 62.84037558685445
type value
precision 59.56572769953053
type value
recall 70.4225352112676
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (orv-eng) orv-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 19.64071856287425
type value
f1 15.125271011207756
type value
precision 13.865019261197494
type value
recall 19.64071856287425
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ind-eng) ind-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 90.2
type value
f1 87.80666666666666
type value
precision 86.70833333333331
type value
recall 90.2
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (tuk-eng) tuk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 23.15270935960591
type value
f1 18.407224958949097
type value
precision 16.982385430661292
type value
recall 23.15270935960591
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (max-eng) max-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 55.98591549295775
type value
f1 49.94718309859154
type value
precision 47.77864154624717
type value
recall 55.98591549295775
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (swh-eng) swh-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 73.07692307692307
type value
f1 66.74358974358974
type value
precision 64.06837606837607
type value
recall 73.07692307692307
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (hin-eng) hin-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 94.89999999999999
type value
f1 93.25
type value
precision 92.43333333333332
type value
recall 94.89999999999999
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (dsb-eng) dsb-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 37.78705636743215
type value
f1 31.63899658680452
type value
precision 29.72264397629742
type value
recall 37.78705636743215
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ber-eng) ber-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 21.6
type value
f1 16.91697302697303
type value
precision 15.71225147075147
type value
recall 21.6
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (tam-eng) tam-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 85.01628664495115
type value
f1 81.38514037536838
type value
precision 79.83170466883823
type value
recall 85.01628664495115
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (slk-eng) slk-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 83.39999999999999
type value
f1 79.96380952380952
type value
precision 78.48333333333333
type value
recall 83.39999999999999
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (tgl-eng) tgl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 83.2
type value
f1 79.26190476190476
type value
precision 77.58833333333334
type value
recall 83.2
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ast-eng) ast-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 75.59055118110236
type value
f1 71.66854143232096
type value
precision 70.30183727034121
type value
recall 75.59055118110236
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (mkd-eng) mkd-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 65.5
type value
f1 59.26095238095238
type value
precision 56.81909090909092
type value
recall 65.5
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (khm-eng) khm-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 55.26315789473685
type value
f1 47.986523325858506
type value
precision 45.33950006595436
type value
recall 55.26315789473685
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ces-eng) ces-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 82.89999999999999
type value
f1 78.835
type value
precision 77.04761904761905
type value
recall 82.89999999999999
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (tzl-eng) tzl-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 43.269230769230774
type value
f1 36.20421245421245
type value
precision 33.57371794871795
type value
recall 43.269230769230774
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (urd-eng) urd-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 88.0
type value
f1 84.70666666666666
type value
precision 83.23166666666665
type value
recall 88.0
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (ara-eng) ara-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 77.4
type value
f1 72.54666666666667
type value
precision 70.54318181818181
type value
recall 77.4
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (kor-eng) kor-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 78.60000000000001
type value
f1 74.1588888888889
type value
precision 72.30250000000001
type value
recall 78.60000000000001
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (yid-eng) yid-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 72.40566037735849
type value
f1 66.82587328813744
type value
precision 64.75039308176099
type value
recall 72.40566037735849
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (fin-eng) fin-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 73.8
type value
f1 68.56357142857144
type value
precision 66.3178822055138
type value
recall 73.8
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (tha-eng) tha-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 91.78832116788321
type value
f1 89.3552311435523
type value
precision 88.20559610705597
type value
recall 91.78832116788321
task dataset metrics
type
BitextMining
type name config split revision
mteb/tatoeba-bitext-mining MTEB Tatoeba (wuu-eng) wuu-eng test 9080400076fbadbb4c4dcb136ff4eddc40b42553
type value
accuracy 74.3
type value
f1 69.05085581085581
type value
precision 66.955
type value
recall 74.3
task dataset metrics
type
Retrieval
type name config split revision
webis-touche2020 MTEB Touche2020 default test None
type value
map_at_1 2.896
type value
map_at_10 8.993
type value
map_at_100 14.133999999999999
type value
map_at_1000 15.668000000000001
type value
map_at_3 5.862
type value
map_at_5 7.17
type value
mrr_at_1 34.694
type value
mrr_at_10 42.931000000000004
type value
mrr_at_100 44.81
type value
mrr_at_1000 44.81
type value
mrr_at_3 38.435
type value
mrr_at_5 41.701
type value
ndcg_at_1 31.633
type value
ndcg_at_10 21.163
type value
ndcg_at_100 33.306000000000004
type value
ndcg_at_1000 45.275999999999996
type value
ndcg_at_3 25.685999999999996
type value
ndcg_at_5 23.732
type value
precision_at_1 34.694
type value
precision_at_10 17.755000000000003
type value
precision_at_100 6.938999999999999
type value
precision_at_1000 1.48
type value
precision_at_3 25.85
type value
precision_at_5 23.265
type value
recall_at_1 2.896
type value
recall_at_10 13.333999999999998
type value
recall_at_100 43.517
type value
recall_at_1000 79.836
type value
recall_at_3 6.306000000000001
type value
recall_at_5 8.825
task dataset metrics
type
Classification
type name config split revision
mteb/toxic_conversations_50k MTEB ToxicConversationsClassification default test d7c0de2777da35d6aae2200a62c6e0e5af397c4c
type value
accuracy 69.3874
type value
ap 13.829909072469423
type value
f1 53.54534203543492
task dataset metrics
type
Classification
type name config split revision
mteb/tweet_sentiment_extraction MTEB TweetSentimentExtractionClassification default test d604517c81ca91fe16a244d1248fc021f9ecee7a
type value
accuracy 62.62026032823995
type value
f1 62.85251350485221
task dataset metrics
type
Clustering
type name config split revision
mteb/twentynewsgroups-clustering MTEB TwentyNewsgroupsClustering default test 6125ec4e24fa026cec8a478383ee943acfbd5449
type value
v_measure 33.21527881409797
task dataset metrics
type
PairClassification
type name config split revision
mteb/twittersemeval2015-pairclassification MTEB TwitterSemEval2015 default test 70970daeab8776df92f5ea462b6173c0b46fd2d1
type value
cos_sim_accuracy 84.97943613280086
type value
cos_sim_ap 70.75454316885921
type value
cos_sim_f1 65.38274012676743
type value
cos_sim_precision 60.761214318078835
type value
cos_sim_recall 70.76517150395777
type value
dot_accuracy 79.0546581629612
type value
dot_ap 47.3197121792147
type value
dot_f1 49.20106524633821
type value
dot_precision 42.45499808502489
type value
dot_recall 58.49604221635884
type value
euclidean_accuracy 85.08076533349228
type value
euclidean_ap 70.95016106374474
type value
euclidean_f1 65.43987900176455
type value
euclidean_precision 62.64478764478765
type value
euclidean_recall 68.49604221635884
type value
manhattan_accuracy 84.93771234428085
type value
manhattan_ap 70.63668388755362
type value
manhattan_f1 65.23895401262398
type value
manhattan_precision 56.946084218811485
type value
manhattan_recall 76.35883905013192
type value
max_accuracy 85.08076533349228
type value
max_ap 70.95016106374474
type value
max_f1 65.43987900176455
task dataset metrics
type
PairClassification
type name config split revision
mteb/twitterurlcorpus-pairclassification MTEB TwitterURLCorpus default test 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
type value
cos_sim_accuracy 88.69096130709822
type value
cos_sim_ap 84.82526278228542
type value
cos_sim_f1 77.65485060585536
type value
cos_sim_precision 75.94582658619167
type value
cos_sim_recall 79.44256236526024
type value
dot_accuracy 80.97954748321496
type value
dot_ap 64.81642914145866
type value
dot_f1 60.631996987229975
type value
dot_precision 54.5897293631712
type value
dot_recall 68.17831844779796
type value
euclidean_accuracy 88.6987231730508
type value
euclidean_ap 84.80003825477253
type value
euclidean_f1 77.67194179854496
type value
euclidean_precision 75.7128235122094
type value
euclidean_recall 79.73514012935017
type value
manhattan_accuracy 88.62692591298949
type value
manhattan_ap 84.80451408255276
type value
manhattan_f1 77.69888949572183
type value
manhattan_precision 73.70311528631622
type value
manhattan_recall 82.15275639051433
type value
max_accuracy 88.6987231730508
type value
max_ap 84.82526278228542
type value
max_f1 77.69888949572183
multilingual
af
am
ar
as
az
be
bg
bn
br
bs
ca
cs
cy
da
de
el
en
eo
es
et
eu
fa
fi
fr
fy
ga
gd
gl
gu
ha
he
hi
hr
hu
hy
id
is
it
ja
jv
ka
kk
km
kn
ko
ku
ky
la
lo
lt
lv
mg
mk
ml
mn
mr
ms
my
ne
nl
no
om
or
pa
pl
ps
pt
ro
ru
sa
sd
si
sk
sl
so
sq
sr
su
sv
sw
ta
te
th
tl
tr
ug
uk
ur
uz
vi
xh
yi
zh
mit

Multilingual-E5-small

Text Embeddings by Weakly-Supervised Contrastive Pre-training. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei, arXiv 2022

This model has 12 layers and the embedding size is 384.

Usage

Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset.

import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def average_pool(last_hidden_states: Tensor,
                 attention_mask: Tensor) -> Tensor:
    last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
    return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]


# Each input text should start with "query: " or "passage: ", even for non-English texts.
# For tasks other than retrieval, you can simply use the "query: " prefix.
input_texts = ['query: how much protein should a female eat',
               'query: 南瓜的家常做法',
               "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
               "passage: 1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"]

tokenizer = AutoTokenizer.from_pretrained('intfloat/multilingual-e5-small')
model = AutoModel.from_pretrained('intfloat/multilingual-e5-small')

# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')

outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())

Supported Languages

This model is initialized from microsoft/Multilingual-MiniLM-L12-H384 and continually trained on a mixture of multilingual datasets. It supports 100 languages from xlm-roberta, but low-resource languages may see performance degradation.

Training Details

Initialization: microsoft/Multilingual-MiniLM-L12-H384

First stage: contrastive pre-training with weak supervision

Dataset Weak supervision # of text pairs
Filtered mC4 (title, page content) 1B
CC News (title, news content) 400M
NLLB translation pairs 2.4B
Wikipedia (hierarchical section title, passage) 150M
Filtered Reddit (comment, response) 800M
S2ORC (title, abstract) and citation pairs 100M
Stackexchange (question, answer) 50M
xP3 (input prompt, response) 80M
Miscellaneous unsupervised SBERT data - 10M

Second stage: supervised fine-tuning

Dataset Language # of text pairs
MS MARCO English 500k
NQ English 70k
Trivia QA English 60k
NLI from SimCSE English <300k
ELI5 English 500k
DuReader Retrieval Chinese 86k
KILT Fever English 70k
KILT HotpotQA English 70k
SQuAD English 87k
Quora English 150k
Mr. TyDi 11 languages 50k
MIRACL 16 languages 40k

For all labeled datasets, we only use its training set for fine-tuning.

For other training details, please refer to our paper at https://arxiv.org/pdf/2212.03533.pdf.

Benchmark Results on Mr. TyDi

Model Avg MRR@10 ar bn en fi id ja ko ru sw te th
BM25 33.3 36.7 41.3 15.1 28.8 38.2 21.7 28.1 32.9 39.6 42.4 41.7
mDPR 16.7 26.0 25.8 16.2 11.3 14.6 18.1 21.9 18.5 7.3 10.6 13.5
BM25 + mDPR 41.7 49.1 53.5 28.4 36.5 45.5 35.5 36.2 42.7 40.5 42.0 49.2
multilingual-e5-small 64.4 71.5 66.3 54.5 57.7 63.2 55.4 54.3 60.8 65.4 89.1 70.1
multilingual-e5-base 65.9 72.3 65.0 58.5 60.8 64.9 56.6 55.8 62.7 69.0 86.6 72.7
multilingual-e5-large 70.5 77.5 73.2 60.8 66.8 68.5 62.5 61.6 65.8 72.7 90.2 76.2

MTEB Benchmark Evaluation

Check out unilm/e5 to reproduce evaluation results on the BEIR and MTEB benchmark.

Support for Sentence Transformers

Below is an example for usage with sentence_transformers.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('intfloat/multilingual-e5-small')
input_texts = [
    'query: how much protein should a female eat',
    'query: 南瓜的家常做法',
    "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 i     s 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or traini     ng for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "passage: 1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮     ,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,     放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油     锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀      6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"
]
embeddings = model.encode(input_texts, normalize_embeddings=True)

Package requirements

pip install sentence_transformers~=2.2.2

Contributors: michaelfeil

FAQ

1. Do I need to add the prefix "query: " and "passage: " to input texts?

Yes, this is how the model is trained, otherwise you will see a performance degradation.

Here are some rules of thumb:

  • Use "query: " and "passage: " correspondingly for asymmetric tasks such as passage retrieval in open QA, ad-hoc information retrieval.

  • Use "query: " prefix for symmetric tasks such as semantic similarity, bitext mining, paraphrase retrieval.

  • Use "query: " prefix if you want to use embeddings as features, such as linear probing classification, clustering.

2. Why are my reproduced results slightly different from reported in the model card?

Different versions of transformers and pytorch could cause negligible but non-zero performance differences.

3. Why does the cosine similarity scores distribute around 0.7 to 1.0?

This is a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss.

For text embedding tasks like text retrieval or semantic similarity, what matters is the relative order of the scores instead of the absolute values, so this should not be an issue.

Citation

If you find our paper or models helpful, please consider cite as follows:

@article{wang2022text,
  title={Text Embeddings by Weakly-Supervised Contrastive Pre-training},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Jiao, Binxing and Yang, Linjun and Jiang, Daxin and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2212.03533},
  year={2022}
}

Limitations

Long texts will be truncated to at most 512 tokens.

S
Description
Model synced from source: maiyad/multilingual-e5-small
Readme
53 KiB
0 Stars 1 Watchers 0 Forks