2668 lines
64 KiB
Markdown
2668 lines
64 KiB
Markdown
---
|
|
pipeline_tag: sentence-similarity
|
|
tags:
|
|
- sentence-transformers
|
|
- feature-extraction
|
|
- sentence-similarity
|
|
- mteb
|
|
model-index:
|
|
- name: SGPT-2.7B-weightedmean-msmarco-specb-bitfit
|
|
results:
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_counterfactual
|
|
name: MTEB AmazonCounterfactualClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
|
|
metrics:
|
|
- type: accuracy
|
|
value: 67.56716417910448
|
|
- type: ap
|
|
value: 30.75574629595259
|
|
- type: f1
|
|
value: 61.805121301858655
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_polarity
|
|
name: MTEB AmazonPolarityClassification
|
|
config: default
|
|
split: test
|
|
revision: 80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1
|
|
metrics:
|
|
- type: accuracy
|
|
value: 71.439575
|
|
- type: ap
|
|
value: 65.91341330532453
|
|
- type: f1
|
|
value: 70.90561852619555
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_reviews_multi
|
|
name: MTEB AmazonReviewsClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: c379a6705fec24a2493fa68e011692605f44e119
|
|
metrics:
|
|
- type: accuracy
|
|
value: 35.748000000000005
|
|
- type: f1
|
|
value: 35.48576287186347
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: arguana
|
|
name: MTEB ArguAna
|
|
config: default
|
|
split: test
|
|
revision: 5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 25.96
|
|
- type: map_at_10
|
|
value: 41.619
|
|
- type: map_at_100
|
|
value: 42.673
|
|
- type: map_at_1000
|
|
value: 42.684
|
|
- type: map_at_3
|
|
value: 36.569
|
|
- type: map_at_5
|
|
value: 39.397
|
|
- type: mrr_at_1
|
|
value: 26.316
|
|
- type: mrr_at_10
|
|
value: 41.772
|
|
- type: mrr_at_100
|
|
value: 42.82
|
|
- type: mrr_at_1000
|
|
value: 42.83
|
|
- type: mrr_at_3
|
|
value: 36.724000000000004
|
|
- type: mrr_at_5
|
|
value: 39.528999999999996
|
|
- type: ndcg_at_1
|
|
value: 25.96
|
|
- type: ndcg_at_10
|
|
value: 50.491
|
|
- type: ndcg_at_100
|
|
value: 54.864999999999995
|
|
- type: ndcg_at_1000
|
|
value: 55.10699999999999
|
|
- type: ndcg_at_3
|
|
value: 40.053
|
|
- type: ndcg_at_5
|
|
value: 45.134
|
|
- type: precision_at_1
|
|
value: 25.96
|
|
- type: precision_at_10
|
|
value: 7.8950000000000005
|
|
- type: precision_at_100
|
|
value: 0.9780000000000001
|
|
- type: precision_at_1000
|
|
value: 0.1
|
|
- type: precision_at_3
|
|
value: 16.714000000000002
|
|
- type: precision_at_5
|
|
value: 12.489
|
|
- type: recall_at_1
|
|
value: 25.96
|
|
- type: recall_at_10
|
|
value: 78.947
|
|
- type: recall_at_100
|
|
value: 97.795
|
|
- type: recall_at_1000
|
|
value: 99.644
|
|
- type: recall_at_3
|
|
value: 50.141999999999996
|
|
- type: recall_at_5
|
|
value: 62.446999999999996
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/arxiv-clustering-p2p
|
|
name: MTEB ArxivClusteringP2P
|
|
config: default
|
|
split: test
|
|
revision: 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8
|
|
metrics:
|
|
- type: v_measure
|
|
value: 44.72125714642202
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/arxiv-clustering-s2s
|
|
name: MTEB ArxivClusteringS2S
|
|
config: default
|
|
split: test
|
|
revision: b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3
|
|
metrics:
|
|
- type: v_measure
|
|
value: 35.081451519142064
|
|
- task:
|
|
type: Reranking
|
|
dataset:
|
|
type: mteb/askubuntudupquestions-reranking
|
|
name: MTEB AskUbuntuDupQuestions
|
|
config: default
|
|
split: test
|
|
revision: 4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c
|
|
metrics:
|
|
- type: map
|
|
value: 59.634661990392054
|
|
- type: mrr
|
|
value: 73.6813525040672
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/biosses-sts
|
|
name: MTEB BIOSSES
|
|
config: default
|
|
split: test
|
|
revision: 9ee918f184421b6bd48b78f6c714d86546106103
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 87.42754550496836
|
|
- type: cos_sim_spearman
|
|
value: 84.84289705838664
|
|
- type: euclidean_pearson
|
|
value: 85.59331970450859
|
|
- type: euclidean_spearman
|
|
value: 85.8525586184271
|
|
- type: manhattan_pearson
|
|
value: 85.41233134466698
|
|
- type: manhattan_spearman
|
|
value: 85.52303303767404
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/banking77
|
|
name: MTEB Banking77Classification
|
|
config: default
|
|
split: test
|
|
revision: 44fa15921b4c889113cc5df03dd4901b49161ab7
|
|
metrics:
|
|
- type: accuracy
|
|
value: 83.21753246753246
|
|
- type: f1
|
|
value: 83.15394543120915
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/biorxiv-clustering-p2p
|
|
name: MTEB BiorxivClusteringP2P
|
|
config: default
|
|
split: test
|
|
revision: 11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55
|
|
metrics:
|
|
- type: v_measure
|
|
value: 34.41414219680629
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/biorxiv-clustering-s2s
|
|
name: MTEB BiorxivClusteringS2S
|
|
config: default
|
|
split: test
|
|
revision: c0fab014e1bcb8d3a5e31b2088972a1e01547dc1
|
|
metrics:
|
|
- type: v_measure
|
|
value: 30.533275862270028
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackAndroidRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 30.808999999999997
|
|
- type: map_at_10
|
|
value: 40.617
|
|
- type: map_at_100
|
|
value: 41.894999999999996
|
|
- type: map_at_1000
|
|
value: 42.025
|
|
- type: map_at_3
|
|
value: 37.0
|
|
- type: map_at_5
|
|
value: 38.993
|
|
- type: mrr_at_1
|
|
value: 37.482
|
|
- type: mrr_at_10
|
|
value: 46.497
|
|
- type: mrr_at_100
|
|
value: 47.144000000000005
|
|
- type: mrr_at_1000
|
|
value: 47.189
|
|
- type: mrr_at_3
|
|
value: 43.705
|
|
- type: mrr_at_5
|
|
value: 45.193
|
|
- type: ndcg_at_1
|
|
value: 37.482
|
|
- type: ndcg_at_10
|
|
value: 46.688
|
|
- type: ndcg_at_100
|
|
value: 51.726000000000006
|
|
- type: ndcg_at_1000
|
|
value: 53.825
|
|
- type: ndcg_at_3
|
|
value: 41.242000000000004
|
|
- type: ndcg_at_5
|
|
value: 43.657000000000004
|
|
- type: precision_at_1
|
|
value: 37.482
|
|
- type: precision_at_10
|
|
value: 8.827
|
|
- type: precision_at_100
|
|
value: 1.393
|
|
- type: precision_at_1000
|
|
value: 0.186
|
|
- type: precision_at_3
|
|
value: 19.361
|
|
- type: precision_at_5
|
|
value: 14.106
|
|
- type: recall_at_1
|
|
value: 30.808999999999997
|
|
- type: recall_at_10
|
|
value: 58.47
|
|
- type: recall_at_100
|
|
value: 80.51899999999999
|
|
- type: recall_at_1000
|
|
value: 93.809
|
|
- type: recall_at_3
|
|
value: 42.462
|
|
- type: recall_at_5
|
|
value: 49.385
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackEnglishRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 26.962000000000003
|
|
- type: map_at_10
|
|
value: 36.93
|
|
- type: map_at_100
|
|
value: 38.102000000000004
|
|
- type: map_at_1000
|
|
value: 38.22
|
|
- type: map_at_3
|
|
value: 34.065
|
|
- type: map_at_5
|
|
value: 35.72
|
|
- type: mrr_at_1
|
|
value: 33.567
|
|
- type: mrr_at_10
|
|
value: 42.269
|
|
- type: mrr_at_100
|
|
value: 42.99
|
|
- type: mrr_at_1000
|
|
value: 43.033
|
|
- type: mrr_at_3
|
|
value: 40.064
|
|
- type: mrr_at_5
|
|
value: 41.258
|
|
- type: ndcg_at_1
|
|
value: 33.567
|
|
- type: ndcg_at_10
|
|
value: 42.405
|
|
- type: ndcg_at_100
|
|
value: 46.847
|
|
- type: ndcg_at_1000
|
|
value: 48.951
|
|
- type: ndcg_at_3
|
|
value: 38.312000000000005
|
|
- type: ndcg_at_5
|
|
value: 40.242
|
|
- type: precision_at_1
|
|
value: 33.567
|
|
- type: precision_at_10
|
|
value: 8.032
|
|
- type: precision_at_100
|
|
value: 1.295
|
|
- type: precision_at_1000
|
|
value: 0.17600000000000002
|
|
- type: precision_at_3
|
|
value: 18.662
|
|
- type: precision_at_5
|
|
value: 13.299
|
|
- type: recall_at_1
|
|
value: 26.962000000000003
|
|
- type: recall_at_10
|
|
value: 52.489
|
|
- type: recall_at_100
|
|
value: 71.635
|
|
- type: recall_at_1000
|
|
value: 85.141
|
|
- type: recall_at_3
|
|
value: 40.28
|
|
- type: recall_at_5
|
|
value: 45.757
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackGamingRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 36.318
|
|
- type: map_at_10
|
|
value: 47.97
|
|
- type: map_at_100
|
|
value: 49.003
|
|
- type: map_at_1000
|
|
value: 49.065999999999995
|
|
- type: map_at_3
|
|
value: 45.031
|
|
- type: map_at_5
|
|
value: 46.633
|
|
- type: mrr_at_1
|
|
value: 41.504999999999995
|
|
- type: mrr_at_10
|
|
value: 51.431000000000004
|
|
- type: mrr_at_100
|
|
value: 52.129000000000005
|
|
- type: mrr_at_1000
|
|
value: 52.161
|
|
- type: mrr_at_3
|
|
value: 48.934
|
|
- type: mrr_at_5
|
|
value: 50.42
|
|
- type: ndcg_at_1
|
|
value: 41.504999999999995
|
|
- type: ndcg_at_10
|
|
value: 53.676
|
|
- type: ndcg_at_100
|
|
value: 57.867000000000004
|
|
- type: ndcg_at_1000
|
|
value: 59.166
|
|
- type: ndcg_at_3
|
|
value: 48.516
|
|
- type: ndcg_at_5
|
|
value: 50.983999999999995
|
|
- type: precision_at_1
|
|
value: 41.504999999999995
|
|
- type: precision_at_10
|
|
value: 8.608
|
|
- type: precision_at_100
|
|
value: 1.1560000000000001
|
|
- type: precision_at_1000
|
|
value: 0.133
|
|
- type: precision_at_3
|
|
value: 21.462999999999997
|
|
- type: precision_at_5
|
|
value: 14.721
|
|
- type: recall_at_1
|
|
value: 36.318
|
|
- type: recall_at_10
|
|
value: 67.066
|
|
- type: recall_at_100
|
|
value: 85.34
|
|
- type: recall_at_1000
|
|
value: 94.491
|
|
- type: recall_at_3
|
|
value: 53.215999999999994
|
|
- type: recall_at_5
|
|
value: 59.214
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackGisRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 22.167
|
|
- type: map_at_10
|
|
value: 29.543999999999997
|
|
- type: map_at_100
|
|
value: 30.579
|
|
- type: map_at_1000
|
|
value: 30.669999999999998
|
|
- type: map_at_3
|
|
value: 26.982
|
|
- type: map_at_5
|
|
value: 28.474
|
|
- type: mrr_at_1
|
|
value: 24.068
|
|
- type: mrr_at_10
|
|
value: 31.237
|
|
- type: mrr_at_100
|
|
value: 32.222
|
|
- type: mrr_at_1000
|
|
value: 32.292
|
|
- type: mrr_at_3
|
|
value: 28.776000000000003
|
|
- type: mrr_at_5
|
|
value: 30.233999999999998
|
|
- type: ndcg_at_1
|
|
value: 24.068
|
|
- type: ndcg_at_10
|
|
value: 33.973
|
|
- type: ndcg_at_100
|
|
value: 39.135
|
|
- type: ndcg_at_1000
|
|
value: 41.443999999999996
|
|
- type: ndcg_at_3
|
|
value: 29.018
|
|
- type: ndcg_at_5
|
|
value: 31.558999999999997
|
|
- type: precision_at_1
|
|
value: 24.068
|
|
- type: precision_at_10
|
|
value: 5.299
|
|
- type: precision_at_100
|
|
value: 0.823
|
|
- type: precision_at_1000
|
|
value: 0.106
|
|
- type: precision_at_3
|
|
value: 12.166
|
|
- type: precision_at_5
|
|
value: 8.767999999999999
|
|
- type: recall_at_1
|
|
value: 22.167
|
|
- type: recall_at_10
|
|
value: 46.115
|
|
- type: recall_at_100
|
|
value: 69.867
|
|
- type: recall_at_1000
|
|
value: 87.234
|
|
- type: recall_at_3
|
|
value: 32.798
|
|
- type: recall_at_5
|
|
value: 38.951
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackMathematicaRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 12.033000000000001
|
|
- type: map_at_10
|
|
value: 19.314
|
|
- type: map_at_100
|
|
value: 20.562
|
|
- type: map_at_1000
|
|
value: 20.695
|
|
- type: map_at_3
|
|
value: 16.946
|
|
- type: map_at_5
|
|
value: 18.076999999999998
|
|
- type: mrr_at_1
|
|
value: 14.801
|
|
- type: mrr_at_10
|
|
value: 22.74
|
|
- type: mrr_at_100
|
|
value: 23.876
|
|
- type: mrr_at_1000
|
|
value: 23.949
|
|
- type: mrr_at_3
|
|
value: 20.211000000000002
|
|
- type: mrr_at_5
|
|
value: 21.573
|
|
- type: ndcg_at_1
|
|
value: 14.801
|
|
- type: ndcg_at_10
|
|
value: 24.038
|
|
- type: ndcg_at_100
|
|
value: 30.186
|
|
- type: ndcg_at_1000
|
|
value: 33.321
|
|
- type: ndcg_at_3
|
|
value: 19.431
|
|
- type: ndcg_at_5
|
|
value: 21.34
|
|
- type: precision_at_1
|
|
value: 14.801
|
|
- type: precision_at_10
|
|
value: 4.776
|
|
- type: precision_at_100
|
|
value: 0.897
|
|
- type: precision_at_1000
|
|
value: 0.133
|
|
- type: precision_at_3
|
|
value: 9.66
|
|
- type: precision_at_5
|
|
value: 7.239
|
|
- type: recall_at_1
|
|
value: 12.033000000000001
|
|
- type: recall_at_10
|
|
value: 35.098
|
|
- type: recall_at_100
|
|
value: 62.175000000000004
|
|
- type: recall_at_1000
|
|
value: 84.17099999999999
|
|
- type: recall_at_3
|
|
value: 22.61
|
|
- type: recall_at_5
|
|
value: 27.278999999999996
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackPhysicsRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 26.651000000000003
|
|
- type: map_at_10
|
|
value: 36.901
|
|
- type: map_at_100
|
|
value: 38.249
|
|
- type: map_at_1000
|
|
value: 38.361000000000004
|
|
- type: map_at_3
|
|
value: 33.891
|
|
- type: map_at_5
|
|
value: 35.439
|
|
- type: mrr_at_1
|
|
value: 32.724
|
|
- type: mrr_at_10
|
|
value: 42.504
|
|
- type: mrr_at_100
|
|
value: 43.391999999999996
|
|
- type: mrr_at_1000
|
|
value: 43.436
|
|
- type: mrr_at_3
|
|
value: 39.989999999999995
|
|
- type: mrr_at_5
|
|
value: 41.347
|
|
- type: ndcg_at_1
|
|
value: 32.724
|
|
- type: ndcg_at_10
|
|
value: 43.007
|
|
- type: ndcg_at_100
|
|
value: 48.601
|
|
- type: ndcg_at_1000
|
|
value: 50.697
|
|
- type: ndcg_at_3
|
|
value: 37.99
|
|
- type: ndcg_at_5
|
|
value: 40.083999999999996
|
|
- type: precision_at_1
|
|
value: 32.724
|
|
- type: precision_at_10
|
|
value: 7.872999999999999
|
|
- type: precision_at_100
|
|
value: 1.247
|
|
- type: precision_at_1000
|
|
value: 0.16199999999999998
|
|
- type: precision_at_3
|
|
value: 18.062
|
|
- type: precision_at_5
|
|
value: 12.666
|
|
- type: recall_at_1
|
|
value: 26.651000000000003
|
|
- type: recall_at_10
|
|
value: 55.674
|
|
- type: recall_at_100
|
|
value: 78.904
|
|
- type: recall_at_1000
|
|
value: 92.55799999999999
|
|
- type: recall_at_3
|
|
value: 41.36
|
|
- type: recall_at_5
|
|
value: 46.983999999999995
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackProgrammersRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 22.589000000000002
|
|
- type: map_at_10
|
|
value: 32.244
|
|
- type: map_at_100
|
|
value: 33.46
|
|
- type: map_at_1000
|
|
value: 33.593
|
|
- type: map_at_3
|
|
value: 29.21
|
|
- type: map_at_5
|
|
value: 31.019999999999996
|
|
- type: mrr_at_1
|
|
value: 28.425
|
|
- type: mrr_at_10
|
|
value: 37.282
|
|
- type: mrr_at_100
|
|
value: 38.187
|
|
- type: mrr_at_1000
|
|
value: 38.248
|
|
- type: mrr_at_3
|
|
value: 34.684
|
|
- type: mrr_at_5
|
|
value: 36.123
|
|
- type: ndcg_at_1
|
|
value: 28.425
|
|
- type: ndcg_at_10
|
|
value: 37.942
|
|
- type: ndcg_at_100
|
|
value: 43.443
|
|
- type: ndcg_at_1000
|
|
value: 45.995999999999995
|
|
- type: ndcg_at_3
|
|
value: 32.873999999999995
|
|
- type: ndcg_at_5
|
|
value: 35.325
|
|
- type: precision_at_1
|
|
value: 28.425
|
|
- type: precision_at_10
|
|
value: 7.1
|
|
- type: precision_at_100
|
|
value: 1.166
|
|
- type: precision_at_1000
|
|
value: 0.158
|
|
- type: precision_at_3
|
|
value: 16.02
|
|
- type: precision_at_5
|
|
value: 11.644
|
|
- type: recall_at_1
|
|
value: 22.589000000000002
|
|
- type: recall_at_10
|
|
value: 50.03999999999999
|
|
- type: recall_at_100
|
|
value: 73.973
|
|
- type: recall_at_1000
|
|
value: 91.128
|
|
- type: recall_at_3
|
|
value: 35.882999999999996
|
|
- type: recall_at_5
|
|
value: 42.187999999999995
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 23.190833333333334
|
|
- type: map_at_10
|
|
value: 31.504916666666666
|
|
- type: map_at_100
|
|
value: 32.64908333333334
|
|
- type: map_at_1000
|
|
value: 32.77075
|
|
- type: map_at_3
|
|
value: 28.82575
|
|
- type: map_at_5
|
|
value: 30.2755
|
|
- type: mrr_at_1
|
|
value: 27.427499999999995
|
|
- type: mrr_at_10
|
|
value: 35.36483333333334
|
|
- type: mrr_at_100
|
|
value: 36.23441666666666
|
|
- type: mrr_at_1000
|
|
value: 36.297583333333336
|
|
- type: mrr_at_3
|
|
value: 32.97966666666667
|
|
- type: mrr_at_5
|
|
value: 34.294583333333335
|
|
- type: ndcg_at_1
|
|
value: 27.427499999999995
|
|
- type: ndcg_at_10
|
|
value: 36.53358333333333
|
|
- type: ndcg_at_100
|
|
value: 41.64508333333333
|
|
- type: ndcg_at_1000
|
|
value: 44.14499999999999
|
|
- type: ndcg_at_3
|
|
value: 31.88908333333333
|
|
- type: ndcg_at_5
|
|
value: 33.98433333333333
|
|
- type: precision_at_1
|
|
value: 27.427499999999995
|
|
- type: precision_at_10
|
|
value: 6.481083333333333
|
|
- type: precision_at_100
|
|
value: 1.0610833333333334
|
|
- type: precision_at_1000
|
|
value: 0.14691666666666667
|
|
- type: precision_at_3
|
|
value: 14.656749999999999
|
|
- type: precision_at_5
|
|
value: 10.493583333333332
|
|
- type: recall_at_1
|
|
value: 23.190833333333334
|
|
- type: recall_at_10
|
|
value: 47.65175
|
|
- type: recall_at_100
|
|
value: 70.41016666666667
|
|
- type: recall_at_1000
|
|
value: 87.82708333333332
|
|
- type: recall_at_3
|
|
value: 34.637583333333325
|
|
- type: recall_at_5
|
|
value: 40.05008333333333
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackStatsRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 20.409
|
|
- type: map_at_10
|
|
value: 26.794
|
|
- type: map_at_100
|
|
value: 27.682000000000002
|
|
- type: map_at_1000
|
|
value: 27.783
|
|
- type: map_at_3
|
|
value: 24.461
|
|
- type: map_at_5
|
|
value: 25.668000000000003
|
|
- type: mrr_at_1
|
|
value: 22.853
|
|
- type: mrr_at_10
|
|
value: 29.296
|
|
- type: mrr_at_100
|
|
value: 30.103
|
|
- type: mrr_at_1000
|
|
value: 30.179000000000002
|
|
- type: mrr_at_3
|
|
value: 27.173000000000002
|
|
- type: mrr_at_5
|
|
value: 28.223
|
|
- type: ndcg_at_1
|
|
value: 22.853
|
|
- type: ndcg_at_10
|
|
value: 31.007
|
|
- type: ndcg_at_100
|
|
value: 35.581
|
|
- type: ndcg_at_1000
|
|
value: 38.147
|
|
- type: ndcg_at_3
|
|
value: 26.590999999999998
|
|
- type: ndcg_at_5
|
|
value: 28.43
|
|
- type: precision_at_1
|
|
value: 22.853
|
|
- type: precision_at_10
|
|
value: 5.031
|
|
- type: precision_at_100
|
|
value: 0.7939999999999999
|
|
- type: precision_at_1000
|
|
value: 0.11
|
|
- type: precision_at_3
|
|
value: 11.401
|
|
- type: precision_at_5
|
|
value: 8.16
|
|
- type: recall_at_1
|
|
value: 20.409
|
|
- type: recall_at_10
|
|
value: 41.766
|
|
- type: recall_at_100
|
|
value: 62.964
|
|
- type: recall_at_1000
|
|
value: 81.682
|
|
- type: recall_at_3
|
|
value: 29.281000000000002
|
|
- type: recall_at_5
|
|
value: 33.83
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackTexRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 14.549000000000001
|
|
- type: map_at_10
|
|
value: 20.315
|
|
- type: map_at_100
|
|
value: 21.301000000000002
|
|
- type: map_at_1000
|
|
value: 21.425
|
|
- type: map_at_3
|
|
value: 18.132
|
|
- type: map_at_5
|
|
value: 19.429
|
|
- type: mrr_at_1
|
|
value: 17.86
|
|
- type: mrr_at_10
|
|
value: 23.860999999999997
|
|
- type: mrr_at_100
|
|
value: 24.737000000000002
|
|
- type: mrr_at_1000
|
|
value: 24.82
|
|
- type: mrr_at_3
|
|
value: 21.685
|
|
- type: mrr_at_5
|
|
value: 23.008
|
|
- type: ndcg_at_1
|
|
value: 17.86
|
|
- type: ndcg_at_10
|
|
value: 24.396
|
|
- type: ndcg_at_100
|
|
value: 29.328
|
|
- type: ndcg_at_1000
|
|
value: 32.486
|
|
- type: ndcg_at_3
|
|
value: 20.375
|
|
- type: ndcg_at_5
|
|
value: 22.411
|
|
- type: precision_at_1
|
|
value: 17.86
|
|
- type: precision_at_10
|
|
value: 4.47
|
|
- type: precision_at_100
|
|
value: 0.8099999999999999
|
|
- type: precision_at_1000
|
|
value: 0.125
|
|
- type: precision_at_3
|
|
value: 9.475
|
|
- type: precision_at_5
|
|
value: 7.170999999999999
|
|
- type: recall_at_1
|
|
value: 14.549000000000001
|
|
- type: recall_at_10
|
|
value: 33.365
|
|
- type: recall_at_100
|
|
value: 55.797
|
|
- type: recall_at_1000
|
|
value: 78.632
|
|
- type: recall_at_3
|
|
value: 22.229
|
|
- type: recall_at_5
|
|
value: 27.339000000000002
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackUnixRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 23.286
|
|
- type: map_at_10
|
|
value: 30.728
|
|
- type: map_at_100
|
|
value: 31.840000000000003
|
|
- type: map_at_1000
|
|
value: 31.953
|
|
- type: map_at_3
|
|
value: 28.302
|
|
- type: map_at_5
|
|
value: 29.615000000000002
|
|
- type: mrr_at_1
|
|
value: 27.239
|
|
- type: mrr_at_10
|
|
value: 34.408
|
|
- type: mrr_at_100
|
|
value: 35.335
|
|
- type: mrr_at_1000
|
|
value: 35.405
|
|
- type: mrr_at_3
|
|
value: 32.151999999999994
|
|
- type: mrr_at_5
|
|
value: 33.355000000000004
|
|
- type: ndcg_at_1
|
|
value: 27.239
|
|
- type: ndcg_at_10
|
|
value: 35.324
|
|
- type: ndcg_at_100
|
|
value: 40.866
|
|
- type: ndcg_at_1000
|
|
value: 43.584
|
|
- type: ndcg_at_3
|
|
value: 30.898999999999997
|
|
- type: ndcg_at_5
|
|
value: 32.812999999999995
|
|
- type: precision_at_1
|
|
value: 27.239
|
|
- type: precision_at_10
|
|
value: 5.896
|
|
- type: precision_at_100
|
|
value: 0.979
|
|
- type: precision_at_1000
|
|
value: 0.133
|
|
- type: precision_at_3
|
|
value: 13.713000000000001
|
|
- type: precision_at_5
|
|
value: 9.683
|
|
- type: recall_at_1
|
|
value: 23.286
|
|
- type: recall_at_10
|
|
value: 45.711
|
|
- type: recall_at_100
|
|
value: 70.611
|
|
- type: recall_at_1000
|
|
value: 90.029
|
|
- type: recall_at_3
|
|
value: 33.615
|
|
- type: recall_at_5
|
|
value: 38.41
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackWebmastersRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 23.962
|
|
- type: map_at_10
|
|
value: 31.942999999999998
|
|
- type: map_at_100
|
|
value: 33.384
|
|
- type: map_at_1000
|
|
value: 33.611000000000004
|
|
- type: map_at_3
|
|
value: 29.243000000000002
|
|
- type: map_at_5
|
|
value: 30.446
|
|
- type: mrr_at_1
|
|
value: 28.458
|
|
- type: mrr_at_10
|
|
value: 36.157000000000004
|
|
- type: mrr_at_100
|
|
value: 37.092999999999996
|
|
- type: mrr_at_1000
|
|
value: 37.163000000000004
|
|
- type: mrr_at_3
|
|
value: 33.86
|
|
- type: mrr_at_5
|
|
value: 35.086
|
|
- type: ndcg_at_1
|
|
value: 28.458
|
|
- type: ndcg_at_10
|
|
value: 37.201
|
|
- type: ndcg_at_100
|
|
value: 42.591
|
|
- type: ndcg_at_1000
|
|
value: 45.539
|
|
- type: ndcg_at_3
|
|
value: 32.889
|
|
- type: ndcg_at_5
|
|
value: 34.483000000000004
|
|
- type: precision_at_1
|
|
value: 28.458
|
|
- type: precision_at_10
|
|
value: 7.332
|
|
- type: precision_at_100
|
|
value: 1.437
|
|
- type: precision_at_1000
|
|
value: 0.233
|
|
- type: precision_at_3
|
|
value: 15.547
|
|
- type: precision_at_5
|
|
value: 11.146
|
|
- type: recall_at_1
|
|
value: 23.962
|
|
- type: recall_at_10
|
|
value: 46.751
|
|
- type: recall_at_100
|
|
value: 71.626
|
|
- type: recall_at_1000
|
|
value: 90.93900000000001
|
|
- type: recall_at_3
|
|
value: 34.138000000000005
|
|
- type: recall_at_5
|
|
value: 38.673
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackWordpressRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 18.555
|
|
- type: map_at_10
|
|
value: 24.759
|
|
- type: map_at_100
|
|
value: 25.732
|
|
- type: map_at_1000
|
|
value: 25.846999999999998
|
|
- type: map_at_3
|
|
value: 22.646
|
|
- type: map_at_5
|
|
value: 23.791999999999998
|
|
- type: mrr_at_1
|
|
value: 20.148
|
|
- type: mrr_at_10
|
|
value: 26.695999999999998
|
|
- type: mrr_at_100
|
|
value: 27.605
|
|
- type: mrr_at_1000
|
|
value: 27.695999999999998
|
|
- type: mrr_at_3
|
|
value: 24.522
|
|
- type: mrr_at_5
|
|
value: 25.715
|
|
- type: ndcg_at_1
|
|
value: 20.148
|
|
- type: ndcg_at_10
|
|
value: 28.746
|
|
- type: ndcg_at_100
|
|
value: 33.57
|
|
- type: ndcg_at_1000
|
|
value: 36.584
|
|
- type: ndcg_at_3
|
|
value: 24.532
|
|
- type: ndcg_at_5
|
|
value: 26.484
|
|
- type: precision_at_1
|
|
value: 20.148
|
|
- type: precision_at_10
|
|
value: 4.529
|
|
- type: precision_at_100
|
|
value: 0.736
|
|
- type: precision_at_1000
|
|
value: 0.108
|
|
- type: precision_at_3
|
|
value: 10.351
|
|
- type: precision_at_5
|
|
value: 7.32
|
|
- type: recall_at_1
|
|
value: 18.555
|
|
- type: recall_at_10
|
|
value: 39.275999999999996
|
|
- type: recall_at_100
|
|
value: 61.511
|
|
- type: recall_at_1000
|
|
value: 84.111
|
|
- type: recall_at_3
|
|
value: 27.778999999999996
|
|
- type: recall_at_5
|
|
value: 32.591
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: climate-fever
|
|
name: MTEB ClimateFEVER
|
|
config: default
|
|
split: test
|
|
revision: 392b78eb68c07badcd7c2cd8f39af108375dfcce
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 10.366999999999999
|
|
- type: map_at_10
|
|
value: 18.953999999999997
|
|
- type: map_at_100
|
|
value: 20.674999999999997
|
|
- type: map_at_1000
|
|
value: 20.868000000000002
|
|
- type: map_at_3
|
|
value: 15.486
|
|
- type: map_at_5
|
|
value: 17.347
|
|
- type: mrr_at_1
|
|
value: 23.257
|
|
- type: mrr_at_10
|
|
value: 35.419
|
|
- type: mrr_at_100
|
|
value: 36.361
|
|
- type: mrr_at_1000
|
|
value: 36.403
|
|
- type: mrr_at_3
|
|
value: 31.747999999999998
|
|
- type: mrr_at_5
|
|
value: 34.077
|
|
- type: ndcg_at_1
|
|
value: 23.257
|
|
- type: ndcg_at_10
|
|
value: 27.11
|
|
- type: ndcg_at_100
|
|
value: 33.981
|
|
- type: ndcg_at_1000
|
|
value: 37.444
|
|
- type: ndcg_at_3
|
|
value: 21.471999999999998
|
|
- type: ndcg_at_5
|
|
value: 23.769000000000002
|
|
- type: precision_at_1
|
|
value: 23.257
|
|
- type: precision_at_10
|
|
value: 8.704
|
|
- type: precision_at_100
|
|
value: 1.606
|
|
- type: precision_at_1000
|
|
value: 0.22499999999999998
|
|
- type: precision_at_3
|
|
value: 16.287
|
|
- type: precision_at_5
|
|
value: 13.068
|
|
- type: recall_at_1
|
|
value: 10.366999999999999
|
|
- type: recall_at_10
|
|
value: 33.706
|
|
- type: recall_at_100
|
|
value: 57.375
|
|
- type: recall_at_1000
|
|
value: 76.79
|
|
- type: recall_at_3
|
|
value: 20.18
|
|
- type: recall_at_5
|
|
value: 26.215
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: dbpedia-entity
|
|
name: MTEB DBPedia
|
|
config: default
|
|
split: test
|
|
revision: f097057d03ed98220bc7309ddb10b71a54d667d6
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 8.246
|
|
- type: map_at_10
|
|
value: 15.979
|
|
- type: map_at_100
|
|
value: 21.025
|
|
- type: map_at_1000
|
|
value: 22.189999999999998
|
|
- type: map_at_3
|
|
value: 11.997
|
|
- type: map_at_5
|
|
value: 13.697000000000001
|
|
- type: mrr_at_1
|
|
value: 60.75000000000001
|
|
- type: mrr_at_10
|
|
value: 68.70100000000001
|
|
- type: mrr_at_100
|
|
value: 69.1
|
|
- type: mrr_at_1000
|
|
value: 69.111
|
|
- type: mrr_at_3
|
|
value: 66.583
|
|
- type: mrr_at_5
|
|
value: 67.87100000000001
|
|
- type: ndcg_at_1
|
|
value: 49.75
|
|
- type: ndcg_at_10
|
|
value: 34.702
|
|
- type: ndcg_at_100
|
|
value: 37.607
|
|
- type: ndcg_at_1000
|
|
value: 44.322
|
|
- type: ndcg_at_3
|
|
value: 39.555
|
|
- type: ndcg_at_5
|
|
value: 36.684
|
|
- type: precision_at_1
|
|
value: 60.75000000000001
|
|
- type: precision_at_10
|
|
value: 26.625
|
|
- type: precision_at_100
|
|
value: 7.969999999999999
|
|
- type: precision_at_1000
|
|
value: 1.678
|
|
- type: precision_at_3
|
|
value: 41.833
|
|
- type: precision_at_5
|
|
value: 34.5
|
|
- type: recall_at_1
|
|
value: 8.246
|
|
- type: recall_at_10
|
|
value: 20.968
|
|
- type: recall_at_100
|
|
value: 42.065000000000005
|
|
- type: recall_at_1000
|
|
value: 63.671
|
|
- type: recall_at_3
|
|
value: 13.039000000000001
|
|
- type: recall_at_5
|
|
value: 16.042
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/emotion
|
|
name: MTEB EmotionClassification
|
|
config: default
|
|
split: test
|
|
revision: 829147f8f75a25f005913200eb5ed41fae320aa1
|
|
metrics:
|
|
- type: accuracy
|
|
value: 49.214999999999996
|
|
- type: f1
|
|
value: 44.85952451163755
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: fever
|
|
name: MTEB FEVER
|
|
config: default
|
|
split: test
|
|
revision: 1429cf27e393599b8b359b9b72c666f96b2525f9
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 56.769000000000005
|
|
- type: map_at_10
|
|
value: 67.30199999999999
|
|
- type: map_at_100
|
|
value: 67.692
|
|
- type: map_at_1000
|
|
value: 67.712
|
|
- type: map_at_3
|
|
value: 65.346
|
|
- type: map_at_5
|
|
value: 66.574
|
|
- type: mrr_at_1
|
|
value: 61.370999999999995
|
|
- type: mrr_at_10
|
|
value: 71.875
|
|
- type: mrr_at_100
|
|
value: 72.195
|
|
- type: mrr_at_1000
|
|
value: 72.206
|
|
- type: mrr_at_3
|
|
value: 70.04
|
|
- type: mrr_at_5
|
|
value: 71.224
|
|
- type: ndcg_at_1
|
|
value: 61.370999999999995
|
|
- type: ndcg_at_10
|
|
value: 72.731
|
|
- type: ndcg_at_100
|
|
value: 74.468
|
|
- type: ndcg_at_1000
|
|
value: 74.91600000000001
|
|
- type: ndcg_at_3
|
|
value: 69.077
|
|
- type: ndcg_at_5
|
|
value: 71.111
|
|
- type: precision_at_1
|
|
value: 61.370999999999995
|
|
- type: precision_at_10
|
|
value: 9.325999999999999
|
|
- type: precision_at_100
|
|
value: 1.03
|
|
- type: precision_at_1000
|
|
value: 0.108
|
|
- type: precision_at_3
|
|
value: 27.303
|
|
- type: precision_at_5
|
|
value: 17.525
|
|
- type: recall_at_1
|
|
value: 56.769000000000005
|
|
- type: recall_at_10
|
|
value: 85.06
|
|
- type: recall_at_100
|
|
value: 92.767
|
|
- type: recall_at_1000
|
|
value: 95.933
|
|
- type: recall_at_3
|
|
value: 75.131
|
|
- type: recall_at_5
|
|
value: 80.17
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: fiqa
|
|
name: MTEB FiQA2018
|
|
config: default
|
|
split: test
|
|
revision: 41b686a7f28c59bcaaa5791efd47c67c8ebe28be
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 15.753
|
|
- type: map_at_10
|
|
value: 25.875999999999998
|
|
- type: map_at_100
|
|
value: 27.415
|
|
- type: map_at_1000
|
|
value: 27.590999999999998
|
|
- type: map_at_3
|
|
value: 22.17
|
|
- type: map_at_5
|
|
value: 24.236
|
|
- type: mrr_at_1
|
|
value: 31.019000000000002
|
|
- type: mrr_at_10
|
|
value: 39.977000000000004
|
|
- type: mrr_at_100
|
|
value: 40.788999999999994
|
|
- type: mrr_at_1000
|
|
value: 40.832
|
|
- type: mrr_at_3
|
|
value: 37.088
|
|
- type: mrr_at_5
|
|
value: 38.655
|
|
- type: ndcg_at_1
|
|
value: 31.019000000000002
|
|
- type: ndcg_at_10
|
|
value: 33.286
|
|
- type: ndcg_at_100
|
|
value: 39.528999999999996
|
|
- type: ndcg_at_1000
|
|
value: 42.934
|
|
- type: ndcg_at_3
|
|
value: 29.29
|
|
- type: ndcg_at_5
|
|
value: 30.615
|
|
- type: precision_at_1
|
|
value: 31.019000000000002
|
|
- type: precision_at_10
|
|
value: 9.383
|
|
- type: precision_at_100
|
|
value: 1.6019999999999999
|
|
- type: precision_at_1000
|
|
value: 0.22200000000000003
|
|
- type: precision_at_3
|
|
value: 19.753
|
|
- type: precision_at_5
|
|
value: 14.815000000000001
|
|
- type: recall_at_1
|
|
value: 15.753
|
|
- type: recall_at_10
|
|
value: 40.896
|
|
- type: recall_at_100
|
|
value: 64.443
|
|
- type: recall_at_1000
|
|
value: 85.218
|
|
- type: recall_at_3
|
|
value: 26.526
|
|
- type: recall_at_5
|
|
value: 32.452999999999996
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: hotpotqa
|
|
name: MTEB HotpotQA
|
|
config: default
|
|
split: test
|
|
revision: 766870b35a1b9ca65e67a0d1913899973551fc6c
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 32.153999999999996
|
|
- type: map_at_10
|
|
value: 43.651
|
|
- type: map_at_100
|
|
value: 44.41
|
|
- type: map_at_1000
|
|
value: 44.487
|
|
- type: map_at_3
|
|
value: 41.239
|
|
- type: map_at_5
|
|
value: 42.659000000000006
|
|
- type: mrr_at_1
|
|
value: 64.30799999999999
|
|
- type: mrr_at_10
|
|
value: 71.22500000000001
|
|
- type: mrr_at_100
|
|
value: 71.57
|
|
- type: mrr_at_1000
|
|
value: 71.59100000000001
|
|
- type: mrr_at_3
|
|
value: 69.95
|
|
- type: mrr_at_5
|
|
value: 70.738
|
|
- type: ndcg_at_1
|
|
value: 64.30799999999999
|
|
- type: ndcg_at_10
|
|
value: 52.835
|
|
- type: ndcg_at_100
|
|
value: 55.840999999999994
|
|
- type: ndcg_at_1000
|
|
value: 57.484
|
|
- type: ndcg_at_3
|
|
value: 49.014
|
|
- type: ndcg_at_5
|
|
value: 51.01599999999999
|
|
- type: precision_at_1
|
|
value: 64.30799999999999
|
|
- type: precision_at_10
|
|
value: 10.77
|
|
- type: precision_at_100
|
|
value: 1.315
|
|
- type: precision_at_1000
|
|
value: 0.153
|
|
- type: precision_at_3
|
|
value: 30.223
|
|
- type: precision_at_5
|
|
value: 19.716
|
|
- type: recall_at_1
|
|
value: 32.153999999999996
|
|
- type: recall_at_10
|
|
value: 53.849000000000004
|
|
- type: recall_at_100
|
|
value: 65.75999999999999
|
|
- type: recall_at_1000
|
|
value: 76.705
|
|
- type: recall_at_3
|
|
value: 45.334
|
|
- type: recall_at_5
|
|
value: 49.291000000000004
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/imdb
|
|
name: MTEB ImdbClassification
|
|
config: default
|
|
split: test
|
|
revision: 8d743909f834c38949e8323a8a6ce8721ea6c7f4
|
|
metrics:
|
|
- type: accuracy
|
|
value: 63.5316
|
|
- type: ap
|
|
value: 58.90084300359825
|
|
- type: f1
|
|
value: 63.35727889030892
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: msmarco
|
|
name: MTEB MSMARCO
|
|
config: default
|
|
split: validation
|
|
revision: e6838a846e2408f22cf5cc337ebc83e0bcf77849
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 20.566000000000003
|
|
- type: map_at_10
|
|
value: 32.229
|
|
- type: map_at_100
|
|
value: 33.445
|
|
- type: map_at_1000
|
|
value: 33.501
|
|
- type: map_at_3
|
|
value: 28.504
|
|
- type: map_at_5
|
|
value: 30.681000000000004
|
|
- type: mrr_at_1
|
|
value: 21.218
|
|
- type: mrr_at_10
|
|
value: 32.816
|
|
- type: mrr_at_100
|
|
value: 33.986
|
|
- type: mrr_at_1000
|
|
value: 34.035
|
|
- type: mrr_at_3
|
|
value: 29.15
|
|
- type: mrr_at_5
|
|
value: 31.290000000000003
|
|
- type: ndcg_at_1
|
|
value: 21.218
|
|
- type: ndcg_at_10
|
|
value: 38.832
|
|
- type: ndcg_at_100
|
|
value: 44.743
|
|
- type: ndcg_at_1000
|
|
value: 46.138
|
|
- type: ndcg_at_3
|
|
value: 31.232
|
|
- type: ndcg_at_5
|
|
value: 35.099999999999994
|
|
- type: precision_at_1
|
|
value: 21.218
|
|
- type: precision_at_10
|
|
value: 6.186
|
|
- type: precision_at_100
|
|
value: 0.914
|
|
- type: precision_at_1000
|
|
value: 0.10300000000000001
|
|
- type: precision_at_3
|
|
value: 13.314
|
|
- type: precision_at_5
|
|
value: 9.943
|
|
- type: recall_at_1
|
|
value: 20.566000000000003
|
|
- type: recall_at_10
|
|
value: 59.192
|
|
- type: recall_at_100
|
|
value: 86.626
|
|
- type: recall_at_1000
|
|
value: 97.283
|
|
- type: recall_at_3
|
|
value: 38.492
|
|
- type: recall_at_5
|
|
value: 47.760000000000005
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_domain
|
|
name: MTEB MTOPDomainClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
|
|
metrics:
|
|
- type: accuracy
|
|
value: 92.56269949840402
|
|
- type: f1
|
|
value: 92.1020975473988
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_intent
|
|
name: MTEB MTOPIntentClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: 6299947a7777084cc2d4b64235bf7190381ce755
|
|
metrics:
|
|
- type: accuracy
|
|
value: 71.8467852257182
|
|
- type: f1
|
|
value: 53.652719348592015
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 69.00806993947546
|
|
- type: f1
|
|
value: 67.41429618885515
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 75.90114324142569
|
|
- type: f1
|
|
value: 76.25183590651454
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/medrxiv-clustering-p2p
|
|
name: MTEB MedrxivClusteringP2P
|
|
config: default
|
|
split: test
|
|
revision: dcefc037ef84348e49b0d29109e891c01067226b
|
|
metrics:
|
|
- type: v_measure
|
|
value: 31.350109978273395
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/medrxiv-clustering-s2s
|
|
name: MTEB MedrxivClusteringS2S
|
|
config: default
|
|
split: test
|
|
revision: 3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc
|
|
metrics:
|
|
- type: v_measure
|
|
value: 28.768923695767327
|
|
- task:
|
|
type: Reranking
|
|
dataset:
|
|
type: mteb/mind_small
|
|
name: MTEB MindSmallReranking
|
|
config: default
|
|
split: test
|
|
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69
|
|
metrics:
|
|
- type: map
|
|
value: 31.716396735210754
|
|
- type: mrr
|
|
value: 32.88970538547634
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: nfcorpus
|
|
name: MTEB NFCorpus
|
|
config: default
|
|
split: test
|
|
revision: 7eb63cc0c1eb59324d709ebed25fcab851fa7610
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 5.604
|
|
- type: map_at_10
|
|
value: 12.379999999999999
|
|
- type: map_at_100
|
|
value: 15.791
|
|
- type: map_at_1000
|
|
value: 17.327
|
|
- type: map_at_3
|
|
value: 9.15
|
|
- type: map_at_5
|
|
value: 10.599
|
|
- type: mrr_at_1
|
|
value: 45.201
|
|
- type: mrr_at_10
|
|
value: 53.374
|
|
- type: mrr_at_100
|
|
value: 54.089
|
|
- type: mrr_at_1000
|
|
value: 54.123
|
|
- type: mrr_at_3
|
|
value: 51.44499999999999
|
|
- type: mrr_at_5
|
|
value: 52.59
|
|
- type: ndcg_at_1
|
|
value: 42.879
|
|
- type: ndcg_at_10
|
|
value: 33.891
|
|
- type: ndcg_at_100
|
|
value: 31.391999999999996
|
|
- type: ndcg_at_1000
|
|
value: 40.36
|
|
- type: ndcg_at_3
|
|
value: 39.076
|
|
- type: ndcg_at_5
|
|
value: 37.047000000000004
|
|
- type: precision_at_1
|
|
value: 44.582
|
|
- type: precision_at_10
|
|
value: 25.294
|
|
- type: precision_at_100
|
|
value: 8.285
|
|
- type: precision_at_1000
|
|
value: 2.1479999999999997
|
|
- type: precision_at_3
|
|
value: 36.120000000000005
|
|
- type: precision_at_5
|
|
value: 31.95
|
|
- type: recall_at_1
|
|
value: 5.604
|
|
- type: recall_at_10
|
|
value: 16.239
|
|
- type: recall_at_100
|
|
value: 32.16
|
|
- type: recall_at_1000
|
|
value: 64.513
|
|
- type: recall_at_3
|
|
value: 10.406
|
|
- type: recall_at_5
|
|
value: 12.684999999999999
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: nq
|
|
name: MTEB NQ
|
|
config: default
|
|
split: test
|
|
revision: 6062aefc120bfe8ece5897809fb2e53bfe0d128c
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 25.881
|
|
- type: map_at_10
|
|
value: 39.501
|
|
- type: map_at_100
|
|
value: 40.615
|
|
- type: map_at_1000
|
|
value: 40.661
|
|
- type: map_at_3
|
|
value: 35.559000000000005
|
|
- type: map_at_5
|
|
value: 37.773
|
|
- type: mrr_at_1
|
|
value: 29.229
|
|
- type: mrr_at_10
|
|
value: 41.955999999999996
|
|
- type: mrr_at_100
|
|
value: 42.86
|
|
- type: mrr_at_1000
|
|
value: 42.893
|
|
- type: mrr_at_3
|
|
value: 38.562000000000005
|
|
- type: mrr_at_5
|
|
value: 40.542
|
|
- type: ndcg_at_1
|
|
value: 29.2
|
|
- type: ndcg_at_10
|
|
value: 46.703
|
|
- type: ndcg_at_100
|
|
value: 51.644
|
|
- type: ndcg_at_1000
|
|
value: 52.771
|
|
- type: ndcg_at_3
|
|
value: 39.141999999999996
|
|
- type: ndcg_at_5
|
|
value: 42.892
|
|
- type: precision_at_1
|
|
value: 29.2
|
|
- type: precision_at_10
|
|
value: 7.920000000000001
|
|
- type: precision_at_100
|
|
value: 1.0659999999999998
|
|
- type: precision_at_1000
|
|
value: 0.117
|
|
- type: precision_at_3
|
|
value: 18.105
|
|
- type: precision_at_5
|
|
value: 13.036
|
|
- type: recall_at_1
|
|
value: 25.881
|
|
- type: recall_at_10
|
|
value: 66.266
|
|
- type: recall_at_100
|
|
value: 88.116
|
|
- type: recall_at_1000
|
|
value: 96.58200000000001
|
|
- type: recall_at_3
|
|
value: 46.526
|
|
- type: recall_at_5
|
|
value: 55.154
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: quora
|
|
name: MTEB QuoraRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 6205996560df11e3a3da9ab4f926788fc30a7db4
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 67.553
|
|
- type: map_at_10
|
|
value: 81.34
|
|
- type: map_at_100
|
|
value: 82.002
|
|
- type: map_at_1000
|
|
value: 82.027
|
|
- type: map_at_3
|
|
value: 78.281
|
|
- type: map_at_5
|
|
value: 80.149
|
|
- type: mrr_at_1
|
|
value: 77.72
|
|
- type: mrr_at_10
|
|
value: 84.733
|
|
- type: mrr_at_100
|
|
value: 84.878
|
|
- type: mrr_at_1000
|
|
value: 84.879
|
|
- type: mrr_at_3
|
|
value: 83.587
|
|
- type: mrr_at_5
|
|
value: 84.32600000000001
|
|
- type: ndcg_at_1
|
|
value: 77.75
|
|
- type: ndcg_at_10
|
|
value: 85.603
|
|
- type: ndcg_at_100
|
|
value: 87.069
|
|
- type: ndcg_at_1000
|
|
value: 87.25
|
|
- type: ndcg_at_3
|
|
value: 82.303
|
|
- type: ndcg_at_5
|
|
value: 84.03699999999999
|
|
- type: precision_at_1
|
|
value: 77.75
|
|
- type: precision_at_10
|
|
value: 13.04
|
|
- type: precision_at_100
|
|
value: 1.5070000000000001
|
|
- type: precision_at_1000
|
|
value: 0.156
|
|
- type: precision_at_3
|
|
value: 35.903
|
|
- type: precision_at_5
|
|
value: 23.738
|
|
- type: recall_at_1
|
|
value: 67.553
|
|
- type: recall_at_10
|
|
value: 93.903
|
|
- type: recall_at_100
|
|
value: 99.062
|
|
- type: recall_at_1000
|
|
value: 99.935
|
|
- type: recall_at_3
|
|
value: 84.58099999999999
|
|
- type: recall_at_5
|
|
value: 89.316
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/reddit-clustering
|
|
name: MTEB RedditClustering
|
|
config: default
|
|
split: test
|
|
revision: b2805658ae38990172679479369a78b86de8c390
|
|
metrics:
|
|
- type: v_measure
|
|
value: 46.46887711230235
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/reddit-clustering-p2p
|
|
name: MTEB RedditClusteringP2P
|
|
config: default
|
|
split: test
|
|
revision: 385e3cb46b4cfa89021f56c4380204149d0efe33
|
|
metrics:
|
|
- type: v_measure
|
|
value: 54.166876298246926
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: scidocs
|
|
name: MTEB SCIDOCS
|
|
config: default
|
|
split: test
|
|
revision: 5c59ef3e437a0a9651c8fe6fde943e7dce59fba5
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 4.053
|
|
- type: map_at_10
|
|
value: 9.693999999999999
|
|
- type: map_at_100
|
|
value: 11.387
|
|
- type: map_at_1000
|
|
value: 11.654
|
|
- type: map_at_3
|
|
value: 7.053
|
|
- type: map_at_5
|
|
value: 8.439
|
|
- type: mrr_at_1
|
|
value: 19.900000000000002
|
|
- type: mrr_at_10
|
|
value: 29.359
|
|
- type: mrr_at_100
|
|
value: 30.484
|
|
- type: mrr_at_1000
|
|
value: 30.553
|
|
- type: mrr_at_3
|
|
value: 26.200000000000003
|
|
- type: mrr_at_5
|
|
value: 28.115000000000002
|
|
- type: ndcg_at_1
|
|
value: 19.900000000000002
|
|
- type: ndcg_at_10
|
|
value: 16.575
|
|
- type: ndcg_at_100
|
|
value: 23.655
|
|
- type: ndcg_at_1000
|
|
value: 28.853
|
|
- type: ndcg_at_3
|
|
value: 15.848
|
|
- type: ndcg_at_5
|
|
value: 14.026
|
|
- type: precision_at_1
|
|
value: 19.900000000000002
|
|
- type: precision_at_10
|
|
value: 8.450000000000001
|
|
- type: precision_at_100
|
|
value: 1.872
|
|
- type: precision_at_1000
|
|
value: 0.313
|
|
- type: precision_at_3
|
|
value: 14.667
|
|
- type: precision_at_5
|
|
value: 12.32
|
|
- type: recall_at_1
|
|
value: 4.053
|
|
- type: recall_at_10
|
|
value: 17.169999999999998
|
|
- type: recall_at_100
|
|
value: 38.025
|
|
- type: recall_at_1000
|
|
value: 63.571999999999996
|
|
- type: recall_at_3
|
|
value: 8.903
|
|
- type: recall_at_5
|
|
value: 12.477
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sickr-sts
|
|
name: MTEB SICK-R
|
|
config: default
|
|
split: test
|
|
revision: 20a6d6f312dd54037fe07a32d58e5e168867909d
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 77.7548748519677
|
|
- type: cos_sim_spearman
|
|
value: 68.19926431966059
|
|
- type: euclidean_pearson
|
|
value: 71.69016204991725
|
|
- type: euclidean_spearman
|
|
value: 66.98099673026834
|
|
- type: manhattan_pearson
|
|
value: 71.62994072488664
|
|
- type: manhattan_spearman
|
|
value: 67.03435950744577
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts12-sts
|
|
name: MTEB STS12
|
|
config: default
|
|
split: test
|
|
revision: fdf84275bb8ce4b49c971d02e84dd1abc677a50f
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 75.91051402657887
|
|
- type: cos_sim_spearman
|
|
value: 66.99390786191645
|
|
- type: euclidean_pearson
|
|
value: 71.54128036454578
|
|
- type: euclidean_spearman
|
|
value: 69.25605675649068
|
|
- type: manhattan_pearson
|
|
value: 71.60981030780171
|
|
- type: manhattan_spearman
|
|
value: 69.27513670128046
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts13-sts
|
|
name: MTEB STS13
|
|
config: default
|
|
split: test
|
|
revision: 1591bfcbe8c69d4bf7fe2a16e2451017832cafb9
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 77.23835466417793
|
|
- type: cos_sim_spearman
|
|
value: 77.57623085766706
|
|
- type: euclidean_pearson
|
|
value: 77.5090992200725
|
|
- type: euclidean_spearman
|
|
value: 77.88601688144924
|
|
- type: manhattan_pearson
|
|
value: 77.39045060647423
|
|
- type: manhattan_spearman
|
|
value: 77.77552718279098
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts14-sts
|
|
name: MTEB STS14
|
|
config: default
|
|
split: test
|
|
revision: e2125984e7df8b7871f6ae9949cf6b6795e7c54b
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 77.91692485139602
|
|
- type: cos_sim_spearman
|
|
value: 72.78258293483495
|
|
- type: euclidean_pearson
|
|
value: 74.64773017077789
|
|
- type: euclidean_spearman
|
|
value: 71.81662299104619
|
|
- type: manhattan_pearson
|
|
value: 74.71043337995533
|
|
- type: manhattan_spearman
|
|
value: 71.83960860845646
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts15-sts
|
|
name: MTEB STS15
|
|
config: default
|
|
split: test
|
|
revision: 1cd7298cac12a96a373b6a2f18738bb3e739a9b6
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 82.13422113617578
|
|
- type: cos_sim_spearman
|
|
value: 82.61707296911949
|
|
- type: euclidean_pearson
|
|
value: 81.42487480400861
|
|
- type: euclidean_spearman
|
|
value: 82.17970991273835
|
|
- type: manhattan_pearson
|
|
value: 81.41985055477845
|
|
- type: manhattan_spearman
|
|
value: 82.15823204362937
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts16-sts
|
|
name: MTEB STS16
|
|
config: default
|
|
split: test
|
|
revision: 360a0b2dff98700d09e634a01e1cc1624d3e42cd
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 79.07989542843826
|
|
- type: cos_sim_spearman
|
|
value: 80.09839524406284
|
|
- type: euclidean_pearson
|
|
value: 76.43186028364195
|
|
- type: euclidean_spearman
|
|
value: 76.76720323266471
|
|
- type: manhattan_pearson
|
|
value: 76.4674747409161
|
|
- type: manhattan_spearman
|
|
value: 76.81797407068667
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (en-en)
|
|
config: en-en
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 87.0420983224933
|
|
- type: cos_sim_spearman
|
|
value: 87.25017540413702
|
|
- type: euclidean_pearson
|
|
value: 84.56384596473421
|
|
- type: euclidean_spearman
|
|
value: 84.72557417564886
|
|
- type: manhattan_pearson
|
|
value: 84.7329954474549
|
|
- type: manhattan_spearman
|
|
value: 84.75071371008909
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (en)
|
|
config: en
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 68.47031320016424
|
|
- type: cos_sim_spearman
|
|
value: 68.7486910762485
|
|
- type: euclidean_pearson
|
|
value: 71.30330985913915
|
|
- type: euclidean_spearman
|
|
value: 71.59666258520735
|
|
- type: manhattan_pearson
|
|
value: 71.4423884279027
|
|
- type: manhattan_spearman
|
|
value: 71.67460706861044
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/stsbenchmark-sts
|
|
name: MTEB STSBenchmark
|
|
config: default
|
|
split: test
|
|
revision: 8913289635987208e6e7c72789e4be2fe94b6abd
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 80.79514366062675
|
|
- type: cos_sim_spearman
|
|
value: 79.20585637461048
|
|
- type: euclidean_pearson
|
|
value: 78.6591557395699
|
|
- type: euclidean_spearman
|
|
value: 77.86455794285718
|
|
- type: manhattan_pearson
|
|
value: 78.67754806486865
|
|
- type: manhattan_spearman
|
|
value: 77.88178687200732
|
|
- task:
|
|
type: Reranking
|
|
dataset:
|
|
type: mteb/scidocs-reranking
|
|
name: MTEB SciDocsRR
|
|
config: default
|
|
split: test
|
|
revision: 56a6d0140cf6356659e2a7c1413286a774468d44
|
|
metrics:
|
|
- type: map
|
|
value: 77.71580844366375
|
|
- type: mrr
|
|
value: 93.04215845882513
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: scifact
|
|
name: MTEB SciFact
|
|
config: default
|
|
split: test
|
|
revision: a75ae049398addde9b70f6b268875f5cbce99089
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 56.39999999999999
|
|
- type: map_at_10
|
|
value: 65.701
|
|
- type: map_at_100
|
|
value: 66.32000000000001
|
|
- type: map_at_1000
|
|
value: 66.34100000000001
|
|
- type: map_at_3
|
|
value: 62.641999999999996
|
|
- type: map_at_5
|
|
value: 64.342
|
|
- type: mrr_at_1
|
|
value: 58.667
|
|
- type: mrr_at_10
|
|
value: 66.45299999999999
|
|
- type: mrr_at_100
|
|
value: 66.967
|
|
- type: mrr_at_1000
|
|
value: 66.988
|
|
- type: mrr_at_3
|
|
value: 64.11099999999999
|
|
- type: mrr_at_5
|
|
value: 65.411
|
|
- type: ndcg_at_1
|
|
value: 58.667
|
|
- type: ndcg_at_10
|
|
value: 70.165
|
|
- type: ndcg_at_100
|
|
value: 72.938
|
|
- type: ndcg_at_1000
|
|
value: 73.456
|
|
- type: ndcg_at_3
|
|
value: 64.79
|
|
- type: ndcg_at_5
|
|
value: 67.28
|
|
- type: precision_at_1
|
|
value: 58.667
|
|
- type: precision_at_10
|
|
value: 9.4
|
|
- type: precision_at_100
|
|
value: 1.087
|
|
- type: precision_at_1000
|
|
value: 0.11299999999999999
|
|
- type: precision_at_3
|
|
value: 24.889
|
|
- type: precision_at_5
|
|
value: 16.667
|
|
- type: recall_at_1
|
|
value: 56.39999999999999
|
|
- type: recall_at_10
|
|
value: 83.122
|
|
- type: recall_at_100
|
|
value: 95.667
|
|
- type: recall_at_1000
|
|
value: 99.667
|
|
- type: recall_at_3
|
|
value: 68.378
|
|
- type: recall_at_5
|
|
value: 74.68299999999999
|
|
- task:
|
|
type: PairClassification
|
|
dataset:
|
|
type: mteb/sprintduplicatequestions-pairclassification
|
|
name: MTEB SprintDuplicateQuestions
|
|
config: default
|
|
split: test
|
|
revision: 5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea
|
|
metrics:
|
|
- type: cos_sim_accuracy
|
|
value: 99.76831683168317
|
|
- type: cos_sim_ap
|
|
value: 93.47124923047998
|
|
- type: cos_sim_f1
|
|
value: 88.06122448979592
|
|
- type: cos_sim_precision
|
|
value: 89.89583333333333
|
|
- type: cos_sim_recall
|
|
value: 86.3
|
|
- type: dot_accuracy
|
|
value: 99.57326732673268
|
|
- type: dot_ap
|
|
value: 84.06577868167207
|
|
- type: dot_f1
|
|
value: 77.82629791363416
|
|
- type: dot_precision
|
|
value: 75.58906691800189
|
|
- type: dot_recall
|
|
value: 80.2
|
|
- type: euclidean_accuracy
|
|
value: 99.74257425742574
|
|
- type: euclidean_ap
|
|
value: 92.1904681653555
|
|
- type: euclidean_f1
|
|
value: 86.74821610601427
|
|
- type: euclidean_precision
|
|
value: 88.46153846153845
|
|
- type: euclidean_recall
|
|
value: 85.1
|
|
- type: manhattan_accuracy
|
|
value: 99.74554455445545
|
|
- type: manhattan_ap
|
|
value: 92.4337790809948
|
|
- type: manhattan_f1
|
|
value: 86.86765457332653
|
|
- type: manhattan_precision
|
|
value: 88.81922675026124
|
|
- type: manhattan_recall
|
|
value: 85.0
|
|
- type: max_accuracy
|
|
value: 99.76831683168317
|
|
- type: max_ap
|
|
value: 93.47124923047998
|
|
- type: max_f1
|
|
value: 88.06122448979592
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/stackexchange-clustering
|
|
name: MTEB StackExchangeClustering
|
|
config: default
|
|
split: test
|
|
revision: 70a89468f6dccacc6aa2b12a6eac54e74328f235
|
|
metrics:
|
|
- type: v_measure
|
|
value: 59.194098673976484
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/stackexchange-clustering-p2p
|
|
name: MTEB StackExchangeClusteringP2P
|
|
config: default
|
|
split: test
|
|
revision: d88009ab563dd0b16cfaf4436abaf97fa3550cf0
|
|
metrics:
|
|
- type: v_measure
|
|
value: 32.5744032578115
|
|
- task:
|
|
type: Reranking
|
|
dataset:
|
|
type: mteb/stackoverflowdupquestions-reranking
|
|
name: MTEB StackOverflowDupQuestions
|
|
config: default
|
|
split: test
|
|
revision: ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9
|
|
metrics:
|
|
- type: map
|
|
value: 49.61186384154483
|
|
- type: mrr
|
|
value: 50.55424253034547
|
|
- task:
|
|
type: Summarization
|
|
dataset:
|
|
type: mteb/summeval
|
|
name: MTEB SummEval
|
|
config: default
|
|
split: test
|
|
revision: 8753c2788d36c01fc6f05d03fe3f7268d63f9122
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 30.027210161713946
|
|
- type: cos_sim_spearman
|
|
value: 31.030178065751735
|
|
- type: dot_pearson
|
|
value: 30.09179785685587
|
|
- type: dot_spearman
|
|
value: 30.408303252207813
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: trec-covid
|
|
name: MTEB TRECCOVID
|
|
config: default
|
|
split: test
|
|
revision: 2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 0.22300000000000003
|
|
- type: map_at_10
|
|
value: 1.762
|
|
- type: map_at_100
|
|
value: 9.984
|
|
- type: map_at_1000
|
|
value: 24.265
|
|
- type: map_at_3
|
|
value: 0.631
|
|
- type: map_at_5
|
|
value: 0.9950000000000001
|
|
- type: mrr_at_1
|
|
value: 88.0
|
|
- type: mrr_at_10
|
|
value: 92.833
|
|
- type: mrr_at_100
|
|
value: 92.833
|
|
- type: mrr_at_1000
|
|
value: 92.833
|
|
- type: mrr_at_3
|
|
value: 92.333
|
|
- type: mrr_at_5
|
|
value: 92.833
|
|
- type: ndcg_at_1
|
|
value: 83.0
|
|
- type: ndcg_at_10
|
|
value: 75.17
|
|
- type: ndcg_at_100
|
|
value: 55.432
|
|
- type: ndcg_at_1000
|
|
value: 49.482
|
|
- type: ndcg_at_3
|
|
value: 82.184
|
|
- type: ndcg_at_5
|
|
value: 79.712
|
|
- type: precision_at_1
|
|
value: 88.0
|
|
- type: precision_at_10
|
|
value: 78.60000000000001
|
|
- type: precision_at_100
|
|
value: 56.56
|
|
- type: precision_at_1000
|
|
value: 22.334
|
|
- type: precision_at_3
|
|
value: 86.667
|
|
- type: precision_at_5
|
|
value: 83.6
|
|
- type: recall_at_1
|
|
value: 0.22300000000000003
|
|
- type: recall_at_10
|
|
value: 1.9879999999999998
|
|
- type: recall_at_100
|
|
value: 13.300999999999998
|
|
- type: recall_at_1000
|
|
value: 46.587
|
|
- type: recall_at_3
|
|
value: 0.6629999999999999
|
|
- type: recall_at_5
|
|
value: 1.079
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: webis-touche2020
|
|
name: MTEB Touche2020
|
|
config: default
|
|
split: test
|
|
revision: 527b7d77e16e343303e68cb6af11d6e18b9f7b3b
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 3.047
|
|
- type: map_at_10
|
|
value: 8.792
|
|
- type: map_at_100
|
|
value: 14.631
|
|
- type: map_at_1000
|
|
value: 16.127
|
|
- type: map_at_3
|
|
value: 4.673
|
|
- type: map_at_5
|
|
value: 5.897
|
|
- type: mrr_at_1
|
|
value: 38.775999999999996
|
|
- type: mrr_at_10
|
|
value: 49.271
|
|
- type: mrr_at_100
|
|
value: 50.181
|
|
- type: mrr_at_1000
|
|
value: 50.2
|
|
- type: mrr_at_3
|
|
value: 44.558
|
|
- type: mrr_at_5
|
|
value: 47.925000000000004
|
|
- type: ndcg_at_1
|
|
value: 35.714
|
|
- type: ndcg_at_10
|
|
value: 23.44
|
|
- type: ndcg_at_100
|
|
value: 35.345
|
|
- type: ndcg_at_1000
|
|
value: 46.495
|
|
- type: ndcg_at_3
|
|
value: 26.146
|
|
- type: ndcg_at_5
|
|
value: 24.878
|
|
- type: precision_at_1
|
|
value: 38.775999999999996
|
|
- type: precision_at_10
|
|
value: 20.816000000000003
|
|
- type: precision_at_100
|
|
value: 7.428999999999999
|
|
- type: precision_at_1000
|
|
value: 1.494
|
|
- type: precision_at_3
|
|
value: 25.85
|
|
- type: precision_at_5
|
|
value: 24.082
|
|
- type: recall_at_1
|
|
value: 3.047
|
|
- type: recall_at_10
|
|
value: 14.975
|
|
- type: recall_at_100
|
|
value: 45.943
|
|
- type: recall_at_1000
|
|
value: 80.31099999999999
|
|
- type: recall_at_3
|
|
value: 5.478000000000001
|
|
- type: recall_at_5
|
|
value: 8.294
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/toxic_conversations_50k
|
|
name: MTEB ToxicConversationsClassification
|
|
config: default
|
|
split: test
|
|
revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de
|
|
metrics:
|
|
- type: accuracy
|
|
value: 68.84080000000002
|
|
- type: ap
|
|
value: 13.135219251019848
|
|
- type: f1
|
|
value: 52.849999421995506
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/tweet_sentiment_extraction
|
|
name: MTEB TweetSentimentExtractionClassification
|
|
config: default
|
|
split: test
|
|
revision: 62146448f05be9e52a36b8ee9936447ea787eede
|
|
metrics:
|
|
- type: accuracy
|
|
value: 56.68647425014149
|
|
- type: f1
|
|
value: 56.97981427365949
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/twentynewsgroups-clustering
|
|
name: MTEB TwentyNewsgroupsClustering
|
|
config: default
|
|
split: test
|
|
revision: 091a54f9a36281ce7d6590ec8c75dd485e7e01d4
|
|
metrics:
|
|
- type: v_measure
|
|
value: 40.8911707239219
|
|
- task:
|
|
type: PairClassification
|
|
dataset:
|
|
type: mteb/twittersemeval2015-pairclassification
|
|
name: MTEB TwitterSemEval2015
|
|
config: default
|
|
split: test
|
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
|
|
metrics:
|
|
- type: cos_sim_accuracy
|
|
value: 83.04226023722954
|
|
- type: cos_sim_ap
|
|
value: 63.681339908301325
|
|
- type: cos_sim_f1
|
|
value: 60.349184470480125
|
|
- type: cos_sim_precision
|
|
value: 53.437754271765655
|
|
- type: cos_sim_recall
|
|
value: 69.31398416886545
|
|
- type: dot_accuracy
|
|
value: 81.46271681468677
|
|
- type: dot_ap
|
|
value: 57.78072296265885
|
|
- type: dot_f1
|
|
value: 56.28769265132901
|
|
- type: dot_precision
|
|
value: 48.7993803253292
|
|
- type: dot_recall
|
|
value: 66.49076517150397
|
|
- type: euclidean_accuracy
|
|
value: 82.16606067830959
|
|
- type: euclidean_ap
|
|
value: 59.974530371203514
|
|
- type: euclidean_f1
|
|
value: 56.856023506366306
|
|
- type: euclidean_precision
|
|
value: 53.037916857012334
|
|
- type: euclidean_recall
|
|
value: 61.2664907651715
|
|
- type: manhattan_accuracy
|
|
value: 82.16606067830959
|
|
- type: manhattan_ap
|
|
value: 59.98962379571767
|
|
- type: manhattan_f1
|
|
value: 56.98153158451947
|
|
- type: manhattan_precision
|
|
value: 51.41158989598811
|
|
- type: manhattan_recall
|
|
value: 63.90501319261214
|
|
- type: max_accuracy
|
|
value: 83.04226023722954
|
|
- type: max_ap
|
|
value: 63.681339908301325
|
|
- type: max_f1
|
|
value: 60.349184470480125
|
|
- task:
|
|
type: PairClassification
|
|
dataset:
|
|
type: mteb/twitterurlcorpus-pairclassification
|
|
name: MTEB TwitterURLCorpus
|
|
config: default
|
|
split: test
|
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
|
|
metrics:
|
|
- type: cos_sim_accuracy
|
|
value: 88.56871191834517
|
|
- type: cos_sim_ap
|
|
value: 84.80240716354544
|
|
- type: cos_sim_f1
|
|
value: 77.07765285922385
|
|
- type: cos_sim_precision
|
|
value: 74.84947406601378
|
|
- type: cos_sim_recall
|
|
value: 79.44256236526024
|
|
- type: dot_accuracy
|
|
value: 86.00923662048356
|
|
- type: dot_ap
|
|
value: 78.6556459012073
|
|
- type: dot_f1
|
|
value: 72.7583749109052
|
|
- type: dot_precision
|
|
value: 67.72823779193206
|
|
- type: dot_recall
|
|
value: 78.59562673236834
|
|
- type: euclidean_accuracy
|
|
value: 87.84103698529127
|
|
- type: euclidean_ap
|
|
value: 83.50424424952834
|
|
- type: euclidean_f1
|
|
value: 75.74496544549307
|
|
- type: euclidean_precision
|
|
value: 73.19402556369381
|
|
- type: euclidean_recall
|
|
value: 78.48013550970127
|
|
- type: manhattan_accuracy
|
|
value: 87.9225365777933
|
|
- type: manhattan_ap
|
|
value: 83.49479248597825
|
|
- type: manhattan_f1
|
|
value: 75.67748162447101
|
|
- type: manhattan_precision
|
|
value: 73.06810035842294
|
|
- type: manhattan_recall
|
|
value: 78.48013550970127
|
|
- type: max_accuracy
|
|
value: 88.56871191834517
|
|
- type: max_ap
|
|
value: 84.80240716354544
|
|
- type: max_f1
|
|
value: 77.07765285922385
|
|
---
|
|
|
|
# SGPT-2.7B-weightedmean-msmarco-specb-bitfit
|
|
|
|
## Usage
|
|
|
|
For usage instructions, refer to our codebase: https://github.com/Muennighoff/sgpt
|
|
|
|
## Evaluation Results
|
|
|
|
For eval results, refer to the eval folder or our paper: https://arxiv.org/abs/2202.08904
|
|
|
|
## Training
|
|
The model was trained with the parameters:
|
|
|
|
**DataLoader**:
|
|
|
|
`torch.utils.data.dataloader.DataLoader` of length 124796 with parameters:
|
|
```
|
|
{'batch_size': 4, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
|
|
```
|
|
|
|
**Loss**:
|
|
|
|
`sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss` with parameters:
|
|
```
|
|
{'scale': 20.0, 'similarity_fct': 'cos_sim'}
|
|
```
|
|
|
|
Parameters of the fit()-Method:
|
|
```
|
|
{
|
|
"epochs": 10,
|
|
"evaluation_steps": 0,
|
|
"evaluator": "NoneType",
|
|
"max_grad_norm": 1,
|
|
"optimizer_class": "<class 'transformers.optimization.AdamW'>",
|
|
"optimizer_params": {
|
|
"lr": 7.5e-05
|
|
},
|
|
"scheduler": "WarmupLinear",
|
|
"steps_per_epoch": null,
|
|
"warmup_steps": 1000,
|
|
"weight_decay": 0.01
|
|
}
|
|
```
|
|
|
|
|
|
## Full Model Architecture
|
|
```
|
|
SentenceTransformer(
|
|
(0): Transformer({'max_seq_length': 300, 'do_lower_case': False}) with Transformer model: GPTNeoModel
|
|
(1): Pooling({'word_embedding_dimension': 2560, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
|
|
)
|
|
```
|
|
|
|
## Citing & Authors
|
|
|
|
```bibtex
|
|
@article{muennighoff2022sgpt,
|
|
title={SGPT: GPT Sentence Embeddings for Semantic Search},
|
|
author={Muennighoff, Niklas},
|
|
journal={arXiv preprint arXiv:2202.08904},
|
|
year={2022}
|
|
}
|
|
``` |