6752 lines
167 KiB
Markdown
6752 lines
167 KiB
Markdown
---
|
|
pipeline_tag: sentence-similarity
|
|
tags:
|
|
- sentence-transformers
|
|
- feature-extraction
|
|
- sentence-similarity
|
|
- mteb
|
|
model-index:
|
|
- name: sgpt-bloom-7b1-msmarco
|
|
results:
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_counterfactual
|
|
name: MTEB AmazonCounterfactualClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
|
|
metrics:
|
|
- type: accuracy
|
|
value: 68.05970149253731
|
|
- type: ap
|
|
value: 31.640363460776193
|
|
- type: f1
|
|
value: 62.50025574145796
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_counterfactual
|
|
name: MTEB AmazonCounterfactualClassification (de)
|
|
config: de
|
|
split: test
|
|
revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
|
|
metrics:
|
|
- type: accuracy
|
|
value: 61.34903640256959
|
|
- type: ap
|
|
value: 75.18797161500426
|
|
- type: f1
|
|
value: 59.04772570730417
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_counterfactual
|
|
name: MTEB AmazonCounterfactualClassification (en-ext)
|
|
config: en-ext
|
|
split: test
|
|
revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
|
|
metrics:
|
|
- type: accuracy
|
|
value: 67.78110944527737
|
|
- type: ap
|
|
value: 19.218916023322706
|
|
- type: f1
|
|
value: 56.24477391445512
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_counterfactual
|
|
name: MTEB AmazonCounterfactualClassification (ja)
|
|
config: ja
|
|
split: test
|
|
revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
|
|
metrics:
|
|
- type: accuracy
|
|
value: 58.23340471092078
|
|
- type: ap
|
|
value: 13.20222967424681
|
|
- type: f1
|
|
value: 47.511718095460296
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_polarity
|
|
name: MTEB AmazonPolarityClassification
|
|
config: default
|
|
split: test
|
|
revision: 80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1
|
|
metrics:
|
|
- type: accuracy
|
|
value: 68.97232499999998
|
|
- type: ap
|
|
value: 63.53632885535693
|
|
- type: f1
|
|
value: 68.62038513152868
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_reviews_multi
|
|
name: MTEB AmazonReviewsClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: c379a6705fec24a2493fa68e011692605f44e119
|
|
metrics:
|
|
- type: accuracy
|
|
value: 33.855999999999995
|
|
- type: f1
|
|
value: 33.43468222830134
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_reviews_multi
|
|
name: MTEB AmazonReviewsClassification (de)
|
|
config: de
|
|
split: test
|
|
revision: c379a6705fec24a2493fa68e011692605f44e119
|
|
metrics:
|
|
- type: accuracy
|
|
value: 29.697999999999997
|
|
- type: f1
|
|
value: 29.39935388885501
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_reviews_multi
|
|
name: MTEB AmazonReviewsClassification (es)
|
|
config: es
|
|
split: test
|
|
revision: c379a6705fec24a2493fa68e011692605f44e119
|
|
metrics:
|
|
- type: accuracy
|
|
value: 35.974000000000004
|
|
- type: f1
|
|
value: 35.25910820714383
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_reviews_multi
|
|
name: MTEB AmazonReviewsClassification (fr)
|
|
config: fr
|
|
split: test
|
|
revision: c379a6705fec24a2493fa68e011692605f44e119
|
|
metrics:
|
|
- type: accuracy
|
|
value: 35.922
|
|
- type: f1
|
|
value: 35.38637028933444
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_reviews_multi
|
|
name: MTEB AmazonReviewsClassification (ja)
|
|
config: ja
|
|
split: test
|
|
revision: c379a6705fec24a2493fa68e011692605f44e119
|
|
metrics:
|
|
- type: accuracy
|
|
value: 27.636
|
|
- type: f1
|
|
value: 27.178349955978266
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_reviews_multi
|
|
name: MTEB AmazonReviewsClassification (zh)
|
|
config: zh
|
|
split: test
|
|
revision: c379a6705fec24a2493fa68e011692605f44e119
|
|
metrics:
|
|
- type: accuracy
|
|
value: 32.632
|
|
- type: f1
|
|
value: 32.08014766494587
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: arguana
|
|
name: MTEB ArguAna
|
|
config: default
|
|
split: test
|
|
revision: 5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 23.684
|
|
- type: map_at_10
|
|
value: 38.507999999999996
|
|
- type: map_at_100
|
|
value: 39.677
|
|
- type: map_at_1000
|
|
value: 39.690999999999995
|
|
- type: map_at_3
|
|
value: 33.369
|
|
- type: map_at_5
|
|
value: 36.15
|
|
- type: mrr_at_1
|
|
value: 24.04
|
|
- type: mrr_at_10
|
|
value: 38.664
|
|
- type: mrr_at_100
|
|
value: 39.833
|
|
- type: mrr_at_1000
|
|
value: 39.847
|
|
- type: mrr_at_3
|
|
value: 33.476
|
|
- type: mrr_at_5
|
|
value: 36.306
|
|
- type: ndcg_at_1
|
|
value: 23.684
|
|
- type: ndcg_at_10
|
|
value: 47.282000000000004
|
|
- type: ndcg_at_100
|
|
value: 52.215
|
|
- type: ndcg_at_1000
|
|
value: 52.551
|
|
- type: ndcg_at_3
|
|
value: 36.628
|
|
- type: ndcg_at_5
|
|
value: 41.653
|
|
- type: precision_at_1
|
|
value: 23.684
|
|
- type: precision_at_10
|
|
value: 7.553
|
|
- type: precision_at_100
|
|
value: 0.97
|
|
- type: precision_at_1000
|
|
value: 0.1
|
|
- type: precision_at_3
|
|
value: 15.363
|
|
- type: precision_at_5
|
|
value: 11.664
|
|
- type: recall_at_1
|
|
value: 23.684
|
|
- type: recall_at_10
|
|
value: 75.533
|
|
- type: recall_at_100
|
|
value: 97.013
|
|
- type: recall_at_1000
|
|
value: 99.57300000000001
|
|
- type: recall_at_3
|
|
value: 46.088
|
|
- type: recall_at_5
|
|
value: 58.321
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/arxiv-clustering-p2p
|
|
name: MTEB ArxivClusteringP2P
|
|
config: default
|
|
split: test
|
|
revision: 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8
|
|
metrics:
|
|
- type: v_measure
|
|
value: 44.59375023881131
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/arxiv-clustering-s2s
|
|
name: MTEB ArxivClusteringS2S
|
|
config: default
|
|
split: test
|
|
revision: b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3
|
|
metrics:
|
|
- type: v_measure
|
|
value: 38.02921907752556
|
|
- task:
|
|
type: Reranking
|
|
dataset:
|
|
type: mteb/askubuntudupquestions-reranking
|
|
name: MTEB AskUbuntuDupQuestions
|
|
config: default
|
|
split: test
|
|
revision: 4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c
|
|
metrics:
|
|
- type: map
|
|
value: 59.97321570342109
|
|
- type: mrr
|
|
value: 73.18284746955106
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/biosses-sts
|
|
name: MTEB BIOSSES
|
|
config: default
|
|
split: test
|
|
revision: 9ee918f184421b6bd48b78f6c714d86546106103
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 89.09091435741429
|
|
- type: cos_sim_spearman
|
|
value: 85.31459455332202
|
|
- type: euclidean_pearson
|
|
value: 79.3587681410798
|
|
- type: euclidean_spearman
|
|
value: 76.8174129874685
|
|
- type: manhattan_pearson
|
|
value: 79.57051762121769
|
|
- type: manhattan_spearman
|
|
value: 76.75837549768094
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/bucc-bitext-mining
|
|
name: MTEB BUCC (de-en)
|
|
config: de-en
|
|
split: test
|
|
revision: d51519689f32196a32af33b075a01d0e7c51e252
|
|
metrics:
|
|
- type: accuracy
|
|
value: 54.27974947807933
|
|
- type: f1
|
|
value: 54.00144411132214
|
|
- type: precision
|
|
value: 53.87119374071357
|
|
- type: recall
|
|
value: 54.27974947807933
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/bucc-bitext-mining
|
|
name: MTEB BUCC (fr-en)
|
|
config: fr-en
|
|
split: test
|
|
revision: d51519689f32196a32af33b075a01d0e7c51e252
|
|
metrics:
|
|
- type: accuracy
|
|
value: 97.3365617433414
|
|
- type: f1
|
|
value: 97.06141316310809
|
|
- type: precision
|
|
value: 96.92567319685965
|
|
- type: recall
|
|
value: 97.3365617433414
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/bucc-bitext-mining
|
|
name: MTEB BUCC (ru-en)
|
|
config: ru-en
|
|
split: test
|
|
revision: d51519689f32196a32af33b075a01d0e7c51e252
|
|
metrics:
|
|
- type: accuracy
|
|
value: 46.05472809144441
|
|
- type: f1
|
|
value: 45.30319274690595
|
|
- type: precision
|
|
value: 45.00015469655234
|
|
- type: recall
|
|
value: 46.05472809144441
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/bucc-bitext-mining
|
|
name: MTEB BUCC (zh-en)
|
|
config: zh-en
|
|
split: test
|
|
revision: d51519689f32196a32af33b075a01d0e7c51e252
|
|
metrics:
|
|
- type: accuracy
|
|
value: 98.10426540284361
|
|
- type: f1
|
|
value: 97.96384061786905
|
|
- type: precision
|
|
value: 97.89362822538178
|
|
- type: recall
|
|
value: 98.10426540284361
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/banking77
|
|
name: MTEB Banking77Classification
|
|
config: default
|
|
split: test
|
|
revision: 44fa15921b4c889113cc5df03dd4901b49161ab7
|
|
metrics:
|
|
- type: accuracy
|
|
value: 84.33441558441558
|
|
- type: f1
|
|
value: 84.31653077470322
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/biorxiv-clustering-p2p
|
|
name: MTEB BiorxivClusteringP2P
|
|
config: default
|
|
split: test
|
|
revision: 11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55
|
|
metrics:
|
|
- type: v_measure
|
|
value: 36.025318694698086
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/biorxiv-clustering-s2s
|
|
name: MTEB BiorxivClusteringS2S
|
|
config: default
|
|
split: test
|
|
revision: c0fab014e1bcb8d3a5e31b2088972a1e01547dc1
|
|
metrics:
|
|
- type: v_measure
|
|
value: 32.484889034590346
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackAndroidRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 30.203999999999997
|
|
- type: map_at_10
|
|
value: 41.314
|
|
- type: map_at_100
|
|
value: 42.66
|
|
- type: map_at_1000
|
|
value: 42.775999999999996
|
|
- type: map_at_3
|
|
value: 37.614999999999995
|
|
- type: map_at_5
|
|
value: 39.643
|
|
- type: mrr_at_1
|
|
value: 37.482
|
|
- type: mrr_at_10
|
|
value: 47.075
|
|
- type: mrr_at_100
|
|
value: 47.845
|
|
- type: mrr_at_1000
|
|
value: 47.887
|
|
- type: mrr_at_3
|
|
value: 44.635000000000005
|
|
- type: mrr_at_5
|
|
value: 45.966
|
|
- type: ndcg_at_1
|
|
value: 37.482
|
|
- type: ndcg_at_10
|
|
value: 47.676
|
|
- type: ndcg_at_100
|
|
value: 52.915
|
|
- type: ndcg_at_1000
|
|
value: 54.82900000000001
|
|
- type: ndcg_at_3
|
|
value: 42.562
|
|
- type: ndcg_at_5
|
|
value: 44.852
|
|
- type: precision_at_1
|
|
value: 37.482
|
|
- type: precision_at_10
|
|
value: 9.142
|
|
- type: precision_at_100
|
|
value: 1.436
|
|
- type: precision_at_1000
|
|
value: 0.189
|
|
- type: precision_at_3
|
|
value: 20.458000000000002
|
|
- type: precision_at_5
|
|
value: 14.821000000000002
|
|
- type: recall_at_1
|
|
value: 30.203999999999997
|
|
- type: recall_at_10
|
|
value: 60.343
|
|
- type: recall_at_100
|
|
value: 82.58
|
|
- type: recall_at_1000
|
|
value: 94.813
|
|
- type: recall_at_3
|
|
value: 45.389
|
|
- type: recall_at_5
|
|
value: 51.800999999999995
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackEnglishRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 30.889
|
|
- type: map_at_10
|
|
value: 40.949999999999996
|
|
- type: map_at_100
|
|
value: 42.131
|
|
- type: map_at_1000
|
|
value: 42.253
|
|
- type: map_at_3
|
|
value: 38.346999999999994
|
|
- type: map_at_5
|
|
value: 39.782000000000004
|
|
- type: mrr_at_1
|
|
value: 38.79
|
|
- type: mrr_at_10
|
|
value: 46.944
|
|
- type: mrr_at_100
|
|
value: 47.61
|
|
- type: mrr_at_1000
|
|
value: 47.650999999999996
|
|
- type: mrr_at_3
|
|
value: 45.053
|
|
- type: mrr_at_5
|
|
value: 46.101
|
|
- type: ndcg_at_1
|
|
value: 38.79
|
|
- type: ndcg_at_10
|
|
value: 46.286
|
|
- type: ndcg_at_100
|
|
value: 50.637
|
|
- type: ndcg_at_1000
|
|
value: 52.649
|
|
- type: ndcg_at_3
|
|
value: 42.851
|
|
- type: ndcg_at_5
|
|
value: 44.311
|
|
- type: precision_at_1
|
|
value: 38.79
|
|
- type: precision_at_10
|
|
value: 8.516
|
|
- type: precision_at_100
|
|
value: 1.3679999999999999
|
|
- type: precision_at_1000
|
|
value: 0.183
|
|
- type: precision_at_3
|
|
value: 20.637
|
|
- type: precision_at_5
|
|
value: 14.318
|
|
- type: recall_at_1
|
|
value: 30.889
|
|
- type: recall_at_10
|
|
value: 55.327000000000005
|
|
- type: recall_at_100
|
|
value: 74.091
|
|
- type: recall_at_1000
|
|
value: 86.75500000000001
|
|
- type: recall_at_3
|
|
value: 44.557
|
|
- type: recall_at_5
|
|
value: 49.064
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackGamingRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 39.105000000000004
|
|
- type: map_at_10
|
|
value: 50.928
|
|
- type: map_at_100
|
|
value: 51.958000000000006
|
|
- type: map_at_1000
|
|
value: 52.017
|
|
- type: map_at_3
|
|
value: 47.638999999999996
|
|
- type: map_at_5
|
|
value: 49.624
|
|
- type: mrr_at_1
|
|
value: 44.639
|
|
- type: mrr_at_10
|
|
value: 54.261
|
|
- type: mrr_at_100
|
|
value: 54.913999999999994
|
|
- type: mrr_at_1000
|
|
value: 54.945
|
|
- type: mrr_at_3
|
|
value: 51.681999999999995
|
|
- type: mrr_at_5
|
|
value: 53.290000000000006
|
|
- type: ndcg_at_1
|
|
value: 44.639
|
|
- type: ndcg_at_10
|
|
value: 56.678
|
|
- type: ndcg_at_100
|
|
value: 60.649
|
|
- type: ndcg_at_1000
|
|
value: 61.855000000000004
|
|
- type: ndcg_at_3
|
|
value: 51.092999999999996
|
|
- type: ndcg_at_5
|
|
value: 54.096999999999994
|
|
- type: precision_at_1
|
|
value: 44.639
|
|
- type: precision_at_10
|
|
value: 9.028
|
|
- type: precision_at_100
|
|
value: 1.194
|
|
- type: precision_at_1000
|
|
value: 0.135
|
|
- type: precision_at_3
|
|
value: 22.508
|
|
- type: precision_at_5
|
|
value: 15.661
|
|
- type: recall_at_1
|
|
value: 39.105000000000004
|
|
- type: recall_at_10
|
|
value: 70.367
|
|
- type: recall_at_100
|
|
value: 87.359
|
|
- type: recall_at_1000
|
|
value: 95.88
|
|
- type: recall_at_3
|
|
value: 55.581
|
|
- type: recall_at_5
|
|
value: 62.821000000000005
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackGisRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 23.777
|
|
- type: map_at_10
|
|
value: 32.297
|
|
- type: map_at_100
|
|
value: 33.516
|
|
- type: map_at_1000
|
|
value: 33.592
|
|
- type: map_at_3
|
|
value: 30.001
|
|
- type: map_at_5
|
|
value: 31.209999999999997
|
|
- type: mrr_at_1
|
|
value: 25.989
|
|
- type: mrr_at_10
|
|
value: 34.472
|
|
- type: mrr_at_100
|
|
value: 35.518
|
|
- type: mrr_at_1000
|
|
value: 35.577
|
|
- type: mrr_at_3
|
|
value: 32.185
|
|
- type: mrr_at_5
|
|
value: 33.399
|
|
- type: ndcg_at_1
|
|
value: 25.989
|
|
- type: ndcg_at_10
|
|
value: 37.037
|
|
- type: ndcg_at_100
|
|
value: 42.699
|
|
- type: ndcg_at_1000
|
|
value: 44.725
|
|
- type: ndcg_at_3
|
|
value: 32.485
|
|
- type: ndcg_at_5
|
|
value: 34.549
|
|
- type: precision_at_1
|
|
value: 25.989
|
|
- type: precision_at_10
|
|
value: 5.718
|
|
- type: precision_at_100
|
|
value: 0.89
|
|
- type: precision_at_1000
|
|
value: 0.11
|
|
- type: precision_at_3
|
|
value: 14.049
|
|
- type: precision_at_5
|
|
value: 9.672
|
|
- type: recall_at_1
|
|
value: 23.777
|
|
- type: recall_at_10
|
|
value: 49.472
|
|
- type: recall_at_100
|
|
value: 74.857
|
|
- type: recall_at_1000
|
|
value: 90.289
|
|
- type: recall_at_3
|
|
value: 37.086000000000006
|
|
- type: recall_at_5
|
|
value: 42.065999999999995
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackMathematicaRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 13.377
|
|
- type: map_at_10
|
|
value: 21.444
|
|
- type: map_at_100
|
|
value: 22.663
|
|
- type: map_at_1000
|
|
value: 22.8
|
|
- type: map_at_3
|
|
value: 18.857
|
|
- type: map_at_5
|
|
value: 20.426
|
|
- type: mrr_at_1
|
|
value: 16.542
|
|
- type: mrr_at_10
|
|
value: 25.326999999999998
|
|
- type: mrr_at_100
|
|
value: 26.323
|
|
- type: mrr_at_1000
|
|
value: 26.406000000000002
|
|
- type: mrr_at_3
|
|
value: 22.823
|
|
- type: mrr_at_5
|
|
value: 24.340999999999998
|
|
- type: ndcg_at_1
|
|
value: 16.542
|
|
- type: ndcg_at_10
|
|
value: 26.479000000000003
|
|
- type: ndcg_at_100
|
|
value: 32.29
|
|
- type: ndcg_at_1000
|
|
value: 35.504999999999995
|
|
- type: ndcg_at_3
|
|
value: 21.619
|
|
- type: ndcg_at_5
|
|
value: 24.19
|
|
- type: precision_at_1
|
|
value: 16.542
|
|
- type: precision_at_10
|
|
value: 5.075
|
|
- type: precision_at_100
|
|
value: 0.9339999999999999
|
|
- type: precision_at_1000
|
|
value: 0.135
|
|
- type: precision_at_3
|
|
value: 10.697
|
|
- type: precision_at_5
|
|
value: 8.134
|
|
- type: recall_at_1
|
|
value: 13.377
|
|
- type: recall_at_10
|
|
value: 38.027
|
|
- type: recall_at_100
|
|
value: 63.439
|
|
- type: recall_at_1000
|
|
value: 86.354
|
|
- type: recall_at_3
|
|
value: 25.0
|
|
- type: recall_at_5
|
|
value: 31.306
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackPhysicsRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 28.368
|
|
- type: map_at_10
|
|
value: 39.305
|
|
- type: map_at_100
|
|
value: 40.637
|
|
- type: map_at_1000
|
|
value: 40.753
|
|
- type: map_at_3
|
|
value: 36.077999999999996
|
|
- type: map_at_5
|
|
value: 37.829
|
|
- type: mrr_at_1
|
|
value: 34.937000000000005
|
|
- type: mrr_at_10
|
|
value: 45.03
|
|
- type: mrr_at_100
|
|
value: 45.78
|
|
- type: mrr_at_1000
|
|
value: 45.827
|
|
- type: mrr_at_3
|
|
value: 42.348
|
|
- type: mrr_at_5
|
|
value: 43.807
|
|
- type: ndcg_at_1
|
|
value: 34.937000000000005
|
|
- type: ndcg_at_10
|
|
value: 45.605000000000004
|
|
- type: ndcg_at_100
|
|
value: 50.941
|
|
- type: ndcg_at_1000
|
|
value: 52.983000000000004
|
|
- type: ndcg_at_3
|
|
value: 40.366
|
|
- type: ndcg_at_5
|
|
value: 42.759
|
|
- type: precision_at_1
|
|
value: 34.937000000000005
|
|
- type: precision_at_10
|
|
value: 8.402
|
|
- type: precision_at_100
|
|
value: 1.2959999999999998
|
|
- type: precision_at_1000
|
|
value: 0.164
|
|
- type: precision_at_3
|
|
value: 19.217000000000002
|
|
- type: precision_at_5
|
|
value: 13.725000000000001
|
|
- type: recall_at_1
|
|
value: 28.368
|
|
- type: recall_at_10
|
|
value: 58.5
|
|
- type: recall_at_100
|
|
value: 80.67999999999999
|
|
- type: recall_at_1000
|
|
value: 93.925
|
|
- type: recall_at_3
|
|
value: 43.956
|
|
- type: recall_at_5
|
|
value: 50.065000000000005
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackProgrammersRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 24.851
|
|
- type: map_at_10
|
|
value: 34.758
|
|
- type: map_at_100
|
|
value: 36.081
|
|
- type: map_at_1000
|
|
value: 36.205999999999996
|
|
- type: map_at_3
|
|
value: 31.678
|
|
- type: map_at_5
|
|
value: 33.398
|
|
- type: mrr_at_1
|
|
value: 31.279
|
|
- type: mrr_at_10
|
|
value: 40.138
|
|
- type: mrr_at_100
|
|
value: 41.005
|
|
- type: mrr_at_1000
|
|
value: 41.065000000000005
|
|
- type: mrr_at_3
|
|
value: 37.519000000000005
|
|
- type: mrr_at_5
|
|
value: 38.986
|
|
- type: ndcg_at_1
|
|
value: 31.279
|
|
- type: ndcg_at_10
|
|
value: 40.534
|
|
- type: ndcg_at_100
|
|
value: 46.093
|
|
- type: ndcg_at_1000
|
|
value: 48.59
|
|
- type: ndcg_at_3
|
|
value: 35.473
|
|
- type: ndcg_at_5
|
|
value: 37.801
|
|
- type: precision_at_1
|
|
value: 31.279
|
|
- type: precision_at_10
|
|
value: 7.477
|
|
- type: precision_at_100
|
|
value: 1.2
|
|
- type: precision_at_1000
|
|
value: 0.159
|
|
- type: precision_at_3
|
|
value: 17.047
|
|
- type: precision_at_5
|
|
value: 12.306000000000001
|
|
- type: recall_at_1
|
|
value: 24.851
|
|
- type: recall_at_10
|
|
value: 52.528
|
|
- type: recall_at_100
|
|
value: 76.198
|
|
- type: recall_at_1000
|
|
value: 93.12
|
|
- type: recall_at_3
|
|
value: 38.257999999999996
|
|
- type: recall_at_5
|
|
value: 44.440000000000005
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 25.289833333333334
|
|
- type: map_at_10
|
|
value: 34.379333333333335
|
|
- type: map_at_100
|
|
value: 35.56916666666666
|
|
- type: map_at_1000
|
|
value: 35.68633333333333
|
|
- type: map_at_3
|
|
value: 31.63916666666666
|
|
- type: map_at_5
|
|
value: 33.18383333333334
|
|
- type: mrr_at_1
|
|
value: 30.081749999999996
|
|
- type: mrr_at_10
|
|
value: 38.53658333333333
|
|
- type: mrr_at_100
|
|
value: 39.37825
|
|
- type: mrr_at_1000
|
|
value: 39.43866666666666
|
|
- type: mrr_at_3
|
|
value: 36.19025
|
|
- type: mrr_at_5
|
|
value: 37.519749999999995
|
|
- type: ndcg_at_1
|
|
value: 30.081749999999996
|
|
- type: ndcg_at_10
|
|
value: 39.62041666666667
|
|
- type: ndcg_at_100
|
|
value: 44.74825
|
|
- type: ndcg_at_1000
|
|
value: 47.11366666666667
|
|
- type: ndcg_at_3
|
|
value: 35.000499999999995
|
|
- type: ndcg_at_5
|
|
value: 37.19283333333333
|
|
- type: precision_at_1
|
|
value: 30.081749999999996
|
|
- type: precision_at_10
|
|
value: 6.940249999999999
|
|
- type: precision_at_100
|
|
value: 1.1164166666666668
|
|
- type: precision_at_1000
|
|
value: 0.15025000000000002
|
|
- type: precision_at_3
|
|
value: 16.110416666666666
|
|
- type: precision_at_5
|
|
value: 11.474416666666668
|
|
- type: recall_at_1
|
|
value: 25.289833333333334
|
|
- type: recall_at_10
|
|
value: 51.01591666666667
|
|
- type: recall_at_100
|
|
value: 73.55275000000002
|
|
- type: recall_at_1000
|
|
value: 90.02666666666667
|
|
- type: recall_at_3
|
|
value: 38.15208333333334
|
|
- type: recall_at_5
|
|
value: 43.78458333333334
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackStatsRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 23.479
|
|
- type: map_at_10
|
|
value: 31.2
|
|
- type: map_at_100
|
|
value: 32.11
|
|
- type: map_at_1000
|
|
value: 32.214
|
|
- type: map_at_3
|
|
value: 29.093999999999998
|
|
- type: map_at_5
|
|
value: 30.415
|
|
- type: mrr_at_1
|
|
value: 26.840000000000003
|
|
- type: mrr_at_10
|
|
value: 34.153
|
|
- type: mrr_at_100
|
|
value: 34.971000000000004
|
|
- type: mrr_at_1000
|
|
value: 35.047
|
|
- type: mrr_at_3
|
|
value: 32.285000000000004
|
|
- type: mrr_at_5
|
|
value: 33.443
|
|
- type: ndcg_at_1
|
|
value: 26.840000000000003
|
|
- type: ndcg_at_10
|
|
value: 35.441
|
|
- type: ndcg_at_100
|
|
value: 40.150000000000006
|
|
- type: ndcg_at_1000
|
|
value: 42.74
|
|
- type: ndcg_at_3
|
|
value: 31.723000000000003
|
|
- type: ndcg_at_5
|
|
value: 33.71
|
|
- type: precision_at_1
|
|
value: 26.840000000000003
|
|
- type: precision_at_10
|
|
value: 5.552
|
|
- type: precision_at_100
|
|
value: 0.859
|
|
- type: precision_at_1000
|
|
value: 0.11499999999999999
|
|
- type: precision_at_3
|
|
value: 13.804
|
|
- type: precision_at_5
|
|
value: 9.600999999999999
|
|
- type: recall_at_1
|
|
value: 23.479
|
|
- type: recall_at_10
|
|
value: 45.442
|
|
- type: recall_at_100
|
|
value: 67.465
|
|
- type: recall_at_1000
|
|
value: 86.53
|
|
- type: recall_at_3
|
|
value: 35.315999999999995
|
|
- type: recall_at_5
|
|
value: 40.253
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackTexRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 16.887
|
|
- type: map_at_10
|
|
value: 23.805
|
|
- type: map_at_100
|
|
value: 24.804000000000002
|
|
- type: map_at_1000
|
|
value: 24.932000000000002
|
|
- type: map_at_3
|
|
value: 21.632
|
|
- type: map_at_5
|
|
value: 22.845
|
|
- type: mrr_at_1
|
|
value: 20.75
|
|
- type: mrr_at_10
|
|
value: 27.686
|
|
- type: mrr_at_100
|
|
value: 28.522
|
|
- type: mrr_at_1000
|
|
value: 28.605000000000004
|
|
- type: mrr_at_3
|
|
value: 25.618999999999996
|
|
- type: mrr_at_5
|
|
value: 26.723999999999997
|
|
- type: ndcg_at_1
|
|
value: 20.75
|
|
- type: ndcg_at_10
|
|
value: 28.233000000000004
|
|
- type: ndcg_at_100
|
|
value: 33.065
|
|
- type: ndcg_at_1000
|
|
value: 36.138999999999996
|
|
- type: ndcg_at_3
|
|
value: 24.361
|
|
- type: ndcg_at_5
|
|
value: 26.111
|
|
- type: precision_at_1
|
|
value: 20.75
|
|
- type: precision_at_10
|
|
value: 5.124
|
|
- type: precision_at_100
|
|
value: 0.8750000000000001
|
|
- type: precision_at_1000
|
|
value: 0.131
|
|
- type: precision_at_3
|
|
value: 11.539000000000001
|
|
- type: precision_at_5
|
|
value: 8.273
|
|
- type: recall_at_1
|
|
value: 16.887
|
|
- type: recall_at_10
|
|
value: 37.774
|
|
- type: recall_at_100
|
|
value: 59.587
|
|
- type: recall_at_1000
|
|
value: 81.523
|
|
- type: recall_at_3
|
|
value: 26.837
|
|
- type: recall_at_5
|
|
value: 31.456
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackUnixRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 25.534000000000002
|
|
- type: map_at_10
|
|
value: 33.495999999999995
|
|
- type: map_at_100
|
|
value: 34.697
|
|
- type: map_at_1000
|
|
value: 34.805
|
|
- type: map_at_3
|
|
value: 31.22
|
|
- type: map_at_5
|
|
value: 32.277
|
|
- type: mrr_at_1
|
|
value: 29.944
|
|
- type: mrr_at_10
|
|
value: 37.723
|
|
- type: mrr_at_100
|
|
value: 38.645
|
|
- type: mrr_at_1000
|
|
value: 38.712999999999994
|
|
- type: mrr_at_3
|
|
value: 35.665
|
|
- type: mrr_at_5
|
|
value: 36.681999999999995
|
|
- type: ndcg_at_1
|
|
value: 29.944
|
|
- type: ndcg_at_10
|
|
value: 38.407000000000004
|
|
- type: ndcg_at_100
|
|
value: 43.877
|
|
- type: ndcg_at_1000
|
|
value: 46.312
|
|
- type: ndcg_at_3
|
|
value: 34.211000000000006
|
|
- type: ndcg_at_5
|
|
value: 35.760999999999996
|
|
- type: precision_at_1
|
|
value: 29.944
|
|
- type: precision_at_10
|
|
value: 6.343
|
|
- type: precision_at_100
|
|
value: 1.023
|
|
- type: precision_at_1000
|
|
value: 0.133
|
|
- type: precision_at_3
|
|
value: 15.360999999999999
|
|
- type: precision_at_5
|
|
value: 10.428999999999998
|
|
- type: recall_at_1
|
|
value: 25.534000000000002
|
|
- type: recall_at_10
|
|
value: 49.204
|
|
- type: recall_at_100
|
|
value: 72.878
|
|
- type: recall_at_1000
|
|
value: 89.95
|
|
- type: recall_at_3
|
|
value: 37.533
|
|
- type: recall_at_5
|
|
value: 41.611
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackWebmastersRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 26.291999999999998
|
|
- type: map_at_10
|
|
value: 35.245
|
|
- type: map_at_100
|
|
value: 36.762
|
|
- type: map_at_1000
|
|
value: 36.983
|
|
- type: map_at_3
|
|
value: 32.439
|
|
- type: map_at_5
|
|
value: 33.964
|
|
- type: mrr_at_1
|
|
value: 31.423000000000002
|
|
- type: mrr_at_10
|
|
value: 39.98
|
|
- type: mrr_at_100
|
|
value: 40.791
|
|
- type: mrr_at_1000
|
|
value: 40.854
|
|
- type: mrr_at_3
|
|
value: 37.451
|
|
- type: mrr_at_5
|
|
value: 38.854
|
|
- type: ndcg_at_1
|
|
value: 31.423000000000002
|
|
- type: ndcg_at_10
|
|
value: 40.848
|
|
- type: ndcg_at_100
|
|
value: 46.35
|
|
- type: ndcg_at_1000
|
|
value: 49.166
|
|
- type: ndcg_at_3
|
|
value: 36.344
|
|
- type: ndcg_at_5
|
|
value: 38.36
|
|
- type: precision_at_1
|
|
value: 31.423000000000002
|
|
- type: precision_at_10
|
|
value: 7.767
|
|
- type: precision_at_100
|
|
value: 1.498
|
|
- type: precision_at_1000
|
|
value: 0.23700000000000002
|
|
- type: precision_at_3
|
|
value: 16.733
|
|
- type: precision_at_5
|
|
value: 12.213000000000001
|
|
- type: recall_at_1
|
|
value: 26.291999999999998
|
|
- type: recall_at_10
|
|
value: 51.184
|
|
- type: recall_at_100
|
|
value: 76.041
|
|
- type: recall_at_1000
|
|
value: 94.11500000000001
|
|
- type: recall_at_3
|
|
value: 38.257000000000005
|
|
- type: recall_at_5
|
|
value: 43.68
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: BeIR/cqadupstack
|
|
name: MTEB CQADupstackWordpressRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 20.715
|
|
- type: map_at_10
|
|
value: 27.810000000000002
|
|
- type: map_at_100
|
|
value: 28.810999999999996
|
|
- type: map_at_1000
|
|
value: 28.904999999999998
|
|
- type: map_at_3
|
|
value: 25.069999999999997
|
|
- type: map_at_5
|
|
value: 26.793
|
|
- type: mrr_at_1
|
|
value: 22.366
|
|
- type: mrr_at_10
|
|
value: 29.65
|
|
- type: mrr_at_100
|
|
value: 30.615
|
|
- type: mrr_at_1000
|
|
value: 30.686999999999998
|
|
- type: mrr_at_3
|
|
value: 27.017999999999997
|
|
- type: mrr_at_5
|
|
value: 28.644
|
|
- type: ndcg_at_1
|
|
value: 22.366
|
|
- type: ndcg_at_10
|
|
value: 32.221
|
|
- type: ndcg_at_100
|
|
value: 37.313
|
|
- type: ndcg_at_1000
|
|
value: 39.871
|
|
- type: ndcg_at_3
|
|
value: 26.918
|
|
- type: ndcg_at_5
|
|
value: 29.813000000000002
|
|
- type: precision_at_1
|
|
value: 22.366
|
|
- type: precision_at_10
|
|
value: 5.139
|
|
- type: precision_at_100
|
|
value: 0.8240000000000001
|
|
- type: precision_at_1000
|
|
value: 0.11199999999999999
|
|
- type: precision_at_3
|
|
value: 11.275
|
|
- type: precision_at_5
|
|
value: 8.540000000000001
|
|
- type: recall_at_1
|
|
value: 20.715
|
|
- type: recall_at_10
|
|
value: 44.023
|
|
- type: recall_at_100
|
|
value: 67.458
|
|
- type: recall_at_1000
|
|
value: 87.066
|
|
- type: recall_at_3
|
|
value: 30.055
|
|
- type: recall_at_5
|
|
value: 36.852000000000004
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: climate-fever
|
|
name: MTEB ClimateFEVER
|
|
config: default
|
|
split: test
|
|
revision: 392b78eb68c07badcd7c2cd8f39af108375dfcce
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 11.859
|
|
- type: map_at_10
|
|
value: 20.625
|
|
- type: map_at_100
|
|
value: 22.5
|
|
- type: map_at_1000
|
|
value: 22.689
|
|
- type: map_at_3
|
|
value: 16.991
|
|
- type: map_at_5
|
|
value: 18.781
|
|
- type: mrr_at_1
|
|
value: 26.906000000000002
|
|
- type: mrr_at_10
|
|
value: 39.083
|
|
- type: mrr_at_100
|
|
value: 39.978
|
|
- type: mrr_at_1000
|
|
value: 40.014
|
|
- type: mrr_at_3
|
|
value: 35.44
|
|
- type: mrr_at_5
|
|
value: 37.619
|
|
- type: ndcg_at_1
|
|
value: 26.906000000000002
|
|
- type: ndcg_at_10
|
|
value: 29.386000000000003
|
|
- type: ndcg_at_100
|
|
value: 36.510999999999996
|
|
- type: ndcg_at_1000
|
|
value: 39.814
|
|
- type: ndcg_at_3
|
|
value: 23.558
|
|
- type: ndcg_at_5
|
|
value: 25.557999999999996
|
|
- type: precision_at_1
|
|
value: 26.906000000000002
|
|
- type: precision_at_10
|
|
value: 9.342
|
|
- type: precision_at_100
|
|
value: 1.6969999999999998
|
|
- type: precision_at_1000
|
|
value: 0.231
|
|
- type: precision_at_3
|
|
value: 17.503
|
|
- type: precision_at_5
|
|
value: 13.655000000000001
|
|
- type: recall_at_1
|
|
value: 11.859
|
|
- type: recall_at_10
|
|
value: 35.929
|
|
- type: recall_at_100
|
|
value: 60.21300000000001
|
|
- type: recall_at_1000
|
|
value: 78.606
|
|
- type: recall_at_3
|
|
value: 21.727
|
|
- type: recall_at_5
|
|
value: 27.349
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: dbpedia-entity
|
|
name: MTEB DBPedia
|
|
config: default
|
|
split: test
|
|
revision: f097057d03ed98220bc7309ddb10b71a54d667d6
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 8.627
|
|
- type: map_at_10
|
|
value: 18.248
|
|
- type: map_at_100
|
|
value: 25.19
|
|
- type: map_at_1000
|
|
value: 26.741
|
|
- type: map_at_3
|
|
value: 13.286000000000001
|
|
- type: map_at_5
|
|
value: 15.126000000000001
|
|
- type: mrr_at_1
|
|
value: 64.75
|
|
- type: mrr_at_10
|
|
value: 71.865
|
|
- type: mrr_at_100
|
|
value: 72.247
|
|
- type: mrr_at_1000
|
|
value: 72.255
|
|
- type: mrr_at_3
|
|
value: 69.958
|
|
- type: mrr_at_5
|
|
value: 71.108
|
|
- type: ndcg_at_1
|
|
value: 53.25
|
|
- type: ndcg_at_10
|
|
value: 39.035
|
|
- type: ndcg_at_100
|
|
value: 42.735
|
|
- type: ndcg_at_1000
|
|
value: 50.166
|
|
- type: ndcg_at_3
|
|
value: 43.857
|
|
- type: ndcg_at_5
|
|
value: 40.579
|
|
- type: precision_at_1
|
|
value: 64.75
|
|
- type: precision_at_10
|
|
value: 30.75
|
|
- type: precision_at_100
|
|
value: 9.54
|
|
- type: precision_at_1000
|
|
value: 2.035
|
|
- type: precision_at_3
|
|
value: 47.333
|
|
- type: precision_at_5
|
|
value: 39.0
|
|
- type: recall_at_1
|
|
value: 8.627
|
|
- type: recall_at_10
|
|
value: 23.413
|
|
- type: recall_at_100
|
|
value: 48.037
|
|
- type: recall_at_1000
|
|
value: 71.428
|
|
- type: recall_at_3
|
|
value: 14.158999999999999
|
|
- type: recall_at_5
|
|
value: 17.002
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/emotion
|
|
name: MTEB EmotionClassification
|
|
config: default
|
|
split: test
|
|
revision: 829147f8f75a25f005913200eb5ed41fae320aa1
|
|
metrics:
|
|
- type: accuracy
|
|
value: 44.865
|
|
- type: f1
|
|
value: 41.56625743266997
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: fever
|
|
name: MTEB FEVER
|
|
config: default
|
|
split: test
|
|
revision: 1429cf27e393599b8b359b9b72c666f96b2525f9
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 57.335
|
|
- type: map_at_10
|
|
value: 68.29499999999999
|
|
- type: map_at_100
|
|
value: 68.69800000000001
|
|
- type: map_at_1000
|
|
value: 68.714
|
|
- type: map_at_3
|
|
value: 66.149
|
|
- type: map_at_5
|
|
value: 67.539
|
|
- type: mrr_at_1
|
|
value: 61.656
|
|
- type: mrr_at_10
|
|
value: 72.609
|
|
- type: mrr_at_100
|
|
value: 72.923
|
|
- type: mrr_at_1000
|
|
value: 72.928
|
|
- type: mrr_at_3
|
|
value: 70.645
|
|
- type: mrr_at_5
|
|
value: 71.938
|
|
- type: ndcg_at_1
|
|
value: 61.656
|
|
- type: ndcg_at_10
|
|
value: 73.966
|
|
- type: ndcg_at_100
|
|
value: 75.663
|
|
- type: ndcg_at_1000
|
|
value: 75.986
|
|
- type: ndcg_at_3
|
|
value: 69.959
|
|
- type: ndcg_at_5
|
|
value: 72.269
|
|
- type: precision_at_1
|
|
value: 61.656
|
|
- type: precision_at_10
|
|
value: 9.581000000000001
|
|
- type: precision_at_100
|
|
value: 1.054
|
|
- type: precision_at_1000
|
|
value: 0.11
|
|
- type: precision_at_3
|
|
value: 27.743000000000002
|
|
- type: precision_at_5
|
|
value: 17.939
|
|
- type: recall_at_1
|
|
value: 57.335
|
|
- type: recall_at_10
|
|
value: 87.24300000000001
|
|
- type: recall_at_100
|
|
value: 94.575
|
|
- type: recall_at_1000
|
|
value: 96.75399999999999
|
|
- type: recall_at_3
|
|
value: 76.44800000000001
|
|
- type: recall_at_5
|
|
value: 82.122
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: fiqa
|
|
name: MTEB FiQA2018
|
|
config: default
|
|
split: test
|
|
revision: 41b686a7f28c59bcaaa5791efd47c67c8ebe28be
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 17.014000000000003
|
|
- type: map_at_10
|
|
value: 28.469
|
|
- type: map_at_100
|
|
value: 30.178
|
|
- type: map_at_1000
|
|
value: 30.369
|
|
- type: map_at_3
|
|
value: 24.63
|
|
- type: map_at_5
|
|
value: 26.891
|
|
- type: mrr_at_1
|
|
value: 34.259
|
|
- type: mrr_at_10
|
|
value: 43.042
|
|
- type: mrr_at_100
|
|
value: 43.91
|
|
- type: mrr_at_1000
|
|
value: 43.963
|
|
- type: mrr_at_3
|
|
value: 40.483999999999995
|
|
- type: mrr_at_5
|
|
value: 42.135
|
|
- type: ndcg_at_1
|
|
value: 34.259
|
|
- type: ndcg_at_10
|
|
value: 35.836
|
|
- type: ndcg_at_100
|
|
value: 42.488
|
|
- type: ndcg_at_1000
|
|
value: 45.902
|
|
- type: ndcg_at_3
|
|
value: 32.131
|
|
- type: ndcg_at_5
|
|
value: 33.697
|
|
- type: precision_at_1
|
|
value: 34.259
|
|
- type: precision_at_10
|
|
value: 10.0
|
|
- type: precision_at_100
|
|
value: 1.699
|
|
- type: precision_at_1000
|
|
value: 0.22999999999999998
|
|
- type: precision_at_3
|
|
value: 21.502
|
|
- type: precision_at_5
|
|
value: 16.296
|
|
- type: recall_at_1
|
|
value: 17.014000000000003
|
|
- type: recall_at_10
|
|
value: 42.832
|
|
- type: recall_at_100
|
|
value: 67.619
|
|
- type: recall_at_1000
|
|
value: 88.453
|
|
- type: recall_at_3
|
|
value: 29.537000000000003
|
|
- type: recall_at_5
|
|
value: 35.886
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: hotpotqa
|
|
name: MTEB HotpotQA
|
|
config: default
|
|
split: test
|
|
revision: 766870b35a1b9ca65e67a0d1913899973551fc6c
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 34.558
|
|
- type: map_at_10
|
|
value: 48.039
|
|
- type: map_at_100
|
|
value: 48.867
|
|
- type: map_at_1000
|
|
value: 48.941
|
|
- type: map_at_3
|
|
value: 45.403
|
|
- type: map_at_5
|
|
value: 46.983999999999995
|
|
- type: mrr_at_1
|
|
value: 69.11500000000001
|
|
- type: mrr_at_10
|
|
value: 75.551
|
|
- type: mrr_at_100
|
|
value: 75.872
|
|
- type: mrr_at_1000
|
|
value: 75.887
|
|
- type: mrr_at_3
|
|
value: 74.447
|
|
- type: mrr_at_5
|
|
value: 75.113
|
|
- type: ndcg_at_1
|
|
value: 69.11500000000001
|
|
- type: ndcg_at_10
|
|
value: 57.25599999999999
|
|
- type: ndcg_at_100
|
|
value: 60.417
|
|
- type: ndcg_at_1000
|
|
value: 61.976
|
|
- type: ndcg_at_3
|
|
value: 53.258
|
|
- type: ndcg_at_5
|
|
value: 55.374
|
|
- type: precision_at_1
|
|
value: 69.11500000000001
|
|
- type: precision_at_10
|
|
value: 11.689
|
|
- type: precision_at_100
|
|
value: 1.418
|
|
- type: precision_at_1000
|
|
value: 0.163
|
|
- type: precision_at_3
|
|
value: 33.018
|
|
- type: precision_at_5
|
|
value: 21.488
|
|
- type: recall_at_1
|
|
value: 34.558
|
|
- type: recall_at_10
|
|
value: 58.447
|
|
- type: recall_at_100
|
|
value: 70.91199999999999
|
|
- type: recall_at_1000
|
|
value: 81.31
|
|
- type: recall_at_3
|
|
value: 49.527
|
|
- type: recall_at_5
|
|
value: 53.72
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/imdb
|
|
name: MTEB ImdbClassification
|
|
config: default
|
|
split: test
|
|
revision: 8d743909f834c38949e8323a8a6ce8721ea6c7f4
|
|
metrics:
|
|
- type: accuracy
|
|
value: 61.772000000000006
|
|
- type: ap
|
|
value: 57.48217702943605
|
|
- type: f1
|
|
value: 61.20495351356274
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: msmarco
|
|
name: MTEB MSMARCO
|
|
config: default
|
|
split: validation
|
|
revision: e6838a846e2408f22cf5cc337ebc83e0bcf77849
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 22.044
|
|
- type: map_at_10
|
|
value: 34.211000000000006
|
|
- type: map_at_100
|
|
value: 35.394
|
|
- type: map_at_1000
|
|
value: 35.443000000000005
|
|
- type: map_at_3
|
|
value: 30.318
|
|
- type: map_at_5
|
|
value: 32.535
|
|
- type: mrr_at_1
|
|
value: 22.722
|
|
- type: mrr_at_10
|
|
value: 34.842
|
|
- type: mrr_at_100
|
|
value: 35.954
|
|
- type: mrr_at_1000
|
|
value: 35.997
|
|
- type: mrr_at_3
|
|
value: 30.991000000000003
|
|
- type: mrr_at_5
|
|
value: 33.2
|
|
- type: ndcg_at_1
|
|
value: 22.722
|
|
- type: ndcg_at_10
|
|
value: 41.121
|
|
- type: ndcg_at_100
|
|
value: 46.841
|
|
- type: ndcg_at_1000
|
|
value: 48.049
|
|
- type: ndcg_at_3
|
|
value: 33.173
|
|
- type: ndcg_at_5
|
|
value: 37.145
|
|
- type: precision_at_1
|
|
value: 22.722
|
|
- type: precision_at_10
|
|
value: 6.516
|
|
- type: precision_at_100
|
|
value: 0.9400000000000001
|
|
- type: precision_at_1000
|
|
value: 0.104
|
|
- type: precision_at_3
|
|
value: 14.093
|
|
- type: precision_at_5
|
|
value: 10.473
|
|
- type: recall_at_1
|
|
value: 22.044
|
|
- type: recall_at_10
|
|
value: 62.382000000000005
|
|
- type: recall_at_100
|
|
value: 88.914
|
|
- type: recall_at_1000
|
|
value: 98.099
|
|
- type: recall_at_3
|
|
value: 40.782000000000004
|
|
- type: recall_at_5
|
|
value: 50.322
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_domain
|
|
name: MTEB MTOPDomainClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
|
|
metrics:
|
|
- type: accuracy
|
|
value: 93.68217054263563
|
|
- type: f1
|
|
value: 93.25810075739523
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_domain
|
|
name: MTEB MTOPDomainClassification (de)
|
|
config: de
|
|
split: test
|
|
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
|
|
metrics:
|
|
- type: accuracy
|
|
value: 82.05409974640745
|
|
- type: f1
|
|
value: 80.42814140324903
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_domain
|
|
name: MTEB MTOPDomainClassification (es)
|
|
config: es
|
|
split: test
|
|
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
|
|
metrics:
|
|
- type: accuracy
|
|
value: 93.54903268845896
|
|
- type: f1
|
|
value: 92.8909878077932
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_domain
|
|
name: MTEB MTOPDomainClassification (fr)
|
|
config: fr
|
|
split: test
|
|
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
|
|
metrics:
|
|
- type: accuracy
|
|
value: 90.98340119010334
|
|
- type: f1
|
|
value: 90.51522537281313
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_domain
|
|
name: MTEB MTOPDomainClassification (hi)
|
|
config: hi
|
|
split: test
|
|
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
|
|
metrics:
|
|
- type: accuracy
|
|
value: 89.33309429903191
|
|
- type: f1
|
|
value: 88.60371305209185
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_domain
|
|
name: MTEB MTOPDomainClassification (th)
|
|
config: th
|
|
split: test
|
|
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
|
|
metrics:
|
|
- type: accuracy
|
|
value: 60.4882459312839
|
|
- type: f1
|
|
value: 59.02590456131682
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_intent
|
|
name: MTEB MTOPIntentClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: 6299947a7777084cc2d4b64235bf7190381ce755
|
|
metrics:
|
|
- type: accuracy
|
|
value: 71.34290925672595
|
|
- type: f1
|
|
value: 54.44803151449109
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_intent
|
|
name: MTEB MTOPIntentClassification (de)
|
|
config: de
|
|
split: test
|
|
revision: 6299947a7777084cc2d4b64235bf7190381ce755
|
|
metrics:
|
|
- type: accuracy
|
|
value: 61.92448577063963
|
|
- type: f1
|
|
value: 43.125939975781854
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_intent
|
|
name: MTEB MTOPIntentClassification (es)
|
|
config: es
|
|
split: test
|
|
revision: 6299947a7777084cc2d4b64235bf7190381ce755
|
|
metrics:
|
|
- type: accuracy
|
|
value: 74.48965977318213
|
|
- type: f1
|
|
value: 51.855353687466696
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_intent
|
|
name: MTEB MTOPIntentClassification (fr)
|
|
config: fr
|
|
split: test
|
|
revision: 6299947a7777084cc2d4b64235bf7190381ce755
|
|
metrics:
|
|
- type: accuracy
|
|
value: 69.11994989038521
|
|
- type: f1
|
|
value: 50.57872704171278
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_intent
|
|
name: MTEB MTOPIntentClassification (hi)
|
|
config: hi
|
|
split: test
|
|
revision: 6299947a7777084cc2d4b64235bf7190381ce755
|
|
metrics:
|
|
- type: accuracy
|
|
value: 64.84761563284331
|
|
- type: f1
|
|
value: 43.61322970761394
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/mtop_intent
|
|
name: MTEB MTOPIntentClassification (th)
|
|
config: th
|
|
split: test
|
|
revision: 6299947a7777084cc2d4b64235bf7190381ce755
|
|
metrics:
|
|
- type: accuracy
|
|
value: 49.35623869801085
|
|
- type: f1
|
|
value: 33.48547326952042
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (af)
|
|
config: af
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 47.85474108944183
|
|
- type: f1
|
|
value: 46.50175016795915
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (am)
|
|
config: am
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 33.29858776059179
|
|
- type: f1
|
|
value: 31.803027601259082
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (ar)
|
|
config: ar
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 59.24680564895763
|
|
- type: f1
|
|
value: 57.037691806846865
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (az)
|
|
config: az
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 45.23537323470073
|
|
- type: f1
|
|
value: 44.81126398428613
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (bn)
|
|
config: bn
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 61.590450571620714
|
|
- type: f1
|
|
value: 59.247442149977104
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (cy)
|
|
config: cy
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 44.9226630800269
|
|
- type: f1
|
|
value: 44.076183379991654
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (da)
|
|
config: da
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 51.23066577000672
|
|
- type: f1
|
|
value: 50.20719330417618
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (de)
|
|
config: de
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 56.0995292535306
|
|
- type: f1
|
|
value: 53.29421532133969
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (el)
|
|
config: el
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 46.12642905178211
|
|
- type: f1
|
|
value: 44.441530267639635
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 69.67047747141896
|
|
- type: f1
|
|
value: 68.38493366054783
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (es)
|
|
config: es
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 66.3483523873571
|
|
- type: f1
|
|
value: 65.13046416817832
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (fa)
|
|
config: fa
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 51.20040349697378
|
|
- type: f1
|
|
value: 49.02889836601541
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (fi)
|
|
config: fi
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 45.33288500336248
|
|
- type: f1
|
|
value: 42.91893101970983
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (fr)
|
|
config: fr
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 66.95359784801613
|
|
- type: f1
|
|
value: 64.98788914810562
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (he)
|
|
config: he
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 43.18090114324143
|
|
- type: f1
|
|
value: 41.31250407417542
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (hi)
|
|
config: hi
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 63.54068594485541
|
|
- type: f1
|
|
value: 61.94829361488948
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (hu)
|
|
config: hu
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 44.7343644922663
|
|
- type: f1
|
|
value: 43.23001702247849
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (hy)
|
|
config: hy
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 38.1271015467384
|
|
- type: f1
|
|
value: 36.94700198241727
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (id)
|
|
config: id
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 64.05514458641561
|
|
- type: f1
|
|
value: 62.35033731674541
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (is)
|
|
config: is
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 44.351042367182245
|
|
- type: f1
|
|
value: 43.13370397574502
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (it)
|
|
config: it
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 60.77000672494955
|
|
- type: f1
|
|
value: 59.71546868957779
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (ja)
|
|
config: ja
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 61.22057834566241
|
|
- type: f1
|
|
value: 59.447639306287044
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (jv)
|
|
config: jv
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 50.9448554135844
|
|
- type: f1
|
|
value: 48.524338247875214
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (ka)
|
|
config: ka
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 33.8399462004035
|
|
- type: f1
|
|
value: 33.518999997305535
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (km)
|
|
config: km
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 37.34028244788165
|
|
- type: f1
|
|
value: 35.6156599064704
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (kn)
|
|
config: kn
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 53.544048419636844
|
|
- type: f1
|
|
value: 51.29299915455352
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (ko)
|
|
config: ko
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 53.35574983187625
|
|
- type: f1
|
|
value: 51.463936565192945
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (lv)
|
|
config: lv
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 46.503026227303295
|
|
- type: f1
|
|
value: 46.049497734375514
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (ml)
|
|
config: ml
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 58.268325487558826
|
|
- type: f1
|
|
value: 56.10849656896158
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (mn)
|
|
config: mn
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 40.27572293207801
|
|
- type: f1
|
|
value: 40.20097238549224
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (ms)
|
|
config: ms
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 59.64694014794889
|
|
- type: f1
|
|
value: 58.39584148789066
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (my)
|
|
config: my
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 37.41761936785474
|
|
- type: f1
|
|
value: 35.04551731363685
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (nb)
|
|
config: nb
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 49.408204438466704
|
|
- type: f1
|
|
value: 48.39369057638714
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (nl)
|
|
config: nl
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 52.09482178883659
|
|
- type: f1
|
|
value: 49.91518031712698
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (pl)
|
|
config: pl
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 50.477471418964356
|
|
- type: f1
|
|
value: 48.429495257184705
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (pt)
|
|
config: pt
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 66.69468728984532
|
|
- type: f1
|
|
value: 65.40306868707009
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (ro)
|
|
config: ro
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 50.52790854068594
|
|
- type: f1
|
|
value: 49.780400354514
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (ru)
|
|
config: ru
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 58.31540013449899
|
|
- type: f1
|
|
value: 56.144142926685134
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (sl)
|
|
config: sl
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 47.74041694687289
|
|
- type: f1
|
|
value: 46.16767322761359
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (sq)
|
|
config: sq
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 48.94418291862811
|
|
- type: f1
|
|
value: 48.445352284756325
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (sv)
|
|
config: sv
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 50.78681909885676
|
|
- type: f1
|
|
value: 49.64882295494536
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (sw)
|
|
config: sw
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 49.811701412239415
|
|
- type: f1
|
|
value: 48.213234514449375
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (ta)
|
|
config: ta
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 56.39542703429725
|
|
- type: f1
|
|
value: 54.031981085233795
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (te)
|
|
config: te
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 54.71082716879623
|
|
- type: f1
|
|
value: 52.513144113474596
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (th)
|
|
config: th
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 44.425016812373904
|
|
- type: f1
|
|
value: 43.96016300057656
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (tl)
|
|
config: tl
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 50.205110961667785
|
|
- type: f1
|
|
value: 48.86669996798709
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (tr)
|
|
config: tr
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 46.56355077336921
|
|
- type: f1
|
|
value: 45.18252022585022
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (ur)
|
|
config: ur
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 56.748486886348346
|
|
- type: f1
|
|
value: 54.29884570375382
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (vi)
|
|
config: vi
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 64.52589105581708
|
|
- type: f1
|
|
value: 62.97947342861603
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (zh-CN)
|
|
config: zh-CN
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 67.06792199058508
|
|
- type: f1
|
|
value: 65.36025601634017
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_intent
|
|
name: MTEB MassiveIntentClassification (zh-TW)
|
|
config: zh-TW
|
|
split: test
|
|
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
|
|
metrics:
|
|
- type: accuracy
|
|
value: 62.89172831203766
|
|
- type: f1
|
|
value: 62.69803707054342
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (af)
|
|
config: af
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 51.47276395427035
|
|
- type: f1
|
|
value: 49.37463208130799
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (am)
|
|
config: am
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 34.86886348352387
|
|
- type: f1
|
|
value: 33.74178074349636
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (ar)
|
|
config: ar
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 65.20511096166778
|
|
- type: f1
|
|
value: 65.85812500602437
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (az)
|
|
config: az
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 45.578345662407536
|
|
- type: f1
|
|
value: 44.44514917028003
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (bn)
|
|
config: bn
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 67.29657027572293
|
|
- type: f1
|
|
value: 67.24477523937466
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (cy)
|
|
config: cy
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 46.29455279085407
|
|
- type: f1
|
|
value: 43.8563839951935
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (da)
|
|
config: da
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 53.52387357094821
|
|
- type: f1
|
|
value: 51.70977848027552
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (de)
|
|
config: de
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 61.741761936785466
|
|
- type: f1
|
|
value: 60.219169644792295
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (el)
|
|
config: el
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 48.957632817753876
|
|
- type: f1
|
|
value: 46.878428264460034
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (en)
|
|
config: en
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 75.33624747814393
|
|
- type: f1
|
|
value: 75.9143846211171
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (es)
|
|
config: es
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 73.34229993275049
|
|
- type: f1
|
|
value: 73.78165397558983
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (fa)
|
|
config: fa
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 53.174176193678555
|
|
- type: f1
|
|
value: 51.709679227778985
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (fi)
|
|
config: fi
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 44.6906523201076
|
|
- type: f1
|
|
value: 41.54881682785664
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (fr)
|
|
config: fr
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 72.9119031607263
|
|
- type: f1
|
|
value: 73.2742013056326
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (he)
|
|
config: he
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 43.10356422326832
|
|
- type: f1
|
|
value: 40.8859122581252
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (hi)
|
|
config: hi
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 69.27370544720914
|
|
- type: f1
|
|
value: 69.39544506405082
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (hu)
|
|
config: hu
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 45.16476126429052
|
|
- type: f1
|
|
value: 42.74022531579054
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (hy)
|
|
config: hy
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 38.73234700739744
|
|
- type: f1
|
|
value: 37.40546754951026
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (id)
|
|
config: id
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 70.12777404169468
|
|
- type: f1
|
|
value: 70.27219152812738
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (is)
|
|
config: is
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 44.21318090114325
|
|
- type: f1
|
|
value: 41.934593213829366
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (it)
|
|
config: it
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 65.57162071284466
|
|
- type: f1
|
|
value: 64.83341759045335
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (ja)
|
|
config: ja
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 65.75991930060525
|
|
- type: f1
|
|
value: 65.16549875504951
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (jv)
|
|
config: jv
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 54.79488903833223
|
|
- type: f1
|
|
value: 54.03616401426859
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (ka)
|
|
config: ka
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 32.992602555480836
|
|
- type: f1
|
|
value: 31.820068470018846
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (km)
|
|
config: km
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 39.34431741761937
|
|
- type: f1
|
|
value: 36.436221665290105
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (kn)
|
|
config: kn
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 60.501008742434436
|
|
- type: f1
|
|
value: 60.051013712579085
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (ko)
|
|
config: ko
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 55.689307330195035
|
|
- type: f1
|
|
value: 53.94058032286942
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (lv)
|
|
config: lv
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 44.351042367182245
|
|
- type: f1
|
|
value: 42.05421666771541
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (ml)
|
|
config: ml
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 65.53127101546738
|
|
- type: f1
|
|
value: 65.98462024333497
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (mn)
|
|
config: mn
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 38.71553463349025
|
|
- type: f1
|
|
value: 37.44327037149584
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (ms)
|
|
config: ms
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 64.98991257565567
|
|
- type: f1
|
|
value: 63.87720198978004
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (my)
|
|
config: my
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 36.839273705447205
|
|
- type: f1
|
|
value: 35.233967279698376
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (nb)
|
|
config: nb
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 51.79892400806993
|
|
- type: f1
|
|
value: 49.66926632125972
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (nl)
|
|
config: nl
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 56.31809011432415
|
|
- type: f1
|
|
value: 53.832185336179826
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (pl)
|
|
config: pl
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 49.979825151311374
|
|
- type: f1
|
|
value: 48.83013175441888
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (pt)
|
|
config: pt
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 71.45595158036315
|
|
- type: f1
|
|
value: 72.08708814699702
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (ro)
|
|
config: ro
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 53.68527236045729
|
|
- type: f1
|
|
value: 52.23278593929981
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (ru)
|
|
config: ru
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 61.60390047074647
|
|
- type: f1
|
|
value: 60.50391482195116
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (sl)
|
|
config: sl
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 48.036314727639535
|
|
- type: f1
|
|
value: 46.43480413383716
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (sq)
|
|
config: sq
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 50.05716207128445
|
|
- type: f1
|
|
value: 48.85821859948888
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (sv)
|
|
config: sv
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 51.728312037659705
|
|
- type: f1
|
|
value: 49.89292996950847
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (sw)
|
|
config: sw
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 54.21990585070613
|
|
- type: f1
|
|
value: 52.8711542984193
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (ta)
|
|
config: ta
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 62.770679219905844
|
|
- type: f1
|
|
value: 63.09441501491594
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (te)
|
|
config: te
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 62.58574310692671
|
|
- type: f1
|
|
value: 61.61370697612978
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (th)
|
|
config: th
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 45.17821116341628
|
|
- type: f1
|
|
value: 43.85143229183324
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (tl)
|
|
config: tl
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 52.064559515803644
|
|
- type: f1
|
|
value: 50.94356892049626
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (tr)
|
|
config: tr
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 47.205783456624076
|
|
- type: f1
|
|
value: 47.04223644120489
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (ur)
|
|
config: ur
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 64.25689307330195
|
|
- type: f1
|
|
value: 63.89944944984115
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (vi)
|
|
config: vi
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 70.60524546065905
|
|
- type: f1
|
|
value: 71.5634157334358
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (zh-CN)
|
|
config: zh-CN
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 73.95427034297242
|
|
- type: f1
|
|
value: 74.39706882311063
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/amazon_massive_scenario
|
|
name: MTEB MassiveScenarioClassification (zh-TW)
|
|
config: zh-TW
|
|
split: test
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
metrics:
|
|
- type: accuracy
|
|
value: 70.29926025554808
|
|
- type: f1
|
|
value: 71.32045932560297
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/medrxiv-clustering-p2p
|
|
name: MTEB MedrxivClusteringP2P
|
|
config: default
|
|
split: test
|
|
revision: dcefc037ef84348e49b0d29109e891c01067226b
|
|
metrics:
|
|
- type: v_measure
|
|
value: 31.054474964883806
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/medrxiv-clustering-s2s
|
|
name: MTEB MedrxivClusteringS2S
|
|
config: default
|
|
split: test
|
|
revision: 3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc
|
|
metrics:
|
|
- type: v_measure
|
|
value: 29.259725940477523
|
|
- task:
|
|
type: Reranking
|
|
dataset:
|
|
type: mteb/mind_small
|
|
name: MTEB MindSmallReranking
|
|
config: default
|
|
split: test
|
|
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69
|
|
metrics:
|
|
- type: map
|
|
value: 31.785007883256572
|
|
- type: mrr
|
|
value: 32.983556622438456
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: nfcorpus
|
|
name: MTEB NFCorpus
|
|
config: default
|
|
split: test
|
|
revision: 7eb63cc0c1eb59324d709ebed25fcab851fa7610
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 5.742
|
|
- type: map_at_10
|
|
value: 13.074
|
|
- type: map_at_100
|
|
value: 16.716
|
|
- type: map_at_1000
|
|
value: 18.238
|
|
- type: map_at_3
|
|
value: 9.600999999999999
|
|
- type: map_at_5
|
|
value: 11.129999999999999
|
|
- type: mrr_at_1
|
|
value: 47.988
|
|
- type: mrr_at_10
|
|
value: 55.958
|
|
- type: mrr_at_100
|
|
value: 56.58800000000001
|
|
- type: mrr_at_1000
|
|
value: 56.620000000000005
|
|
- type: mrr_at_3
|
|
value: 54.025
|
|
- type: mrr_at_5
|
|
value: 55.31
|
|
- type: ndcg_at_1
|
|
value: 46.44
|
|
- type: ndcg_at_10
|
|
value: 35.776
|
|
- type: ndcg_at_100
|
|
value: 32.891999999999996
|
|
- type: ndcg_at_1000
|
|
value: 41.835
|
|
- type: ndcg_at_3
|
|
value: 41.812
|
|
- type: ndcg_at_5
|
|
value: 39.249
|
|
- type: precision_at_1
|
|
value: 48.297000000000004
|
|
- type: precision_at_10
|
|
value: 26.687
|
|
- type: precision_at_100
|
|
value: 8.511000000000001
|
|
- type: precision_at_1000
|
|
value: 2.128
|
|
- type: precision_at_3
|
|
value: 39.009
|
|
- type: precision_at_5
|
|
value: 33.994
|
|
- type: recall_at_1
|
|
value: 5.742
|
|
- type: recall_at_10
|
|
value: 16.993
|
|
- type: recall_at_100
|
|
value: 33.69
|
|
- type: recall_at_1000
|
|
value: 66.75
|
|
- type: recall_at_3
|
|
value: 10.817
|
|
- type: recall_at_5
|
|
value: 13.256
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: nq
|
|
name: MTEB NQ
|
|
config: default
|
|
split: test
|
|
revision: 6062aefc120bfe8ece5897809fb2e53bfe0d128c
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 30.789
|
|
- type: map_at_10
|
|
value: 45.751999999999995
|
|
- type: map_at_100
|
|
value: 46.766000000000005
|
|
- type: map_at_1000
|
|
value: 46.798
|
|
- type: map_at_3
|
|
value: 41.746
|
|
- type: map_at_5
|
|
value: 44.046
|
|
- type: mrr_at_1
|
|
value: 34.618
|
|
- type: mrr_at_10
|
|
value: 48.288
|
|
- type: mrr_at_100
|
|
value: 49.071999999999996
|
|
- type: mrr_at_1000
|
|
value: 49.094
|
|
- type: mrr_at_3
|
|
value: 44.979
|
|
- type: mrr_at_5
|
|
value: 46.953
|
|
- type: ndcg_at_1
|
|
value: 34.589
|
|
- type: ndcg_at_10
|
|
value: 53.151
|
|
- type: ndcg_at_100
|
|
value: 57.537000000000006
|
|
- type: ndcg_at_1000
|
|
value: 58.321999999999996
|
|
- type: ndcg_at_3
|
|
value: 45.628
|
|
- type: ndcg_at_5
|
|
value: 49.474000000000004
|
|
- type: precision_at_1
|
|
value: 34.589
|
|
- type: precision_at_10
|
|
value: 8.731
|
|
- type: precision_at_100
|
|
value: 1.119
|
|
- type: precision_at_1000
|
|
value: 0.11900000000000001
|
|
- type: precision_at_3
|
|
value: 20.819
|
|
- type: precision_at_5
|
|
value: 14.728
|
|
- type: recall_at_1
|
|
value: 30.789
|
|
- type: recall_at_10
|
|
value: 73.066
|
|
- type: recall_at_100
|
|
value: 92.27
|
|
- type: recall_at_1000
|
|
value: 98.18
|
|
- type: recall_at_3
|
|
value: 53.632999999999996
|
|
- type: recall_at_5
|
|
value: 62.476
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: quora
|
|
name: MTEB QuoraRetrieval
|
|
config: default
|
|
split: test
|
|
revision: 6205996560df11e3a3da9ab4f926788fc30a7db4
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 54.993
|
|
- type: map_at_10
|
|
value: 69.07600000000001
|
|
- type: map_at_100
|
|
value: 70.05799999999999
|
|
- type: map_at_1000
|
|
value: 70.09
|
|
- type: map_at_3
|
|
value: 65.456
|
|
- type: map_at_5
|
|
value: 67.622
|
|
- type: mrr_at_1
|
|
value: 63.07000000000001
|
|
- type: mrr_at_10
|
|
value: 72.637
|
|
- type: mrr_at_100
|
|
value: 73.029
|
|
- type: mrr_at_1000
|
|
value: 73.033
|
|
- type: mrr_at_3
|
|
value: 70.572
|
|
- type: mrr_at_5
|
|
value: 71.86399999999999
|
|
- type: ndcg_at_1
|
|
value: 63.07000000000001
|
|
- type: ndcg_at_10
|
|
value: 74.708
|
|
- type: ndcg_at_100
|
|
value: 77.579
|
|
- type: ndcg_at_1000
|
|
value: 77.897
|
|
- type: ndcg_at_3
|
|
value: 69.69999999999999
|
|
- type: ndcg_at_5
|
|
value: 72.321
|
|
- type: precision_at_1
|
|
value: 63.07000000000001
|
|
- type: precision_at_10
|
|
value: 11.851
|
|
- type: precision_at_100
|
|
value: 1.481
|
|
- type: precision_at_1000
|
|
value: 0.156
|
|
- type: precision_at_3
|
|
value: 30.747000000000003
|
|
- type: precision_at_5
|
|
value: 20.830000000000002
|
|
- type: recall_at_1
|
|
value: 54.993
|
|
- type: recall_at_10
|
|
value: 87.18900000000001
|
|
- type: recall_at_100
|
|
value: 98.137
|
|
- type: recall_at_1000
|
|
value: 99.833
|
|
- type: recall_at_3
|
|
value: 73.654
|
|
- type: recall_at_5
|
|
value: 80.36
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/reddit-clustering
|
|
name: MTEB RedditClustering
|
|
config: default
|
|
split: test
|
|
revision: b2805658ae38990172679479369a78b86de8c390
|
|
metrics:
|
|
- type: v_measure
|
|
value: 35.53178375429036
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/reddit-clustering-p2p
|
|
name: MTEB RedditClusteringP2P
|
|
config: default
|
|
split: test
|
|
revision: 385e3cb46b4cfa89021f56c4380204149d0efe33
|
|
metrics:
|
|
- type: v_measure
|
|
value: 54.520782970558265
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: scidocs
|
|
name: MTEB SCIDOCS
|
|
config: default
|
|
split: test
|
|
revision: 5c59ef3e437a0a9651c8fe6fde943e7dce59fba5
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 4.3229999999999995
|
|
- type: map_at_10
|
|
value: 10.979999999999999
|
|
- type: map_at_100
|
|
value: 12.867
|
|
- type: map_at_1000
|
|
value: 13.147
|
|
- type: map_at_3
|
|
value: 7.973
|
|
- type: map_at_5
|
|
value: 9.513
|
|
- type: mrr_at_1
|
|
value: 21.3
|
|
- type: mrr_at_10
|
|
value: 32.34
|
|
- type: mrr_at_100
|
|
value: 33.428999999999995
|
|
- type: mrr_at_1000
|
|
value: 33.489999999999995
|
|
- type: mrr_at_3
|
|
value: 28.999999999999996
|
|
- type: mrr_at_5
|
|
value: 31.019999999999996
|
|
- type: ndcg_at_1
|
|
value: 21.3
|
|
- type: ndcg_at_10
|
|
value: 18.619
|
|
- type: ndcg_at_100
|
|
value: 26.108999999999998
|
|
- type: ndcg_at_1000
|
|
value: 31.253999999999998
|
|
- type: ndcg_at_3
|
|
value: 17.842
|
|
- type: ndcg_at_5
|
|
value: 15.673
|
|
- type: precision_at_1
|
|
value: 21.3
|
|
- type: precision_at_10
|
|
value: 9.55
|
|
- type: precision_at_100
|
|
value: 2.0340000000000003
|
|
- type: precision_at_1000
|
|
value: 0.327
|
|
- type: precision_at_3
|
|
value: 16.667
|
|
- type: precision_at_5
|
|
value: 13.76
|
|
- type: recall_at_1
|
|
value: 4.3229999999999995
|
|
- type: recall_at_10
|
|
value: 19.387
|
|
- type: recall_at_100
|
|
value: 41.307
|
|
- type: recall_at_1000
|
|
value: 66.475
|
|
- type: recall_at_3
|
|
value: 10.143
|
|
- type: recall_at_5
|
|
value: 14.007
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sickr-sts
|
|
name: MTEB SICK-R
|
|
config: default
|
|
split: test
|
|
revision: 20a6d6f312dd54037fe07a32d58e5e168867909d
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 78.77975189382573
|
|
- type: cos_sim_spearman
|
|
value: 69.81522686267631
|
|
- type: euclidean_pearson
|
|
value: 71.37617936889518
|
|
- type: euclidean_spearman
|
|
value: 65.71738481148611
|
|
- type: manhattan_pearson
|
|
value: 71.58222165832424
|
|
- type: manhattan_spearman
|
|
value: 65.86851365286654
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts12-sts
|
|
name: MTEB STS12
|
|
config: default
|
|
split: test
|
|
revision: fdf84275bb8ce4b49c971d02e84dd1abc677a50f
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 77.75509450443367
|
|
- type: cos_sim_spearman
|
|
value: 69.66180222442091
|
|
- type: euclidean_pearson
|
|
value: 74.98512779786111
|
|
- type: euclidean_spearman
|
|
value: 69.5997451409469
|
|
- type: manhattan_pearson
|
|
value: 75.50135090962459
|
|
- type: manhattan_spearman
|
|
value: 69.94984748475302
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts13-sts
|
|
name: MTEB STS13
|
|
config: default
|
|
split: test
|
|
revision: 1591bfcbe8c69d4bf7fe2a16e2451017832cafb9
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 79.42363892383264
|
|
- type: cos_sim_spearman
|
|
value: 79.66529244176742
|
|
- type: euclidean_pearson
|
|
value: 79.50429208135942
|
|
- type: euclidean_spearman
|
|
value: 80.44767586416276
|
|
- type: manhattan_pearson
|
|
value: 79.58563944997708
|
|
- type: manhattan_spearman
|
|
value: 80.51452267103
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts14-sts
|
|
name: MTEB STS14
|
|
config: default
|
|
split: test
|
|
revision: e2125984e7df8b7871f6ae9949cf6b6795e7c54b
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 79.2749401478149
|
|
- type: cos_sim_spearman
|
|
value: 74.6076920702392
|
|
- type: euclidean_pearson
|
|
value: 73.3302002952881
|
|
- type: euclidean_spearman
|
|
value: 70.67029803077013
|
|
- type: manhattan_pearson
|
|
value: 73.52699344010296
|
|
- type: manhattan_spearman
|
|
value: 70.8517556194297
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts15-sts
|
|
name: MTEB STS15
|
|
config: default
|
|
split: test
|
|
revision: 1cd7298cac12a96a373b6a2f18738bb3e739a9b6
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 83.20884740785921
|
|
- type: cos_sim_spearman
|
|
value: 83.80600789090722
|
|
- type: euclidean_pearson
|
|
value: 74.9154089816344
|
|
- type: euclidean_spearman
|
|
value: 75.69243899592276
|
|
- type: manhattan_pearson
|
|
value: 75.0312832634451
|
|
- type: manhattan_spearman
|
|
value: 75.78324960357642
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts16-sts
|
|
name: MTEB STS16
|
|
config: default
|
|
split: test
|
|
revision: 360a0b2dff98700d09e634a01e1cc1624d3e42cd
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 79.63194141000497
|
|
- type: cos_sim_spearman
|
|
value: 80.40118418350866
|
|
- type: euclidean_pearson
|
|
value: 72.07354384551088
|
|
- type: euclidean_spearman
|
|
value: 72.28819150373845
|
|
- type: manhattan_pearson
|
|
value: 72.08736119834145
|
|
- type: manhattan_spearman
|
|
value: 72.28347083261288
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (ko-ko)
|
|
config: ko-ko
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 66.78512789499386
|
|
- type: cos_sim_spearman
|
|
value: 66.89125587193288
|
|
- type: euclidean_pearson
|
|
value: 58.74535708627959
|
|
- type: euclidean_spearman
|
|
value: 59.62103716794647
|
|
- type: manhattan_pearson
|
|
value: 59.00494529143961
|
|
- type: manhattan_spearman
|
|
value: 59.832257846799806
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (ar-ar)
|
|
config: ar-ar
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 75.48960503523992
|
|
- type: cos_sim_spearman
|
|
value: 76.4223037534204
|
|
- type: euclidean_pearson
|
|
value: 64.93966381820944
|
|
- type: euclidean_spearman
|
|
value: 62.39697395373789
|
|
- type: manhattan_pearson
|
|
value: 65.54480770061505
|
|
- type: manhattan_spearman
|
|
value: 62.944204863043105
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (en-ar)
|
|
config: en-ar
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 77.7331440643619
|
|
- type: cos_sim_spearman
|
|
value: 78.0748413292835
|
|
- type: euclidean_pearson
|
|
value: 38.533108233460304
|
|
- type: euclidean_spearman
|
|
value: 35.37638615280026
|
|
- type: manhattan_pearson
|
|
value: 41.0639726746513
|
|
- type: manhattan_spearman
|
|
value: 37.688161243671765
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (en-de)
|
|
config: en-de
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 58.4628923720782
|
|
- type: cos_sim_spearman
|
|
value: 59.10093128795948
|
|
- type: euclidean_pearson
|
|
value: 30.422902393436836
|
|
- type: euclidean_spearman
|
|
value: 27.837806030497457
|
|
- type: manhattan_pearson
|
|
value: 32.51576984630963
|
|
- type: manhattan_spearman
|
|
value: 29.181887010982514
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (en-en)
|
|
config: en-en
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 86.87447904613737
|
|
- type: cos_sim_spearman
|
|
value: 87.06554974065622
|
|
- type: euclidean_pearson
|
|
value: 76.82669047851108
|
|
- type: euclidean_spearman
|
|
value: 75.45711985511991
|
|
- type: manhattan_pearson
|
|
value: 77.46644556452847
|
|
- type: manhattan_spearman
|
|
value: 76.0249120007112
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (en-tr)
|
|
config: en-tr
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 17.784495723497468
|
|
- type: cos_sim_spearman
|
|
value: 11.79629537128697
|
|
- type: euclidean_pearson
|
|
value: -4.354328445994008
|
|
- type: euclidean_spearman
|
|
value: -6.984566116230058
|
|
- type: manhattan_pearson
|
|
value: -4.166751901507852
|
|
- type: manhattan_spearman
|
|
value: -6.984143198323786
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (es-en)
|
|
config: es-en
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 76.9009642643449
|
|
- type: cos_sim_spearman
|
|
value: 78.21764726338341
|
|
- type: euclidean_pearson
|
|
value: 50.578959144342925
|
|
- type: euclidean_spearman
|
|
value: 51.664379260719606
|
|
- type: manhattan_pearson
|
|
value: 53.95690880393329
|
|
- type: manhattan_spearman
|
|
value: 54.910058464050785
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (es-es)
|
|
config: es-es
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 86.41638022270219
|
|
- type: cos_sim_spearman
|
|
value: 86.00477030366811
|
|
- type: euclidean_pearson
|
|
value: 79.7224037788285
|
|
- type: euclidean_spearman
|
|
value: 79.21417626867616
|
|
- type: manhattan_pearson
|
|
value: 80.29412412756984
|
|
- type: manhattan_spearman
|
|
value: 79.49460867616206
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (fr-en)
|
|
config: fr-en
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 79.90432664091082
|
|
- type: cos_sim_spearman
|
|
value: 80.46007940700204
|
|
- type: euclidean_pearson
|
|
value: 49.25348015214428
|
|
- type: euclidean_spearman
|
|
value: 47.13113020475859
|
|
- type: manhattan_pearson
|
|
value: 54.57291204043908
|
|
- type: manhattan_spearman
|
|
value: 51.98559736896087
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (it-en)
|
|
config: it-en
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 52.55164822309034
|
|
- type: cos_sim_spearman
|
|
value: 51.57629192137736
|
|
- type: euclidean_pearson
|
|
value: 16.63360593235354
|
|
- type: euclidean_spearman
|
|
value: 14.479679923782912
|
|
- type: manhattan_pearson
|
|
value: 18.524867185117472
|
|
- type: manhattan_spearman
|
|
value: 16.65940056664755
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts17-crosslingual-sts
|
|
name: MTEB STS17 (nl-en)
|
|
config: nl-en
|
|
split: test
|
|
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 46.83690919715875
|
|
- type: cos_sim_spearman
|
|
value: 45.84993650002922
|
|
- type: euclidean_pearson
|
|
value: 6.173128686815117
|
|
- type: euclidean_spearman
|
|
value: 6.260781946306191
|
|
- type: manhattan_pearson
|
|
value: 7.328440452367316
|
|
- type: manhattan_spearman
|
|
value: 7.370842306497447
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (en)
|
|
config: en
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 64.97916914277232
|
|
- type: cos_sim_spearman
|
|
value: 66.13392188807865
|
|
- type: euclidean_pearson
|
|
value: 65.3921146908468
|
|
- type: euclidean_spearman
|
|
value: 65.8381588635056
|
|
- type: manhattan_pearson
|
|
value: 65.8866165769975
|
|
- type: manhattan_spearman
|
|
value: 66.27774050472219
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (de)
|
|
config: de
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 25.605130445111545
|
|
- type: cos_sim_spearman
|
|
value: 30.054844562369254
|
|
- type: euclidean_pearson
|
|
value: 23.890611005408196
|
|
- type: euclidean_spearman
|
|
value: 29.07902600726761
|
|
- type: manhattan_pearson
|
|
value: 24.239478426621833
|
|
- type: manhattan_spearman
|
|
value: 29.48547576782375
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (es)
|
|
config: es
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 61.6665616159781
|
|
- type: cos_sim_spearman
|
|
value: 65.41310206289988
|
|
- type: euclidean_pearson
|
|
value: 68.38805493215008
|
|
- type: euclidean_spearman
|
|
value: 65.22777377603435
|
|
- type: manhattan_pearson
|
|
value: 69.37445390454346
|
|
- type: manhattan_spearman
|
|
value: 66.02437701858754
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (pl)
|
|
config: pl
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 15.302891825626372
|
|
- type: cos_sim_spearman
|
|
value: 31.134517255070097
|
|
- type: euclidean_pearson
|
|
value: 12.672592658843143
|
|
- type: euclidean_spearman
|
|
value: 29.14881036784207
|
|
- type: manhattan_pearson
|
|
value: 13.528545327757735
|
|
- type: manhattan_spearman
|
|
value: 29.56217928148797
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (tr)
|
|
config: tr
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 28.79299114515319
|
|
- type: cos_sim_spearman
|
|
value: 47.135864983626206
|
|
- type: euclidean_pearson
|
|
value: 40.66410787594309
|
|
- type: euclidean_spearman
|
|
value: 45.09585593138228
|
|
- type: manhattan_pearson
|
|
value: 42.02561630700308
|
|
- type: manhattan_spearman
|
|
value: 45.43979983670554
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (ar)
|
|
config: ar
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 46.00096625052943
|
|
- type: cos_sim_spearman
|
|
value: 58.67147426715496
|
|
- type: euclidean_pearson
|
|
value: 54.7154367422438
|
|
- type: euclidean_spearman
|
|
value: 59.003235142442634
|
|
- type: manhattan_pearson
|
|
value: 56.3116235357115
|
|
- type: manhattan_spearman
|
|
value: 60.12956331404423
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (ru)
|
|
config: ru
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 29.3396354650316
|
|
- type: cos_sim_spearman
|
|
value: 43.3632935734809
|
|
- type: euclidean_pearson
|
|
value: 31.18506539466593
|
|
- type: euclidean_spearman
|
|
value: 37.531745324803815
|
|
- type: manhattan_pearson
|
|
value: 32.829038232529015
|
|
- type: manhattan_spearman
|
|
value: 38.04574361589953
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (zh)
|
|
config: zh
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 62.9596148375188
|
|
- type: cos_sim_spearman
|
|
value: 66.77653412402461
|
|
- type: euclidean_pearson
|
|
value: 64.53156585980886
|
|
- type: euclidean_spearman
|
|
value: 66.2884373036083
|
|
- type: manhattan_pearson
|
|
value: 65.2831035495143
|
|
- type: manhattan_spearman
|
|
value: 66.83641945244322
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (fr)
|
|
config: fr
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 79.9138821493919
|
|
- type: cos_sim_spearman
|
|
value: 80.38097535004677
|
|
- type: euclidean_pearson
|
|
value: 76.2401499094322
|
|
- type: euclidean_spearman
|
|
value: 77.00897050735907
|
|
- type: manhattan_pearson
|
|
value: 76.69531453728563
|
|
- type: manhattan_spearman
|
|
value: 77.83189696428695
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (de-en)
|
|
config: de-en
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 51.27009640779202
|
|
- type: cos_sim_spearman
|
|
value: 51.16120562029285
|
|
- type: euclidean_pearson
|
|
value: 52.20594985566323
|
|
- type: euclidean_spearman
|
|
value: 52.75331049709882
|
|
- type: manhattan_pearson
|
|
value: 52.2725118792549
|
|
- type: manhattan_spearman
|
|
value: 53.614847968995115
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (es-en)
|
|
config: es-en
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 70.46044814118835
|
|
- type: cos_sim_spearman
|
|
value: 75.05760236668672
|
|
- type: euclidean_pearson
|
|
value: 72.80128921879461
|
|
- type: euclidean_spearman
|
|
value: 73.81164755219257
|
|
- type: manhattan_pearson
|
|
value: 72.7863795809044
|
|
- type: manhattan_spearman
|
|
value: 73.65932033818906
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (it)
|
|
config: it
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 61.89276840435938
|
|
- type: cos_sim_spearman
|
|
value: 65.65042955732055
|
|
- type: euclidean_pearson
|
|
value: 61.22969491863841
|
|
- type: euclidean_spearman
|
|
value: 63.451215637904724
|
|
- type: manhattan_pearson
|
|
value: 61.16138956945465
|
|
- type: manhattan_spearman
|
|
value: 63.34966179331079
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (pl-en)
|
|
config: pl-en
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 56.377577221753626
|
|
- type: cos_sim_spearman
|
|
value: 53.31223653270353
|
|
- type: euclidean_pearson
|
|
value: 26.488793041564307
|
|
- type: euclidean_spearman
|
|
value: 19.524551741701472
|
|
- type: manhattan_pearson
|
|
value: 24.322868054606474
|
|
- type: manhattan_spearman
|
|
value: 19.50371443994939
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (zh-en)
|
|
config: zh-en
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 69.3634693673425
|
|
- type: cos_sim_spearman
|
|
value: 68.45051245419702
|
|
- type: euclidean_pearson
|
|
value: 56.1417414374769
|
|
- type: euclidean_spearman
|
|
value: 55.89891749631458
|
|
- type: manhattan_pearson
|
|
value: 57.266417430882925
|
|
- type: manhattan_spearman
|
|
value: 56.57927102744128
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (es-it)
|
|
config: es-it
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 60.04169437653179
|
|
- type: cos_sim_spearman
|
|
value: 65.49531007553446
|
|
- type: euclidean_pearson
|
|
value: 58.583860732586324
|
|
- type: euclidean_spearman
|
|
value: 58.80034792537441
|
|
- type: manhattan_pearson
|
|
value: 59.02513161664622
|
|
- type: manhattan_spearman
|
|
value: 58.42942047904558
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (de-fr)
|
|
config: de-fr
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 48.81035211493999
|
|
- type: cos_sim_spearman
|
|
value: 53.27599246786967
|
|
- type: euclidean_pearson
|
|
value: 52.25710699032889
|
|
- type: euclidean_spearman
|
|
value: 55.22995695529873
|
|
- type: manhattan_pearson
|
|
value: 51.894901893217884
|
|
- type: manhattan_spearman
|
|
value: 54.95919975149795
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (de-pl)
|
|
config: de-pl
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 36.75993101477816
|
|
- type: cos_sim_spearman
|
|
value: 43.050156692479355
|
|
- type: euclidean_pearson
|
|
value: 51.49021084746248
|
|
- type: euclidean_spearman
|
|
value: 49.54771253090078
|
|
- type: manhattan_pearson
|
|
value: 54.68410760796417
|
|
- type: manhattan_spearman
|
|
value: 48.19277197691717
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/sts22-crosslingual-sts
|
|
name: MTEB STS22 (fr-pl)
|
|
config: fr-pl
|
|
split: test
|
|
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 48.553763306386486
|
|
- type: cos_sim_spearman
|
|
value: 28.17180849095055
|
|
- type: euclidean_pearson
|
|
value: 17.50739087826514
|
|
- type: euclidean_spearman
|
|
value: 16.903085094570333
|
|
- type: manhattan_pearson
|
|
value: 20.750046512534112
|
|
- type: manhattan_spearman
|
|
value: 5.634361698190111
|
|
- task:
|
|
type: STS
|
|
dataset:
|
|
type: mteb/stsbenchmark-sts
|
|
name: MTEB STSBenchmark
|
|
config: default
|
|
split: test
|
|
revision: 8913289635987208e6e7c72789e4be2fe94b6abd
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 82.17107190594417
|
|
- type: cos_sim_spearman
|
|
value: 80.89611873505183
|
|
- type: euclidean_pearson
|
|
value: 71.82491561814403
|
|
- type: euclidean_spearman
|
|
value: 70.33608835403274
|
|
- type: manhattan_pearson
|
|
value: 71.89538332420133
|
|
- type: manhattan_spearman
|
|
value: 70.36082395775944
|
|
- task:
|
|
type: Reranking
|
|
dataset:
|
|
type: mteb/scidocs-reranking
|
|
name: MTEB SciDocsRR
|
|
config: default
|
|
split: test
|
|
revision: 56a6d0140cf6356659e2a7c1413286a774468d44
|
|
metrics:
|
|
- type: map
|
|
value: 79.77047154974562
|
|
- type: mrr
|
|
value: 94.25887021475256
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: scifact
|
|
name: MTEB SciFact
|
|
config: default
|
|
split: test
|
|
revision: a75ae049398addde9b70f6b268875f5cbce99089
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 56.328
|
|
- type: map_at_10
|
|
value: 67.167
|
|
- type: map_at_100
|
|
value: 67.721
|
|
- type: map_at_1000
|
|
value: 67.735
|
|
- type: map_at_3
|
|
value: 64.20400000000001
|
|
- type: map_at_5
|
|
value: 65.904
|
|
- type: mrr_at_1
|
|
value: 59.667
|
|
- type: mrr_at_10
|
|
value: 68.553
|
|
- type: mrr_at_100
|
|
value: 68.992
|
|
- type: mrr_at_1000
|
|
value: 69.004
|
|
- type: mrr_at_3
|
|
value: 66.22200000000001
|
|
- type: mrr_at_5
|
|
value: 67.739
|
|
- type: ndcg_at_1
|
|
value: 59.667
|
|
- type: ndcg_at_10
|
|
value: 72.111
|
|
- type: ndcg_at_100
|
|
value: 74.441
|
|
- type: ndcg_at_1000
|
|
value: 74.90599999999999
|
|
- type: ndcg_at_3
|
|
value: 67.11399999999999
|
|
- type: ndcg_at_5
|
|
value: 69.687
|
|
- type: precision_at_1
|
|
value: 59.667
|
|
- type: precision_at_10
|
|
value: 9.733
|
|
- type: precision_at_100
|
|
value: 1.09
|
|
- type: precision_at_1000
|
|
value: 0.11299999999999999
|
|
- type: precision_at_3
|
|
value: 26.444000000000003
|
|
- type: precision_at_5
|
|
value: 17.599999999999998
|
|
- type: recall_at_1
|
|
value: 56.328
|
|
- type: recall_at_10
|
|
value: 85.8
|
|
- type: recall_at_100
|
|
value: 96.167
|
|
- type: recall_at_1000
|
|
value: 100.0
|
|
- type: recall_at_3
|
|
value: 72.433
|
|
- type: recall_at_5
|
|
value: 78.972
|
|
- task:
|
|
type: PairClassification
|
|
dataset:
|
|
type: mteb/sprintduplicatequestions-pairclassification
|
|
name: MTEB SprintDuplicateQuestions
|
|
config: default
|
|
split: test
|
|
revision: 5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea
|
|
metrics:
|
|
- type: cos_sim_accuracy
|
|
value: 99.8019801980198
|
|
- type: cos_sim_ap
|
|
value: 94.92527097094644
|
|
- type: cos_sim_f1
|
|
value: 89.91935483870968
|
|
- type: cos_sim_precision
|
|
value: 90.65040650406505
|
|
- type: cos_sim_recall
|
|
value: 89.2
|
|
- type: dot_accuracy
|
|
value: 99.51782178217822
|
|
- type: dot_ap
|
|
value: 81.30756869559929
|
|
- type: dot_f1
|
|
value: 75.88235294117648
|
|
- type: dot_precision
|
|
value: 74.42307692307692
|
|
- type: dot_recall
|
|
value: 77.4
|
|
- type: euclidean_accuracy
|
|
value: 99.73069306930694
|
|
- type: euclidean_ap
|
|
value: 91.05040371796932
|
|
- type: euclidean_f1
|
|
value: 85.7889237199582
|
|
- type: euclidean_precision
|
|
value: 89.82494529540482
|
|
- type: euclidean_recall
|
|
value: 82.1
|
|
- type: manhattan_accuracy
|
|
value: 99.73762376237623
|
|
- type: manhattan_ap
|
|
value: 91.4823412839869
|
|
- type: manhattan_f1
|
|
value: 86.39836984207845
|
|
- type: manhattan_precision
|
|
value: 88.05815160955348
|
|
- type: manhattan_recall
|
|
value: 84.8
|
|
- type: max_accuracy
|
|
value: 99.8019801980198
|
|
- type: max_ap
|
|
value: 94.92527097094644
|
|
- type: max_f1
|
|
value: 89.91935483870968
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/stackexchange-clustering
|
|
name: MTEB StackExchangeClustering
|
|
config: default
|
|
split: test
|
|
revision: 70a89468f6dccacc6aa2b12a6eac54e74328f235
|
|
metrics:
|
|
- type: v_measure
|
|
value: 55.13046832022158
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/stackexchange-clustering-p2p
|
|
name: MTEB StackExchangeClusteringP2P
|
|
config: default
|
|
split: test
|
|
revision: d88009ab563dd0b16cfaf4436abaf97fa3550cf0
|
|
metrics:
|
|
- type: v_measure
|
|
value: 34.31252463546675
|
|
- task:
|
|
type: Reranking
|
|
dataset:
|
|
type: mteb/stackoverflowdupquestions-reranking
|
|
name: MTEB StackOverflowDupQuestions
|
|
config: default
|
|
split: test
|
|
revision: ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9
|
|
metrics:
|
|
- type: map
|
|
value: 51.06639688231414
|
|
- type: mrr
|
|
value: 51.80205415499534
|
|
- task:
|
|
type: Summarization
|
|
dataset:
|
|
type: mteb/summeval
|
|
name: MTEB SummEval
|
|
config: default
|
|
split: test
|
|
revision: 8753c2788d36c01fc6f05d03fe3f7268d63f9122
|
|
metrics:
|
|
- type: cos_sim_pearson
|
|
value: 31.963331462886957
|
|
- type: cos_sim_spearman
|
|
value: 33.59510652629926
|
|
- type: dot_pearson
|
|
value: 29.033733540882123
|
|
- type: dot_spearman
|
|
value: 31.550290638315504
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: trec-covid
|
|
name: MTEB TRECCOVID
|
|
config: default
|
|
split: test
|
|
revision: 2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 0.23600000000000002
|
|
- type: map_at_10
|
|
value: 2.09
|
|
- type: map_at_100
|
|
value: 12.466000000000001
|
|
- type: map_at_1000
|
|
value: 29.852
|
|
- type: map_at_3
|
|
value: 0.6859999999999999
|
|
- type: map_at_5
|
|
value: 1.099
|
|
- type: mrr_at_1
|
|
value: 88.0
|
|
- type: mrr_at_10
|
|
value: 94.0
|
|
- type: mrr_at_100
|
|
value: 94.0
|
|
- type: mrr_at_1000
|
|
value: 94.0
|
|
- type: mrr_at_3
|
|
value: 94.0
|
|
- type: mrr_at_5
|
|
value: 94.0
|
|
- type: ndcg_at_1
|
|
value: 86.0
|
|
- type: ndcg_at_10
|
|
value: 81.368
|
|
- type: ndcg_at_100
|
|
value: 61.879
|
|
- type: ndcg_at_1000
|
|
value: 55.282
|
|
- type: ndcg_at_3
|
|
value: 84.816
|
|
- type: ndcg_at_5
|
|
value: 82.503
|
|
- type: precision_at_1
|
|
value: 88.0
|
|
- type: precision_at_10
|
|
value: 85.6
|
|
- type: precision_at_100
|
|
value: 63.85999999999999
|
|
- type: precision_at_1000
|
|
value: 24.682000000000002
|
|
- type: precision_at_3
|
|
value: 88.667
|
|
- type: precision_at_5
|
|
value: 86.0
|
|
- type: recall_at_1
|
|
value: 0.23600000000000002
|
|
- type: recall_at_10
|
|
value: 2.25
|
|
- type: recall_at_100
|
|
value: 15.488
|
|
- type: recall_at_1000
|
|
value: 52.196
|
|
- type: recall_at_3
|
|
value: 0.721
|
|
- type: recall_at_5
|
|
value: 1.159
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (sqi-eng)
|
|
config: sqi-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 12.7
|
|
- type: f1
|
|
value: 10.384182044950325
|
|
- type: precision
|
|
value: 9.805277385275312
|
|
- type: recall
|
|
value: 12.7
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (fry-eng)
|
|
config: fry-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 30.63583815028902
|
|
- type: f1
|
|
value: 24.623726947426373
|
|
- type: precision
|
|
value: 22.987809919828013
|
|
- type: recall
|
|
value: 30.63583815028902
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (kur-eng)
|
|
config: kur-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 10.487804878048781
|
|
- type: f1
|
|
value: 8.255945048627975
|
|
- type: precision
|
|
value: 7.649047253615001
|
|
- type: recall
|
|
value: 10.487804878048781
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (tur-eng)
|
|
config: tur-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 8.5
|
|
- type: f1
|
|
value: 6.154428783776609
|
|
- type: precision
|
|
value: 5.680727638128585
|
|
- type: recall
|
|
value: 8.5
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (deu-eng)
|
|
config: deu-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 73.0
|
|
- type: f1
|
|
value: 70.10046605876393
|
|
- type: precision
|
|
value: 69.0018253968254
|
|
- type: recall
|
|
value: 73.0
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (nld-eng)
|
|
config: nld-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 32.7
|
|
- type: f1
|
|
value: 29.7428583868239
|
|
- type: precision
|
|
value: 28.81671359506905
|
|
- type: recall
|
|
value: 32.7
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ron-eng)
|
|
config: ron-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 31.5
|
|
- type: f1
|
|
value: 27.228675552174003
|
|
- type: precision
|
|
value: 25.950062299847747
|
|
- type: recall
|
|
value: 31.5
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ang-eng)
|
|
config: ang-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 35.82089552238806
|
|
- type: f1
|
|
value: 28.75836980510979
|
|
- type: precision
|
|
value: 26.971643613434658
|
|
- type: recall
|
|
value: 35.82089552238806
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ido-eng)
|
|
config: ido-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 49.8
|
|
- type: f1
|
|
value: 43.909237401451776
|
|
- type: precision
|
|
value: 41.944763440988936
|
|
- type: recall
|
|
value: 49.8
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (jav-eng)
|
|
config: jav-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 18.536585365853657
|
|
- type: f1
|
|
value: 15.020182570246751
|
|
- type: precision
|
|
value: 14.231108073213337
|
|
- type: recall
|
|
value: 18.536585365853657
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (isl-eng)
|
|
config: isl-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 8.7
|
|
- type: f1
|
|
value: 6.2934784902885355
|
|
- type: precision
|
|
value: 5.685926293425392
|
|
- type: recall
|
|
value: 8.7
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (slv-eng)
|
|
config: slv-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 12.879708383961116
|
|
- type: f1
|
|
value: 10.136118341751114
|
|
- type: precision
|
|
value: 9.571444036679436
|
|
- type: recall
|
|
value: 12.879708383961116
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (cym-eng)
|
|
config: cym-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 9.217391304347826
|
|
- type: f1
|
|
value: 6.965003297761793
|
|
- type: precision
|
|
value: 6.476093529199119
|
|
- type: recall
|
|
value: 9.217391304347826
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (kaz-eng)
|
|
config: kaz-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 4.3478260869565215
|
|
- type: f1
|
|
value: 3.3186971707677397
|
|
- type: precision
|
|
value: 3.198658632552104
|
|
- type: recall
|
|
value: 4.3478260869565215
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (est-eng)
|
|
config: est-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 6.9
|
|
- type: f1
|
|
value: 4.760708297894056
|
|
- type: precision
|
|
value: 4.28409511756074
|
|
- type: recall
|
|
value: 6.9
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (heb-eng)
|
|
config: heb-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 2.1999999999999997
|
|
- type: f1
|
|
value: 1.6862703878117107
|
|
- type: precision
|
|
value: 1.6048118233915603
|
|
- type: recall
|
|
value: 2.1999999999999997
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (gla-eng)
|
|
config: gla-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 3.0156815440289506
|
|
- type: f1
|
|
value: 2.0913257250659134
|
|
- type: precision
|
|
value: 1.9072775486461648
|
|
- type: recall
|
|
value: 3.0156815440289506
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (mar-eng)
|
|
config: mar-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 49.0
|
|
- type: f1
|
|
value: 45.5254456536713
|
|
- type: precision
|
|
value: 44.134609250398725
|
|
- type: recall
|
|
value: 49.0
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (lat-eng)
|
|
config: lat-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 33.5
|
|
- type: f1
|
|
value: 28.759893973182564
|
|
- type: precision
|
|
value: 27.401259116024836
|
|
- type: recall
|
|
value: 33.5
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (bel-eng)
|
|
config: bel-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 10.2
|
|
- type: f1
|
|
value: 8.030039981676275
|
|
- type: precision
|
|
value: 7.548748077210127
|
|
- type: recall
|
|
value: 10.2
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (pms-eng)
|
|
config: pms-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 38.095238095238095
|
|
- type: f1
|
|
value: 31.944999250262406
|
|
- type: precision
|
|
value: 30.04452690166976
|
|
- type: recall
|
|
value: 38.095238095238095
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (gle-eng)
|
|
config: gle-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 4.8
|
|
- type: f1
|
|
value: 3.2638960786708067
|
|
- type: precision
|
|
value: 3.0495382950729644
|
|
- type: recall
|
|
value: 4.8
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (pes-eng)
|
|
config: pes-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 15.8
|
|
- type: f1
|
|
value: 12.131087470371275
|
|
- type: precision
|
|
value: 11.141304011547815
|
|
- type: recall
|
|
value: 15.8
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (nob-eng)
|
|
config: nob-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 23.3
|
|
- type: f1
|
|
value: 21.073044636921384
|
|
- type: precision
|
|
value: 20.374220568287285
|
|
- type: recall
|
|
value: 23.3
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (bul-eng)
|
|
config: bul-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 24.9
|
|
- type: f1
|
|
value: 20.091060685364987
|
|
- type: precision
|
|
value: 18.899700591081224
|
|
- type: recall
|
|
value: 24.9
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (cbk-eng)
|
|
config: cbk-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 70.1
|
|
- type: f1
|
|
value: 64.62940836940835
|
|
- type: precision
|
|
value: 62.46559523809524
|
|
- type: recall
|
|
value: 70.1
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (hun-eng)
|
|
config: hun-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 7.199999999999999
|
|
- type: f1
|
|
value: 5.06613460576115
|
|
- type: precision
|
|
value: 4.625224463391809
|
|
- type: recall
|
|
value: 7.199999999999999
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (uig-eng)
|
|
config: uig-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 1.7999999999999998
|
|
- type: f1
|
|
value: 1.2716249514772895
|
|
- type: precision
|
|
value: 1.2107445914723798
|
|
- type: recall
|
|
value: 1.7999999999999998
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (rus-eng)
|
|
config: rus-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 65.5
|
|
- type: f1
|
|
value: 59.84399711399712
|
|
- type: precision
|
|
value: 57.86349567099567
|
|
- type: recall
|
|
value: 65.5
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (spa-eng)
|
|
config: spa-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 95.7
|
|
- type: f1
|
|
value: 94.48333333333333
|
|
- type: precision
|
|
value: 93.89999999999999
|
|
- type: recall
|
|
value: 95.7
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (hye-eng)
|
|
config: hye-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 0.8086253369272237
|
|
- type: f1
|
|
value: 0.4962046191492002
|
|
- type: precision
|
|
value: 0.47272438578554393
|
|
- type: recall
|
|
value: 0.8086253369272237
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (tel-eng)
|
|
config: tel-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 69.23076923076923
|
|
- type: f1
|
|
value: 64.6227941099736
|
|
- type: precision
|
|
value: 63.03795877325289
|
|
- type: recall
|
|
value: 69.23076923076923
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (afr-eng)
|
|
config: afr-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 20.599999999999998
|
|
- type: f1
|
|
value: 16.62410040660465
|
|
- type: precision
|
|
value: 15.598352437967069
|
|
- type: recall
|
|
value: 20.599999999999998
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (mon-eng)
|
|
config: mon-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 4.318181818181818
|
|
- type: f1
|
|
value: 2.846721192535661
|
|
- type: precision
|
|
value: 2.6787861417537147
|
|
- type: recall
|
|
value: 4.318181818181818
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (arz-eng)
|
|
config: arz-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 74.84276729559748
|
|
- type: f1
|
|
value: 70.6638714185884
|
|
- type: precision
|
|
value: 68.86792452830188
|
|
- type: recall
|
|
value: 74.84276729559748
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (hrv-eng)
|
|
config: hrv-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 15.9
|
|
- type: f1
|
|
value: 12.793698974586706
|
|
- type: precision
|
|
value: 12.088118017657736
|
|
- type: recall
|
|
value: 15.9
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (nov-eng)
|
|
config: nov-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 59.92217898832685
|
|
- type: f1
|
|
value: 52.23086900129701
|
|
- type: precision
|
|
value: 49.25853869433636
|
|
- type: recall
|
|
value: 59.92217898832685
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (gsw-eng)
|
|
config: gsw-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 27.350427350427353
|
|
- type: f1
|
|
value: 21.033781033781032
|
|
- type: precision
|
|
value: 19.337955491801644
|
|
- type: recall
|
|
value: 27.350427350427353
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (nds-eng)
|
|
config: nds-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 29.299999999999997
|
|
- type: f1
|
|
value: 23.91597452425777
|
|
- type: precision
|
|
value: 22.36696598364942
|
|
- type: recall
|
|
value: 29.299999999999997
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ukr-eng)
|
|
config: ukr-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 27.3
|
|
- type: f1
|
|
value: 22.059393517688886
|
|
- type: precision
|
|
value: 20.503235534170887
|
|
- type: recall
|
|
value: 27.3
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (uzb-eng)
|
|
config: uzb-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 8.177570093457943
|
|
- type: f1
|
|
value: 4.714367017906037
|
|
- type: precision
|
|
value: 4.163882933965758
|
|
- type: recall
|
|
value: 8.177570093457943
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (lit-eng)
|
|
config: lit-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 5.800000000000001
|
|
- type: f1
|
|
value: 4.4859357432293825
|
|
- type: precision
|
|
value: 4.247814465614043
|
|
- type: recall
|
|
value: 5.800000000000001
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ina-eng)
|
|
config: ina-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 78.4
|
|
- type: f1
|
|
value: 73.67166666666667
|
|
- type: precision
|
|
value: 71.83285714285714
|
|
- type: recall
|
|
value: 78.4
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (lfn-eng)
|
|
config: lfn-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 50.3
|
|
- type: f1
|
|
value: 44.85221545883311
|
|
- type: precision
|
|
value: 43.04913026243909
|
|
- type: recall
|
|
value: 50.3
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (zsm-eng)
|
|
config: zsm-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 83.5
|
|
- type: f1
|
|
value: 79.95151515151515
|
|
- type: precision
|
|
value: 78.53611111111111
|
|
- type: recall
|
|
value: 83.5
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ita-eng)
|
|
config: ita-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 69.89999999999999
|
|
- type: f1
|
|
value: 65.03756269256269
|
|
- type: precision
|
|
value: 63.233519536019536
|
|
- type: recall
|
|
value: 69.89999999999999
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (cmn-eng)
|
|
config: cmn-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 93.2
|
|
- type: f1
|
|
value: 91.44666666666666
|
|
- type: precision
|
|
value: 90.63333333333333
|
|
- type: recall
|
|
value: 93.2
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (lvs-eng)
|
|
config: lvs-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 8.3
|
|
- type: f1
|
|
value: 6.553388144729963
|
|
- type: precision
|
|
value: 6.313497782829976
|
|
- type: recall
|
|
value: 8.3
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (glg-eng)
|
|
config: glg-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 83.6
|
|
- type: f1
|
|
value: 79.86243107769424
|
|
- type: precision
|
|
value: 78.32555555555555
|
|
- type: recall
|
|
value: 83.6
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ceb-eng)
|
|
config: ceb-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 9.166666666666666
|
|
- type: f1
|
|
value: 6.637753604420271
|
|
- type: precision
|
|
value: 6.10568253585495
|
|
- type: recall
|
|
value: 9.166666666666666
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (bre-eng)
|
|
config: bre-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 7.3999999999999995
|
|
- type: f1
|
|
value: 4.6729483612322165
|
|
- type: precision
|
|
value: 4.103844520292658
|
|
- type: recall
|
|
value: 7.3999999999999995
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ben-eng)
|
|
config: ben-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 80.30000000000001
|
|
- type: f1
|
|
value: 75.97666666666667
|
|
- type: precision
|
|
value: 74.16
|
|
- type: recall
|
|
value: 80.30000000000001
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (swg-eng)
|
|
config: swg-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 23.214285714285715
|
|
- type: f1
|
|
value: 16.88988095238095
|
|
- type: precision
|
|
value: 15.364937641723353
|
|
- type: recall
|
|
value: 23.214285714285715
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (arq-eng)
|
|
config: arq-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 33.15038419319429
|
|
- type: f1
|
|
value: 27.747873024072415
|
|
- type: precision
|
|
value: 25.99320572578704
|
|
- type: recall
|
|
value: 33.15038419319429
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (kab-eng)
|
|
config: kab-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 2.6
|
|
- type: f1
|
|
value: 1.687059048752127
|
|
- type: precision
|
|
value: 1.5384884521299
|
|
- type: recall
|
|
value: 2.6
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (fra-eng)
|
|
config: fra-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 93.30000000000001
|
|
- type: f1
|
|
value: 91.44000000000001
|
|
- type: precision
|
|
value: 90.59166666666667
|
|
- type: recall
|
|
value: 93.30000000000001
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (por-eng)
|
|
config: por-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 94.1
|
|
- type: f1
|
|
value: 92.61666666666667
|
|
- type: precision
|
|
value: 91.88333333333333
|
|
- type: recall
|
|
value: 94.1
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (tat-eng)
|
|
config: tat-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 5.0
|
|
- type: f1
|
|
value: 3.589591971281927
|
|
- type: precision
|
|
value: 3.3046491614532854
|
|
- type: recall
|
|
value: 5.0
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (oci-eng)
|
|
config: oci-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 45.9
|
|
- type: f1
|
|
value: 40.171969141969136
|
|
- type: precision
|
|
value: 38.30764368870302
|
|
- type: recall
|
|
value: 45.9
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (pol-eng)
|
|
config: pol-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 16.900000000000002
|
|
- type: f1
|
|
value: 14.094365204207351
|
|
- type: precision
|
|
value: 13.276519841269844
|
|
- type: recall
|
|
value: 16.900000000000002
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (war-eng)
|
|
config: war-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 12.8
|
|
- type: f1
|
|
value: 10.376574912567156
|
|
- type: precision
|
|
value: 9.758423963284509
|
|
- type: recall
|
|
value: 12.8
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (aze-eng)
|
|
config: aze-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 8.1
|
|
- type: f1
|
|
value: 6.319455355175778
|
|
- type: precision
|
|
value: 5.849948830628881
|
|
- type: recall
|
|
value: 8.1
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (vie-eng)
|
|
config: vie-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 95.5
|
|
- type: f1
|
|
value: 94.19666666666667
|
|
- type: precision
|
|
value: 93.60000000000001
|
|
- type: recall
|
|
value: 95.5
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (nno-eng)
|
|
config: nno-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 19.1
|
|
- type: f1
|
|
value: 16.280080686081906
|
|
- type: precision
|
|
value: 15.451573089395668
|
|
- type: recall
|
|
value: 19.1
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (cha-eng)
|
|
config: cha-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 30.656934306569344
|
|
- type: f1
|
|
value: 23.2568647897115
|
|
- type: precision
|
|
value: 21.260309034031664
|
|
- type: recall
|
|
value: 30.656934306569344
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (mhr-eng)
|
|
config: mhr-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 2.1999999999999997
|
|
- type: f1
|
|
value: 1.556861047295521
|
|
- type: precision
|
|
value: 1.4555993437238521
|
|
- type: recall
|
|
value: 2.1999999999999997
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (dan-eng)
|
|
config: dan-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 27.500000000000004
|
|
- type: f1
|
|
value: 23.521682636223492
|
|
- type: precision
|
|
value: 22.345341306967683
|
|
- type: recall
|
|
value: 27.500000000000004
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ell-eng)
|
|
config: ell-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 7.3999999999999995
|
|
- type: f1
|
|
value: 5.344253880846173
|
|
- type: precision
|
|
value: 4.999794279068863
|
|
- type: recall
|
|
value: 7.3999999999999995
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (amh-eng)
|
|
config: amh-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 0.5952380952380952
|
|
- type: f1
|
|
value: 0.026455026455026457
|
|
- type: precision
|
|
value: 0.013528138528138528
|
|
- type: recall
|
|
value: 0.5952380952380952
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (pam-eng)
|
|
config: pam-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 7.3
|
|
- type: f1
|
|
value: 5.853140211779251
|
|
- type: precision
|
|
value: 5.505563080945322
|
|
- type: recall
|
|
value: 7.3
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (hsb-eng)
|
|
config: hsb-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 13.250517598343686
|
|
- type: f1
|
|
value: 9.676349506190704
|
|
- type: precision
|
|
value: 8.930392053553216
|
|
- type: recall
|
|
value: 13.250517598343686
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (srp-eng)
|
|
config: srp-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 14.499999999999998
|
|
- type: f1
|
|
value: 11.68912588067557
|
|
- type: precision
|
|
value: 11.024716513105519
|
|
- type: recall
|
|
value: 14.499999999999998
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (epo-eng)
|
|
config: epo-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 30.099999999999998
|
|
- type: f1
|
|
value: 26.196880936315146
|
|
- type: precision
|
|
value: 25.271714086169478
|
|
- type: recall
|
|
value: 30.099999999999998
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (kzj-eng)
|
|
config: kzj-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 6.4
|
|
- type: f1
|
|
value: 5.1749445942023335
|
|
- type: precision
|
|
value: 4.975338142029625
|
|
- type: recall
|
|
value: 6.4
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (awa-eng)
|
|
config: awa-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 39.39393939393939
|
|
- type: f1
|
|
value: 35.005707393767096
|
|
- type: precision
|
|
value: 33.64342032053631
|
|
- type: recall
|
|
value: 39.39393939393939
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (fao-eng)
|
|
config: fao-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 18.3206106870229
|
|
- type: f1
|
|
value: 12.610893447220345
|
|
- type: precision
|
|
value: 11.079228765297467
|
|
- type: recall
|
|
value: 18.3206106870229
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (mal-eng)
|
|
config: mal-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 85.58951965065502
|
|
- type: f1
|
|
value: 83.30363944928548
|
|
- type: precision
|
|
value: 82.40026591554977
|
|
- type: recall
|
|
value: 85.58951965065502
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ile-eng)
|
|
config: ile-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 65.7
|
|
- type: f1
|
|
value: 59.589642857142856
|
|
- type: precision
|
|
value: 57.392826797385624
|
|
- type: recall
|
|
value: 65.7
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (bos-eng)
|
|
config: bos-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 18.07909604519774
|
|
- type: f1
|
|
value: 13.65194306689995
|
|
- type: precision
|
|
value: 12.567953943826327
|
|
- type: recall
|
|
value: 18.07909604519774
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (cor-eng)
|
|
config: cor-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 4.6
|
|
- type: f1
|
|
value: 2.8335386392505013
|
|
- type: precision
|
|
value: 2.558444143575722
|
|
- type: recall
|
|
value: 4.6
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (cat-eng)
|
|
config: cat-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 90.7
|
|
- type: f1
|
|
value: 88.30666666666666
|
|
- type: precision
|
|
value: 87.195
|
|
- type: recall
|
|
value: 90.7
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (eus-eng)
|
|
config: eus-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 57.699999999999996
|
|
- type: f1
|
|
value: 53.38433067253876
|
|
- type: precision
|
|
value: 51.815451335350346
|
|
- type: recall
|
|
value: 57.699999999999996
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (yue-eng)
|
|
config: yue-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 80.60000000000001
|
|
- type: f1
|
|
value: 77.0290354090354
|
|
- type: precision
|
|
value: 75.61685897435898
|
|
- type: recall
|
|
value: 80.60000000000001
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (swe-eng)
|
|
config: swe-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 24.6
|
|
- type: f1
|
|
value: 19.52814960069739
|
|
- type: precision
|
|
value: 18.169084599880502
|
|
- type: recall
|
|
value: 24.6
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (dtp-eng)
|
|
config: dtp-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 5.0
|
|
- type: f1
|
|
value: 3.4078491753102376
|
|
- type: precision
|
|
value: 3.1757682319102387
|
|
- type: recall
|
|
value: 5.0
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (kat-eng)
|
|
config: kat-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 1.2064343163538873
|
|
- type: f1
|
|
value: 0.4224313053283095
|
|
- type: precision
|
|
value: 0.3360484946842894
|
|
- type: recall
|
|
value: 1.2064343163538873
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (jpn-eng)
|
|
config: jpn-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 76.1
|
|
- type: f1
|
|
value: 71.36246031746032
|
|
- type: precision
|
|
value: 69.5086544011544
|
|
- type: recall
|
|
value: 76.1
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (csb-eng)
|
|
config: csb-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 14.229249011857709
|
|
- type: f1
|
|
value: 10.026578603653704
|
|
- type: precision
|
|
value: 9.09171178352764
|
|
- type: recall
|
|
value: 14.229249011857709
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (xho-eng)
|
|
config: xho-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 8.450704225352112
|
|
- type: f1
|
|
value: 5.51214407186151
|
|
- type: precision
|
|
value: 4.928281812084629
|
|
- type: recall
|
|
value: 8.450704225352112
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (orv-eng)
|
|
config: orv-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 7.664670658682635
|
|
- type: f1
|
|
value: 5.786190079917295
|
|
- type: precision
|
|
value: 5.3643643579244
|
|
- type: recall
|
|
value: 7.664670658682635
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ind-eng)
|
|
config: ind-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 90.5
|
|
- type: f1
|
|
value: 88.03999999999999
|
|
- type: precision
|
|
value: 86.94833333333334
|
|
- type: recall
|
|
value: 90.5
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (tuk-eng)
|
|
config: tuk-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 7.389162561576355
|
|
- type: f1
|
|
value: 5.482366349556517
|
|
- type: precision
|
|
value: 5.156814449917898
|
|
- type: recall
|
|
value: 7.389162561576355
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (max-eng)
|
|
config: max-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 41.54929577464789
|
|
- type: f1
|
|
value: 36.13520282534367
|
|
- type: precision
|
|
value: 34.818226488560995
|
|
- type: recall
|
|
value: 41.54929577464789
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (swh-eng)
|
|
config: swh-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 20.76923076923077
|
|
- type: f1
|
|
value: 16.742497560177643
|
|
- type: precision
|
|
value: 15.965759712090138
|
|
- type: recall
|
|
value: 20.76923076923077
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (hin-eng)
|
|
config: hin-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 88.1
|
|
- type: f1
|
|
value: 85.23176470588236
|
|
- type: precision
|
|
value: 84.04458333333334
|
|
- type: recall
|
|
value: 88.1
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (dsb-eng)
|
|
config: dsb-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 11.899791231732777
|
|
- type: f1
|
|
value: 8.776706659565102
|
|
- type: precision
|
|
value: 8.167815946521582
|
|
- type: recall
|
|
value: 11.899791231732777
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ber-eng)
|
|
config: ber-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 6.1
|
|
- type: f1
|
|
value: 4.916589537178435
|
|
- type: precision
|
|
value: 4.72523017415345
|
|
- type: recall
|
|
value: 6.1
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (tam-eng)
|
|
config: tam-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 76.54723127035831
|
|
- type: f1
|
|
value: 72.75787187839306
|
|
- type: precision
|
|
value: 71.43338442869005
|
|
- type: recall
|
|
value: 76.54723127035831
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (slk-eng)
|
|
config: slk-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 11.700000000000001
|
|
- type: f1
|
|
value: 9.975679190026007
|
|
- type: precision
|
|
value: 9.569927715653522
|
|
- type: recall
|
|
value: 11.700000000000001
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (tgl-eng)
|
|
config: tgl-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 13.100000000000001
|
|
- type: f1
|
|
value: 10.697335850115408
|
|
- type: precision
|
|
value: 10.113816082086341
|
|
- type: recall
|
|
value: 13.100000000000001
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ast-eng)
|
|
config: ast-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 76.37795275590551
|
|
- type: f1
|
|
value: 71.12860892388451
|
|
- type: precision
|
|
value: 68.89763779527559
|
|
- type: recall
|
|
value: 76.37795275590551
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (mkd-eng)
|
|
config: mkd-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 13.700000000000001
|
|
- type: f1
|
|
value: 10.471861684067568
|
|
- type: precision
|
|
value: 9.602902567641697
|
|
- type: recall
|
|
value: 13.700000000000001
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (khm-eng)
|
|
config: khm-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 0.554016620498615
|
|
- type: f1
|
|
value: 0.37034084643642423
|
|
- type: precision
|
|
value: 0.34676040281208437
|
|
- type: recall
|
|
value: 0.554016620498615
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ces-eng)
|
|
config: ces-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 12.4
|
|
- type: f1
|
|
value: 9.552607451092534
|
|
- type: precision
|
|
value: 8.985175505050504
|
|
- type: recall
|
|
value: 12.4
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (tzl-eng)
|
|
config: tzl-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 33.65384615384615
|
|
- type: f1
|
|
value: 27.820512820512818
|
|
- type: precision
|
|
value: 26.09432234432234
|
|
- type: recall
|
|
value: 33.65384615384615
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (urd-eng)
|
|
config: urd-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 74.5
|
|
- type: f1
|
|
value: 70.09686507936507
|
|
- type: precision
|
|
value: 68.3117857142857
|
|
- type: recall
|
|
value: 74.5
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (ara-eng)
|
|
config: ara-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 88.3
|
|
- type: f1
|
|
value: 85.37333333333333
|
|
- type: precision
|
|
value: 84.05833333333334
|
|
- type: recall
|
|
value: 88.3
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (kor-eng)
|
|
config: kor-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 25.0
|
|
- type: f1
|
|
value: 22.393124632031995
|
|
- type: precision
|
|
value: 21.58347686592367
|
|
- type: recall
|
|
value: 25.0
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (yid-eng)
|
|
config: yid-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 0.589622641509434
|
|
- type: f1
|
|
value: 0.15804980033762941
|
|
- type: precision
|
|
value: 0.1393275384872965
|
|
- type: recall
|
|
value: 0.589622641509434
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (fin-eng)
|
|
config: fin-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 4.1000000000000005
|
|
- type: f1
|
|
value: 3.4069011332551775
|
|
- type: precision
|
|
value: 3.1784507042253516
|
|
- type: recall
|
|
value: 4.1000000000000005
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (tha-eng)
|
|
config: tha-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 3.102189781021898
|
|
- type: f1
|
|
value: 2.223851811694751
|
|
- type: precision
|
|
value: 2.103465682299194
|
|
- type: recall
|
|
value: 3.102189781021898
|
|
- task:
|
|
type: BitextMining
|
|
dataset:
|
|
type: mteb/tatoeba-bitext-mining
|
|
name: MTEB Tatoeba (wuu-eng)
|
|
config: wuu-eng
|
|
split: test
|
|
revision: ed9e4a974f867fd9736efcf222fc3a26487387a5
|
|
metrics:
|
|
- type: accuracy
|
|
value: 83.1
|
|
- type: f1
|
|
value: 79.58255835667599
|
|
- type: precision
|
|
value: 78.09708333333333
|
|
- type: recall
|
|
value: 83.1
|
|
- task:
|
|
type: Retrieval
|
|
dataset:
|
|
type: webis-touche2020
|
|
name: MTEB Touche2020
|
|
config: default
|
|
split: test
|
|
revision: 527b7d77e16e343303e68cb6af11d6e18b9f7b3b
|
|
metrics:
|
|
- type: map_at_1
|
|
value: 2.322
|
|
- type: map_at_10
|
|
value: 8.959999999999999
|
|
- type: map_at_100
|
|
value: 15.136
|
|
- type: map_at_1000
|
|
value: 16.694
|
|
- type: map_at_3
|
|
value: 4.837000000000001
|
|
- type: map_at_5
|
|
value: 6.196
|
|
- type: mrr_at_1
|
|
value: 28.571
|
|
- type: mrr_at_10
|
|
value: 47.589999999999996
|
|
- type: mrr_at_100
|
|
value: 48.166
|
|
- type: mrr_at_1000
|
|
value: 48.169000000000004
|
|
- type: mrr_at_3
|
|
value: 43.197
|
|
- type: mrr_at_5
|
|
value: 45.646
|
|
- type: ndcg_at_1
|
|
value: 26.531
|
|
- type: ndcg_at_10
|
|
value: 23.982
|
|
- type: ndcg_at_100
|
|
value: 35.519
|
|
- type: ndcg_at_1000
|
|
value: 46.878
|
|
- type: ndcg_at_3
|
|
value: 26.801000000000002
|
|
- type: ndcg_at_5
|
|
value: 24.879
|
|
- type: precision_at_1
|
|
value: 28.571
|
|
- type: precision_at_10
|
|
value: 22.041
|
|
- type: precision_at_100
|
|
value: 7.4079999999999995
|
|
- type: precision_at_1000
|
|
value: 1.492
|
|
- type: precision_at_3
|
|
value: 28.571
|
|
- type: precision_at_5
|
|
value: 25.306
|
|
- type: recall_at_1
|
|
value: 2.322
|
|
- type: recall_at_10
|
|
value: 15.443999999999999
|
|
- type: recall_at_100
|
|
value: 45.918
|
|
- type: recall_at_1000
|
|
value: 79.952
|
|
- type: recall_at_3
|
|
value: 6.143
|
|
- type: recall_at_5
|
|
value: 8.737
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/toxic_conversations_50k
|
|
name: MTEB ToxicConversationsClassification
|
|
config: default
|
|
split: test
|
|
revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de
|
|
metrics:
|
|
- type: accuracy
|
|
value: 66.5452
|
|
- type: ap
|
|
value: 12.99191723223892
|
|
- type: f1
|
|
value: 51.667665096195734
|
|
- task:
|
|
type: Classification
|
|
dataset:
|
|
type: mteb/tweet_sentiment_extraction
|
|
name: MTEB TweetSentimentExtractionClassification
|
|
config: default
|
|
split: test
|
|
revision: 62146448f05be9e52a36b8ee9936447ea787eede
|
|
metrics:
|
|
- type: accuracy
|
|
value: 55.854555744199196
|
|
- type: f1
|
|
value: 56.131766302254185
|
|
- task:
|
|
type: Clustering
|
|
dataset:
|
|
type: mteb/twentynewsgroups-clustering
|
|
name: MTEB TwentyNewsgroupsClustering
|
|
config: default
|
|
split: test
|
|
revision: 091a54f9a36281ce7d6590ec8c75dd485e7e01d4
|
|
metrics:
|
|
- type: v_measure
|
|
value: 37.27891385518074
|
|
- task:
|
|
type: PairClassification
|
|
dataset:
|
|
type: mteb/twittersemeval2015-pairclassification
|
|
name: MTEB TwitterSemEval2015
|
|
config: default
|
|
split: test
|
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
|
|
metrics:
|
|
- type: cos_sim_accuracy
|
|
value: 83.53102461703523
|
|
- type: cos_sim_ap
|
|
value: 65.30753664579191
|
|
- type: cos_sim_f1
|
|
value: 61.739943872778305
|
|
- type: cos_sim_precision
|
|
value: 55.438891222175556
|
|
- type: cos_sim_recall
|
|
value: 69.65699208443272
|
|
- type: dot_accuracy
|
|
value: 80.38981939560112
|
|
- type: dot_ap
|
|
value: 53.52081118421347
|
|
- type: dot_f1
|
|
value: 54.232957844617346
|
|
- type: dot_precision
|
|
value: 48.43393486828459
|
|
- type: dot_recall
|
|
value: 61.60949868073878
|
|
- type: euclidean_accuracy
|
|
value: 82.23758717291531
|
|
- type: euclidean_ap
|
|
value: 60.361102792772535
|
|
- type: euclidean_f1
|
|
value: 57.50518791791561
|
|
- type: euclidean_precision
|
|
value: 51.06470106470107
|
|
- type: euclidean_recall
|
|
value: 65.8047493403694
|
|
- type: manhattan_accuracy
|
|
value: 82.14221851344102
|
|
- type: manhattan_ap
|
|
value: 60.341937223793366
|
|
- type: manhattan_f1
|
|
value: 57.53803596127247
|
|
- type: manhattan_precision
|
|
value: 51.08473188702415
|
|
- type: manhattan_recall
|
|
value: 65.85751978891821
|
|
- type: max_accuracy
|
|
value: 83.53102461703523
|
|
- type: max_ap
|
|
value: 65.30753664579191
|
|
- type: max_f1
|
|
value: 61.739943872778305
|
|
- task:
|
|
type: PairClassification
|
|
dataset:
|
|
type: mteb/twitterurlcorpus-pairclassification
|
|
name: MTEB TwitterURLCorpus
|
|
config: default
|
|
split: test
|
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
|
|
metrics:
|
|
- type: cos_sim_accuracy
|
|
value: 88.75305623471883
|
|
- type: cos_sim_ap
|
|
value: 85.46387153880272
|
|
- type: cos_sim_f1
|
|
value: 77.91527673159008
|
|
- type: cos_sim_precision
|
|
value: 72.93667315828353
|
|
- type: cos_sim_recall
|
|
value: 83.62334462580844
|
|
- type: dot_accuracy
|
|
value: 85.08169363915086
|
|
- type: dot_ap
|
|
value: 74.96808060965559
|
|
- type: dot_f1
|
|
value: 71.39685033990366
|
|
- type: dot_precision
|
|
value: 64.16948111759288
|
|
- type: dot_recall
|
|
value: 80.45888512473051
|
|
- type: euclidean_accuracy
|
|
value: 85.84235650250321
|
|
- type: euclidean_ap
|
|
value: 78.42045145247211
|
|
- type: euclidean_f1
|
|
value: 70.32669630775179
|
|
- type: euclidean_precision
|
|
value: 70.6298050788227
|
|
- type: euclidean_recall
|
|
value: 70.02617801047121
|
|
- type: manhattan_accuracy
|
|
value: 85.86176116738464
|
|
- type: manhattan_ap
|
|
value: 78.54012451558276
|
|
- type: manhattan_f1
|
|
value: 70.56508080693389
|
|
- type: manhattan_precision
|
|
value: 69.39626293456413
|
|
- type: manhattan_recall
|
|
value: 71.77394518016631
|
|
- type: max_accuracy
|
|
value: 88.75305623471883
|
|
- type: max_ap
|
|
value: 85.46387153880272
|
|
- type: max_f1
|
|
value: 77.91527673159008
|
|
---
|
|
|
|
## Usage
|
|
|
|
For usage instructions, refer to: https://github.com/Muennighoff/sgpt#asymmetric-semantic-search-be
|
|
|
|
The model was trained with the command
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch examples/training/ms_marco/train_bi-encoder_mnrl.py --model_name bigscience/bloom-7b1 --train_batch_size 32 --eval_batch_size 16 --freezenonbias --specb --lr 4e-4 --wandb --wandbwatchlog gradients --pooling weightedmean --gradcache --chunksize 8
|
|
```
|
|
|
|
## Evaluation Results
|
|
|
|
|
|
`{"ndcgs": {"sgpt-bloom-7b1-msmarco": {"scifact": {"NDCG@10": 0.71824}, "nfcorpus": {"NDCG@10": 0.35748}, "arguana": {"NDCG@10": 0.47281}, "scidocs": {"NDCG@10": 0.18435}, "fiqa": {"NDCG@10": 0.35736}, "cqadupstack": {"NDCG@10": 0.3708525}, "quora": {"NDCG@10": 0.74655}, "trec-covid": {"NDCG@10": 0.82731}, "webis-touche2020": {"NDCG@10": 0.2365}}}`
|
|
|
|
See the evaluation folder or [MTEB](https://huggingface.co/spaces/mteb/leaderboard) for more results.
|
|
|
|
## Training
|
|
The model was trained with the parameters:
|
|
|
|
**DataLoader**:
|
|
|
|
`torch.utils.data.dataloader.DataLoader` of length 15600 with parameters:
|
|
```
|
|
{'batch_size': 32, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
|
|
```
|
|
|
|
The model uses BitFit, weighted-mean pooling & GradCache, for details see: https://arxiv.org/abs/2202.08904
|
|
|
|
**Loss**:
|
|
|
|
`sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache`
|
|
|
|
Parameters of the fit()-Method:
|
|
```
|
|
{
|
|
"epochs": 10,
|
|
"evaluation_steps": 0,
|
|
"evaluator": "NoneType",
|
|
"max_grad_norm": 1,
|
|
"optimizer_class": "<class 'transformers.optimization.AdamW'>",
|
|
"optimizer_params": {
|
|
"lr": 0.0004
|
|
},
|
|
"scheduler": "WarmupLinear",
|
|
"steps_per_epoch": null,
|
|
"warmup_steps": 1000,
|
|
"weight_decay": 0.01
|
|
}
|
|
```
|
|
|
|
|
|
## Full Model Architecture
|
|
```
|
|
SentenceTransformer(
|
|
(0): Transformer({'max_seq_length': 300, 'do_lower_case': False}) with Transformer model: BloomModel
|
|
(1): Pooling({'word_embedding_dimension': 4096, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
|
|
)
|
|
```
|
|
|
|
## Citing & Authors
|
|
|
|
```bibtex
|
|
@article{muennighoff2022sgpt,
|
|
title={SGPT: GPT Sentence Embeddings for Semantic Search},
|
|
author={Muennighoff, Niklas},
|
|
journal={arXiv preprint arXiv:2202.08904},
|
|
year={2022}
|
|
}
|
|
``` |