sbert-chinese-general-v1/README.md

pipeline_tag, tags, model-index, license, language

pipeline_tag	tags	model-index	license	language
sentence-similarity	sentence-transformers feature-extraction sentence-similarity transformers semantic-search chinese mteb	name results sbert-chinese-general-v1 task dataset metrics type STS type name config split revision C-MTEB/AFQMC MTEB AFQMC default validation None type value cos_sim_pearson 22.293919432958074 type value cos_sim_spearman 22.56718923553609 type value euclidean_pearson 22.525656322797026 type value euclidean_spearman 22.56718923553609 type value manhattan_pearson 22.501773028824065 type value manhattan_spearman 22.536992587828397 task dataset metrics type STS type name config split revision C-MTEB/ATEC MTEB ATEC default test None type value cos_sim_pearson 30.33575274463879 type value cos_sim_spearman 30.298708742167772 type value euclidean_pearson 32.33094743729218 type value euclidean_spearman 30.298710993858734 type value manhattan_pearson 32.31155376195945 type value manhattan_spearman 30.267669681690744 task dataset metrics type Classification type name config split revision mteb/amazon_reviews_multi MTEB AmazonReviewsClassification (zh) zh test 1399c76144fd37290681b995c656ef9b2e06e26d type value accuracy 37.507999999999996 type value f1 36.436808400753286 task dataset metrics type STS type name config split revision C-MTEB/BQ MTEB BQ default test None type value cos_sim_pearson 41.493256724214255 type value cos_sim_spearman 40.98395961967895 type value euclidean_pearson 41.12345737966565 type value euclidean_spearman 40.983959619555996 type value manhattan_pearson 41.02584539471014 type value manhattan_spearman 40.87549513383032 task dataset metrics type BitextMining type name config split revision mteb/bucc-bitext-mining MTEB BUCC (zh-en) zh-en test d51519689f32196a32af33b075a01d0e7c51e252 type value accuracy 9.794628751974724 type value f1 9.350535369492716 type value precision 9.179392662804986 type value recall 9.794628751974724 task dataset metrics type Clustering type name config split revision C-MTEB/CLSClusteringP2P MTEB CLSClusteringP2P default test None type value v_measure 34.984726547788284 task dataset metrics type Clustering type name config split revision C-MTEB/CLSClusteringS2S MTEB CLSClusteringS2S default test None type value v_measure 27.81945732281589 task dataset metrics type Reranking type name config split revision C-MTEB/CMedQAv1-reranking MTEB CMedQAv1 default test None type value map 53.06586280826805 type value mrr 59.58781746031746 task dataset metrics type Reranking type name config split revision C-MTEB/CMedQAv2-reranking MTEB CMedQAv2 default test None type value map 52.83635946154306 type value mrr 59.315079365079356 task dataset metrics type Retrieval type name config split revision C-MTEB/CmedqaRetrieval MTEB CmedqaRetrieval default dev None type value map_at_1 5.721 type value map_at_10 8.645 type value map_at_100 9.434 type value map_at_1000 9.586 type value map_at_3 7.413 type value map_at_5 8.05 type value mrr_at_1 9.626999999999999 type value mrr_at_10 13.094 type value mrr_at_100 13.854 type value mrr_at_1000 13.958 type value mrr_at_3 11.724 type value mrr_at_5 12.409 type value ndcg_at_1 9.626999999999999 type value ndcg_at_10 11.35 type value ndcg_at_100 15.593000000000002 type value ndcg_at_1000 19.619 type value ndcg_at_3 9.317 type value ndcg_at_5 10.049 type value precision_at_1 9.626999999999999 type value precision_at_10 2.796 type value precision_at_100 0.629 type value precision_at_1000 0.11800000000000001 type value precision_at_3 5.476 type value precision_at_5 4.1209999999999996 type value recall_at_1 5.721 type value recall_at_10 15.190000000000001 type value recall_at_100 33.633 type value recall_at_1000 62.019999999999996 type value recall_at_3 9.099 type value recall_at_5 11.423 task dataset metrics type PairClassification type name config split revision C-MTEB/CMNLI MTEB Cmnli default validation None type value cos_sim_accuracy 77.36620565243535 type value cos_sim_ap 85.92291866877001 type value cos_sim_f1 78.19390231037029 type value cos_sim_precision 71.24183006535948 type value cos_sim_recall 86.64952069207388 type value dot_accuracy 77.36620565243535 type value dot_ap 85.94113738490068 type value dot_f1 78.19390231037029 type value dot_precision 71.24183006535948 type value dot_recall 86.64952069207388 type value euclidean_accuracy 77.36620565243535 type value euclidean_ap 85.92291893444687 type value euclidean_f1 78.19390231037029 type value euclidean_precision 71.24183006535948 type value euclidean_recall 86.64952069207388 type value manhattan_accuracy 77.29404690318701 type value manhattan_ap 85.88284362100919 type value manhattan_f1 78.17836812144213 type value manhattan_precision 71.18448838548666 type value manhattan_recall 86.69628244096329 type value max_accuracy 77.36620565243535 type value max_ap 85.94113738490068 type value max_f1 78.19390231037029 task dataset metrics type Retrieval type name config split revision C-MTEB/CovidRetrieval MTEB CovidRetrieval default dev None type value map_at_1 26.976 type value map_at_10 35.18 type value map_at_100 35.921 type value map_at_1000 35.998999999999995 type value map_at_3 32.763 type value map_at_5 34.165 type value mrr_at_1 26.976 type value mrr_at_10 35.234 type value mrr_at_100 35.939 type value mrr_at_1000 36.016 type value mrr_at_3 32.771 type value mrr_at_5 34.172999999999995 type value ndcg_at_1 26.976 type value ndcg_at_10 39.635 type value ndcg_at_100 43.54 type value ndcg_at_1000 45.723 type value ndcg_at_3 34.652 type value ndcg_at_5 37.186 type value precision_at_1 26.976 type value precision_at_10 5.406 type value precision_at_100 0.736 type value precision_at_1000 0.091 type value precision_at_3 13.418 type value precision_at_5 9.293999999999999 type value recall_at_1 26.976 type value recall_at_10 53.766999999999996 type value recall_at_100 72.761 type value recall_at_1000 90.148 type value recall_at_3 40.095 type value recall_at_5 46.233000000000004 task dataset metrics type Retrieval type name config split revision C-MTEB/DuRetrieval MTEB DuRetrieval default dev None type value map_at_1 11.285 type value map_at_10 30.259000000000004 type value map_at_100 33.772000000000006 type value map_at_1000 34.037 type value map_at_3 21.038999999999998 type value map_at_5 25.939 type value mrr_at_1 45.1 type value mrr_at_10 55.803999999999995 type value mrr_at_100 56.301 type value mrr_at_1000 56.330999999999996 type value mrr_at_3 53.333 type value mrr_at_5 54.798 type value ndcg_at_1 45.1 type value ndcg_at_10 41.156 type value ndcg_at_100 49.518 type value ndcg_at_1000 52.947 type value ndcg_at_3 39.708 type value ndcg_at_5 38.704 type value precision_at_1 45.1 type value precision_at_10 20.75 type value precision_at_100 3.424 type value precision_at_1000 0.42700000000000005 type value precision_at_3 35.632999999999996 type value precision_at_5 30.080000000000002 type value recall_at_1 11.285 type value recall_at_10 43.242000000000004 type value recall_at_100 68.604 type value recall_at_1000 85.904 type value recall_at_3 24.404 type value recall_at_5 32.757 task dataset metrics type Retrieval type name config split revision C-MTEB/EcomRetrieval MTEB EcomRetrieval default dev None type value map_at_1 21 type value map_at_10 28.364 type value map_at_100 29.199 type value map_at_1000 29.265 type value map_at_3 25.717000000000002 type value map_at_5 27.311999999999998 type value mrr_at_1 21 type value mrr_at_10 28.364 type value mrr_at_100 29.199 type value mrr_at_1000 29.265 type value mrr_at_3 25.717000000000002 type value mrr_at_5 27.311999999999998 type value ndcg_at_1 21 type value ndcg_at_10 32.708 type value ndcg_at_100 37.184 type value ndcg_at_1000 39.273 type value ndcg_at_3 27.372000000000003 type value ndcg_at_5 30.23 type value precision_at_1 21 type value precision_at_10 4.66 type value precision_at_100 0.685 type value precision_at_1000 0.086 type value precision_at_3 10.732999999999999 type value precision_at_5 7.82 type value recall_at_1 21 type value recall_at_10 46.6 type value recall_at_100 68.5 type value recall_at_1000 85.6 type value recall_at_3 32.2 type value recall_at_5 39.1 task dataset metrics type Classification type name config split revision C-MTEB/IFlyTek-classification MTEB IFlyTek default validation None type value accuracy 44.878799538283964 type value f1 33.84678310261366 task dataset metrics type Classification type name config split revision C-MTEB/JDReview-classification MTEB JDReview default test None type value accuracy 82.1951219512195 type value ap 46.78292030042397 type value f1 76.20482468514128 task dataset metrics type STS type name config split revision C-MTEB/LCQMC MTEB LCQMC default test None type value cos_sim_pearson 62.84331627244547 type value cos_sim_spearman 68.39990265073726 type value euclidean_pearson 66.87431827169324 type value euclidean_spearman 68.39990264979167 type value manhattan_pearson 66.89702078900328 type value manhattan_spearman 68.42107302159141 task dataset metrics type Reranking type name config split revision C-MTEB/Mmarco-reranking MTEB MMarcoReranking default dev None type value map 9.28600891904827 type value mrr 8.057936507936509 task dataset metrics type Retrieval type name config split revision C-MTEB/MMarcoRetrieval MTEB MMarcoRetrieval default dev None type value map_at_1 22.820999999999998 type value map_at_10 30.44 type value map_at_100 31.35 type value map_at_1000 31.419000000000004 type value map_at_3 28.134999999999998 type value map_at_5 29.482000000000003 type value mrr_at_1 23.782 type value mrr_at_10 31.141999999999996 type value mrr_at_100 32.004 type value mrr_at_1000 32.068000000000005 type value mrr_at_3 28.904000000000003 type value mrr_at_5 30.214999999999996 type value ndcg_at_1 23.782 type value ndcg_at_10 34.625 type value ndcg_at_100 39.226 type value ndcg_at_1000 41.128 type value ndcg_at_3 29.968 type value ndcg_at_5 32.35 type value precision_at_1 23.782 type value precision_at_10 4.994 type value precision_at_100 0.736 type value precision_at_1000 0.09 type value precision_at_3 12.13 type value precision_at_5 8.495999999999999 type value recall_at_1 22.820999999999998 type value recall_at_10 47.141 type value recall_at_100 68.952 type value recall_at_1000 83.985 type value recall_at_3 34.508 type value recall_at_5 40.232 task dataset metrics type Classification type name config split revision mteb/amazon_massive_intent MTEB MassiveIntentClassification (zh-CN) zh-CN test 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 type value accuracy 57.343644922663074 type value f1 56.744802953803486 task dataset metrics type Classification type name config split revision mteb/amazon_massive_scenario MTEB MassiveScenarioClassification (zh-CN) zh-CN test 7d571f92784cd94a019292a1f45445077d0ef634 type value accuracy 62.363819771351714 type value f1 62.15920863434656 task dataset metrics type Retrieval type name config split revision C-MTEB/MedicalRetrieval MTEB MedicalRetrieval default dev None type value map_at_1 14.6 type value map_at_10 18.231 type value map_at_100 18.744 type value map_at_1000 18.811 type value map_at_3 17.133000000000003 type value map_at_5 17.663 type value mrr_at_1 14.6 type value mrr_at_10 18.231 type value mrr_at_100 18.744 type value mrr_at_1000 18.811 type value mrr_at_3 17.133000000000003 type value mrr_at_5 17.663 type value ndcg_at_1 14.6 type value ndcg_at_10 20.349 type value ndcg_at_100 23.204 type value ndcg_at_1000 25.44 type value ndcg_at_3 17.995 type value ndcg_at_5 18.945999999999998 type value precision_at_1 14.6 type value precision_at_10 2.7199999999999998 type value precision_at_100 0.414 type value precision_at_1000 0.06 type value precision_at_3 6.833 type value precision_at_5 4.5600000000000005 type value recall_at_1 14.6 type value recall_at_10 27.200000000000003 type value recall_at_100 41.4 type value recall_at_1000 60 type value recall_at_3 20.5 type value recall_at_5 22.8 task dataset metrics type Classification type name config split revision C-MTEB/MultilingualSentiment-classification MTEB MultilingualSentiment default validation None type value accuracy 66.58333333333333 type value f1 66.26700927460007 task dataset metrics type PairClassification type name config split revision C-MTEB/OCNLI MTEB Ocnli default validation None type value cos_sim_accuracy 72.00866269626421 type value cos_sim_ap 77.00520104243304 type value cos_sim_f1 74.39303710490151 type value cos_sim_precision 65.69579288025889 type value cos_sim_recall 85.74445617740233 type value dot_accuracy 72.00866269626421 type value dot_ap 77.00520104243304 type value dot_f1 74.39303710490151 type value dot_precision 65.69579288025889 type value dot_recall 85.74445617740233 type value euclidean_accuracy 72.00866269626421 type value euclidean_ap 77.00520104243304 type value euclidean_f1 74.39303710490151 type value euclidean_precision 65.69579288025889 type value euclidean_recall 85.74445617740233 type value manhattan_accuracy 72.1710882512182 type value manhattan_ap 77.00551017913976 type value manhattan_f1 74.23423423423424 type value manhattan_precision 64.72898664571878 type value manhattan_recall 87.0116156282999 type value max_accuracy 72.1710882512182 type value max_ap 77.00551017913976 type value max_f1 74.39303710490151 task dataset metrics type Classification type name config split revision C-MTEB/OnlineShopping-classification MTEB OnlineShopping default test None type value accuracy 88.19000000000001 type value ap 85.13415594781077 type value f1 88.17344156114062 task dataset metrics type STS type name config split revision C-MTEB/PAWSX MTEB PAWSX default test None type value cos_sim_pearson 13.70522140998517 type value cos_sim_spearman 15.07546667334743 type value euclidean_pearson 17.49511420225285 type value euclidean_spearman 15.093970931789618 type value manhattan_pearson 17.44069961390521 type value manhattan_spearman 15.076029291596962 task dataset metrics type STS type name config split revision C-MTEB/QBQTC MTEB QBQTC default test None type value cos_sim_pearson 26.835294224547155 type value cos_sim_spearman 27.920204597498856 type value euclidean_pearson 26.153796707702803 type value euclidean_spearman 27.920971379720548 type value manhattan_pearson 26.21954147857523 type value manhattan_spearman 27.996860049937478 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh) zh test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 55.15901259718581 type value cos_sim_spearman 61.57967880874167 type value euclidean_pearson 53.83523291596683 type value euclidean_spearman 61.57967880874167 type value manhattan_pearson 54.99971428907956 type value manhattan_spearman 61.61229543613867 task dataset metrics type STS type name config split revision mteb/sts22-crosslingual-sts MTEB STS22 (zh-en) zh-en test 6d1ba47164174a496b7fa5d3569dae26a6813b80 type value cos_sim_pearson 34.20930208460845 type value cos_sim_spearman 33.879011104224524 type value euclidean_pearson 35.08526425284862 type value euclidean_spearman 33.879011104224524 type value manhattan_pearson 35.509419089701275 type value manhattan_spearman 33.30035487147621 task dataset metrics type STS type name config split revision C-MTEB/STSB MTEB STSB default test None type value cos_sim_pearson 82.30068282185835 type value cos_sim_spearman 82.16763221361724 type value euclidean_pearson 80.52772752433374 type value euclidean_spearman 82.16797037220333 type value manhattan_pearson 80.51093859500105 type value manhattan_spearman 82.17643310049654 task dataset metrics type Reranking type name config split revision C-MTEB/T2Reranking MTEB T2Reranking default dev None type value map 65.14113035189213 type value mrr 74.9589270937443 task dataset metrics type Retrieval type name config split revision C-MTEB/T2Retrieval MTEB T2Retrieval default dev None type value map_at_1 12.013 type value map_at_10 30.885 type value map_at_100 34.643 type value map_at_1000 34.927 type value map_at_3 21.901 type value map_at_5 26.467000000000002 type value mrr_at_1 49.623 type value mrr_at_10 58.05200000000001 type value mrr_at_100 58.61300000000001 type value mrr_at_1000 58.643 type value mrr_at_3 55.947 type value mrr_at_5 57.229 type value ndcg_at_1 49.623 type value ndcg_at_10 41.802 type value ndcg_at_100 49.975 type value ndcg_at_1000 53.504 type value ndcg_at_3 43.515 type value ndcg_at_5 41.576 type value precision_at_1 49.623 type value precision_at_10 22.052 type value precision_at_100 3.6450000000000005 type value precision_at_1000 0.45399999999999996 type value precision_at_3 38.616 type value precision_at_5 31.966 type value recall_at_1 12.013 type value recall_at_10 41.891 type value recall_at_100 67.096 type value recall_at_1000 84.756 type value recall_at_3 24.695 type value recall_at_5 32.09 task dataset metrics type Classification type name config split revision C-MTEB/TNews-classification MTEB TNews default validation None type value accuracy 39.800999999999995 type value f1 38.5345899934575 task dataset metrics type Clustering type name config split revision C-MTEB/ThuNewsClusteringP2P MTEB ThuNewsClusteringP2P default test None type value v_measure 40.16574242797479 task dataset metrics type Clustering type name config split revision C-MTEB/ThuNewsClusteringS2S MTEB ThuNewsClusteringS2S default test None type value v_measure 24.232617974671754 task dataset metrics type Retrieval type name config split revision C-MTEB/VideoRetrieval MTEB VideoRetrieval default dev None type value map_at_1 24.6 type value map_at_10 31.328 type value map_at_100 32.088 type value map_at_1000 32.164 type value map_at_3 29.133 type value map_at_5 30.358 type value mrr_at_1 24.6 type value mrr_at_10 31.328 type value mrr_at_100 32.088 type value mrr_at_1000 32.164 type value mrr_at_3 29.133 type value mrr_at_5 30.358 type value ndcg_at_1 24.6 type value ndcg_at_10 35.150999999999996 type value ndcg_at_100 39.024 type value ndcg_at_1000 41.157 type value ndcg_at_3 30.637999999999998 type value ndcg_at_5 32.833 type value precision_at_1 24.6 type value precision_at_10 4.74 type value precision_at_100 0.66 type value precision_at_1000 0.083 type value precision_at_3 11.667 type value precision_at_5 8.06 type value recall_at_1 24.6 type value recall_at_10 47.4 type value recall_at_100 66 type value recall_at_1000 83 type value recall_at_3 35 type value recall_at_5 40.300000000000004 task dataset metrics type Classification type name config split revision C-MTEB/waimai-classification MTEB Waimai default test None type value accuracy 83.96000000000001 type value ap 65.11027167433211 type value f1 82.03549710974653	apache-2.0	zh

DMetaSoul/sbert-chinese-general-v1

此模型基于 bert-base-chinese 版本 BERT 模型，在 NLI、PAWS-X、PKU-Paraphrase-Bank、STS 等语义相似数据集上进行训练，适用于通用语义匹配场景（此模型在 Chinese-STS 任务上效果较好，但在其它任务上效果并非最优，存在一定过拟合风险），比如文本特征抽取、文本向量聚类、文本语义搜索等业务场景。

注：此模型的轻量化版本，也已经开源啦！

Usage

1. Sentence-Transformers

通过 sentence-transformers 框架来使用该模型，首先进行安装：

pip install -U sentence-transformers

然后使用下面的代码来载入该模型并进行文本表征向量的提取：

from sentence_transformers import SentenceTransformer
sentences = ["我的儿子！他猛然间喊道，我的儿子在哪儿？", "我的儿子呢！他突然喊道，我的儿子在哪里？"]

model = SentenceTransformer('DMetaSoul/sbert-chinese-general-v1')
embeddings = model.encode(sentences)
print(embeddings)

2. HuggingFace Transformers

如果不想使用 sentence-transformers 的话，也可以通过 HuggingFace Transformers 来载入该模型并进行文本向量抽取：

from transformers import AutoTokenizer, AutoModel
import torch


#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


# Sentences we want sentence embeddings for
sentences = ["我的儿子！他猛然间喊道，我的儿子在哪儿？", "我的儿子呢！他突然喊道，我的儿子在哪里？"]

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('DMetaSoul/sbert-chinese-general-v1')
model = AutoModel.from_pretrained('DMetaSoul/sbert-chinese-general-v1')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, mean pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)

Evaluation

该模型在公开的几个语义匹配数据集上进行了评测，计算了向量相似度跟真实标签之间的相关性系数：

	csts_dev	csts_test	afqmc	lcqmc	bqcorpus	pawsx	xiaobu
spearman	84.54%	82.17%	23.80%	65.94%	45.52%	11.52%	48.51%

Citing & Authors

E-mail: xiaowenbin@dmetasoul.com