tags, pipeline_tag, model-index

tags	pipeline_tag	model-index
mteb sentence-transformers transformers	text-ranking	name results ListConRanker task dataset metrics type Reranking type name config split revision C-MTEB/CMedQAv1-reranking MTEB CMedQAv1 default test None type value map 90.55366308098787 type value mrr_1 87.8 type value mrr_10 92.45134920634919 type value mrr_5 92.325 task dataset metrics type Reranking type name config split revision C-MTEB/CMedQAv2-reranking MTEB CMedQAv2 default test None type value map 89.38076135722042 type value mrr_1 85.9 type value mrr_10 91.28769841269842 type value mrr_5 91.08999999999999 task dataset metrics type Reranking type name config split revision C-MTEB/Mmarco-reranking MTEB MMarcoReranking default dev None type value map 43.881461866703894 type value mrr_1 32.0 type value mrr_10 44.534126984126985 type value mrr_5 43.45 task dataset metrics type Reranking type name config split revision C-MTEB/T2Reranking MTEB T2Reranking default dev None type value map 69.16513825032682 type value mrr_1 67.36628300609343 type value mrr_10 80.06914890758831 type value mrr_5 79.69137892123675

ListConRanker

Model

• We propose a Listwise-encoded Contrastive text reRanker (ListConRanker), includes a ListTransformer module for listwise encoding. The ListTransformer can facilitate global contrastive information learning between passage features, including the clustering of similar passages, the clustering between dissimilar passages, and the distinction between similar and dissimilar passages. Besides, we propose ListAttention to help ListTransformer maintain the features of the query while learning global comparative information.
• The training loss function is Circle Loss[1]. Compared with cross-entropy loss and ranking loss, it can solve the problems of low data efficiency and unsmooth gradient change.

Data

The training data consists of approximately 2.6 million queries, each corresponding to multiple passages. The data comes from the training sets of several datasets, including cMedQA1.0, cMedQA2.0, MMarcoReranking, T2Reranking, huatuo, MARC, XL-sum, CSL and so on.

Training

We trained the model in two stages. In the first stage, we freeze the parameters of embedding model and only train the ListTransformer for 4 epochs with a batch size of 1024. In the second stage, we do not freeze any parameter and train for another 2 epochs with a batch size of 256.

Inference

Due to the limited memory of GPUs, we input about 20 passages at a time for each query during training. However, during actual use, there may be situations where far more than 20 passages are input at the same time (e.g, MMarcoReranking).

To reduce the discrepancy between training and inference, we propose iterative inference. The iterative inference feeds the passages into the ListConRanker multiple times, and each time it only decides the ranking of the passage at the end of the list.

Performance

Model	cMedQA1.0	cMedQA2.0	MMarcoReranking	T2Reranking	Avg.
LdIR-Qwen2-reranker-1.5B	86.50	87.11	39.35	68.84	70.45
zpoint-large-embedding-zh	91.11	90.07	38.87	69.29	72.34
xiaobu-embedding-v2	90.96	90.41	39.91	69.03	72.58
Conan-embedding-v1	91.39	89.72	41.58	68.36	72.76
ListConRanker	90.55	89.38	43.88	69.17	73.25
- w/o Iterative Inference	90.20	89.98	37.52	69.17	71.72

How to use

from transfoermers import AutoModelForSequenceClassification, AutoTokenizer

reranker = AutoModelForSequenceClassification.from_pretrained('ByteDance/ListConRanker', trust_remote_code=True)

# [query, passages_1, passage_2, ..., passage_n]
batch = [
    [
        '皮蛋是寒性的食物吗', # query
        '营养医师介绍皮蛋是属于凉性的食物,中医认为皮蛋可治眼疼、牙疼、高血压、耳鸣眩晕等疾病。体虚者要少吃。',  # passage_1
        '皮蛋这种食品是在中国地域才常见的传统食品,它的生长汗青也是非常的悠长。',  # passage_2
        '喜欢皮蛋的人会觉得皮蛋是最美味的食物,不喜欢皮蛋的人则觉得皮蛋是黑暗料理,尤其很多外国朋友都不理解我们吃皮蛋的习惯' # passage_3
    ],
    [
        '月有阴晴圆缺的意义', # query
        '形容的是月所有的状态,晴朗明媚,阴沉混沌,有月圆时,但多数时总是有缺陷。',  # passage_1
        '人有悲欢离合,月有阴晴圆缺这句话意思是人有悲欢离合的变迁,月有阴晴圆缺的转换。',  # passage_2
        '既然是诗歌,又哪里会有真正含义呢? 大概可以说:人生有太多坎坷,苦难,从容坦荡面对就好。', # passage_3
        '一零七六年苏轼贬官密州,时年四十一岁的他政治上很不得志,时值中秋佳节,非常想念自己的弟弟子由内心颇感忧郁,情绪低沉,有感而发写了这首词。' # passage_4
    ]
]

# for conventional inference, please manage the batch size by yourself
scores = reranker.multi_passage(batch)
print(scores)
# [0.5126814246177673, 0.33125635981559753, 0.3642643094062805, 0.6367220282554626, 0.7166246175765991, 0.4281482696533203, 0.3530198335647583]

# for iterative inferfence, only a batch size of 1 is supported
# the scores do not carry similarity meaning and are only used for ranking
scores = reranker.multi_passage_in_iterative_inference(batch[0])
print(scores)
# [0.5126813650131226, 0.3312564790248871, 0.3642643094062805]

tokenizer = AutoTokenizer.from_pretrained('ByteDance/ListConRanker')
inputs = tokenizer(
    [
        [
            "皮蛋是寒性的食物吗",
            "营养医师介绍皮蛋是属于凉性的食物,中医认为皮蛋可治眼疼、牙疼、高血压、耳鸣眩晕等疾病。体虚者要少吃。",
        ],
        [
            "皮蛋是寒性的食物吗",
            "皮蛋这种食品是在中国地域才常见的传统食品,它的生长汗青也是非常的悠长。",
        ],
        [
            "月有阴晴圆缺的意义",
            "形容的是月所有的状态,晴朗明媚,阴沉混沌,有月圆时,但多数时总是有缺陷。",
        ],
    ],
    return_tensors="pt",
    padding=True,
    truncation=False
)
# tensor([[0.5070], [0.3334], [0.6294]], device='cuda:0', dtype=torch.float16, grad_fn=<ViewBackward0>)

or using the sentence_transformers library (We do not recommend using sentence_transformers. Because its truncation strategy may not match the model design, which may lead to performance degradation.):

from sentence_transformers import CrossEncoder

model = CrossEncoder('ByteDance/ListConRanker', trust_remote_code=True)

inputs = [
  [
    "皮蛋是寒性的食物吗",
    "营养医师介绍皮蛋是属于凉性的食物,中医认为皮蛋可治眼疼、牙疼、高血压、耳鸣眩晕等疾病。体虚者要少吃。",
  ],
  [
    "皮蛋是寒性的食物吗",
    "皮蛋这种食品是在中国地域才常见的传统食品,它的生长汗青也是非常的悠长。",
  ],
  [
    "月有阴晴圆缺的意义",
    "形容的是月所有的状态,晴朗明媚,阴沉混沌,有月圆时,但多数时总是有缺陷。",
  ],
]
scores = model.predict(inputs)
print(scores)

To reproduce the results with iterative inference, please run:

python3 eval_listconranker_iterative_inference.py

To reproduce the results without iterative inference, please run:

python3 eval_listconranker.py

Reference

1. https://arxiv.org/abs/2002.10857
2. https://github.com/FlagOpen/FlagEmbedding
3. https://arxiv.org/abs/2408.15710

License

This work is licensed under a MIT License and the weight of models is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.