66 lines
2.5 KiB
Markdown
66 lines
2.5 KiB
Markdown
|
---
|
||
|
|
license: mit
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
base_model:
|
||
|
|
- mistralai/Mistral-7B-Instruct-v0.3
|
||
|
|
---
|
||
|
|
# Model Card for `first_mistral`
|
||
|
|
`first_mistral` is a language model trained to act as a listwise reranker, decoding from the first-token logits only to improve efficiency while maintaining effectiveness.
|
||
|
|
`first_mistral` is built on Mistral-7B-Instruct-v0.3, following [FIRST](https://arxiv.org/abs/2406.15657)'s strategy, trained using 40K GPT-4 labeled rerank instances from RankZephyr.
|
||
|
|
More details can be found in the paper.
|
||
|
|
|
||
|
|
## Model description
|
||
|
|
|
||
|
|
- **Model type:** A 7B parameter listwise reranker fine-tuned from Mistral-7B-Instruct-v0.3
|
||
|
|
- **Language(s) (NLP):** Primarily English
|
||
|
|
- **License:** MIT
|
||
|
|
- **Fine-tuned from model:** [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)
|
||
|
|
|
||
|
|
### Model Sources
|
||
|
|
|
||
|
|
- **Repository:** https://github.com/castorini/rank_llm
|
||
|
|
- **Paper:** https://arxiv.org/abs/2411.05508
|
||
|
|
|
||
|
|
## Evaluation
|
||
|
|
|
||
|
|
At the time of release, `first_mistral` outperforms the original FIRST implementation on most subsets of the BEIR benchmark.
|
||
|
|
More details that compare other LLM rerankers on more datasets can be found in the paper.
|
||
|
|
|
||
|
|
| Dataset | FIRST (original) | first_mistral |
|
||
|
|
|----------------|-------------|--------------|
|
||
|
|
| climate-fever | **0.2672** | 0.2417 |
|
||
|
|
| dbpedia-entity | **0.5092** | 0.5033 |
|
||
|
|
| fever | 0.8164 | **0.8413** |
|
||
|
|
| fiqa | 0.4223 | **0.4778** |
|
||
|
|
| hotpotqa | 0.7424 | **0.7705** |
|
||
|
|
| msmarco | 0.4425 | **0.4512** |
|
||
|
|
| nfcorpus | 0.3725 | **0.3816** |
|
||
|
|
| nq | 0.6638 | **0.6985** |
|
||
|
|
| scidocs | 0.2047 | **0.2110** |
|
||
|
|
| scifact | 0.7459 | **0.7769** |
|
||
|
|
| trec-covid | **0.7913** | 0.7666 |
|
||
|
|
| Average | 0.5435 | **0.5564** |
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
If you find `first_mistral` useful for your work, please consider citing:
|
||
|
|
|
||
|
|
```
|
||
|
|
@ARTICLE{chen2024firstrepro,
|
||
|
|
title = title={An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking},
|
||
|
|
author = {Zijian Chen and Ronak Pradeep and Jimmy Lin},
|
||
|
|
year = {2024},
|
||
|
|
journal = {arXiv:2411.05508}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
If you would like to cite the FIRST methodology, please consider citing:
|
||
|
|
|
||
|
|
```
|
||
|
|
@ARTICLE{reddy2024first,
|
||
|
|
title = {FIRST: Faster Improved Listwise Reranking with Single Token Decoding},
|
||
|
|
author = {Reddy, Revanth Gangi and Doo, JaeHyeok and Xu, Yifei and Sultan, Md Arafat and Swain, Deevya and Sil, Avirup and Ji, Heng},
|
||
|
|
year = {2024}
|
||
|
|
journal = {arXiv:2406.15657},
|
||
|
|
}
|
||
|
|
```
|