133 lines
5.1 KiB
Markdown
133 lines
5.1 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: Qwen/Qwen3-8B
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- table-ranking
|
|
- listwise-reranking
|
|
- table-retrieval
|
|
- table-question-answering
|
|
- qwen3
|
|
language:
|
|
- en
|
|
---
|
|
# TabRankMultiTableCoTGen
|
|
|
|
|
|
|
|
A single call generative listwise reranker for table retrieval built on Qwen3 8B.
|
|
Given a question and a set of candidate tables the model returns the tables ordered
|
|
from most to least useful for answering the question. It reads all candidates in one
|
|
prompt and emits the full ranking in a single generation.
|
|
|
|
This checkpoint is the **Single plus Multi Table Reasoning Generation** variant.
|
|
|
|
## What this variant does
|
|
|
|
The model is trained to first write a chain of thought inside a think block and then output the ranking. It generates its own reasoning at inference time. Use this variant when you want an explanation alongside the ranking.
|
|
|
|
## Training data
|
|
|
|
This model is trained on a mix of **NQ Tables** and **MultiTabQA** which adds multi table retrieval questions. The sibling checkpoint [TabRankSingleTableCoTGen](https://huggingface.co/AdarshSingh7647/TabRankSingleTableCoTGen)
|
|
uses the other training mix with the same objective.
|
|
|
|
## Input and output format
|
|
|
|
The user message lists candidate tables as blocks headed by `### Table 1` `### Table 2`
|
|
and so on. The model returns a JSON object whose value is the ranked list of one based
|
|
candidate positions best first:
|
|
|
|
```json
|
|
{"ranked_tables": [3, 1, 5, 2, 4]}
|
|
```
|
|
|
|
Map those positions back to your table ids to obtain the reranked list.
|
|
|
|
## Evaluation
|
|
|
|
Scored as a listwise reranker that reorders a first stage top 25 candidate list on four
|
|
table question answering benchmarks. SQA and TAT QA use the full test split. HybridQA and
|
|
TabFact use a fixed shared 500 query sample. `acc@10` counts a query correct only when every
|
|
gold table falls inside the top 10.
|
|
|
|
| Dataset | n | recall@10 | ndcg@10 | acc@10 | MRR |
|
|
|---|---|---|---|---|---|
|
|
| SQA | 148 | 0.861 | 0.736 | 0.784 | 0.715 |
|
|
| TAT QA | 362 | 0.657 | 0.540 | 0.425 | 0.632 |
|
|
| HybridQA | 500 | 0.875 | 0.791 | 0.772 | 0.849 |
|
|
| TabFact | 500 | 0.734 | 0.670 | 0.514 | 0.785 |
|
|
| **Mean** | | **0.782** | **0.684** | **0.624** | **0.745** |
|
|
|
|
## Usage with vLLM
|
|
|
|
```python
|
|
from vllm import LLM, SamplingParams
|
|
from transformers import AutoTokenizer
|
|
|
|
repo = "AdarshSingh7647/TabRankMultiTableCoTGen"
|
|
tok = AutoTokenizer.from_pretrained(repo)
|
|
llm = LLM(model=repo, dtype="bfloat16", max_model_len=32768)
|
|
|
|
system = ("You are a table relevance expert. Given a question and a set of candidate tables "
|
|
"rank them from most to least useful for answering the question. Reason in a "
|
|
"<think>...</think> block then output exactly JSON with key ranked_tables.")
|
|
user = "Question: ...\n\n### Table 1\n...\n\n### Table 2\n...\n"
|
|
|
|
msgs = [{"role": "system", "content": system}, {"role": "user", "content": user}]
|
|
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
|
|
# the model writes a <think> block then the ranking json
|
|
out = llm.generate([text], SamplingParams(temperature=0.6, top_p=0.95, max_tokens=8192))
|
|
print(out[0].outputs[0].text)
|
|
```
|
|
|
|
## Usage with Transformers
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
repo = "AdarshSingh7647/TabRankMultiTableCoTGen"
|
|
tok = AutoTokenizer.from_pretrained(repo)
|
|
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
|
|
|
|
system = ("You are a table relevance expert. Given a question and a set of candidate tables "
|
|
"rank them from most to least useful for answering the question. Reason in a "
|
|
"<think>...</think> block then output exactly JSON with key ranked_tables.")
|
|
user = "Question: ...\n\n### Table 1\n...\n\n### Table 2\n...\n"
|
|
|
|
msgs = [{"role": "system", "content": system}, {"role": "user", "content": user}]
|
|
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
|
|
inputs = tok(text, return_tensors="pt").to(model.device)
|
|
out = model.generate(**inputs, max_new_tokens=8192, temperature=0.6, top_p=0.95, do_sample=True)
|
|
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
|
|
```
|
|
|
|
## Model details
|
|
|
|
* Base model Qwen3 8B
|
|
* Method LoRA rank 16 fine tuning merged into the base weights so it loads directly
|
|
* Precision bfloat16 single file safetensors near 16 GB
|
|
* Family the six TabRank checkpoints span three objectives (Answer Only, Reasoning Generation,
|
|
Reasoning Conditioned) across two training mixes (Single Table, Single plus Multi Table)
|
|
|
|
## Citation
|
|
|
|
The MultiTabQA data comes from RAG over Tables. Please cite it when using the Single plus
|
|
Multi Table checkpoints:
|
|
|
|
```bibtex
|
|
@misc{zou2025ragtableshierarchicalmemory,
|
|
title={RAG over Tables: Hierarchical Memory Index, Multi-Stage Retrieval, and Benchmarking},
|
|
author={Jiaru Zou and Dongqi Fu and Sirui Chen and Xinrui He and Zihao Li and Yada Zhu and Jiawei Han and Jingrui He},
|
|
year={2025},
|
|
eprint={2504.01346},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CL},
|
|
url={https://arxiv.org/abs/2504.01346}
|
|
}
|
|
```
|
|
|
|
The accompanying TabRank paper is currently under review. A citation will be added here once it
|
|
is available on arXiv.
|