153 lines
6.6 KiB
Markdown
153 lines
6.6 KiB
Markdown
---
|
|
license: llama3
|
|
language:
|
|
- en
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
base_model: meta-llama/Meta-Llama-3-8B-Instruct
|
|
base_model_relation: finetune
|
|
datasets:
|
|
- bactrianus/bactrainus-hotpotqa
|
|
- bactrianus/bactrainus-hotpotqa-teacher-traces
|
|
tags:
|
|
- bactrainus
|
|
- bactrianus
|
|
- llama-3
|
|
- hotpotqa
|
|
- multi-hop-qa
|
|
- rationale-supervision
|
|
- answer-generation
|
|
- legacy
|
|
---
|
|
|
|
# Bactrainus HotpotQA Rationale Reader — Llama 3 8B Instruct
|
|
|
|
<p align="center">
|
|
<img src="assets/models.png" alt="Bactrainus Llama 3 model collection artwork" width="720">
|
|
</p>
|
|
|
|
## Artifact identity
|
|
|
|
- **Status:** complete merged causal-language-model checkpoint
|
|
- **Base model:** `meta-llama/Meta-Llama-3-8B-Instruct`
|
|
- **Audited Hub revision:** `852277e5b9534ff51a66adbad1ad43b7a3ef4457`
|
|
- **Public artifact date:** August 2024
|
|
- **Role:** rationale-plus-answer generation from supplied evidence
|
|
|
|
This is a historical **Llama 3** artifact. It must not be represented as either revised Llama 3.1 rationale-reader variant described in the updated manuscript.
|
|
|
|
## Model summary
|
|
|
|
This reader is adapted to generate an intermediate natural-language rationale followed by an answer. The rationale is process supervision generated for task adaptation; it is not a hidden trace recovered from the base model and is not a gold supporting-fact annotation.
|
|
|
|
## Intended use
|
|
|
|
- Studying natural-language rationale supervision for HotpotQA readers.
|
|
- Qualitative inspection of an evidence-conditioned answer path.
|
|
- Reader-stage comparisons where evidence is supplied independently.
|
|
|
|
### Out-of-scope use
|
|
|
|
- Treating generated rationales as faithful explanations or verified proofs.
|
|
- Using rationale text as a substitute for HotpotQA supporting-fact labels.
|
|
- Open-domain retrieval, safety-critical decisions, or factual verification.
|
|
- Associating revised Llama 3.1 rationale results with this legacy checkpoint.
|
|
|
|
## Input and output contract
|
|
|
|
Input should contain a question and selected, title-preserving evidence. Output is expected to contain rationale text and a final answer. Downstream code must parse the final answer explicitly and must keep rationale evaluation separate from answer/evidence metrics.
|
|
|
|
The public legacy configuration does not preserve a complete prompt-version manifest or an independently verified rationale delimiter. Do not assume that a newly invented delimiter exactly matches historical training.
|
|
|
|
## Loading
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
MODEL_ID = "bactrianus/HotpotQA-Reader-CoT-Llama-3-8B-Instruct"
|
|
REVISION = "852277e5b9534ff51a66adbad1ad43b7a3ef4457"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, revision=REVISION)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
MODEL_ID,
|
|
revision=REVISION,
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto",
|
|
)
|
|
model.eval()
|
|
```
|
|
|
|
## Training data and lineage
|
|
|
|
The checkpoint derives from Meta Llama 3 8B Instruct and HotpotQA-based reader/rationale supervision. Two separately versioned dataset resources are relevant to this task:
|
|
|
|
- [`cot-reader-sft`](https://huggingface.co/datasets/bactrianus/bactrainus-hotpotqa/tree/v1.0.0/data/cot-reader-sft) in the canonical dataset at `v1.0.0` is a complete deterministic view with 90,447 unique training source IDs. Its assistant targets serialize indexed gold evidence followed by the reference answer; they are not model-generated rationales.
|
|
- [`teacher-cot-llama31-8b-sft`](https://huggingface.co/datasets/bactrianus/bactrainus-hotpotqa-teacher-traces/tree/v1.0.0/data/teacher-cot-llama31-8b-sft) in the teacher-trace dataset at `v1.0.0` contains one resolved Llama 3.1 8B-labelled rationale per source ID for 71,238 training examples.
|
|
- [`teacher-cot-llama31-70b-sft`](https://huggingface.co/datasets/bactrianus/bactrainus-hotpotqa-teacher-traces/tree/v1.0.0/data/teacher-cot-llama31-70b-sft) in the teacher-trace dataset at `v1.0.0` contains 28,176 recovered Llama 3.1 70B-labelled records: all 15,661 hard examples and 12,515 medium examples. Each row includes the archived annotation and an SFT conversation joined to the canonical dataset through `source_id`.
|
|
|
|
```python
|
|
from datasets import load_dataset
|
|
|
|
deterministic_train = load_dataset(
|
|
"bactrianus/bactrainus-hotpotqa",
|
|
"cot-reader-sft",
|
|
split="train",
|
|
revision="v1.0.0",
|
|
)
|
|
|
|
teacher_8b_train = load_dataset(
|
|
"bactrianus/bactrainus-hotpotqa-teacher-traces",
|
|
"teacher-cot-llama31-8b-sft",
|
|
split="train",
|
|
revision="v1.0.0",
|
|
)
|
|
|
|
teacher_70b_train = load_dataset(
|
|
"bactrianus/bactrainus-hotpotqa-teacher-traces",
|
|
"teacher-cot-llama31-70b-sft",
|
|
split="train",
|
|
revision="v1.0.0",
|
|
)
|
|
```
|
|
|
|
These resources document the current dataset release; neither is asserted to be byte-identical to the historical training serialization for this legacy Llama 3 checkpoint. The teacher configuration's Llama 3.1 label describes its archived generator record and must not be used to relabel these Llama 3 weights or infer an unrecorded training dependency.
|
|
|
|
Likewise, the revised `reader_8b_rationale_8b.yaml` and `reader_8b_rationale_70b.yaml` files describe Llama 3.1 experiments, not this Llama 3 weight artifact.
|
|
|
|
## Evaluation boundary
|
|
|
|
No predictions or evaluation results are bundled with this card. The [Bactrainus paper](https://arxiv.org/abs/2501.06286) reports rationale-supervision experiments with explicit recipe caveats. A correct final answer does not establish rationale faithfulness.
|
|
|
|
## Limitations
|
|
|
|
- Generated rationales can be post-hoc, incomplete, contradictory, or unsupported.
|
|
- Longer outputs increase parsing and truncation risk.
|
|
- Evidence omissions propagate to both rationale and answer.
|
|
- The model is specialized for English HotpotQA-style inputs.
|
|
- Wikipedia-derived data carries temporal and representational biases.
|
|
- Historical prompt, generator, and environment details are incomplete.
|
|
|
|
## License and attribution
|
|
|
|
The weights remain subject to the [Meta Llama 3 Community License](https://github.com/meta-llama/llama3/blob/main/LICENSE) and Acceptable Use Policy.
|
|
|
|
> Meta Llama 3 is licensed under the Meta Llama 3 Community License, Copyright Meta Platforms, Inc. All Rights Reserved.
|
|
|
|
**Built with Meta Llama 3.**
|
|
|
|
HotpotQA-derived data is licensed under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). Bactrainus code is Apache-2.0 licensed.
|
|
|
|
## Citation
|
|
|
|
```bibtex
|
|
@article{barati2025bactrainus,
|
|
title = {Bactrainus: Optimizing Large Language Models for Multi-hop Complex Question Answering Tasks},
|
|
author = {Barati, Iman and Ghafouri, Arash and Minaei-Bidgoli, Behrouz},
|
|
journal = {arXiv preprint arXiv:2501.06286},
|
|
year = {2025},
|
|
doi = {10.48550/arXiv.2501.06286},
|
|
url = {https://arxiv.org/abs/2501.06286}
|
|
}
|
|
```
|