mistral-7b-arc-qlora-exp7-3/README.md

base_model, library_name, pipeline_tag, tags

base_model	library_name	pipeline_tag	tags
mistralai/Mistral-7B-v0.1	transformers	text-generation	qlora arc-challenge mistral science-qa

mistral-7b-arc-qlora-exp7-3

This model is a QLoRA task-adapted Mistral-7B-v0.1 model for ARC-Challenge science multiple-choice QA.

Training setup

• Base model: mistralai/Mistral-7B-v0.1
• Method: 4-bit NF4 QLoRA + response-only loss
• Training data: ARC-Challenge train, ARC-Easy train subset, OpenBookQA train
• ARC-Easy ratio: 0.3
• Learning rate: 3e-05
• Epochs: 4
• LoRA r/alpha/dropout: 64 / 128 / 0.05

Evaluation

Evaluated with lm-evaluation-harness on arc_challenge using 25-shot prompting.