29 lines
688 B
Markdown
29 lines
688 B
Markdown
|
---
|
||
|
|
base_model: mistralai/Mistral-7B-v0.1
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags:
|
||
|
|
- qlora
|
||
|
|
- arc-challenge
|
||
|
|
- mistral
|
||
|
|
- science-qa
|
||
|
|
---
|
||
|
|
|
||
|
|
# mistral-7b-arc-qlora-exp7-3
|
||
|
|
|
||
|
|
This model is a QLoRA task-adapted Mistral-7B-v0.1 model for ARC-Challenge science multiple-choice QA.
|
||
|
|
|
||
|
|
## Training setup
|
||
|
|
|
||
|
|
- Base model: `mistralai/Mistral-7B-v0.1`
|
||
|
|
- Method: 4-bit NF4 QLoRA + response-only loss
|
||
|
|
- Training data: ARC-Challenge train, ARC-Easy train subset, OpenBookQA train
|
||
|
|
- ARC-Easy ratio: `0.3`
|
||
|
|
- Learning rate: `3e-05`
|
||
|
|
- Epochs: `4`
|
||
|
|
- LoRA r/alpha/dropout: `64` / `128` / `0.05`
|
||
|
|
|
||
|
|
## Evaluation
|
||
|
|
|
||
|
|
Evaluated with `lm-evaluation-harness` on `arc_challenge` using `25`-shot prompting.
|