mistral-7b-qlora-dpo/README.md

---
license: apache-2.0
language:
- en
base_model: mistralai/Mistral-7B-v0.1
library_name: transformers
pipeline_tag: text-generation
tags:
- mistral
- qlora
- lora
- peft
- trl
- dpo
- bitsandbytes
- 4-bit
- question-answering
- arc
datasets:
- tau/commonsense_qa
- allenai/qasc
- allenai/openbookqa
metrics:
- accuracy
model-index:
- name: mistral-7b-qlora-dpo
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: AI2 Reasoning Challenge (25-Shot)
      type: allenai/ai2_arc
      config: ARC-Challenge
      split: test
      args:
        num_few_shot: 25
    metrics:
    - type: acc_norm
      value: 65.27
      name: normalized accuracy
---

# mistral-7b-qlora-dpo

**Mistral-7B-v0.1 fine-tuned with QLoRA + DPO on multiple-choice science/commonsense QA, improving ARC-Challenge (25-shot) from 61.35 → 65.27 acc_norm.**

This repository contains a **merged, bitsandbytes 4-bit (NF4) checkpoint**: the LoRA adapter was merged into the 4-bit quantized base model with `merge_and_unload()` and pushed as-is, so the safetensors weights are stored quantized (~4.5 GB) and loading requires `bitsandbytes` on a CUDA device.

## Approach

ARC-Challenge is evaluated (via `lm-evaluation-harness`) by computing the log-likelihood of each answer choice and picking the highest — no instruction following is involved. Standard supervised instruction tuning (e.g., on Alpaca-style data) actually *degraded* the base model's ARC score in our ablations, because the training objective was not aligned with the likelihood-ranking evaluation.

Instead, this model uses **Direct Preference Optimization (DPO)** to directly increase the likelihood of the correct answer relative to distractors: every multiple-choice question is converted into preference pairs (chosen = correct answer, rejected = each wrong answer), which matches the evaluation protocol exactly.

## Results

ARC-Challenge, 25-shot, `acc_norm`, evaluated with EleutherAI `lm-evaluation-harness` (`--batch_size 8`, `dtype=float16`). The base-model row was measured with the same harness command on the unquantized fp16 model, which is why it can differ slightly from other published numbers for Mistral-7B-v0.1:

| Setup | acc_norm |
|---|---|
| Mistral-7B-v0.1 (base) | 61.35 |
| SFT (QLoRA) on Alpaca 0.5k | 58.96 |
| SFT (QLoRA) on SciQ | 55.38 |
| SFT (QLoRA) on QASC | 43.17 |
| SFT (QLoRA) on CommonsenseQA | 56.83 |
| DPO on CommonsenseQA only | 64.16 |
| **DPO on multi-dataset curriculum (this model)** | **65.27** |
| SimPO on the same multi-dataset mix | 62.12 |

Notes from the experiment series:

- All SFT variants scored *below* the base model — evidence that the failure mode was the training/evaluation mismatch (and likely catastrophic forgetting), not data quality.
- SimPO trained in about half the time of DPO (44 min 03 s vs 1 h 28 min 13 s) but scored ~3.2 points lower under this setup.

## Training data

Three multiple-choice QA datasets (`train + validation` splits) were normalized to a common 4-choice format:

| Dataset | Source | Questions used | Preprocessing |
|---|---|---|---|
| CommonsenseQA | `tau/commonsense_qa` | 1,500 | 5 choices → 4: gold answer + 3 randomly sampled distractors, order shuffled |
| QASC | `allenai/qasc` | 500 | 8 choices → 4: gold answer + 3 randomly sampled distractors, order shuffled |
| OpenBookQA | `allenai/openbookqa` (config `additional`) | 1,500 | native 4 choices kept as-is |

Each dataset was shuffled with `seed=42` before subsampling. The three subsets were concatenated **in the order above**, following a curriculum-inspired design: datasets on which the base model scored higher in a no-training difficulty probe come first, the hardest last. (Since the trainer's default sampler shuffles examples each epoch, this should be read as the composition of the data mixture rather than a strict ordering guarantee.) SciQ and MMLU science subsets were also explored but excluded from the final mix.

Every question then yields **3 preference pairs** (one per distractor), for a total of **10,500 DPO pairs** from 3,500 questions:

```
prompt   = "Question: {question}\nChoices: (A) {a} (B) {b} (C) {c} (D) {d}\nAnswer:"
chosen   = " {correct answer text}"
rejected = " {distractor text}"
```

## Training procedure

### Quantization (QLoRA)

| Setting | Value |
|---|---|
| Quantization | bitsandbytes 4-bit, NF4 |
| Double quantization | off |
| Compute dtype | float16 (as recorded in the released `config.json`) |
| Attention | FlashAttention-2 |

### LoRA

| Setting | Value |
|---|---|
| r | 8 |
| lora_alpha | 16 |
| target_modules | `q_proj`, `v_proj` |
| lora_dropout | 0.05 |
| bias | none |
| task_type | CAUSAL_LM |

The model was prepared with `peft.prepare_model_for_kbit_training` before wrapping.

### DPO (TRL `DPOTrainer`)

| Hyperparameter | Value |
|---|---|
| beta | 0.1 |
| epochs | 2 |
| per_device_train_batch_size | 32 |
| gradient_accumulation_steps | 1 |
| learning_rate | 2e-4 |
| lr_scheduler | cosine |
| warmup_ratio | 0.03 |
| weight_decay | 0.001 |
| max_prompt_length | 256 |
| max_completion_length | 128 |

Tokenizer setup: `add_eos_token=True`, `padding_side="left"` during training, and `pad_token = eos_token` (`</s>`, id 2) since the Mistral tokenizer has no dedicated pad token.

Training ran on a single CUDA GPU (`CUDA_VISIBLE_DEVICES=0`) and took ~1 h 28 min. After training, the LoRA adapter was merged into the 4-bit base (`merge_and_unload()`) and the merged quantized model was pushed. Note that merging LoRA weights into a 4-bit quantized model introduces small rounding differences relative to merging in full precision.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "nevertmr/mistral-7b-qlora-dpo"

# Weights are stored 4-bit (bitsandbytes); requires a CUDA GPU and `pip install bitsandbytes`
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

prompt = (
    "Question: Which property of a mineral can be determined just by looking at it?\n"
    "Choices: (A) luster (B) mass (C) weight (D) hardness\n"
    "Answer:"
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=8, use_cache=True)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

Tips:

- The released `config.json` has `use_cache: false` (a training-time leftover) — pass `use_cache=True` to `generate()` for fast decoding.
- The model is optimized for **likelihood-based multiple-choice scoring** in the prompt format shown above. It is *not* instruction-tuned or chat-aligned; treat it as a base-style LM specialized for MCQA.
- For evaluation-style usage, score each candidate answer's log-likelihood after `"Answer:"` and pick the argmax, rather than free-form generation.

## Evaluation setup

```
lm_eval --model hf \
    --model_args pretrained=nevertmr/mistral-7b-qlora-dpo,dtype=float16 \
    --tasks arc_challenge \
    --device cuda:0 \
    --batch_size 8 \
    --num_fewshot 25
```

## Framework versions

Pinned environment used for training:

- `torch` (cu126 build), `transformers==4.50.0`, `accelerate==1.5.2`, `bitsandbytes==0.45.3`, `datasets==3.4.1`, `flash_attn==2.7.4.post1`, `safetensors==0.5.3`, `tokenizers==0.21.1`
- `peft @ git+https://github.com/huggingface/peft.git@e79fdd78f63ca274174fa973d79779107448e660`
- `trl @ git+https://github.com/huggingface/trl.git@a0a53171cce596642c5fbb9e7b88b8f26a44ecad`
- `lm-evaluation-harness @ git+https://github.com/EleutherAI/lm-evaluation-harness@fd93c6c4fcb008b348fe21049a7b8b46d2bdc88e`

## Limitations

- Specialized for 4-choice science/commonsense QA likelihood ranking; general instruction following, chat, and open-ended generation are out of scope.
- Inherits all limitations and biases of Mistral-7B-v0.1 and of the training datasets; no safety alignment was performed.
- The checkpoint is only usable with `bitsandbytes` on CUDA hardware (weights are stored 4-bit); there is no full-precision variant of the merged model.