初始化项目，由ModelHub XC社区提供模型

Model: nevertmr/mistral-7b-qlora-dpo Source: Original Platform
2026-10-17 01:46:18 +08:00
commit 77bbddcf3f
8 changed files with 268433 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,192 @@
+---
+license: apache-2.0
+language:
+- en
+base_model: mistralai/Mistral-7B-v0.1
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- mistral
+- qlora
+- lora
+- peft
+- trl
+- dpo
+- bitsandbytes
+- 4-bit
+- question-answering
+- arc
+datasets:
+- tau/commonsense_qa
+- allenai/qasc
+- allenai/openbookqa
+metrics:
+- accuracy
+model-index:
+- name: mistral-7b-qlora-dpo
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: AI2 Reasoning Challenge (25-Shot)
+      type: allenai/ai2_arc
+      config: ARC-Challenge
+      split: test
+      args:
+        num_few_shot: 25
+    metrics:
+    - type: acc_norm
+      value: 65.27
+      name: normalized accuracy
+---
+
+# mistral-7b-qlora-dpo
+
+**Mistral-7B-v0.1 fine-tuned with QLoRA + DPO on multiple-choice science/commonsense QA, improving ARC-Challenge (25-shot) from 61.35 → 65.27 acc_norm.**
+
+This repository contains a **merged, bitsandbytes 4-bit (NF4) checkpoint**: the LoRA adapter was merged into the 4-bit quantized base model with `merge_and_unload()` and pushed as-is, so the safetensors weights are stored quantized (~4.5 GB) and loading requires `bitsandbytes` on a CUDA device.
+
+## Approach
+
+ARC-Challenge is evaluated (via `lm-evaluation-harness`) by computing the log-likelihood of each answer choice and picking the highest — no instruction following is involved. Standard supervised instruction tuning (e.g., on Alpaca-style data) actually *degraded* the base model's ARC score in our ablations, because the training objective was not aligned with the likelihood-ranking evaluation.
+
+Instead, this model uses **Direct Preference Optimization (DPO)** to directly increase the likelihood of the correct answer relative to distractors: every multiple-choice question is converted into preference pairs (chosen = correct answer, rejected = each wrong answer), which matches the evaluation protocol exactly.
+
+## Results
+
+ARC-Challenge, 25-shot, `acc_norm`, evaluated with EleutherAI `lm-evaluation-harness` (`--batch_size 8`, `dtype=float16`). The base-model row was measured with the same harness command on the unquantized fp16 model, which is why it can differ slightly from other published numbers for Mistral-7B-v0.1:
+
+| Setup | acc_norm |
+|---|---|
+| Mistral-7B-v0.1 (base) | 61.35 |
+| SFT (QLoRA) on Alpaca 0.5k | 58.96 |
+| SFT (QLoRA) on SciQ | 55.38 |
+| SFT (QLoRA) on QASC | 43.17 |
+| SFT (QLoRA) on CommonsenseQA | 56.83 |
+| DPO on CommonsenseQA only | 64.16 |
+| **DPO on multi-dataset curriculum (this model)** | **65.27** |
+| SimPO on the same multi-dataset mix | 62.12 |
+
+Notes from the experiment series:
+
+- All SFT variants scored *below* the base model — evidence that the failure mode was the training/evaluation mismatch (and likely catastrophic forgetting), not data quality.
+- SimPO trained in about half the time of DPO (44 min 03 s vs 1 h 28 min 13 s) but scored ~3.2 points lower under this setup.
+
+## Training data
+
+Three multiple-choice QA datasets (`train + validation` splits) were normalized to a common 4-choice format:
+
+| Dataset | Source | Questions used | Preprocessing |
+|---|---|---|---|
+| CommonsenseQA | `tau/commonsense_qa` | 1,500 | 5 choices → 4: gold answer + 3 randomly sampled distractors, order shuffled |
+| QASC | `allenai/qasc` | 500 | 8 choices → 4: gold answer + 3 randomly sampled distractors, order shuffled |
+| OpenBookQA | `allenai/openbookqa` (config `additional`) | 1,500 | native 4 choices kept as-is |
+
+Each dataset was shuffled with `seed=42` before subsampling. The three subsets were concatenated **in the order above**, following a curriculum-inspired design: datasets on which the base model scored higher in a no-training difficulty probe come first, the hardest last. (Since the trainer's default sampler shuffles examples each epoch, this should be read as the composition of the data mixture rather than a strict ordering guarantee.) SciQ and MMLU science subsets were also explored but excluded from the final mix.
+
+Every question then yields **3 preference pairs** (one per distractor), for a total of **10,500 DPO pairs** from 3,500 questions:
+
+```
+prompt   = "Question: {question}\nChoices: (A) {a} (B) {b} (C) {c} (D) {d}\nAnswer:"
+chosen   = " {correct answer text}"
+rejected = " {distractor text}"
+```
+
+## Training procedure
+
+### Quantization (QLoRA)
+
+| Setting | Value |
+|---|---|
+| Quantization | bitsandbytes 4-bit, NF4 |
+| Double quantization | off |
+| Compute dtype | float16 (as recorded in the released `config.json`) |
+| Attention | FlashAttention-2 |
+
+### LoRA
+
+| Setting | Value |
+|---|---|
+| r | 8 |
+| lora_alpha | 16 |
+| target_modules | `q_proj`, `v_proj` |
+| lora_dropout | 0.05 |
+| bias | none |
+| task_type | CAUSAL_LM |
+
+The model was prepared with `peft.prepare_model_for_kbit_training` before wrapping.
+
+### DPO (TRL `DPOTrainer`)
+
+| Hyperparameter | Value |
+|---|---|
+| beta | 0.1 |
+| epochs | 2 |
+| per_device_train_batch_size | 32 |
+| gradient_accumulation_steps | 1 |
+| learning_rate | 2e-4 |
+| lr_scheduler | cosine |
+| warmup_ratio | 0.03 |
+| weight_decay | 0.001 |
+| max_prompt_length | 256 |
+| max_completion_length | 128 |
+
+Tokenizer setup: `add_eos_token=True`, `padding_side="left"` during training, and `pad_token = eos_token` (`</s>`, id 2) since the Mistral tokenizer has no dedicated pad token.
+
+Training ran on a single CUDA GPU (`CUDA_VISIBLE_DEVICES=0`) and took ~1 h 28 min. After training, the LoRA adapter was merged into the 4-bit base (`merge_and_unload()`) and the merged quantized model was pushed. Note that merging LoRA weights into a 4-bit quantized model introduces small rounding differences relative to merging in full precision.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "nevertmr/mistral-7b-qlora-dpo"
+
+# Weights are stored 4-bit (bitsandbytes); requires a CUDA GPU and `pip install bitsandbytes`
+model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+prompt = (
+    "Question: Which property of a mineral can be determined just by looking at it?\n"
+    "Choices: (A) luster (B) mass (C) weight (D) hardness\n"
+    "Answer:"
+)
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+with torch.inference_mode():
+    out = model.generate(**inputs, max_new_tokens=8, use_cache=True)
+print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
+```
+
+Tips:
+
+- The released `config.json` has `use_cache: false` (a training-time leftover) — pass `use_cache=True` to `generate()` for fast decoding.
+- The model is optimized for **likelihood-based multiple-choice scoring** in the prompt format shown above. It is *not* instruction-tuned or chat-aligned; treat it as a base-style LM specialized for MCQA.
+- For evaluation-style usage, score each candidate answer's log-likelihood after `"Answer:"` and pick the argmax, rather than free-form generation.
+
+## Evaluation setup
+
+```
+lm_eval --model hf \
+    --model_args pretrained=nevertmr/mistral-7b-qlora-dpo,dtype=float16 \
+    --tasks arc_challenge \
+    --device cuda:0 \
+    --batch_size 8 \
+    --num_fewshot 25
+```
+
+## Framework versions
+
+Pinned environment used for training:
+
+- `torch` (cu126 build), `transformers==4.50.0`, `accelerate==1.5.2`, `bitsandbytes==0.45.3`, `datasets==3.4.1`, `flash_attn==2.7.4.post1`, `safetensors==0.5.3`, `tokenizers==0.21.1`
+- `peft @ git+https://github.com/huggingface/peft.git@e79fdd78f63ca274174fa973d79779107448e660`
+- `trl @ git+https://github.com/huggingface/trl.git@a0a53171cce596642c5fbb9e7b88b8f26a44ecad`
+- `lm-evaluation-harness @ git+https://github.com/EleutherAI/lm-evaluation-harness@fd93c6c4fcb008b348fe21049a7b8b46d2bdc88e`
+
+## Limitations
+
+- Specialized for 4-choice science/commonsense QA likelihood ranking; general instruction following, chat, and open-ended generation are out of scope.
+- Inherits all limitations and biases of Mistral-7B-v0.1 and of the training datasets; no safety alignment was performed.
+- The checkpoint is only usable with `bitsandbytes` on CUDA hardware (weights are stored 4-bit); there is no full-precision variant of the merged model.
--- a/config.json
+++ b/config.json
@@ -0,0 +1,43 @@
+{
+  "architectures": [
+    "MistralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "model_type": "mistral",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "pretraining_tp": 1,
+  "quantization_config": {
+    "_load_in_4bit": true,
+    "_load_in_8bit": false,
+    "bnb_4bit_compute_dtype": "float16",
+    "bnb_4bit_quant_storage": "uint8",
+    "bnb_4bit_quant_type": "nf4",
+    "bnb_4bit_use_double_quant": false,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": null,
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": true,
+    "load_in_8bit": false,
+    "quant_method": "bitsandbytes"
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.50.0",
+  "use_cache": false,
+  "vocab_size": 32000
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.50.0"
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13fdaac499061d6cd135952f6d5a811d823046dcbe1522233a9c2a098a7fb62c
+size 4450837496
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,44 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": true,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizerFast",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}