psymon/mistral-7b-mio-arc-fp16
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: psymon/mistral-7b-mio-arc-fp16
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-21 06:28:11 +08:00
commit 62bcb05435
12 changed files with 268775 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

290
README.md Normal file
View File

@@ -0,0 +1,290 @@
---
language:
- en
license: other
license_name: apache-2.0-mixed-data-terms
library_name: transformers
pipeline_tag: text-generation
base_model: mistralai/Mistral-7B-v0.1
datasets:
- allenai/ai2_arc
- allenai/openbookqa
- allenai/qasc
- allenai/sciq
tags:
- mistral
- qlora
- preference-learning
- mutual-information-optimization
- mio
- hard-negative-mining
- arc-challenge
- science-qa
- multiple-choice
- fp16
model-index:
- name: AIRE-ARC-Mistral-7B-SFT-MIO-FP16
results:
- task:
type: question-answering
name: ARC-Challenge 25-shot
dataset:
type: allenai/ai2_arc
name: AI2 ARC-Challenge
config: ARC-Challenge
split: test
metrics:
- type: acc_norm
name: Normalized Accuracy
value: 0.765358361774744
- type: acc
name: Accuracy
value: 0.7474402730375427
---
# AIRE-ARC-Mistral-7B-SFT-MIO-FP16
A standalone FP16 fine-tune of [`mistralai/Mistral-7B-v0.1`](https://huggingface.co/mistralai/Mistral-7B-v0.1) for ARC-Challenge and related science multiple-choice reasoning. The model was trained in two stages:
1. completion-only QLoRA SFT on 27,294 public science-QA training examples;
2. MIO preference fine-tuning on 47,087 train-only pairs, where the rejected answers were the incorrect choices that the SFT model found most plausible.
The final model reached **76.54% `acc_norm`** and **74.74% raw accuracy** on the 1,172-example ARC-Challenge test split with 25-shot evaluation. Model selection was completed on the validation split before the test split was opened.
> **Important:** this is a domain-specialized completion model, not a chat or general instruction model. It improves science and commonsense multiple-choice reasoning, but capability-retention audits found regressions on MMLU Humanities, MMLU Social Sciences, and WikiText perplexity.
## Model summary
| Item | Value |
|---|---|
| Architecture | `MistralForCausalLM` |
| Parameters | 7B |
| Parent model | `mistralai/Mistral-7B-v0.1` |
| Parent revision | `27d67f1b5f57dc0953326b2601d68371d40ea8da` |
| Repository format | Standalone merged FP16 weights |
| Adapter required | No |
| Primary language | English |
| Primary task | Science multiple-choice QA |
| Training hardware | One NVIDIA A100-SXM4 40GB |
The repository contains the tokenizer and merged model weights. PEFT and bitsandbytes are not required for inference. `merge_summary.json` records the merge provenance used for the submitted checkpoint.
## Intended use
Suitable uses include:
- evaluating science and commonsense multiple-choice reasoning;
- studying model-scored hard-negative selection;
- reproducing the reported ARC-Challenge evaluation;
- research on preference optimization for short-answer QA.
The model is not intended for safety-critical use, factual deployment without verification, general chat, or instruction following. No additional safety alignment was performed.
## Prompt format
Training and evaluation use answer-text completions rather than answer-position labels:
```text
Question: {question}
Answer:
```
The expected completion is the answer text, including its leading space. No chat template is used.
## Quick start
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "psymon/mistral-7b-mio-arc-fp16"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
if tokenizer.pad_token_id is None:
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
repo_id,
dtype=torch.float16,
device_map="auto",
)
model.eval()
prompt = "Question: What is a worldwide increase in temperature called?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=24,
do_sample=False,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
completion = tokenizer.decode(
output[0, inputs["input_ids"].shape[1]:],
skip_special_tokens=True,
)
print(completion)
```
For multiple-choice evaluation, compare the conditional log-likelihood of each answer text. Do not convert the choices to `A`, `B`, `C`, and `D` unless the evaluation protocol is changed accordingly.
## Training data
Only public **training splits** were used for optimization. ARC-Challenge validation was used for deduplication, failure analysis, and model selection. ARC-Challenge test was not used during training or model selection.
### SFT corpus
| Source | Examples after filtering | Role |
|---|---:|---|
| ARC-Challenge | 1,117 | SFT and preference candidates |
| ARC-Easy | 2,241 | SFT and preference candidates |
| OpenBookQA | 4,826 | SFT and preference candidates |
| QASC | 7,514 | SFT and preference candidates |
| SciQ | 11,596 | SFT only |
| **Total** | **27,294** | |
The SFT prompt contains the question only; the target is the correct answer text. Choices were retained for data normalization and duplicate checks but were not included in the SFT prompt.
### Preference corpus
The SFT model scored every answer choice for 15,698 train-only prompts from ARC-Challenge, ARC-Easy, OpenBookQA, and QASC. Each score was the answer-token log-likelihood sum divided by the answer's character length. For each question, up to three highest-scoring incorrect choices were paired with the correct answer:
```text
prompt = "Question: {question}\nAnswer:"
chosen = " {correct answer text}"
rejected = " {model-selected incorrect answer text}"
```
This produced 47,087 preference pairs. Seven three-choice questions contributed two pairs each; the remaining questions contributed three pairs each. In 3,081 prompts, at least one incorrect choice received a higher selection score than the correct answer.
## Training procedure
### Stage 1: completion-only SFT
| Hyperparameter | Value |
|---|---:|
| Quantization | 4-bit NF4 QLoRA |
| Compute dtype | BF16 |
| LoRA target | all linear layers |
| LoRA rank / alpha / dropout | 16 / 32 / 0.05 |
| Learning rate | `1e-4` |
| Scheduler | cosine |
| Effective batch size | 16 |
| Maximum sequence length | 384 |
| Epochs / optimizer steps | 1 / 1,706 |
Loss was computed only on the answer completion; prompt tokens were excluded from the labels.
### Stage 2: MIO preference fine-tuning
MIO (Mutual Information Optimization) contrasts a chosen and rejected completion against a frozen reference model. This run used the merged SFT checkpoint as the reference and initialized a new LoRA policy from the same weights. Completion log-probabilities were averaged over response tokens, including EOS.
| Hyperparameter | Value |
|---|---:|
| Preference pairs | 47,087 |
| Beta | 0.5 |
| Learning rate | `7.5e-6` |
| LoRA rank / alpha / dropout | 16 / 32 / 0.0 |
| Effective batch size | 32 |
| Maximum sequence length | 256 |
| Epochs / optimizer steps | 1 / 1,472 |
The initial policy/reference log-ratios were exactly zero and the initial loss was `1.386294`, matching `ln(4)`. Reference inference was executed first under `no_grad`; its activations were released before the policy forward pass to fit training on a 40GB A100 without changing the objective.
The result should be interpreted as the effect of the **complete procedure**: SFT reference, model-scored hard negatives, mean completion log-probabilities, and MIO. This experiment did not isolate the causal contribution of each component.
## Evaluation
Evaluation used EleutherAI's `lm-evaluation-harness` protocol with 25 few-shot examples and answer-choice likelihood normalization.
### ARC-Challenge validation
| Model stage | `acc` | `acc_norm` |
|---|---:|---:|
| Mistral-7B-v0.1 | 51.51% | 56.52% |
| Completion-only SFT | 57.86% | 59.20% |
| SFT + MIO | 71.57% | **73.24%** |
From SFT to MIO, 51 normalized predictions changed from incorrect to correct and 9 changed from correct to incorrect, for a net gain of 42 correct answers on 299 validation examples.
### One-time held-out test
| Split | Documents | Few-shot | `acc` | `acc_norm` |
|---|---:|---:|---:|---:|
| ARC-Challenge test | 1,172 | 25 | 74.74% | **76.54%** |
The candidate model was locked before this evaluation. The test split was evaluated once, and no post-test training, hyperparameter tuning, or model reselection was performed.
### Post-test overlap audit
The test split was compared with the training ledger only after final evaluation:
- exact Test/SFT ID overlap: 0;
- exact question plus complete choice-set overlap: 0;
- exact normalized question-stem overlap: 9 test rows;
- character similarity of at least 0.90: 18 test rows.
Removing the 18 near-overlap rows changed test `acc_norm` from 76.5358% to **76.4298%** (-0.1060 percentage points). This is a sensitivity analysis, not proof that semantic contamination is absent; paraphrases and shared underlying facts may remain undetected.
## Capability-retention audit
After model selection and the one-time ARC test, the standalone Hub checkpoint was downloaded again and compared with the base model. These results were not used for training or selection.
| Benchmark | Metric | Base | Final | Change |
|---|---|---:|---:|---:|
| HellaSwag | `acc_norm` | 81.17% | 85.43% | +4.26 pp |
| PIQA | `acc_norm` | 82.48% | 85.75% | +3.26 pp |
| WinoGrande | `acc` | 75.37% | 80.03% | +4.66 pp |
| MMLU Humanities | `acc` | 56.43% | 53.18% | -3.25 pp |
| MMLU Social Sciences | `acc` | 73.81% | 70.46% | -3.35 pp |
| WikiText | word perplexity | 8.0848 | 8.7194 | +7.85% (worse) |
The MMLU regressions partly depend on answer-position labels: accuracy fell much more for answers at positions `B` and `D` than for `A` and `C`. This suggests that output-label calibration changed in addition to any knowledge loss. WikiText does not use answer labels and also regressed, so format calibration alone cannot explain all of the degradation.
## Reproducing the ARC evaluation
```bash
pip install "lm-eval==0.4.12"
python -m lm_eval \
--model hf \
--model_args "pretrained=psymon/mistral-7b-mio-arc-fp16,dtype=float16" \
--tasks arc_challenge \
--num_fewshot 25 \
--batch_size 8
```
The reported submission result used the official ARC-Challenge parquet files in an offline custom task, with hashes checked before evaluation. The submitted Colab notebook contains the exact data validation, model-selection lock, and one-time test procedure.
## Limitations
- **Domain specialization:** optimized for short English science-QA completions.
- **Not a chat model:** no instruction-following or conversational alignment was added.
- **Single training seed:** training variance was not estimated across seeds.
- **Public-corpus overlap risk:** post-test string audits cannot exclude semantic overlap or paraphrases.
- **Capability trade-offs:** MMLU and WikiText regressions show that the model did not preserve all base capabilities.
- **Incomplete coverage:** code, mathematics, safety, long-form generation, summarization, and multilingual behavior were not evaluated.
- **No safety guarantee:** outputs may be incorrect, biased, or unsafe.
## License and data terms
The parent model is released under Apache-2.0. The training sources have separate terms:
| Source | Terms shown by the source dataset card |
|---|---|
| ARC | CC BY-SA 4.0 |
| QASC | CC BY 4.0 |
| SciQ | CC BY-NC 3.0 |
| OpenBookQA | No license declared in the pinned Hugging Face dataset card |
Because the SFT stage includes SciQ and the OpenBookQA card does not declare complete licensing information, this model card intentionally uses `license: other` rather than presenting the repository as unconditionally Apache-2.0. The repository is shared for research and coding-test evaluation. Users are responsible for reviewing the parent-model license and each source dataset's terms before redistribution or downstream use, especially commercial use. This section is informational and not legal advice.
## References
- A. Q. Jiang et al., [Mistral 7B](https://arxiv.org/abs/2310.06825), 2023.
- T. Dettmers et al., [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314), 2023.
- X. Lv et al., [The Hidden Link Between RLHF and Contrastive Learning](https://arxiv.org/abs/2506.22578), ICLR 2026.
- P. Clark et al., [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://arxiv.org/abs/1803.05457), 2018.

30
config.json Normal file
View File

@@ -0,0 +1,30 @@
{
"architectures": [
"MistralForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 1,
"dtype": "float16",
"eos_token_id": 2,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"max_position_embeddings": 32768,
"model_type": "mistral",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"pad_token_id": 0,
"rms_norm_eps": 1e-05,
"rope_parameters": {
"rope_theta": 10000.0,
"rope_type": "default"
},
"sliding_window": 4096,
"tie_word_embeddings": false,
"transformers_version": "5.13.1",
"use_cache": true,
"vocab_size": 32000
}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 1,
"eos_token_id": 2,
"transformers_version": "5.13.1"
}

32
merge_summary.json Normal file
View File

@@ -0,0 +1,32 @@
{
"summary_created_at_utc": "2026-07-15T12:40:51.080088+00:00",
"format": "standalone_fp16_transformers",
"base_model": "mistralai/Mistral-7B-v0.1",
"base_revision": "27d67f1b5f57dc0953326b2601d68371d40ea8da",
"tokenizer_source": "mistralai/Mistral-7B-v0.1",
"merge_order": [
{
"stage": "SFT",
"method": "safe_merge",
"adapter_sha256": "6ca48a1ccc3ee2cecef8d5175a1bc9f1129336b41f5443197433b067d6030fff"
},
{
"stage": "MIO",
"method": "safe_merge",
"adapter_sha256": "46bf0d88c5979bd47bb57542229d668522adf75b1822f9ad89a45703047a785b"
}
],
"output_dtype": "float16",
"quantization_in_uploaded_model": null,
"safe_serialization": true,
"shard_count": 4,
"candidate_lock_sha256": "ecfc899fc05133805425f73639b8c4a32a2e3c632846d4e4cae089b8fada93a7",
"evaluation": {
"protocol": "ARC-Challenge 25-shot, lm-eval 0.4.12",
"validation_acc_norm": 0.7324414715719063,
"test_acc": 0.7474402730375427,
"test_acc_norm": 0.765358361774744,
"test_evaluation_count": 1,
"post_test_training_or_selection": false
}
}

3
model-00001-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b27a0c714a52a947affee696cdf8db845ae212255ed43123c8a6010e36354f94
size 3972145200

3
model-00002-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2fc221a78a2ea2db8856de2b4d43b69fd3b5dbd17d4dbd095b74460c6482c3ef
size 3967976920

3
model-00003-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0a121da9ce6cef757f7d95737921678b8a4e559864ec0702d24abb7de24898eb
size 3926025376

3
model-00004-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:618a9008ff99c84e8c123eefe329ebb49b153ea6b19b010ef54ae1a47360dde0
size 2617350224

299
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,299 @@
{
"metadata": {
"total_parameters": 7241732096,
"total_size": 14483464192
},
"weight_map": {
"lm_head.weight": "model-00001-of-00004.safetensors",
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.17.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.17.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.17.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.17.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.18.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.26.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.26.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.26.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.26.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.26.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.26.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.26.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.26.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.input_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.27.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.27.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.input_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.28.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.28.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.input_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.29.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.29.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.30.input_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.30.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.30.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.30.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.30.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.30.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.30.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.30.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.30.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.input_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.31.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.31.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.7.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.7.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
"model.norm.weight": "model-00004-of-00004.safetensors"
}
}

268053
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

18
tokenizer_config.json Normal file
View File

@@ -0,0 +1,18 @@
{
"add_prefix_space": null,
"backend": "tokenizers",
"bos_token": "<s>",
"clean_up_tokenization_spaces": false,
"eos_token": "</s>",
"extra_special_tokens": [],
"is_local": true,
"legacy": false,
"local_files_only": true,
"model_max_length": 1000000000000000019884624838656,
"pad_token": "<unk>",
"sp_model_kwargs": {},
"spaces_between_special_tokens": false,
"tokenizer_class": "TokenizersBackend",
"unk_token": "<unk>",
"use_default_system_prompt": false
}