legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-09-09 13:20:17 +08:00
commit ca6bf29376
30 changed files with 2944 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
.gitattributes vendored Normal file
View File

@@ -0,0 +1,61 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.BF16.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.FP16.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.Q5_K.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
imatrix.dat filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.Q4_K.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text

172
README.md Normal file
View File

@@ -0,0 +1,172 @@
---
base_model: cognitivecomputations/dolphin-2.9.2-Phi-3-Medium-abliterated
datasets:
- cognitivecomputations/Dolphin-2.9.2
- teknium/OpenHermes-2.5
- m-a-p/CodeFeedback-Filtered-Instruction
- cognitivecomputations/dolphin-coder
- cognitivecomputations/samantha-data
- microsoft/orca-math-word-problems-200k
- internlm/Agent-FLAN
- cognitivecomputations/SystemChat-2.0
inference: false
language:
- en
library_name: gguf
license: mit
pipeline_tag: text-generation
quantized_by: legraphista
tags:
- quantized
- GGUF
- imatrix
- quantization
- imat
- imatrix
- static
- 16bit
- 8bit
- 6bit
- 5bit
- 4bit
- 3bit
- 2bit
- 1bit
---
# dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF
_Llama.cpp imatrix quantization of cognitivecomputations/dolphin-2.9.2-Phi-3-Medium-abliterated_
Original Model: [cognitivecomputations/dolphin-2.9.2-Phi-3-Medium-abliterated](https://huggingface.co/cognitivecomputations/dolphin-2.9.2-Phi-3-Medium-abliterated)
Original dtype: `BF16` (`bfloat16`)
Quantized by: llama.cpp [b3072](https://github.com/ggerganov/llama.cpp/releases/tag/b3072)
IMatrix dataset: [here](https://gist.githubusercontent.com/bartowski1182/eb213dccb3571f863da82e99418f81e8/raw/b2869d80f5c16fd7082594248e80144677736635/calibration_datav3.txt)
- [Files](#files)
- [IMatrix](#imatrix)
- [Common Quants](#common-quants)
- [All Quants](#all-quants)
- [Downloading using huggingface-cli](#downloading-using-huggingface-cli)
- [Inference](#inference)
- [Simple chat template](#simple-chat-template)
- [Chat template with system prompt](#chat-template-with-system-prompt)
- [Llama.cpp](#llama-cpp)
- [FAQ](#faq)
- [Why is the IMatrix not applied everywhere?](#why-is-the-imatrix-not-applied-everywhere)
- [How do I merge a split GGUF?](#how-do-i-merge-a-split-gguf)
---
## Files
### IMatrix
Status: ✅ Available
Link: [here](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/imatrix.dat)
### Common Quants
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
| -------- | ---------- | --------- | ------ | ------------ | -------- |
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0.gguf) | Q8_0 | 14.83GB | ✅ Available | ⚪ Static | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q6_K.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q6_K.gguf) | Q6_K | 11.45GB | ✅ Available | ⚪ Static | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q4_K.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q4_K.gguf) | Q4_K | 8.40GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K.gguf) | Q3_K | 6.75GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q2_K.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q2_K.gguf) | Q2_K | 5.20GB | ✅ Available | 🟢 IMatrix | 📦 No
### All Quants
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
| -------- | ---------- | --------- | ------ | ------------ | -------- |
| [dolphin-2.9.2-Phi-3-Medium-abliterated.BF16.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.BF16.gguf) | BF16 | 27.92GB | ✅ Available | ⚪ Static | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.FP16.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.FP16.gguf) | F16 | 27.92GB | ✅ Available | ⚪ Static | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0.gguf) | Q8_0 | 14.83GB | ✅ Available | ⚪ Static | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q6_K.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q6_K.gguf) | Q6_K | 11.45GB | ✅ Available | ⚪ Static | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q5_K.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q5_K.gguf) | Q5_K | 9.88GB | ✅ Available | ⚪ Static | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q5_K_S.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q5_K_S.gguf) | Q5_K_S | 9.62GB | ✅ Available | ⚪ Static | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q4_K.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q4_K.gguf) | Q4_K | 8.40GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q4_K_S.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q4_K_S.gguf) | Q4_K_S | 7.96GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ4_NL.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ4_NL.gguf) | IQ4_NL | 7.93GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ4_XS.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ4_XS.gguf) | IQ4_XS | 7.51GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K.gguf) | Q3_K | 6.75GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K_L.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K_L.gguf) | Q3_K_L | 7.35GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K_S.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K_S.gguf) | Q3_K_S | 6.06GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_M.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_M.gguf) | IQ3_M | 6.30GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_S.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_S.gguf) | IQ3_S | 6.10GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_XS.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_XS.gguf) | IQ3_XS | 5.78GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_XXS.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_XXS.gguf) | IQ3_XXS | 5.42GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q2_K.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q2_K.gguf) | Q2_K | 5.20GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.Q2_K_S.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.Q2_K_S.gguf) | Q2_K_S | 4.83GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_M.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_M.gguf) | IQ2_M | 4.78GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_S.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_S.gguf) | IQ2_S | 4.41GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_XS.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_XS.gguf) | IQ2_XS | 4.20GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_XXS.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_XXS.gguf) | IQ2_XXS | 3.80GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ1_M.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ1_M.gguf) | IQ1_M | 3.33GB | ✅ Available | 🟢 IMatrix | 📦 No
| [dolphin-2.9.2-Phi-3-Medium-abliterated.IQ1_S.gguf](https://huggingface.co/legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/blob/main/dolphin-2.9.2-Phi-3-Medium-abliterated.IQ1_S.gguf) | IQ1_S | 3.05GB | ✅ Available | 🟢 IMatrix | 📦 No
## Downloading using huggingface-cli
If you do not have hugginface-cli installed:
```
pip install -U "huggingface_hub[cli]"
```
Download the specific file you want:
```
huggingface-cli download legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF --include "dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0.gguf" --local-dir ./
```
If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
```
huggingface-cli download legraphista/dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF --include "dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0/*" --local-dir ./
# see FAQ for merging GGUF's
```
---
## Inference
### Simple chat template
```
<|im_start|>user
{user_prompt}<|im_end|>
<|im_start|>assistant
{assistant_response}<|im_end|>
<|im_start|>user
{next_user_prompt}<|im_end|>
```
### Chat template with system prompt
```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_prompt}<|im_end|>
<|im_start|>assistant
{assistant_response}<|im_end|>
<|im_start|>user
{next_user_prompt}<|im_end|>
```
### Llama.cpp
```
llama.cpp/main -m dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"
```
---
## FAQ
### Why is the IMatrix not applied everywhere?
According to [this investigation](https://www.reddit.com/r/LocalLLaMA/comments/1993iro/ggufs_quants_can_punch_above_their_weights_now/), it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).
### How do I merge a split GGUF?
1. Make sure you have `gguf-split` available
- To get hold of `gguf-split`, navigate to https://github.com/ggerganov/llama.cpp/releases
- Download the appropriate zip for your system from the latest release
- Unzip the archive and you should be able to find `gguf-split`
2. Locate your GGUF chunks folder (ex: `dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0`)
3. Run `gguf-split --merge dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0/dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0-00001-of-XXXXX.gguf dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0.gguf`
- Make sure to point `gguf-split` to the first chunk of the split.
---
Got a suggestion? Ping me [@legraphista](https://x.com/legraphista)!

3
dolphin-2.9.2-Phi-3-Medium-abliterated.BF16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3d26d05aaa9a25266bc51f9aa62b32a89277a59db57a09eec94d2a23aaa52be5
size 27922053280

3
dolphin-2.9.2-Phi-3-Medium-abliterated.FP16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e3a23c64d6ac1a6505312fcf44b1dd34db8c312b934fadd0dce1ff152b2e691d
size 27922053280

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ1_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2a0d074324500518e47d1e4ea386a39e0583eddf899e6c034ee84a23bc2fc4d4
size 3332275648

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ1_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3a56f1c93ada6ba20c5aad2f31a4564ffc1b6f9bb9b1d96c3ba9e32cc7c036ad
size 3054259648

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:85cd8442eb91a94a100bcf5c33f09ead70521063ce7add11590ffd8b5e61d902
size 4780500928

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d357e2042a3eb933279e44c6e19f08a70499fe01496833a2526445d79ce274a3
size 4409812928

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a7b7ded0561209a4cf653fef5309dbf3dab366b4089fc4e987f353f3387617b7
size 4199091648

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ2_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:85db22d2ee8f971235a8314516793f9ba7f30b238918ec78366e274574f326f6
size 3795635648

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eee471387af45538026bd27d90b6b064faf2384dfa56f4629a5379d589cf37f2
size 6299904448

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eb1abc3ef8e897f024971f3255d246a460fc38431460dfbf5e303e1d2b5fb592
size 6099712448

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e5e80117ccbbff5aaa2eca9d357e88739c8d1723fe1a3478aee4f233b168fffe
size 5780224448

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ3_XXS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a54a1174fef7867ded3d19ca1eb20d9ce1687579cc428f463ab36c9161fa13f6
size 5418452928

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ4_NL.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bd3b49211f065e179259152d1b0a7b604ae4b74fbc7dcd6246e5fd58f7967636
size 7929899968

3
dolphin-2.9.2-Phi-3-Medium-abliterated.IQ4_XS.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b37f36340c9ea6e933468f2d2f6d68e366d3490b13c2bd6364d17a1e5ee04d04
size 7506977728

3
dolphin-2.9.2-Phi-3-Medium-abliterated.Q2_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e26bbfb5bea89fd1d7d3249d69e7cb1cf9c476b434e2bb526aa612ed731e0005
size 5204447168

3
dolphin-2.9.2-Phi-3-Medium-abliterated.Q2_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4eefd4e4d8d63365d020c5c4fa8db3357550f2bdd2c26401d6f9666b3b76a83f
size 4832735168

3
dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3a30bbd57522f12dc81e2e5f464588bba316cd41c5d78e5d737c6b06ee0d6eba
size 6750976448

3
dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K_L.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0a9e7915f6d66af98e9320517ee460b21663cc2c293e6b0319525e7fbdfa6df0
size 7348992448

3
dolphin-2.9.2-Phi-3-Medium-abliterated.Q3_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:80b2a07af8e4a40ca6430ac89e054840d26b1d28df55dcfb943fe845258bd100
size 6064896448

3
dolphin-2.9.2-Phi-3-Medium-abliterated.Q4_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d90cf09cda2ec712bf35b1ad870e8adf6b31e52938136a7a3f0e55688245a0d8
size 8404011968

3
dolphin-2.9.2-Phi-3-Medium-abliterated.Q4_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6cef75251e55f33cf5c76b757604c52fa862ec4cb2e3aa5244094fe9246669b8
size 7957752768

3
dolphin-2.9.2-Phi-3-Medium-abliterated.Q5_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4a734358d9803eee7e06bab02746542476197917afaa81bf8fc0d0b3d7b5b6bb
size 9882708640

3
dolphin-2.9.2-Phi-3-Medium-abliterated.Q5_K_S.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b88bd8a2526d22a59e0654170af62250bd1888f8f2070dd7f0d2e90f9bbac024
size 9621588640

3
dolphin-2.9.2-Phi-3-Medium-abliterated.Q6_K.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:05b529b8bdf8c61871bb09a142dbf8127b509e0b1faba70f7d55a8bc4aa4dd88
size 11453824160

3
dolphin-2.9.2-Phi-3-Medium-abliterated.Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6ae0d2e537a66220afdd67da5c4732c9b08971fdf3a7a8e43d029e57a94d13b6
size 14834718880

3
imatrix.dat Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b1cd9a6915d52ebda3b3c08753e9aa46365939872f540a08278e121acb734f68
size 7791726

2482
imatrix.dataset Normal file
View File

File diff suppressed because one or more lines are too long

151
imatrix.log Normal file
View File

@@ -0,0 +1,151 @@
main: build = 3072 (549279d8)
main: built with cc (Ubuntu 13.2.0-23ubuntu4) 13.2.0 for x86_64-linux-gnu
main: seed = 1717403919
llama_model_loader: loaded meta data with 26 key-value pairs and 363 tensors from dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/dolphin-2.9.2-Phi-3-Medium-abliterated.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = dolphin-2.9.2-Phi-3-Medium-abliterated
llama_model_loader: - kv 2: llama.block_count u32 = 40
llama_model_loader: - kv 3: llama.context_length u32 = 4096
llama_model_loader: - kv 4: llama.embedding_length u32 = 5120
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 17920
llama_model_loader: - kv 6: llama.attention.head_count u32 = 40
llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 10
llama_model_loader: - kv 8: llama.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: general.file_type u32 = 0
llama_model_loader: - kv 11: llama.vocab_size u32 = 32064
llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 13: tokenizer.ggml.model str = llama
llama_model_loader: - kv 14: tokenizer.ggml.pre str = default
llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32064] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32064] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32064] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 32011
llama_model_loader: - kv 20: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 32009
llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 24: tokenizer.chat_template str = {% if not add_generation_prompt is de...
llama_model_loader: - kv 25: general.quantization_version u32 = 2
llama_model_loader: - type f32: 363 tensors
llm_load_vocab: special tokens cache size = 323
llm_load_vocab: token to piece cache size = 0.1687 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32064
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 4096
llm_load_print_meta: n_embd = 5120
llm_load_print_meta: n_head = 40
llm_load_print_meta: n_head_kv = 10
llm_load_print_meta: n_layer = 40
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1280
llm_load_print_meta: n_embd_v_gqa = 1280
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 17920
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 4096
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 13B
llm_load_print_meta: model ftype = all F32
llm_load_print_meta: model params = 13.96 B
llm_load_print_meta: model size = 52.01 GiB (32.00 BPW)
llm_load_print_meta: general.name = dolphin-2.9.2-Phi-3-Medium-abliterated
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 32011 '<|im_end|>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 32009 '<|placeholder6|>'
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_print_meta: EOT token = 32007 '<|end|>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
llm_load_tensors: ggml ctx size = 0.37 MiB
llm_load_tensors: offloading 16 repeating layers to GPU
llm_load_tensors: offloaded 16/41 layers to GPU
llm_load_tensors: CPU buffer size = 53254.08 MiB
llm_load_tensors: CUDA0 buffer size = 20800.62 MiB
....................................................................................................
llama_new_context_with_model: n_ctx = 512
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA_Host KV buffer size = 60.00 MiB
llama_kv_cache_init: CUDA0 KV buffer size = 40.00 MiB
llama_new_context_with_model: KV self size = 100.00 MiB, K (f16): 50.00 MiB, V (f16): 50.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.12 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 698.88 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 11.01 MiB
llama_new_context_with_model: graph nodes = 1286
llama_new_context_with_model: graph splits = 268
system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
compute_imatrix: tokenizing the input ..
compute_imatrix: tokenization took 103.872 ms
compute_imatrix: computing over 151 chunks with batch_size 512
compute_imatrix: 30.08 seconds per pass - ETA 1 hours 15.68 minutes
[1]5.5209,[2]4.2369,[3]4.1303,[4]4.5098,[5]4.9651,[6]5.0892,[7]4.5975,[8]5.0183,[9]5.3814,
save_imatrix: stored collected data after 10 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[10]5.7366,[11]5.6958,[12]5.3171,[13]5.2155,[14]5.3907,[15]5.8044,[16]5.9303,[17]6.2729,[18]6.4081,[19]6.5270,
save_imatrix: stored collected data after 20 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[20]6.6309,[21]6.8403,[22]6.6006,[23]6.3256,[24]6.4012,[25]6.4397,[26]6.3371,[27]6.2671,[28]6.3576,[29]6.5325,
save_imatrix: stored collected data after 30 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[30]6.6488,[31]6.6023,[32]6.7093,[33]6.8190,[34]6.9812,[35]6.9775,[36]6.9287,[37]6.6669,[38]6.4958,[39]6.3857,
save_imatrix: stored collected data after 40 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[40]6.2716,[41]6.1735,[42]6.1914,[43]6.1399,[44]6.1571,[45]6.1433,[46]6.0983,[47]6.0972,[48]6.1712,[49]6.2425,
save_imatrix: stored collected data after 50 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[50]6.2475,[51]6.4119,[52]6.5501,[53]6.7235,[54]6.8715,[55]6.9538,[56]6.8636,[57]6.7603,[58]6.7651,[59]6.8131,
save_imatrix: stored collected data after 60 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[60]6.8962,[61]6.7693,[62]6.7858,[63]6.8348,[64]6.9229,[65]6.9803,[66]7.0066,[67]7.0609,[68]7.1048,[69]7.0949,
save_imatrix: stored collected data after 70 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[70]7.0890,[71]7.0915,[72]7.1405,[73]7.0880,[74]7.0181,[75]6.9959,[76]7.0408,[77]7.0332,[78]6.9995,[79]6.9902,
save_imatrix: stored collected data after 80 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[80]7.0031,[81]6.9634,[82]6.9378,[83]6.9098,[84]6.9072,[85]6.9110,[86]6.8996,[87]6.9036,[88]6.8956,[89]6.8828,
save_imatrix: stored collected data after 90 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[90]6.8658,[91]6.8735,[92]6.8379,[93]6.8379,[94]6.8107,[95]6.7672,[96]6.7778,[97]6.7565,[98]6.7636,[99]6.7329,
save_imatrix: stored collected data after 100 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[100]6.7254,[101]6.7334,[102]6.7096,[103]6.6804,[104]6.6572,[105]6.6656,[106]6.6603,[107]6.6857,[108]6.7052,[109]6.6576,
save_imatrix: stored collected data after 110 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[110]6.6115,[111]6.5634,[112]6.5150,[113]6.4675,[114]6.4279,[115]6.3860,[116]6.3417,[117]6.3150,[118]6.3233,[119]6.3293,
save_imatrix: stored collected data after 120 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[120]6.3715,[121]6.4091,[122]6.4562,[123]6.5050,[124]6.5819,[125]6.6454,[126]6.6561,[127]6.6646,[128]6.6121,[129]6.6050,
save_imatrix: stored collected data after 130 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[130]6.5712,[131]6.5364,[132]6.4850,[133]6.4141,[134]6.4276,[135]6.4500,[136]6.4480,[137]6.4417,[138]6.4512,[139]6.4614,
save_imatrix: stored collected data after 140 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[140]6.4745,[141]6.4761,[142]6.4800,[143]6.4706,[144]6.4510,[145]6.4725,[146]6.5015,[147]6.5357,[148]6.5681,[149]6.6067,
save_imatrix: stored collected data after 150 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
[150]6.6392,[151]6.6730,
save_imatrix: stored collected data after 151 chunks in dolphin-2.9.2-Phi-3-Medium-abliterated-IMat-GGUF/imatrix.dat
llama_print_timings: load time = 82696.61 ms
llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: prompt eval time = 560820.89 ms / 77312 tokens ( 7.25 ms per token, 137.86 tokens per second)
llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 615183.59 ms / 77313 tokens
Final estimate: PPL = 6.6730 +/- 0.08813