61 lines
1.9 KiB
Markdown
61 lines
1.9 KiB
Markdown
|
---
|
||
|
|
tags:
|
||
|
|
- experimental
|
||
|
|
- testing
|
||
|
|
- gguf
|
||
|
|
- roleplay
|
||
|
|
- quantized
|
||
|
|
- mistral
|
||
|
|
- text-generation-inference
|
||
|
|
---
|
||
|
|
**These are quants for an experimental model.**
|
||
|
|
|
||
|
|
"Q4_K_M", "Q4_K_S", "IQ4_XS", "Q5_K_M", "Q5_K_S",
|
||
|
|
"Q6_K", "Q8_0", "IQ3_M", "IQ3_S", "IQ3_XXS"
|
||
|
|
|
||
|
|
Original model weights: <br> https://huggingface.co/Nitral-AI/Eris_PrimeV4-Vision-7B
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
# Vision/multimodal capabilities:
|
||
|
|
|
||
|
|
<details><summary>
|
||
|
|
Click here to see how this would work in practice in a roleplay chat.
|
||
|
|
</summary>
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
</details><br>
|
||
|
|
|
||
|
|
<details><summary>
|
||
|
|
Click here to see what your SillyTavern Image Captions extension settings should look like.
|
||
|
|
</summary>
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
</details><br>
|
||
|
|
|
||
|
|
**If you want to use vision functionality:**
|
||
|
|
|
||
|
|
* Make sure you are using the latest version of [KoboldCpp](https://github.com/LostRuins/koboldcpp).
|
||
|
|
|
||
|
|
To use the multimodal capabilities of this model, such as **vision**, you also need to load the specified **mmproj** file, you can get it [here](https://huggingface.co/cjpais/llava-1.6-mistral-7b-gguf/blob/main/mmproj-model-f16.gguf), it's also hosted in this repository inside the **mmproj** folder.
|
||
|
|
|
||
|
|
* You can load the **mmproj** by using the corresponding section in the interface:
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
* For CLI users, you can load the **mmproj file** by adding the respective flag to your usual command:
|
||
|
|
|
||
|
|
```
|
||
|
|
--mmproj your-mmproj-file.gguf
|
||
|
|
```
|
||
|
|
|
||
|
|
# Quantization information:
|
||
|
|
|
||
|
|
**Steps performed:**
|
||
|
|
|
||
|
|
```
|
||
|
|
Base⇢ GGUF(F16)⇢ Imatrix-Data(F16)⇢ GGUF(Imatrix-Quants)
|
||
|
|
```
|
||
|
|
*Using the latest llama.cpp at the time.*
|