32 lines
926 B
Markdown
32 lines
926 B
Markdown
|
---
|
||
|
|
license: mit
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
base_model:
|
||
|
|
- microsoft/Phi-3-mini-128k-instruct
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
## GGUF quantized version of Phi-3 Model (128k-instruct mini)
|
||
|
|
|
||
|
|
project original [source](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) (base model)
|
||
|
|
|
||
|
|
Q_2 (not nice)
|
||
|
|
|
||
|
|
Q_3 (acceptable)
|
||
|
|
|
||
|
|
Q_4 family is recommanded (good for running with CPU as well)
|
||
|
|
|
||
|
|
Q_5 (good in general)
|
||
|
|
|
||
|
|
Q_6 is good also; if you want a better result; take this one instead of Q_5
|
||
|
|
|
||
|
|
Q_8 which is very good; need a reasonable size of RAM otherwise you might expect a long wait
|
||
|
|
|
||
|
|
16-bit and 32-bit are also provided here for research perspectives; since the file size (16bit) is similar to the original safetensors; once you have a GPU, go ahead with the safetensors, pretty much the same
|
||
|
|
|
||
|
|
### how to run it
|
||
|
|
|
||
|
|
use any connector for interacting with gguf; i.e., [gguf-connector](https://pypi.org/project/gguf-connector/)
|
||
|
|
|
||
|
|
welcome to ai era
|