jsincn/phi-3-mini-128k-instruct-awq
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
1 Commit 1 Branch 0 Tags
9d2ef0e92057e1cd1338b6f9119b90735c686996
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE

README.md

license
license
mit

This repo includes a Version of Phi-3 that was quantized to AWQ using AutoAWQ. Currently hosting via the TGI docker image fails due to its fallback on AutoModel and that not being compatible with AWQ. Hosting on vLLM is recommended.

To run the model you need to set the trust-remote-code (or similar) flag. While the remote code comes from microsoft (see LICENSE information in the file) you should validate the code yourself before deployment.

Description
Model synced from source: jsincn/phi-3-mini-128k-instruct-awq
Readme 602 KiB
Languages
Python 100%