99 lines
3.5 KiB
Markdown
99 lines
3.5 KiB
Markdown
---
|
|
license: other
|
|
language:
|
|
- en
|
|
pipeline_tag: text-generation
|
|
inference: false
|
|
tags:
|
|
- transformers
|
|
- gguf
|
|
- imatrix
|
|
- Phi-3.5-mini-instruct
|
|
---
|
|
Quantizations of https://huggingface.co/microsoft/Phi-3.5-mini-instruct
|
|
|
|
|
|
### Inference Clients/UIs
|
|
* [llama.cpp](https://github.com/ggerganov/llama.cpp)
|
|
* [JanAI](https://github.com/janhq/jan)
|
|
* [KoboldCPP](https://github.com/LostRuins/koboldcpp)
|
|
* [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
|
|
* [ollama](https://github.com/ollama/ollama)
|
|
* [GPT4All](https://github.com/nomic-ai/gpt4all)
|
|
|
|
---
|
|
|
|
# From original readme
|
|
|
|
Phi-3.5-mini is a lightweight, state-of-the-art open model built upon datasets used for Phi-3 - synthetic data and filtered publicly available websites - with a focus on very high-quality, reasoning dense data. The model belongs to the Phi-3 model family and supports 128K token context length. The model underwent a rigorous enhancement process, incorporating both supervised fine-tuning, proximal policy optimization, and direct preference optimization to ensure precise instruction adherence and robust safety measures.
|
|
|
|
## Usage
|
|
|
|
### Requirements
|
|
Phi-3 family has been integrated in the `4.43.0` version of `transformers`. The current `transformers` version can be verified with: `pip list | grep transformers`.
|
|
|
|
Examples of required packages:
|
|
```
|
|
flash_attn==2.5.8
|
|
torch==2.3.1
|
|
accelerate==0.31.0
|
|
transformers==4.43.0
|
|
```
|
|
|
|
Phi-3.5-mini-instruct is also available in [Azure AI Studio](https://aka.ms/try-phi3.5mini)
|
|
|
|
### Tokenizer
|
|
|
|
Phi-3.5-mini-Instruct supports a vocabulary size of up to `32064` tokens. The [tokenizer files](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/blob/main/added_tokens.json) already provide placeholder tokens that can be used for downstream fine-tuning, but they can also be extended up to the model's vocabulary size.
|
|
|
|
### Input Formats
|
|
Given the nature of the training data, the Phi-3.5-mini-instruct model is best suited for prompts using the chat format as follows:
|
|
|
|
```
|
|
<|system|>
|
|
You are a helpful assistant.<|end|>
|
|
<|user|>
|
|
How to explain Internet for a medieval knight?<|end|>
|
|
<|assistant|>
|
|
```
|
|
|
|
### Loading the model locally
|
|
After obtaining the Phi-3.5-mini-instruct model checkpoint, users can use this sample code for inference.
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
|
|
|
|
torch.random.manual_seed(0)
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
"microsoft/Phi-3.5-mini-instruct",
|
|
device_map="cuda",
|
|
torch_dtype="auto",
|
|
trust_remote_code=True,
|
|
)
|
|
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3.5-mini-instruct")
|
|
|
|
messages = [
|
|
{"role": "system", "content": "You are a helpful AI assistant."},
|
|
{"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"},
|
|
{"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."},
|
|
{"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"},
|
|
]
|
|
|
|
pipe = pipeline(
|
|
"text-generation",
|
|
model=model,
|
|
tokenizer=tokenizer,
|
|
)
|
|
|
|
generation_args = {
|
|
"max_new_tokens": 500,
|
|
"return_full_text": False,
|
|
"temperature": 0.0,
|
|
"do_sample": False,
|
|
}
|
|
|
|
output = pipe(messages, **generation_args)
|
|
print(output[0]['generated_text'])
|
|
``` |