phi4-mm-gguf/README.md

---
license: mit
base_model: microsoft/Phi-4-multimodal-instruct
tags:
  - phi4
  - phi4mm
  - gguf
  - quantized
  - vision-language
  - audio
  - multimodal
  - llama-cpp
  - lm-studio
language:
  - en
pipeline_tag: visual-question-answering
library_name: gguf
---

# Phi-4-multimodal-instruct GGUF Quantizations

GGUF quantizations of [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) —
a **5.6 B parameter** multimodal model by Microsoft supporting **text, vision (images), and audio** inputs.

Produced with [llama.cpp](https://github.com/ggerganov/llama.cpp) build **b8347** on RTX 5090.

> **License:** [MIT](https://huggingface.co/microsoft/Phi-4-multimodal-instruct/blob/main/LICENSE) — © Microsoft Corporation.
> These quantizations carry the same MIT license as the original model.

---

## Available Files

| File | Quant | Size | BPW | Best for |
|------|-------|------|-----|----------|
| `phi4-mm-f16.gguf` | F16 | 7.17 GB | 16.0 | Re-quantization base, maximum quality |
| `phi4-mm-Q8_0.gguf` | Q8_0 | 3.90 GB | 8.0 | High-end GPU — near-lossless |
| `phi4-mm-Q4_K_M.gguf` | Q4_K_M | 2.37 GB | 5.18 | CPU / constrained VRAM — good quality |
| `mmproj-phi4-mm-f16.gguf` | F16 | 825 MB | 16.0 | Vision encoder — **required for image input** |

> **One mmproj for all:** `mmproj-phi4-mm-f16.gguf` works with every text GGUF above.
> It cannot be quantized further — the CLIP FF layer dimension (4304) is not divisible by 32.

---

## VRAM Requirements

Full GPU offload (`-ngl 99`):

| Configuration | VRAM |
|---------------|------|
| F16 + mmproj-F16 | ~10,000 MiB |
| Q8_0 + mmproj-F16 | ~5,400 MiB |
| Q4_K_M + mmproj-F16 | ~3,500 MiB |

---

## Quality Metrics

### Perplexity (wikitext-2-raw test set, context 512)

| Model | PPL | vs F16 |
|-------|-----|--------|
| F16 (baseline) | 14.9338 ± 0.107 | — |
| Q8_0 | 14.9107 ± 0.106 | −0.15% ✅ lossless |
| Q4_K_M | 16.3183 ± 0.121 | +9.3% |

### Throughput (llama-bench, RTX 5090, pp512 / tg128)

| Model | Prompt (t/s) | Generation (t/s) |
|-------|-------------|-----------------|
| Q8_0 | 21,352 | 247 |
| Q4_K_M | 19,904 | 324 |

### Multimodal Benchmarks (lmms-eval)

*VQA evaluation in progress — will be added when complete. Suite: MMStar, OCRBench, AI2D, MathVista, HallusionBench.*

---

## Usage

### LM Studio (Recommended for Desktop)

> **LM Studio** has native GGUF support including multimodal vision. No command-line needed.

**Text-only:**
1. Open LM Studio → search `Swicked86/phi4-mm-gguf`
2. Download `phi4-mm-Q8_0.gguf` (GPU) or `phi4-mm-Q4_K_M.gguf` (CPU / low VRAM)
3. Load the model → Chat

**With vision (image input):**
1. Download both `phi4-mm-Q8_0.gguf` **and** `mmproj-phi4-mm-f16.gguf`
2. Load the main model in LM Studio
3. In **Model Settings → Multimodal → Vision Model (mmproj)**, browse to `mmproj-phi4-mm-f16.gguf`
4. In Chat, click the image icon to attach a photo and ask questions about it

> The mmproj file is the vision encoder. Without it the model runs text-only.  
> `mmproj-phi4-mm-f16.gguf` is compatible with **all three** text GGUFs.

---

### llama.cpp CLI

#### Step 1 — Download the files

```bash
# Install huggingface-cli if needed
pip install huggingface_hub

# Text + vision (recommended)
huggingface-cli download Swicked86/phi4-mm-gguf phi4-mm-Q8_0.gguf mmproj-phi4-mm-f16.gguf --local-dir ./phi4-mm

# CPU / low-VRAM variant
huggingface-cli download Swicked86/phi4-mm-gguf phi4-mm-Q4_K_M.gguf mmproj-phi4-mm-f16.gguf --local-dir ./phi4-mm
```

Build llama.cpp if you haven't already:
```bash
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON    # omit DGGML_CUDA=ON for CPU-only
cmake --build build --config Release -j$(nproc)
```

---

#### Step 2 — Interactive multimodal chat (images + text)

`llama-mtmd-cli` launches an interactive session. Type your prompt, or prefix it with an image path using `/image`:

```bash
./build/bin/llama-mtmd-cli \
  -m ./phi4-mm/phi4-mm-Q8_0.gguf \
  --mmproj ./phi4-mm/mmproj-phi4-mm-f16.gguf \
  -ngl 99 --threads 16 --ctx-size 8192
```

Inside the session:
```
> /image photo.jpg
Image loaded.
> What is in this image?
[model describes the image]

> /image chart.png
Image loaded.
> Summarise the trend shown in this chart.
[model analyses the chart]

> Explain the previous image again but in French.
[responds without re-loading the image]
```

Single-shot (non-interactive):
```bash
./build/bin/llama-mtmd-cli \
  -m ./phi4-mm/phi4-mm-Q8_0.gguf \
  --mmproj ./phi4-mm/mmproj-phi4-mm-f16.gguf \
  --image photo.jpg \
  -p "Describe this image in detail." \
  -ngl 99 --threads 16 --no-display-prompt
```

CPU (no GPU):
```bash
./build/bin/llama-mtmd-cli \
  -m ./phi4-mm/phi4-mm-Q4_K_M.gguf \
  --mmproj ./phi4-mm/mmproj-phi4-mm-f16.gguf \
  --image photo.jpg \
  -p "What objects are in this photo?" \
  --threads 8
```

---

#### Text-only (no image)

```bash
./build/bin/llama-cli \
  -m ./phi4-mm/phi4-mm-Q8_0.gguf \
  --ctx-size 65536 --flash-attn on --kv-offload -ngl 99 --threads 16 \
  -p "<|system|>You are a helpful assistant.<|end|><|user|>Hello<|end|><|assistant|>"
```

---

### llama-server — OpenAI-compatible API

Serves both text and vision via `/v1/chat/completions`. Useful for integrations (Open WebUI, SillyTavern, Continue.dev, etc.):

```bash
./build/bin/llama-server \
  -m ./phi4-mm/phi4-mm-Q8_0.gguf \
  --mmproj ./phi4-mm/mmproj-phi4-mm-f16.gguf \
  -ngl 99 --threads 16 \
  --ctx-size 8192 --parallel 4 \
  --port 8080 --host 127.0.0.1
```

**Text query (curl):**
```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "phi4-mm",
    "messages": [{"role": "user", "content": "Explain the Pythagorean theorem."}],
    "max_tokens": 300
  }'
```

**Image query (curl, base64):**
```bash
IMAGE_B64=$(base64 -w0 photo.jpg)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"phi4-mm\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/jpeg;base64,${IMAGE_B64}\"}},
        {\"type\": \"text\", \"text\": \"What is in this image?\"}
      ]
    }],
    \"max_tokens\": 300
  }"
```

**Image query (Python, openai SDK):**
```python
import base64, httpx
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

with open("photo.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
            {"type": "text", "text": "Describe this image."},
        ],
    }],
    max_tokens=300,
)
print(response.choices[0].message.content)
```

#### Modality support summary for GGUF (llama.cpp / LM Studio)

| Modality | Supported | Notes |
|----------|-----------|-------|
| Text | ✅ | All three GGUFs |
| Vision (images) | ✅ | Requires `mmproj-phi4-mm-f16.gguf` + `--mmproj` flag |
| Audio / Speech | ❌ | Not available — see below |

> **Audio is not supported in the GGUF files.** phi4-mm's speech capability uses a custom conformer-based audio encoder (24 conformer blocks, initialized from a proprietary AED ASR model) plus a rank-320 speech LoRA applied to the language decoder. The GGUF conversion pipeline (`convert_hf_to_gguf.py`) only exports the text transformer and the SigLIP vision encoder (mmproj) — the audio encoder tensors are not extracted. There is currently no `audioproj` equivalent in llama.cpp for phi4-mm.
>
> **To use audio/speech transcription, use the vLLM path below** with the original bf16 safetensors model.

---

### vLLM — Full Multimodal (Image + Audio, bf16 Safetensors)

> **Use vLLM when you want maximum quality and full multimodal support** (images + audio) from the original bf16 safetensors weights. For constrained hardware, use the GGUF options above instead.

**Requirements:** Python 3.10+, CUDA GPU with ~16 GB VRAM, vLLM 0.7.0+

#### 1 — Install vLLM and download the model

```bash
python3 -m venv ~/.vllm-env
source ~/.vllm-env/bin/activate
pip install --upgrade pip
pip install vllm

# Download the original safetensors model (~14 GB, 3 shards)
huggingface-cli login          # paste your HF token if the model is gated
huggingface-cli download microsoft/Phi-4-multimodal-instruct \
    --local-dir ~/phi4-mm-hf
```

#### 2 — Launch the vLLM server

```bash
source ~/.vllm-env/bin/activate
python -m vllm.entrypoints.openai.api_server \
  --model ~/phi4-mm-hf \
  --dtype bfloat16 \
  --trust-remote-code \
  --max-model-len 65520 \
  --kv-cache-memory-bytes 8G \
  --limit-mm-per-prompt '{"image": 3, "audio": 3}' \
  --enable-auto-tool-choice \
  --tool-call-parser phi4_mini_json \
  --port 8080 \
  --host 127.0.0.1 \
  --served-model-name phi4-mm
```

| Flag | Value | Notes |
|------|-------|-------|
| `--dtype bfloat16` | — | Native dtype — do not change to float16 |
| `--max-model-len 65520` | — | Stable context ceiling for phi4-mm (131 K nominal) |
| `--kv-cache-memory-bytes` | `8G` | Tune down to `4G` on 12–16 GB GPUs |
| `--limit-mm-per-prompt` | `{"image":3,"audio":3}` | Max attachments per request |
| `--tool-call-parser phi4_mini_json` | — | phi4-mm emits `functools[...]` not Hermes — required for tool calling |
| `--trust-remote-code` | — | Required for phi4-mm's custom modelling code |

> **Official vLLM LoRA flags:** Microsoft's published vLLM command includes explicit LoRA adapter flags to activate
> the rank-320 vision and speech adapters stored in separate subfolders of the model directory:
> ```bash
> --enable-lora \
> --max-lora-rank 320 \
> --lora-extra-vocab-size 0 \
> --max-loras 2 \
> --lora-modules speech=~/phi4-mm-hf/speech-lora vision=~/phi4-mm-hf/vision-lora
> ```
> If you experience degraded vision or audio quality, add these flags to the launch command above.

Wait for the server to finish loading (~60 s):
```bash
curl http://localhost:8080/health   # → {"status":"ok"}
```

#### 3 — Send requests (OpenAI-compatible API)

**Text (Python):**
```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")
response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=256,
)
print(response.choices[0].message.content)
```

**Image (Python):**
```python
import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

with open("photo.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
            {"type": "text", "text": "Describe this image in detail."},
        ],
    }],
    max_tokens=512,
)
print(response.choices[0].message.content)
```

**Image (curl, base64):**
```bash
IMAGE_B64=$(base64 -w0 photo.jpg)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"phi4-mm\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/jpeg;base64,${IMAGE_B64}\"}},
        {\"type\": \"text\", \"text\": \"What is in this image?\"}
      ]
    }],
    \"max_tokens\": 300
  }"
```

**Audio — speech transcription / understanding (Python):**
```python
import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

with open("audio.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{
        "role": "user",
        "content": [
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
            {"type": "text", "text": "Transcribe this audio."},
        ],
    }],
    max_tokens=512,
)
print(response.choices[0].message.content)
```

> phi4-mm uses a custom conformer-based audio encoder with rank-320 speech LoRA — no separate ASR model needed.
> Supported formats: `wav`, `mp3`, `ogg`, `flac`.

#### Tool calling

phi4-mm emits tool calls as `functools[{"name":"...","arguments":{...}}]`.
The `--tool-call-parser phi4_mini_json` flag (vLLM 0.7+) handles this format automatically.
For a complete chat template that injects tools into phi4-mm's native `<|tool|>...<|/tool|>` block,
see [`deploy/wsl-vllm/phi4-mm-tool-template.jinja`](deploy/wsl-vllm/phi4-mm-tool-template.jinja)
in the [companion repo](https://github.com/Swicked86/LLM_Creation).

---

### Ollama (CPU / NUC / edge)

See the [`deploy/`](deploy/) folder for a complete `Modelfile`, NUC install script, and OpenClaw integration config.

```
FROM ./phi4-mm-Q4_K_M.gguf
PARAMETER num_ctx 8192
PARAMETER num_thread 8
PARAMETER num_gpu 0
PARAMETER flash_attn false
PARAMETER temperature 0.7
```

---

## Architecture

| Property | Value |
|----------|-------|
| Base model | Phi-4-Mini (3.8 B LLM backbone) |
| Total parameters | ~5.6 B |
| GGUF arch | `phi3` |
| Context length | 128 K tokens (131,072) |
| Modalities | Text, Vision (SigLIP-400M), Audio/Speech |

The vision encoder (`mmproj-phi4-mm-f16.gguf`) is a SigLIP-400M encoder finetuned
with LLM2CLIP, with a 2-layer MLP projector. Audio/speech is **not embedded in the GGUF** — see the audio limitation callout above.

---

## Conversion Details

| Item | Value |
|------|-------|
| Converter | `llama.cpp convert_hf_to_gguf.py` (text) + custom mmproj converter |
| llama.cpp build | b8347 / `fc350fdf9` |
| Source | [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) |
| Hardware | RTX 5090 32 GB, CUDA 12.0, WSL2 Ubuntu 24.04 |

Quantization commands:

```bash
# Q8_0
llama-quantize phi4-mm-f16.gguf phi4-mm-Q8_0.gguf Q8_0

# Q4_K_M
llama-quantize phi4-mm-f16.gguf phi4-mm-Q4_K_M.gguf Q4_K_M
```

---

## NUC / Edge Deployment

The `Q4_K_M + mmproj-phi4-mm-f16.gguf` combination (~3,500 MiB VRAM) fits on:

- Intel NUC 13/14 Pro (Intel Arc iGPU, 4–8 GB shared VRAM)
- Systems with 8 GB unified memory (Apple Silicon M-series, etc.)

See [`deploy/`](deploy/) for install scripts and configuration.

---

## Related

- [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) — original model weights
- [llama.cpp](https://github.com/ggerganov/llama.cpp) — inference engine
- [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval) — multimodal evaluation harness