63 lines
2.7 KiB
Markdown
63 lines
2.7 KiB
Markdown
# Model Quantization with llama.cpp
|
|
|
|
This README explains how to use the `quantize_models.sh` script to create quantized versions of your GGUF models.
|
|
|
|
## Prerequisites
|
|
|
|
- llama.cpp must be cloned and built in this directory
|
|
- You need a base GGUF model (default is `osmosis-mcp-4B-BF16.gguf`)
|
|
|
|
## How to Use
|
|
|
|
1. Make sure your base model is in the current directory
|
|
2. Run the script:
|
|
|
|
```bash
|
|
./quantize_models.sh
|
|
```
|
|
|
|
## Supported Quantization Formats
|
|
|
|
The script will create the following quantized versions:
|
|
|
|
| Format | Description | Approximate Size |
|
|
|----------|-----------------------------------------------------------------|-------------------|
|
|
| Q4_K_S | 4-bit quantization, smaller size | ~29% of original |
|
|
| Q5_K_M | 5-bit quantization, medium size | ~34% of original |
|
|
| Q5_K_S | 5-bit quantization, smaller size | ~33% of original |
|
|
| Q6_K | 6-bit quantization, balanced quality and size | ~38% of original |
|
|
| IQ4_XS | Improved 4-bit non-linear quantization, extra small | ~27% of original |
|
|
| Q8_0 | 8-bit quantization, highest quality | ~50% of original |
|
|
| Q2_K | 2-bit quantization, extremely small | ~18% of original |
|
|
| Q3_K_L | 3-bit quantization, larger size | ~25% of original |
|
|
| Q3_K_M | 3-bit quantization, medium size | ~23% of original |
|
|
| Q3_K_S | 3-bit quantization, smaller size | ~21% of original |
|
|
| Q4_K_M | 4-bit quantization, medium size | ~28% of original |
|
|
|
|
## Customizing the Script
|
|
|
|
If you want to quantize a different base model, edit the `INPUT_MODEL` variable in the script:
|
|
|
|
```bash
|
|
# Input model file
|
|
INPUT_MODEL="your-model-file.gguf"
|
|
```
|
|
|
|
## Time and Space Requirements
|
|
|
|
- Quantization can take from several minutes to an hour depending on your hardware
|
|
- Make sure you have enough free disk space for all the output models
|
|
- The total disk space required will be approximately 3x the size of the original model
|
|
|
|
## Using the Quantized Models
|
|
|
|
Each quantized model can be used with llama.cpp tools:
|
|
|
|
```bash
|
|
llama.cpp/build/bin/llama-cli -m osmosis-mcp-4b.Q4_K_S.gguf -p "Your prompt here"
|
|
```
|
|
|
|
Choose the quantization format based on your needs:
|
|
- Smaller quantization (Q2_K, Q3_K_S) for limited hardware resources
|
|
- Medium quantization (Q4_K_M, Q5_K_S) for balanced performance
|
|
- Larger quantization (Q6_K, Q8_0) for highest quality with sufficient hardware |