133 lines
4.3 KiB
Markdown
133 lines
4.3 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- hi
|
||
- en
|
||
base_model: pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct
|
||
tags:
|
||
- hindi
|
||
- indic
|
||
- gguf
|
||
- quantized
|
||
- llama.cpp
|
||
- ollama
|
||
- lm-studio
|
||
- minicpm5
|
||
- instruction-tuned
|
||
library_name: gguf
|
||
pipeline_tag: text-generation
|
||
base_model_relation: quantized
|
||
---
|
||
|
||
# MiniCPM5-1B-Hindi-Instruct v1 — GGUF Quantizations
|
||
|
||
GGUF quantized versions of [`pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct`](https://huggingface.co/pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct) for efficient local inference with [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://ollama.ai), [LM Studio](https://lmstudio.ai), and other GGUF-compatible runtimes.
|
||
|
||
Part of the [🇮🇳 Hindi LLM Series](https://huggingface.co/collections/pankajpandey-dev) by [@pankajpandey-dev](https://huggingface.co/pankajpandey-dev).
|
||
|
||
## Available Quantizations
|
||
|
||
| File | Quant | Size | Recommended Use |
|
||
|------|-------|------|-----------------|
|
||
| `MiniCPM5-1B-Hindi-Instruct-Q3_K_M.gguf` | Q3_K_M | ~560 MB | Mobile, low-RAM devices, fast inference |
|
||
| `MiniCPM5-1B-Hindi-Instruct-Q4_K_M.gguf` | Q4_K_M | ~670 MB | **Recommended** — best size/quality balance |
|
||
| `MiniCPM5-1B-Hindi-Instruct-Q5_K_M.gguf` | Q5_K_M | ~790 MB | Better quality, slightly larger |
|
||
| `MiniCPM5-1B-Hindi-Instruct-Q6_K.gguf` | Q6_K | ~900 MB | Near-lossless quality |
|
||
| `MiniCPM5-1B-Hindi-Instruct-Q8_0.gguf` | Q8_0 | ~1.2 GB | Highest quality, essentially full precision |
|
||
|
||
## Quick Start
|
||
|
||
### llama.cpp
|
||
|
||
```bash
|
||
huggingface-cli download pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct-v1-GGUF MiniCPM5-1B-Hindi-Instruct-Q4_K_M.gguf --local-dir .
|
||
|
||
./llama-cli -m MiniCPM5-1B-Hindi-Instruct-Q4_K_M.gguf \
|
||
-p "नमस्ते! बारिश के दिन पर एक छोटी कविता लिखो।" \
|
||
-n 256 --temp 0.7 --top-p 0.9 --repeat-penalty 1.1
|
||
```
|
||
|
||
### Ollama
|
||
|
||
Create a `Modelfile`:
|
||
FROM ./MiniCPM5-1B-Hindi-Instruct-Q4_K_M.gguf
|
||
PARAMETER temperature 0.7
|
||
PARAMETER top_p 0.9
|
||
PARAMETER repeat_penalty 1.1
|
||
|
||
Then run:
|
||
|
||
```bash
|
||
ollama create hindi-minicpm5 -f Modelfile
|
||
ollama run hindi-minicpm5 "मशीन लर्निंग क्या है?"
|
||
```
|
||
|
||
### LM Studio
|
||
|
||
1. Download any `.gguf` file from this repo
|
||
2. Open LM Studio → Local Models → load the file
|
||
3. Use chat template: ChatML (`<|im_start|>` / `<|im_end|>`)
|
||
|
||
### Python (llama-cpp-python)
|
||
|
||
```python
|
||
from llama_cpp import Llama
|
||
|
||
llm = Llama(
|
||
model_path = "MiniCPM5-1B-Hindi-Instruct-Q4_K_M.gguf",
|
||
n_ctx = 2048,
|
||
n_threads = 4,
|
||
)
|
||
|
||
response = llm.create_chat_completion(
|
||
messages = [
|
||
{"role": "user", "content": "भारत के बारे में एक रोचक तथ्य बताइए।"}
|
||
],
|
||
temperature = 0.7,
|
||
top_p = 0.9,
|
||
max_tokens = 256,
|
||
)
|
||
print(response["choices"][0]["message"]["content"])
|
||
```
|
||
|
||
## Recommended Generation Parameters
|
||
|
||
- **temperature:** 0.7 (range 0.5–0.9)
|
||
- **top_p:** 0.9
|
||
- **repeat_penalty:** 1.1
|
||
- **max_tokens:** 256–512 depending on task
|
||
|
||
## Choosing the Right Quant
|
||
|
||
- **Phone / Raspberry Pi / 2GB RAM:** Q3_K_M or Q4_K_M
|
||
- **Laptop / desktop CPU:** Q4_K_M or Q5_K_M (best default)
|
||
- **Quality-focused workflows:** Q6_K or Q8_0
|
||
- **Research / reproducibility:** Q8_0
|
||
|
||
## Base Model & Training
|
||
|
||
These quants are derived from the full-precision merged 16-bit model at [`pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct`](https://huggingface.co/pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct), which was fine-tuned from [`openbmb/MiniCPM5-1B`](https://huggingface.co/openbmb/MiniCPM5-1B) on AI4Bharat's Hindi instruction datasets.
|
||
|
||
See the [main model card](https://huggingface.co/pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct) for full training details.
|
||
|
||
## Acknowledgements
|
||
|
||
- [OpenBMB](https://huggingface.co/openbmb) — MiniCPM5-1B base model
|
||
- [AI4Bharat](https://huggingface.co/ai4bharat) — `indic-instruct-data-v0.1` (anudesh, dolly)
|
||
- [ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp) — GGUF format & quantization tools
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{pandey2026minicpm5hindigguf,
|
||
title = {MiniCPM5-1B-Hindi-Instruct v1 GGUF},
|
||
author = {Pankaj Pandey},
|
||
year = {2026},
|
||
url = {https://huggingface.co/pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct-v1-GGUF}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
*Part of an ongoing effort to bring strong open-source LLMs to Indian languages. Feedback welcome via the community tab.*
|