MiniCPM5-1B-Hindi-Instruct/README.md

---
license: apache-2.0
language:
- hi
- en
base_model: openbmb/MiniCPM5-1B
tags:
- hindi
- indic
- instruction-tuned
- minicpm5
- text-generation
- conversational
- lora
- unsloth
library_name: transformers
pipeline_tag: text-generation
---

# MiniCPM5-1B-Hindi-Instruct

A Hindi instruction-tuned variant of [openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B), fine-tuned for Hindi (हिंदी) conversational and instruction-following tasks.

Part of the [🇮🇳 Hindi LLM Series](https://huggingface.co/collections/pankajpandey-dev) by [@pankajpandey-dev](https://huggingface.co/pankajpandey-dev).

## Model Details

- **Base model:** openbmb/MiniCPM5-1B (1.1B parameters)
- **Language:** Hindi (हिंदी), with English understanding retained from the base
- **Fine-tuning method:** LoRA (r=32, alpha=64) merged into base weights
- **Training framework:** [Unsloth](https://github.com/unslothai/unsloth) + TRL
- **License:** Apache 2.0

## Training Data

Fine-tuned on **4,000 high-quality Hindi instruction examples** sampled from:

- [`ai4bharat/indic-instruct-data-v0.1`](https://huggingface.co/datasets/ai4bharat/indic-instruct-data-v0.1) — `anudesh` (Hindi split): native crowd-sourced Hindi instructions
- [`ai4bharat/indic-instruct-data-v0.1`](https://huggingface.co/datasets/ai4bharat/indic-instruct-data-v0.1) — `dolly` (Hindi split, filtered to chrF ≥ 60): broad instruction variety

All examples ≤ 2048 tokens, formatted with the MiniCPM5 ChatML template.

## Training Configuration

| Hyperparameter | Value |
|----------------|-------|
| LoRA rank | 32 |
| LoRA alpha | 64 |
| LoRA dropout | 0.0 |
| Target modules | q, k, v, o, gate, up, down |
| Batch size (effective) | 16 |
| Learning rate | 2e-4 |
| LR scheduler | cosine |
| Warmup steps | 15 |
| Epochs | 2 |
| Total steps | 500 |
| Precision | fp16 (4-bit base) |
| Hardware | NVIDIA Tesla T4 (Colab) |
| Training time | ~60 minutes |
| Final training loss | 1.108 |

## Usage

### With Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {"role": "user", "content": "नमस्ते! बारिश के दिन पर एक छोटी कविता लिखो।"}
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    repetition_penalty=1.1,
)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
```

### Recommended Generation Parameters

- **temperature:** 0.7 (lower = more focused, higher = more creative)
- **top_p:** 0.9
- **repetition_penalty:** 1.1
- **max_new_tokens:** 256–512 depending on task

### LoRA Adapter Only

If you prefer to load the LoRA adapter on top of the base model (~85 MB vs 2.2 GB), it's available in the `lora_adapter/` folder of this repo:

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("openbmb/MiniCPM5-1B", trust_remote_code=True)
model = PeftModel.from_pretrained(base, "pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct", subfolder="lora_adapter")
```

## Example Outputs

**Prompt:** बारिश के दिन पर एक छोटी कविता लिखो।
**Response:** *(creative Hindi poetry generation)*

**Prompt:** मशीन लर्निंग क्या है? सरल हिंदी में समझाइए।
**Response:** *(simplified Hindi explanation of ML)*

**Prompt:** नमस्ते! अपना परिचय दीजिए।
**Response:** *(conversational Hindi self-introduction)*

## Quantized Versions (GGUF)

For running locally with llama.cpp, Ollama, LM Studio, or other GGUF-compatible inference engines.

## Acknowledgements

- [OpenBMB](https://huggingface.co/openbmb) for the MiniCPM5-1B base model
- [AI4Bharat](https://huggingface.co/ai4bharat) (IIT Madras) for the indic-instruct-data dataset
- [Unsloth](https://github.com/unslothai/unsloth) for the training framework

## Citation

If you use this model in your work, please cite:

```bibtex
@misc{pandey2026minicpm5hindi,
  title  = {MiniCPM5-1B-Hindi-Instruct},
  author = {Pankaj Pandey},
  year   = {2026},
  url    = {https://huggingface.co/pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct}
}
```

---

*Part of an ongoing effort to bring strong open-source LLMs to Indian languages. Feedback and contributions welcome via the community tab.*