157 lines
4.8 KiB
Markdown
157 lines
4.8 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- hi
|
||
- en
|
||
base_model: openbmb/MiniCPM5-1B
|
||
tags:
|
||
- hindi
|
||
- indic
|
||
- instruction-tuned
|
||
- minicpm5
|
||
- text-generation
|
||
- conversational
|
||
- lora
|
||
- unsloth
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# MiniCPM5-1B-Hindi-Instruct
|
||
|
||
A Hindi instruction-tuned variant of [openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B), fine-tuned for Hindi (हिंदी) conversational and instruction-following tasks.
|
||
|
||
Part of the [🇮🇳 Hindi LLM Series](https://huggingface.co/collections/pankajpandey-dev) by [@pankajpandey-dev](https://huggingface.co/pankajpandey-dev).
|
||
|
||
## Model Details
|
||
|
||
- **Base model:** openbmb/MiniCPM5-1B (1.1B parameters)
|
||
- **Language:** Hindi (हिंदी), with English understanding retained from the base
|
||
- **Fine-tuning method:** LoRA (r=32, alpha=64) merged into base weights
|
||
- **Training framework:** [Unsloth](https://github.com/unslothai/unsloth) + TRL
|
||
- **License:** Apache 2.0
|
||
|
||
## Training Data
|
||
|
||
Fine-tuned on **4,000 high-quality Hindi instruction examples** sampled from:
|
||
|
||
- [`ai4bharat/indic-instruct-data-v0.1`](https://huggingface.co/datasets/ai4bharat/indic-instruct-data-v0.1) — `anudesh` (Hindi split): native crowd-sourced Hindi instructions
|
||
- [`ai4bharat/indic-instruct-data-v0.1`](https://huggingface.co/datasets/ai4bharat/indic-instruct-data-v0.1) — `dolly` (Hindi split, filtered to chrF ≥ 60): broad instruction variety
|
||
|
||
All examples ≤ 2048 tokens, formatted with the MiniCPM5 ChatML template.
|
||
|
||
## Training Configuration
|
||
|
||
| Hyperparameter | Value |
|
||
|----------------|-------|
|
||
| LoRA rank | 32 |
|
||
| LoRA alpha | 64 |
|
||
| LoRA dropout | 0.0 |
|
||
| Target modules | q, k, v, o, gate, up, down |
|
||
| Batch size (effective) | 16 |
|
||
| Learning rate | 2e-4 |
|
||
| LR scheduler | cosine |
|
||
| Warmup steps | 15 |
|
||
| Epochs | 2 |
|
||
| Total steps | 500 |
|
||
| Precision | fp16 (4-bit base) |
|
||
| Hardware | NVIDIA Tesla T4 (Colab) |
|
||
| Training time | ~60 minutes |
|
||
| Final training loss | 1.108 |
|
||
|
||
## Usage
|
||
|
||
### With Transformers
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
model_id = "pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct"
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_id,
|
||
torch_dtype=torch.float16,
|
||
device_map="auto",
|
||
trust_remote_code=True,
|
||
)
|
||
|
||
messages = [
|
||
{"role": "user", "content": "नमस्ते! बारिश के दिन पर एक छोटी कविता लिखो।"}
|
||
]
|
||
|
||
inputs = tokenizer.apply_chat_template(
|
||
messages,
|
||
add_generation_prompt=True,
|
||
return_tensors="pt",
|
||
).to(model.device)
|
||
|
||
outputs = model.generate(
|
||
inputs,
|
||
max_new_tokens=256,
|
||
temperature=0.7,
|
||
top_p=0.9,
|
||
do_sample=True,
|
||
repetition_penalty=1.1,
|
||
)
|
||
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
### Recommended Generation Parameters
|
||
|
||
- **temperature:** 0.7 (lower = more focused, higher = more creative)
|
||
- **top_p:** 0.9
|
||
- **repetition_penalty:** 1.1
|
||
- **max_new_tokens:** 256–512 depending on task
|
||
|
||
### LoRA Adapter Only
|
||
|
||
If you prefer to load the LoRA adapter on top of the base model (~85 MB vs 2.2 GB), it's available in the `lora_adapter/` folder of this repo:
|
||
|
||
```python
|
||
from peft import PeftModel
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
base = AutoModelForCausalLM.from_pretrained("openbmb/MiniCPM5-1B", trust_remote_code=True)
|
||
model = PeftModel.from_pretrained(base, "pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct", subfolder="lora_adapter")
|
||
```
|
||
|
||
## Example Outputs
|
||
|
||
**Prompt:** बारिश के दिन पर एक छोटी कविता लिखो।
|
||
**Response:** *(creative Hindi poetry generation)*
|
||
|
||
**Prompt:** मशीन लर्निंग क्या है? सरल हिंदी में समझाइए।
|
||
**Response:** *(simplified Hindi explanation of ML)*
|
||
|
||
**Prompt:** नमस्ते! अपना परिचय दीजिए।
|
||
**Response:** *(conversational Hindi self-introduction)*
|
||
|
||
## Quantized Versions (GGUF)
|
||
|
||
For running locally with llama.cpp, Ollama, LM Studio, or other GGUF-compatible inference engines.
|
||
|
||
## Acknowledgements
|
||
|
||
- [OpenBMB](https://huggingface.co/openbmb) for the MiniCPM5-1B base model
|
||
- [AI4Bharat](https://huggingface.co/ai4bharat) (IIT Madras) for the indic-instruct-data dataset
|
||
- [Unsloth](https://github.com/unslothai/unsloth) for the training framework
|
||
|
||
## Citation
|
||
|
||
If you use this model in your work, please cite:
|
||
|
||
```bibtex
|
||
@misc{pandey2026minicpm5hindi,
|
||
title = {MiniCPM5-1B-Hindi-Instruct},
|
||
author = {Pankaj Pandey},
|
||
year = {2026},
|
||
url = {https://huggingface.co/pankajpandey-dev/MiniCPM5-1B-Hindi-Instruct}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
*Part of an ongoing effort to bring strong open-source LLMs to Indian languages. Feedback and contributions welcome via the community tab.*
|