86 lines
2.8 KiB
Markdown
86 lines
2.8 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- hi
|
||
|
|
- en
|
||
|
|
- hne
|
||
|
|
tags:
|
||
|
|
- hinglish
|
||
|
|
- hindi
|
||
|
|
- indian-languages
|
||
|
|
- fused
|
||
|
|
- pytorch
|
||
|
|
- fp16
|
||
|
|
- qwen
|
||
|
|
- qwen2
|
||
|
|
- bharat
|
||
|
|
- indic-nlp
|
||
|
|
base_model: Qwen/Qwen2.5-1.5B
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
# Bharat-Tiny-LLM (fused · fp16)
|
||
|
|
|
||
|
|
This is the **full-precision fused model** for [Bharat-Tiny-LLM](https://huggingface.co/eulogik/Bharat-Tiny-LLM) —
|
||
|
|
the LoRA adapter merged into the base [Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) weights,
|
||
|
|
in PyTorch `float16`.
|
||
|
|
|
||
|
|
Use this repo when you want to:
|
||
|
|
- run inference on CPU / CUDA with `transformers`,
|
||
|
|
- fine-tune further, or
|
||
|
|
- produce your own quantized builds (GGUF, MLX, etc.).
|
||
|
|
|
||
|
|
> Built by [eulogik](https://eulogik.com)
|
||
|
|
|
||
|
|
## For most users
|
||
|
|
|
||
|
|
You probably want a smaller, ready-to-run build instead:
|
||
|
|
|
||
|
|
| Build | Repo | Size | Use |
|
||
|
|
|-------|------|------|-----|
|
||
|
|
| **MLX 4-bit** (edge / Apple Silicon) | [`eulogik/Bharat-Tiny-LLM`](https://huggingface.co/eulogik/Bharat-Tiny-LLM) | ~880 MB | Recommended for Mac / on-device |
|
||
|
|
| **GGUF Q4_K_M** (llama.cpp, Android / Pi / CPU) | [`eulogik/Bharat-Tiny-LLM-GGUF`](https://huggingface.co/eulogik/Bharat-Tiny-LLM-GGUF) | ~1.06 GB | Cross-platform, llama.cpp |
|
||
|
|
| **PyTorch fp16** (this repo) | `eulogik/Bharat-Tiny-LLM-fused` | ~3.3 GB | Server / fine-tuning base |
|
||
|
|
|
||
|
|
## Quick start (transformers)
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("eulogik/Bharat-Tiny-LLM-fused")
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained("eulogik/Bharat-Tiny-LLM-fused")
|
||
|
|
|
||
|
|
messages = [{"role": "user", "content": "Chai peete hain?"}]
|
||
|
|
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
|
||
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
||
|
|
out = model.generate(
|
||
|
|
**inputs,
|
||
|
|
max_new_tokens=256,
|
||
|
|
temperature=0.3,
|
||
|
|
top_p=0.85,
|
||
|
|
repetition_penalty=1.25,
|
||
|
|
no_repeat_ngram_size=3,
|
||
|
|
do_sample=True,
|
||
|
|
)
|
||
|
|
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
|
||
|
|
```
|
||
|
|
|
||
|
|
> ⚠️ **Generation config matters.** The base Qwen2.5-1.5B emits garbled out-of-script tokens
|
||
|
|
> at high temperature. Always use `temperature ≈ 0.3` + `repetition_penalty ≥ 1.25` +
|
||
|
|
> `no_repeat_ngram_size = 3`. The [`bharat-tiny-llm`](https://pypi.org/project/bharat-tiny-llm/)
|
||
|
|
> PyPI package applies these for you.
|
||
|
|
|
||
|
|
## Links
|
||
|
|
|
||
|
|
- 🤗 Edge model (MLX): https://huggingface.co/eulogik/Bharat-Tiny-LLM
|
||
|
|
- 🤗 GGUF (llama.cpp): https://huggingface.co/eulogik/Bharat-Tiny-LLM-GGUF
|
||
|
|
- 🚀 Demo: https://huggingface.co/spaces/eulogik/Bharat-Tiny-LLM
|
||
|
|
- 💻 Source: https://github.com/eulogik/Bharat-Tiny-LLM
|
||
|
|
- 📦 PyPI: https://pypi.org/project/bharat-tiny-llm/
|
||
|
|
- 🏢 Built by [eulogik](https://eulogik.com)
|
||
|
|
|
||
|
|
## License
|
||
|
|
|
||
|
|
Apache-2.0 (base Qwen2.5-1.5B weights Apache-2.0; LoRA adapter Apache-2.0).
|