58 lines
1.5 KiB
Markdown
58 lines
1.5 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
- hi
|
||
|
|
- te
|
||
|
|
- kn
|
||
|
|
- ta
|
||
|
|
base_model: Qwen/Qwen2.5-3B-Instruct
|
||
|
|
tags:
|
||
|
|
- laabam-ai
|
||
|
|
- qwen2.5
|
||
|
|
- multilingual
|
||
|
|
- indic
|
||
|
|
- fine-tuned
|
||
|
|
- qlora
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
# Laabam AI 3B v1
|
||
|
|
|
||
|
|
A multilingual AI assistant fine-tuned from Qwen2.5-3B-Instruct using QLoRA.
|
||
|
|
|
||
|
|
## Training Details
|
||
|
|
|
||
|
|
- **Base model**: Qwen2.5-3B-Instruct (4-bit quantized)
|
||
|
|
- **Method**: QLoRA (r=16, alpha=32)
|
||
|
|
- **Training**: 4 epochs on ~98K samples (final train loss 0.465)
|
||
|
|
- **Languages**: English, Hindi, Telugu, Kannada, Tamil
|
||
|
|
- **Domains**: General instruction following, coding, reasoning, safety alignment, Indic languages
|
||
|
|
|
||
|
|
## Training Epochs
|
||
|
|
|
||
|
|
| Epoch | Dataset Size | Learning Rate | Focus |
|
||
|
|
|-------|-------------|---------------|-------|
|
||
|
|
| 1 | 36K | 2e-4 | Core instruction following |
|
||
|
|
| 2 | 36K | 5e-5 | Continued refinement |
|
||
|
|
| 3 | 98K | 2e-5 | Expanded: safety, Indic languages, clean instructions |
|
||
|
|
| 4 | 98K | 1e-5 | Careful refinement (low LR, anti-forgetting) |
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("laabamone/laabam-ai-3b-v1")
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained("laabamone/laabam-ai-3b-v1")
|
||
|
|
|
||
|
|
messages = [{"role": "user", "content": "Hello, who are you?"}]
|
||
|
|
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
|
||
|
|
outputs = model.generate(inputs, max_new_tokens=256)
|
||
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
|
|
```
|
||
|
|
|
||
|
|
## License
|
||
|
|
|
||
|
|
Apache 2.0
|