初始化项目,由ModelHub XC社区提供模型
Model: Tamil-ai/tamil-qwen25-7b-instruct Source: Original Platform
This commit is contained in:
125
README.md
Normal file
125
README.md
Normal file
@@ -0,0 +1,125 @@
|
||||
---
|
||||
language:
|
||||
- ta
|
||||
- en
|
||||
license: apache-2.0
|
||||
base_model: Qwen/Qwen2.5-7B-Instruct
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- tamil
|
||||
- qwen2
|
||||
- qlora
|
||||
- instruction-tuning
|
||||
- morphology
|
||||
- dravidian
|
||||
datasets:
|
||||
- Tamil-ai/samacheer-kalvi-tamil
|
||||
model-index:
|
||||
- name: Tamil-Qwen2.5-7B-Instruct
|
||||
results: []
|
||||
---
|
||||
|
||||
# Tamil-Qwen2.5-7B-Instruct
|
||||
|
||||
A Tamil-specialized instruction-tuned LLM built on Qwen2.5-7B-Instruct using QLoRA fine-tuning on 150K deduplicated Tamil instruction pairs.
|
||||
|
||||
**Paper:** *"A Thousand Language Problem: Morphological Understanding in Linguistic AI"*
|
||||
|
||||
## Model Details
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Base model | [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) |
|
||||
| Parameters | 7.6B |
|
||||
| Method | QLoRA (r=64, alpha=128, dropout=0.05) |
|
||||
| Training data | 150K deduplicated Tamil instruction-response pairs |
|
||||
| Tokenizer efficiency | 4.62x ratio (best among tested models for Tamil) |
|
||||
| Compute | RunPod RTX 5090, ~$5 total cost |
|
||||
| Sequence length | 1024 |
|
||||
| Batch size | 32 (effective) |
|
||||
| Epochs | 1 |
|
||||
|
||||
## Training Data
|
||||
|
||||
150,000 deduplicated instruction-response pairs from 5 Tamil datasets:
|
||||
- Tamil Alpaca
|
||||
- Tamil Orca
|
||||
- Tamil Dolly
|
||||
- Tamil-ai/samacheer-kalvi-tamil (morphological drills + grammar QA)
|
||||
- Additional Tamil instruction sets
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "Tamil-ai/tamil-qwen25-7b-instruct"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype="auto",
|
||||
device_map="auto",
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "You are a helpful Tamil language assistant."},
|
||||
{"role": "user", "content": "வீடு என்ற சொல்லின் வேற்றுமை வடிவங்களைக் கூறுக."},
|
||||
]
|
||||
|
||||
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
||||
outputs = model.generate(**inputs, max_new_tokens=256)
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
### 4-bit Quantized (for limited VRAM)
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
"Tamil-ai/tamil-qwen25-7b-instruct",
|
||||
quantization_config=BitsAndBytesConfig(load_in_4bit=True),
|
||||
device_map="auto",
|
||||
)
|
||||
```
|
||||
|
||||
## Why Qwen2.5?
|
||||
|
||||
Tokenizer analysis across 6 base models showed Qwen2.5 has the best Tamil tokenization efficiency:
|
||||
|
||||
| Model | Tamil Token Ratio | Verdict |
|
||||
|-------|------------------|---------|
|
||||
| **Qwen2.5** | **4.62x** | Best for Tamil |
|
||||
| Llama 3.1 | 5.8x | |
|
||||
| Gemma 2 | 6.1x | |
|
||||
| Mistral | 7.2x | |
|
||||
| Falcon | 10.5x | Worst |
|
||||
|
||||
Lower ratio = fewer tokens per Tamil word = more efficient training and inference.
|
||||
|
||||
## Intended Use
|
||||
|
||||
- Tamil question answering and instruction following
|
||||
- Tamil morphological analysis
|
||||
- Tamil grammar and linguistics tasks
|
||||
- Research on low-resource language LLMs
|
||||
|
||||
## Limitations
|
||||
|
||||
- Trained primarily on instructional Tamil; may underperform on colloquial/slang
|
||||
- Morphological accuracy varies by category (see benchmark results)
|
||||
- English capabilities may degrade compared to base Qwen2.5
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{tamilai2026,
|
||||
title={A Thousand Language Problem: Morphological Understanding in Linguistic AI},
|
||||
author={Tamil-AI},
|
||||
year={2026},
|
||||
publisher={HuggingFace},
|
||||
url={https://huggingface.co/Tamil-ai/tamil-qwen25-7b-instruct}
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user