188 lines
5.1 KiB
Markdown
188 lines
5.1 KiB
Markdown
---
|
||
library_name: transformers
|
||
language:
|
||
- kn
|
||
tags:
|
||
- kannada
|
||
- gpt2
|
||
- language-model
|
||
- low-resource-language
|
||
- dravidian
|
||
license: mit
|
||
---
|
||
|
||
# Kannada GPT-2 Small (kannada-gpt2-32m)
|
||
|
||
A **31.6M parameter GPT-2 style autoregressive language model** trained entirely from scratch on Kannada text. Everything — data pipeline, BPE tokenizer, model weights — built from the ground up on a single NVIDIA RTX 5070.
|
||
|
||
**No pretrained initialization. No fine-tuning. Pure Kannada.**
|
||
|
||
## Model Details
|
||
|
||
### Model Description
|
||
|
||
This is a small GPT-2 model trained from scratch on Kannada text. It uses a custom BPE tokenizer also trained from scratch on the same data. The model can generate coherent Kannada text and produces useful representations for downstream tasks.
|
||
|
||
- **Developed by:** AbhiDS16
|
||
- **Model type:** GPT-2 (decoder-only transformer)
|
||
- **Language:** Kannada (kn)
|
||
- **License:** MIT
|
||
- **Parameters:** 31,626,240
|
||
- **Context length:** 512 tokens
|
||
- **Vocabulary size:** 12,000
|
||
- **Trained from scratch:** Yes (no pretrained initialization)
|
||
|
||
### Model Sources
|
||
|
||
- **Repository:** https://github.com/thorOdinson16/KanLM
|
||
- **Demo:** Use the Quick Start code below
|
||
|
||
## Uses
|
||
|
||
### Direct Use
|
||
|
||
The model can be used for:
|
||
- Kannada text generation
|
||
- Extracting embeddings for downstream tasks (classification, clustering)
|
||
- Fine-tuning on task-specific Kannada datasets
|
||
- Studying low-resource language model training
|
||
|
||
### Downstream Use
|
||
|
||
The model's frozen embeddings achieve **73.5% accuracy** on Kannada sentiment classification with a simple logistic regression head — demonstrating transferable representations.
|
||
|
||
### Out-of-Scope Use
|
||
|
||
- Chat/instruction-following (model is not instruction-tuned)
|
||
- Production systems requiring high factual accuracy
|
||
- Sensitive content generation without safeguards
|
||
|
||
## Bias, Risks, and Limitations
|
||
|
||
- **Small model size:** 31.6M parameters limits factual knowledge and reasoning
|
||
- **Repetition:** Tends to repeat phrases in longer generations
|
||
- **Training data bias:** Web text (news, blogs) reflects biases and code-mixing present in online Kannada
|
||
- **Not instruction-tuned:** Raw causal LM — not suitable for chat/QA without fine-tuning
|
||
- **Data recency:** Training data from mC4 (2011–2022)
|
||
|
||
## How to Get Started with the Model
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model = AutoModelForCausalLM.from_pretrained("AbhiDS16/kannada-gpt2-32m")
|
||
tokenizer = AutoTokenizer.from_pretrained("AbhiDS16/kannada-gpt2-32m")
|
||
|
||
prompt = "ನಾನು ಇಂದು ಬೆಳಿಗ್ಗೆ"
|
||
inputs = tokenizer(prompt, return_tensors="pt")
|
||
outputs = model.generate(
|
||
**inputs,
|
||
max_new_tokens=80,
|
||
temperature=0.7,
|
||
do_sample=True,
|
||
top_p=0.9,
|
||
pad_token_id=tokenizer.pad_token_id,
|
||
)
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
```
|
||
|
||
## Training Details
|
||
|
||
### Training Data
|
||
|
||
**CulturaX-Kn** — 1.35M documents (~4GB) of Kannada web text from mC4. After filtering (Kannada script ratio ≥ 60%, deduplication, length filtering), **12.6M clean sentences** were used for training.
|
||
|
||
### Training Procedure
|
||
|
||
- **Precision:** fp16 mixed
|
||
- **Batch size:** 16 (effective 32 with gradient accumulation)
|
||
- **Learning rate:** 5e-4 with cosine decay and 1,000 step warmup
|
||
- **Optimizer:** AdamW (β₁=0.9, β₂=0.95, weight decay=0.01)
|
||
- **Gradient clipping:** 1.0
|
||
- **Epochs:** 3
|
||
- **Total steps:** 83,874
|
||
- **Training tokens:** ~463M
|
||
|
||
### Speeds, Sizes, Times
|
||
|
||
- **Hardware:** NVIDIA RTX 5070 (8GB VRAM)
|
||
- **Training time:** 7 hours 16 minutes
|
||
- **Model size on disk:** ~126MB (safetensors)
|
||
- **Throughput:** ~3.2 steps/second
|
||
|
||
## Evaluation
|
||
|
||
### Perplexity
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Validation loss | 3.4594 |
|
||
| Perplexity | **31.80** |
|
||
| Evaluation tokens | 4,626,944 |
|
||
|
||
### Sentiment Classification
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Method | Frozen LM + Logistic Regression |
|
||
| Accuracy | **73.5%** |
|
||
| F1 (macro) | **0.735** |
|
||
|
||
### Tokenizer Efficiency
|
||
|
||
Custom BPE tokenizer trained from scratch on Kannada text:
|
||
|
||
| Tokenizer | Tokens/Word | Improvement |
|
||
|-----------|-------------|-------------|
|
||
| **Our BPE** | **1.91** | — |
|
||
| XLM-R | 2.43 | 21.5% |
|
||
| mBERT | 4.00 | 52.2% |
|
||
|
||
## Environmental Impact
|
||
|
||
- **Hardware:** NVIDIA RTX 5070 (125W TDP under load)
|
||
- **Hours used:** ~7.3 hours
|
||
- **Estimated carbon:** ~0.35 kg CO2eq (assuming 0.4 kg/kWh grid average)
|
||
- **Cloud provider:** N/A (local desktop)
|
||
|
||
## Technical Specifications
|
||
|
||
### Model Architecture
|
||
|
||
- 8 transformer layers
|
||
- 512 hidden dimension
|
||
- 8 attention heads
|
||
- 2,048 feed-forward dimension
|
||
- GELU activation
|
||
- 0.1 dropout
|
||
|
||
### Compute Infrastructure
|
||
|
||
- **GPU:** NVIDIA RTX 5070 (8GB VRAM)
|
||
- **CPU:** Intel Core Ultra 9 285H
|
||
- **RAM:** 32GB
|
||
|
||
### Software
|
||
|
||
- Python 3.10
|
||
- PyTorch 2.10
|
||
- Transformers 4.x
|
||
- Datasets 3.x
|
||
- Tokenizers 0.19
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{kannada-gpt2-32m,
|
||
author = {AbhiDS16},
|
||
title = {Kannada GPT-2 Small: A From-Scratch Language Model for Kannada},
|
||
year = {2026},
|
||
publisher = {HuggingFace},
|
||
howpublished = {\url{https://huggingface.co/AbhiDS16/kannada-gpt2-32m}},
|
||
note = {Trained entirely from scratch with custom BPE tokenizer}
|
||
}
|
||
```
|
||
|
||
## Model Card Contact
|
||
|
||
Open an issue on GitHub: https://github.com/thorOdinson16/KanLM |