初始化项目，由ModelHub XC社区提供模型
Model: AbhiDS16/kannada-gpt2-32m Source: Original Platform
This commit is contained in:
ModelHub XC
188
README.md
Normal file
188
README.md
Normal file
@@ -0,0 +1,188 @@
|
||||
---
|
||||
library_name: transformers
|
||||
language:
|
||||
- kn
|
||||
tags:
|
||||
- kannada
|
||||
- gpt2
|
||||
- language-model
|
||||
- low-resource-language
|
||||
- dravidian
|
||||
license: mit
|
||||
---
|
||||
|
||||
# Kannada GPT-2 Small (kannada-gpt2-32m)
|
||||
|
||||
A **31.6M parameter GPT-2 style autoregressive language model** trained entirely from scratch on Kannada text. Everything — data pipeline, BPE tokenizer, model weights — built from the ground up on a single NVIDIA RTX 5070.
|
||||
|
||||
**No pretrained initialization. No fine-tuning. Pure Kannada.**
|
||||
|
||||
## Model Details
|
||||
|
||||
### Model Description
|
||||
|
||||
This is a small GPT-2 model trained from scratch on Kannada text. It uses a custom BPE tokenizer also trained from scratch on the same data. The model can generate coherent Kannada text and produces useful representations for downstream tasks.
|
||||
|
||||
- **Developed by:** AbhiDS16
|
||||
- **Model type:** GPT-2 (decoder-only transformer)
|
||||
- **Language:** Kannada (kn)
|
||||
- **License:** MIT
|
||||
- **Parameters:** 31,626,240
|
||||
- **Context length:** 512 tokens
|
||||
- **Vocabulary size:** 12,000
|
||||
- **Trained from scratch:** Yes (no pretrained initialization)
|
||||
|
||||
### Model Sources
|
||||
|
||||
- **Repository:** https://github.com/thorOdinson16/KanLM
|
||||
- **Demo:** Use the Quick Start code below
|
||||
|
||||
## Uses
|
||||
|
||||
### Direct Use
|
||||
|
||||
The model can be used for:
|
||||
- Kannada text generation
|
||||
- Extracting embeddings for downstream tasks (classification, clustering)
|
||||
- Fine-tuning on task-specific Kannada datasets
|
||||
- Studying low-resource language model training
|
||||
|
||||
### Downstream Use
|
||||
|
||||
The model's frozen embeddings achieve **73.5% accuracy** on Kannada sentiment classification with a simple logistic regression head — demonstrating transferable representations.
|
||||
|
||||
### Out-of-Scope Use
|
||||
|
||||
- Chat/instruction-following (model is not instruction-tuned)
|
||||
- Production systems requiring high factual accuracy
|
||||
- Sensitive content generation without safeguards
|
||||
|
||||
## Bias, Risks, and Limitations
|
||||
|
||||
- **Small model size:** 31.6M parameters limits factual knowledge and reasoning
|
||||
- **Repetition:** Tends to repeat phrases in longer generations
|
||||
- **Training data bias:** Web text (news, blogs) reflects biases and code-mixing present in online Kannada
|
||||
- **Not instruction-tuned:** Raw causal LM — not suitable for chat/QA without fine-tuning
|
||||
- **Data recency:** Training data from mC4 (2011–2022)
|
||||
|
||||
## How to Get Started with the Model
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained("AbhiDS16/kannada-gpt2-32m")
|
||||
tokenizer = AutoTokenizer.from_pretrained("AbhiDS16/kannada-gpt2-32m")
|
||||
|
||||
prompt = "ನಾನು ಇಂದು ಬೆಳಿಗ್ಗೆ"
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=80,
|
||||
temperature=0.7,
|
||||
do_sample=True,
|
||||
top_p=0.9,
|
||||
pad_token_id=tokenizer.pad_token_id,
|
||||
)
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## Training Details
|
||||
|
||||
### Training Data
|
||||
|
||||
**CulturaX-Kn** — 1.35M documents (~4GB) of Kannada web text from mC4. After filtering (Kannada script ratio ≥ 60%, deduplication, length filtering), **12.6M clean sentences** were used for training.
|
||||
|
||||
### Training Procedure
|
||||
|
||||
- **Precision:** fp16 mixed
|
||||
- **Batch size:** 16 (effective 32 with gradient accumulation)
|
||||
- **Learning rate:** 5e-4 with cosine decay and 1,000 step warmup
|
||||
- **Optimizer:** AdamW (β₁=0.9, β₂=0.95, weight decay=0.01)
|
||||
- **Gradient clipping:** 1.0
|
||||
- **Epochs:** 3
|
||||
- **Total steps:** 83,874
|
||||
- **Training tokens:** ~463M
|
||||
|
||||
### Speeds, Sizes, Times
|
||||
|
||||
- **Hardware:** NVIDIA RTX 5070 (8GB VRAM)
|
||||
- **Training time:** 7 hours 16 minutes
|
||||
- **Model size on disk:** ~126MB (safetensors)
|
||||
- **Throughput:** ~3.2 steps/second
|
||||
|
||||
## Evaluation
|
||||
|
||||
### Perplexity
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Validation loss | 3.4594 |
|
||||
| Perplexity | **31.80** |
|
||||
| Evaluation tokens | 4,626,944 |
|
||||
|
||||
### Sentiment Classification
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Method | Frozen LM + Logistic Regression |
|
||||
| Accuracy | **73.5%** |
|
||||
| F1 (macro) | **0.735** |
|
||||
|
||||
### Tokenizer Efficiency
|
||||
|
||||
Custom BPE tokenizer trained from scratch on Kannada text:
|
||||
|
||||
| Tokenizer | Tokens/Word | Improvement |
|
||||
|-----------|-------------|-------------|
|
||||
| **Our BPE** | **1.91** | — |
|
||||
| XLM-R | 2.43 | 21.5% |
|
||||
| mBERT | 4.00 | 52.2% |
|
||||
|
||||
## Environmental Impact
|
||||
|
||||
- **Hardware:** NVIDIA RTX 5070 (125W TDP under load)
|
||||
- **Hours used:** ~7.3 hours
|
||||
- **Estimated carbon:** ~0.35 kg CO2eq (assuming 0.4 kg/kWh grid average)
|
||||
- **Cloud provider:** N/A (local desktop)
|
||||
|
||||
## Technical Specifications
|
||||
|
||||
### Model Architecture
|
||||
|
||||
- 8 transformer layers
|
||||
- 512 hidden dimension
|
||||
- 8 attention heads
|
||||
- 2,048 feed-forward dimension
|
||||
- GELU activation
|
||||
- 0.1 dropout
|
||||
|
||||
### Compute Infrastructure
|
||||
|
||||
- **GPU:** NVIDIA RTX 5070 (8GB VRAM)
|
||||
- **CPU:** Intel Core Ultra 9 285H
|
||||
- **RAM:** 32GB
|
||||
|
||||
### Software
|
||||
|
||||
- Python 3.10
|
||||
- PyTorch 2.10
|
||||
- Transformers 4.x
|
||||
- Datasets 3.x
|
||||
- Tokenizers 0.19
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{kannada-gpt2-32m,
|
||||
author = {AbhiDS16},
|
||||
title = {Kannada GPT-2 Small: A From-Scratch Language Model for Kannada},
|
||||
year = {2026},
|
||||
publisher = {HuggingFace},
|
||||
howpublished = {\url{https://huggingface.co/AbhiDS16/kannada-gpt2-32m}},
|
||||
note = {Trained entirely from scratch with custom BPE tokenizer}
|
||||
}
|
||||
```
|
||||
|
||||
## Model Card Contact
|
||||
|
||||
Open an issue on GitHub: https://github.com/thorOdinson16/KanLM
|
||||
Reference in New Issue
Block a user