初始化项目，由ModelHub XC社区提供模型

Model: AbhiDS16/kannada-gpt2-32m Source: Original Platform
2026-06-29 19:34:17 +08:00
commit c7658b3716
7 changed files with 46244 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,188 @@
+---
+library_name: transformers
+language:
+  - kn
+tags:
+  - kannada
+  - gpt2
+  - language-model
+  - low-resource-language
+  - dravidian
+license: mit
+---
+
+# Kannada GPT-2 Small (kannada-gpt2-32m)
+
+A **31.6M parameter GPT-2 style autoregressive language model** trained entirely from scratch on Kannada text. Everything — data pipeline, BPE tokenizer, model weights — built from the ground up on a single NVIDIA RTX 5070.
+
+**No pretrained initialization. No fine-tuning. Pure Kannada.**
+
+## Model Details
+
+### Model Description
+
+This is a small GPT-2 model trained from scratch on Kannada text. It uses a custom BPE tokenizer also trained from scratch on the same data. The model can generate coherent Kannada text and produces useful representations for downstream tasks.
+
+- **Developed by:** AbhiDS16
+- **Model type:** GPT-2 (decoder-only transformer)
+- **Language:** Kannada (kn)
+- **License:** MIT
+- **Parameters:** 31,626,240
+- **Context length:** 512 tokens
+- **Vocabulary size:** 12,000
+- **Trained from scratch:** Yes (no pretrained initialization)
+
+### Model Sources
+
+- **Repository:** https://github.com/thorOdinson16/KanLM
+- **Demo:** Use the Quick Start code below
+
+## Uses
+
+### Direct Use
+
+The model can be used for:
+- Kannada text generation
+- Extracting embeddings for downstream tasks (classification, clustering)
+- Fine-tuning on task-specific Kannada datasets
+- Studying low-resource language model training
+
+### Downstream Use
+
+The model's frozen embeddings achieve **73.5% accuracy** on Kannada sentiment classification with a simple logistic regression head — demonstrating transferable representations.
+
+### Out-of-Scope Use
+
+- Chat/instruction-following (model is not instruction-tuned)
+- Production systems requiring high factual accuracy
+- Sensitive content generation without safeguards
+
+## Bias, Risks, and Limitations
+
+- **Small model size:** 31.6M parameters limits factual knowledge and reasoning
+- **Repetition:** Tends to repeat phrases in longer generations
+- **Training data bias:** Web text (news, blogs) reflects biases and code-mixing present in online Kannada
+- **Not instruction-tuned:** Raw causal LM — not suitable for chat/QA without fine-tuning
+- **Data recency:** Training data from mC4 (2011–2022)
+
+## How to Get Started with the Model
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("AbhiDS16/kannada-gpt2-32m")
+tokenizer = AutoTokenizer.from_pretrained("AbhiDS16/kannada-gpt2-32m")
+
+prompt = "ನಾನು ಇಂದು ಬೆಳಿಗ್ಗೆ"
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=80,
+    temperature=0.7,
+    do_sample=True,
+    top_p=0.9,
+    pad_token_id=tokenizer.pad_token_id,
+)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Training Details
+
+### Training Data
+
+**CulturaX-Kn** — 1.35M documents (~4GB) of Kannada web text from mC4. After filtering (Kannada script ratio ≥ 60%, deduplication, length filtering), **12.6M clean sentences** were used for training.
+
+### Training Procedure
+
+- **Precision:** fp16 mixed
+- **Batch size:** 16 (effective 32 with gradient accumulation)
+- **Learning rate:** 5e-4 with cosine decay and 1,000 step warmup
+- **Optimizer:** AdamW (β₁=0.9, β₂=0.95, weight decay=0.01)
+- **Gradient clipping:** 1.0
+- **Epochs:** 3
+- **Total steps:** 83,874
+- **Training tokens:** ~463M
+
+### Speeds, Sizes, Times
+
+- **Hardware:** NVIDIA RTX 5070 (8GB VRAM)
+- **Training time:** 7 hours 16 minutes
+- **Model size on disk:** ~126MB (safetensors)
+- **Throughput:** ~3.2 steps/second
+
+## Evaluation
+
+### Perplexity
+
+| Metric | Value |
+|--------|-------|
+| Validation loss | 3.4594 |
+| Perplexity | **31.80** |
+| Evaluation tokens | 4,626,944 |
+
+### Sentiment Classification
+
+| Metric | Value |
+|--------|-------|
+| Method | Frozen LM + Logistic Regression |
+| Accuracy | **73.5%** |
+| F1 (macro) | **0.735** |
+
+### Tokenizer Efficiency
+
+Custom BPE tokenizer trained from scratch on Kannada text:
+
+| Tokenizer | Tokens/Word | Improvement |
+|-----------|-------------|-------------|
+| **Our BPE** | **1.91** | — |
+| XLM-R | 2.43 | 21.5% |
+| mBERT | 4.00 | 52.2% |
+
+## Environmental Impact
+
+- **Hardware:** NVIDIA RTX 5070 (125W TDP under load)
+- **Hours used:** ~7.3 hours
+- **Estimated carbon:** ~0.35 kg CO2eq (assuming 0.4 kg/kWh grid average)
+- **Cloud provider:** N/A (local desktop)
+
+## Technical Specifications
+
+### Model Architecture
+
+- 8 transformer layers
+- 512 hidden dimension
+- 8 attention heads
+- 2,048 feed-forward dimension
+- GELU activation
+- 0.1 dropout
+
+### Compute Infrastructure
+
+- **GPU:** NVIDIA RTX 5070 (8GB VRAM)
+- **CPU:** Intel Core Ultra 9 285H
+- **RAM:** 32GB
+
+### Software
+
+- Python 3.10
+- PyTorch 2.10
+- Transformers 4.x
+- Datasets 3.x
+- Tokenizers 0.19
+
+## Citation
+
+```bibtex
+@misc{kannada-gpt2-32m,
+  author = {AbhiDS16},
+  title = {Kannada GPT-2 Small: A From-Scratch Language Model for Kannada},
+  year = {2026},
+  publisher = {HuggingFace},
+  howpublished = {\url{https://huggingface.co/AbhiDS16/kannada-gpt2-32m}},
+  note = {Trained entirely from scratch with custom BPE tokenizer}
+}
+```
+
+## Model Card Contact
+
+Open an issue on GitHub: https://github.com/thorOdinson16/KanLM
--- a/config.json
+++ b/config.json
@@ -0,0 +1,34 @@
+{
+  "activation_function": "gelu",
+  "add_cross_attention": false,
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 1,
+  "dtype": "float32",
+  "embd_pdrop": 0.1,
+  "eos_token_id": 2,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_embd": 512,
+  "n_head": 8,
+  "n_inner": null,
+  "n_layer": 8,
+  "n_positions": 512,
+  "pad_token_id": 0,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.3.0",
+  "use_cache": false,
+  "vocab_size": 12000
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.3.0",
+  "use_cache": true
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:85862f1526df8e00fcb069be9d6bc6faf1264ffc78a8e90cfdb308b01f38241f
+size 126514920
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,11 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<bos>",
+  "eos_token": "<eos>",
+  "is_local": true,
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}