初始化项目，由ModelHub XC社区提供模型
Model: Chamaka8/Serendip-LLM-CPT-SFT-v2 Source: Original Platform
This commit is contained in:
ModelHub XC
166
README.md
Normal file
166
README.md
Normal file
@@ -0,0 +1,166 @@
|
||||
---
|
||||
language:
|
||||
- si
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- sinhala
|
||||
- llama-3
|
||||
- instruction-following
|
||||
- news-classification
|
||||
- text-generation
|
||||
datasets:
|
||||
- Chamaka8/Serendip-sft-sinhala
|
||||
base_model: Chamaka8/serendib-llm-cpt-llama3-8b
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# SerendipLLM V2 🇱🇰
|
||||
|
||||
**The largest Sinhala instruction-following language model trained on 309,328 examples**
|
||||
|
||||
SerendipLLM V2 is a specialized Sinhala language model with exceptional capabilities in news classification, question answering, and general Sinhala text generation. Built on Llama-3-8B with continued pre-training and instruction fine-tuning, it represents a significant advancement in Sinhala NLP.
|
||||
|
||||
## 🏆 Key Achievements
|
||||
|
||||
- ✅ **6.2x larger dataset** than existing Sinhala models (309K vs ~50K examples)
|
||||
- ✅ **45,080 news classification examples** for specialized Sinhala news categorization
|
||||
- ✅ **50% training loss reduction** (0.54 → 0.27) over 3 epochs
|
||||
- ✅ **Comprehensive training** on diverse Sinhala tasks
|
||||
- ✅ **Open-source** - Complete pipeline and dataset available
|
||||
|
||||
## 📊 Model Details
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| **Base Model** | Meta Llama-3-8B |
|
||||
| **CPT Foundation** | [serendib-llm-cpt-llama3-8b](https://huggingface.co/Chamaka8/serendib-llm-cpt-llama3-8b) |
|
||||
| **Parameters** | 8.16B total, 130M trainable (1.59%) |
|
||||
| **Training Examples** | 309,328 |
|
||||
| **Training Method** | LoRA fine-tuning |
|
||||
| **Training Duration** | 26.5 hours on A100 80GB |
|
||||
| **Final Loss** | 0.27 |
|
||||
| **License** | Apache 2.0 |
|
||||
|
||||
## 🎯 Specialized Capabilities
|
||||
|
||||
### News Classification (Our Strength!)
|
||||
Trained on **45,080 Sinhala news examples** - the largest news classification dataset for Sinhala.
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
Input: "ශ්රී ලංකා ක්රිකට් කණ්ඩායම අද ඉන්දියාවට එරෙහිව තරගයක් ආරම්භ කළේය"
|
||||
Output: "මෙය ක්රීඩා පුවතකි" ✅
|
||||
```
|
||||
|
||||
### Question Answering
|
||||
**29,390 QA pairs** covering geography, history, culture, and general knowledge.
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
Input: "ශ්රී ලංකාවේ අගනුවර කුමක්ද?"
|
||||
Output: "ශ්රී ලංකාවේ අගනුවර කොළඹයි" ✅
|
||||
```
|
||||
|
||||
## 📈 Dataset Composition
|
||||
|
||||
| Category | Examples | Percentage |
|
||||
|----------|----------|------------|
|
||||
| **General Sinhala** | 205,403 | 66.4% |
|
||||
| **News Classification** | 45,080 | 14.6% |
|
||||
| **QA Pairs** | 29,390 | 9.5% |
|
||||
| **Summarization** | 19,593 | 6.3% |
|
||||
| **Rewrite/Formatting** | 9,862 | 3.2% |
|
||||
| **TOTAL** | **309,328** | **100%** |
|
||||
|
||||
## 🚀 Usage
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
import torch
|
||||
|
||||
# Load model
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
"Chamaka8/Serendip-LLM-CPT-SFT-v2",
|
||||
torch_dtype=torch.float16,
|
||||
device_map="auto"
|
||||
)
|
||||
tokenizer = AutoTokenizer.from_pretrained("Chamaka8/Serendip-LLM-CPT-SFT-v2")
|
||||
|
||||
# Format prompt
|
||||
prompt = """### Instruction:
|
||||
පහත පුවත් ලිපිය වර්ගීකරණය කරන්න
|
||||
|
||||
### Input:
|
||||
ශ්රී ලංකා ක්රිකට් කණ්ඩායම අද ඉන්දියාවට එරෙහිව තරගයක් ආරම්භ කළේය.
|
||||
|
||||
### Response:
|
||||
"""
|
||||
|
||||
# Generate
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=150,
|
||||
temperature=0.7,
|
||||
top_p=0.9
|
||||
)
|
||||
|
||||
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||||
print(response.split("### Response:")[-1].strip())
|
||||
```
|
||||
|
||||
## ⚙️ Training Configuration
|
||||
|
||||
### Hardware
|
||||
- **GPU:** NVIDIA A100 SXM 80GB
|
||||
- **Training Time:** 26.5 hours
|
||||
- **Cost:** ~$37 USD
|
||||
|
||||
### Hyperparameters
|
||||
```python
|
||||
num_train_epochs = 3
|
||||
per_device_train_batch_size = 8
|
||||
gradient_accumulation_steps = 4
|
||||
learning_rate = 2e-5
|
||||
max_seq_length = 384
|
||||
lora_r = 64
|
||||
lora_alpha = 128
|
||||
```
|
||||
|
||||
### Training Loss
|
||||
|
||||
| Epoch | Loss |
|
||||
|-------|------|
|
||||
| 1.0 | 0.28 |
|
||||
| 2.0 | 0.24 |
|
||||
| 3.0 | 0.27 |
|
||||
|
||||
## 📊 Comparison
|
||||
|
||||
| Model | Training Data | News Examples |
|
||||
|-------|--------------|---------------|
|
||||
| SinLlama | ~50,000 | Limited |
|
||||
| **SerendipLLM V2** | **309,328** | **45,080** ✅ |
|
||||
|
||||
## 🔗 Resources
|
||||
|
||||
- **Dataset:** [Serendip-sft-sinhala](https://huggingface.co/datasets/Chamaka8/Serendip-sft-sinhala)
|
||||
- **Base CPT:** [serendib-llm-cpt-llama3-8b](https://huggingface.co/Chamaka8/serendib-llm-cpt-llama3-8b)
|
||||
- **Training Script:** See `training_scripts/` folder
|
||||
|
||||
## 📚 Citation
|
||||
```bibtex
|
||||
@model{serendipllm2026,
|
||||
title={SerendipLLM V2: Large-Scale Instruction-Tuning for Sinhala},
|
||||
author={Chamaka Alwis},
|
||||
year={2026},
|
||||
url={https://huggingface.co/Chamaka8/Serendip-LLM-CPT-SFT-v2}
|
||||
}
|
||||
```
|
||||
|
||||
## 📄 License
|
||||
|
||||
Apache 2.0
|
||||
|
||||
---
|
||||
|
||||
**Built with ❤️ for the Sinhala NLP community**
|
||||
Reference in New Issue
Block a user