42 lines
1.1 KiB
Markdown
42 lines
1.1 KiB
Markdown
|
# Training Scripts
|
||
|
|
|
||
|
|
This folder contains all training scripts used in the SerendipLLM V2 project.
|
||
|
|
|
||
|
|
## ✅ Final Script Used (ACTUAL TRAINING)
|
||
|
|
|
||
|
|
**`train_v2_fast.py`** ← This is the exact script that trained the final model!
|
||
|
|
|
||
|
|
- Training time: 26.5 hours
|
||
|
|
- Loss: 0.54 → 0.27 (50% improvement)
|
||
|
|
- Epochs: 3
|
||
|
|
- Batch size: 8 (effective 32)
|
||
|
|
- LoRA rank: 64
|
||
|
|
- Dataset: 309,328 examples
|
||
|
|
|
||
|
|
## 📝 Other Scripts (Development/Testing)
|
||
|
|
|
||
|
|
- `train_phase1_fixed.py` - Initial attempt (slower, 512 tokens)
|
||
|
|
- `continue_training.py` - Script for resuming training (not used)
|
||
|
|
|
||
|
|
## 🎯 To Reproduce
|
||
|
|
|
||
|
|
Use `train_v2_fast.py` with these settings:
|
||
|
|
- GPU: A100 80GB
|
||
|
|
- Dataset: Chamaka8/Serendip-sft-sinhala (serendipllm_sft_final_train_v2.json)
|
||
|
|
- Time: ~27 hours
|
||
|
|
- Cost: ~$37
|
||
|
|
|
||
|
|
## 📊 Training Results
|
||
|
|
```
|
||
|
|
Epoch 1: Loss 0.28
|
||
|
|
Epoch 2: Loss 0.24
|
||
|
|
Epoch 3: Loss 0.27
|
||
|
|
Final average loss: 0.27
|
||
|
|
```
|
||
|
|
|
||
|
|
## 🔗 Related Resources
|
||
|
|
|
||
|
|
- Model: https://huggingface.co/Chamaka8/Serendip-LLM-CPT-SFT-v2
|
||
|
|
- Dataset: https://huggingface.co/datasets/Chamaka8/Serendip-sft-sinhala
|
||
|
|
- Base CPT: https://huggingface.co/Chamaka8/serendib-llm-cpt-llama3-8b
|