446 lines
15 KiB
Markdown
446 lines
15 KiB
Markdown
---
|
|
library_name: transformers
|
|
license: apache-2.0
|
|
base_model: diabolic6045/Sanskrit-Qwen2.5-7B-base
|
|
tags:
|
|
- axolotl
|
|
- transformers
|
|
- sanskrit
|
|
- translation
|
|
- transliteration
|
|
- chat
|
|
datasets:
|
|
- diabolic6045/Sanskrit-transliteration-chat-dataset
|
|
pipeline_tag: text-generation
|
|
model-index:
|
|
- name: sanskrit-qwen2.5-7b-chat
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
dataset:
|
|
name: Sanskrit Translation & Transliteration Test Set
|
|
type: sanskrit-translation
|
|
metrics:
|
|
- name: Sanskrit to English Translation - Jaccard Similarity
|
|
type: jaccard_similarity
|
|
value: 30.9
|
|
- name: Sanskrit to English Translation - BLEU Score
|
|
type: bleu
|
|
value: 10.9
|
|
- name: English to Sanskrit Translation - Jaccard Similarity
|
|
type: jaccard_similarity
|
|
value: 10.3
|
|
- name: English to Sanskrit Translation - BLEU Score
|
|
type: bleu
|
|
value: 4.1
|
|
- name: Transliteration - Exact Match Accuracy
|
|
type: exact_match
|
|
value: 29.1
|
|
- name: Transliteration - Character Accuracy
|
|
type: character_accuracy
|
|
value: 96.7
|
|
- name: Success Rate
|
|
type: success_rate
|
|
value: 100
|
|
source:
|
|
name: Sanskrit Translation & Transliteration Evaluation
|
|
url: >-
|
|
https://huggingface.co/datasets/diabolic6045/Sanskrit-transliteration-chat-dataset/viewer/default/test
|
|
language:
|
|
- sa
|
|
- en
|
|
---
|
|
|
|
> **Note**: This project represents a series of experimental attempts to create specialized Sanskrit language models. While significant effort was invested, the results did not meet the quality standards required for practical use. This repository is maintained for research documentation and learning purposes.
|
|
|
|
|
|
# Sanskrit Qwen2.5-7B Chat Model
|
|
|
|
This model is a fine-tuned version of [diabolic6045/Sanskrit-Qwen2.5-7B-base](https://huggingface.co/diabolic6045/Sanskrit-Qwen2.5-7B-base) specialized for Sanskrit language tasks including bidirectional translation (Sanskrit ↔ English) and transliteration (Devanagari to IAST). The model has been trained in two phases: initial Sanskrit language modeling followed by chat template enhancement for translation and transliteration capabilities.
|
|
|
|
## Model Description
|
|
|
|
This is a specialized version of Qwen2.5-7B that has been enhanced for Sanskrit language processing through a two-phase training approach:
|
|
|
|
**Phase 1: Sanskrit Language Model Training**
|
|
- Trained on 664,104+ clean Sanskrit texts from 9 different sources
|
|
- Enhanced with Sanskrit-specific punctuation tokens (। ॥)
|
|
- Custom tokenizer with proper Devanagari script support
|
|
|
|
**Phase 2: Chat Template Enhancement**
|
|
- Post-trained on 94,363+ bidirectional translation and transliteration examples
|
|
- Chat template format for conversation-style interactions
|
|
- Specialized system prompts for different task types
|
|
|
|
**Key Features:**
|
|
- **Base Model**: Qwen/Qwen2.5-7B (7 billion parameters)
|
|
- **Training Method**: Two-phase LoRA fine-tuning
|
|
- **Languages**: Sanskrit (Devanagari script) and English
|
|
- **Tasks**: Translation, Transliteration, Text Generation
|
|
- **Architecture**: Transformer-based with LoRA adapters
|
|
|
|
**Capabilities:**
|
|
- **Bidirectional Translation**: Sanskrit ↔ English translation
|
|
- **Transliteration**: Devanagari script to IAST format
|
|
- **Sanskrit Text Generation**: Coherent Sanskrit verse and prose generation
|
|
- **Chat Interface**: Conversational responses with task understanding
|
|
- **Context Preservation**: Maintains meaning during translation tasks
|
|
|
|
## Training and Evaluation Data
|
|
|
|
### Training Datasets
|
|
|
|
**Phase 1 - Sanskrit Language Model:**
|
|
- **Dataset**: [diabolic6045/Sanskrit-shlok-collection](https://huggingface.co/datasets/diabolic6045/Sanskrit-shlok-collection)
|
|
- **Size**: 664,104 unique Sanskrit texts
|
|
- **Sources**: 9 different Sanskrit literature sources
|
|
- **Format**: Clean text completion format
|
|
- **Content**: Classical Sanskrit texts, verses, and prose
|
|
|
|
**Phase 2 - Chat Enhancement:**
|
|
- **Dataset**: [diabolic6045/Sanskrit-transliteration-chat-dataset](https://huggingface.co/datasets/diabolic6045/Sanskrit-transliteration-chat-dataset)
|
|
- **Size**: 94,363 conversation examples
|
|
- **Tasks**:
|
|
- Sanskrit to English translation: 20,205 examples
|
|
- English to Sanskrit translation: 20,205 examples
|
|
- Transliteration (Devanagari to IAST): 53,953 examples
|
|
- **Format**: Chat template with system prompts
|
|
|
|
### Evaluation Results
|
|
|
|
The model was evaluated on a comprehensive test set of 500 samples from [Test Dataset](https://huggingface.co/datasets/diabolic6045/Sanskrit-transliteration-chat-dataset/viewer/default/test) across three task types:
|
|
|
|
| Task Type | Samples | Exact Match | Jaccard Similarity | BLEU Score | Character Accuracy |
|
|
|:---------:|:-------:|:-----------:|:------------------:|:----------:|:------------------:|
|
|
| **Sanskrit → English** | 114 | 0.0% | 30.9% | 10.9% | - |
|
|
| **English → Sanskrit** | 101 | 0.0% | 10.3% | 4.1% | - |
|
|
| **Transliteration** | 285 | 29.1% | - | - | 96.7% |
|
|
| **Overall** | 500 | - | - | - | - |
|
|
|
|
**Key Insights:**
|
|
- **100% Success Rate**: All test samples processed without failures
|
|
- **High Transliteration Accuracy**: 96.7% character-level accuracy for Devanagari to IAST conversion
|
|
- **Moderate Translation Performance**: Room for improvement in exact translation matching
|
|
- **Strong Character Recognition**: Excellent handling of Sanskrit script and punctuation
|
|
|
|
## Usage
|
|
|
|
### Basic Usage
|
|
|
|
```python
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
import torch
|
|
|
|
# Load model and tokenizer
|
|
model_name = "diabolic6045/Sanskrit-Qwen2.5-7B-chat"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_name,
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto",
|
|
trust_remote_code=True
|
|
)
|
|
|
|
# Sanskrit to English Translation
|
|
def translate_sanskrit_to_english(sanskrit_text):
|
|
messages = [
|
|
{"role": "system", "content": "You are a Sanskrit to English translation expert. Translate the given Sanskrit text accurately while preserving the meaning and context."},
|
|
{"role": "user", "content": f"Translate this Sanskrit text to English: {sanskrit_text}"}
|
|
]
|
|
|
|
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
|
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
|
|
|
with torch.no_grad():
|
|
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
|
|
|
|
response = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
|
|
return response.strip()
|
|
|
|
# English to Sanskrit Translation
|
|
def translate_english_to_sanskrit(english_text):
|
|
messages = [
|
|
{"role": "system", "content": "You are an English to Sanskrit translation expert. Translate the given English text accurately into Sanskrit while preserving the meaning and context."},
|
|
{"role": "user", "content": f"Translate this English text to Sanskrit: {english_text}"}
|
|
]
|
|
|
|
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
|
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
|
|
|
with torch.no_grad():
|
|
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
|
|
|
|
response = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
|
|
return response.strip()
|
|
|
|
# Transliteration (Devanagari to IAST)
|
|
def transliterate_to_iast(devanagari_text):
|
|
messages = [
|
|
{"role": "system", "content": "You are a Sanskrit transliteration expert. Convert the given Sanskrit text from Devanagari script to IAST (International Alphabet of Sanskrit Transliteration) format."},
|
|
{"role": "user", "content": f"Transliterate this Sanskrit text to IAST: {devanagari_text}"}
|
|
]
|
|
|
|
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
|
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
|
|
|
with torch.no_grad():
|
|
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
|
|
|
|
response = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
|
|
return response.strip()
|
|
|
|
# Example usage
|
|
sanskrit_text = "नमस्ते संस्कृत भाषा ।"
|
|
english_translation = translate_sanskrit_to_english(sanskrit_text)
|
|
print(f"Sanskrit: {sanskrit_text}")
|
|
print(f"English: {english_translation}")
|
|
|
|
iast_transliteration = transliterate_to_iast(sanskrit_text)
|
|
print(f"IAST: {iast_transliteration}")
|
|
```
|
|
|
|
### Chat Interface
|
|
|
|
```python
|
|
def chat_with_sanskrit_model(user_input, task_type="translation"):
|
|
if task_type == "translation":
|
|
system_prompt = "You are a Sanskrit to English translation expert. Translate the given Sanskrit text accurately while preserving the meaning and context."
|
|
elif task_type == "transliteration":
|
|
system_prompt = "You are a Sanskrit transliteration expert. Convert the given Sanskrit text from Devanagari script to IAST format."
|
|
else:
|
|
system_prompt = "You are a Sanskrit language expert. Help with Sanskrit text processing tasks."
|
|
|
|
messages = [
|
|
{"role": "system", "content": system_prompt},
|
|
{"role": "user", "content": user_input}
|
|
]
|
|
|
|
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
|
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
|
|
|
with torch.no_grad():
|
|
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
|
|
|
|
response = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
|
|
return response.strip()
|
|
```
|
|
|
|
## Training Procedure
|
|
|
|
### Phase 1: Sanskrit Language Model Training
|
|
|
|
**Hyperparameters:**
|
|
- learning_rate: 0.0003
|
|
- train_batch_size: 2
|
|
- gradient_accumulation_steps: 2
|
|
- num_epochs: 3
|
|
- sequence_len: 1024
|
|
- lora_r: 32
|
|
- lora_alpha: 64
|
|
- optimizer: adamw_bnb_8bit
|
|
- lr_scheduler: cosine
|
|
|
|
### Phase 2: Chat Template Enhancement
|
|
|
|
**Hyperparameters:**
|
|
- learning_rate: 0.0002
|
|
- train_batch_size: 2
|
|
- gradient_accumulation_steps: 4
|
|
- num_epochs: 3
|
|
- sequence_len: 1024
|
|
- lora_r: 32
|
|
- lora_alpha: 16
|
|
- optimizer: adamw_bnb_8bit
|
|
- lr_scheduler: cosine
|
|
|
|
### Training Results
|
|
|
|
**Phase 1 Results:**
|
|
- Final Loss: ~0.15
|
|
- Training Time: ~8 hours (2x RTX 4090)
|
|
- Memory Usage: ~24GB per GPU
|
|
|
|
**Phase 2 Results:**
|
|
- Final Loss: ~0.12
|
|
- Training Time: ~4 hours (2x RTX 4090)
|
|
- Memory Usage: ~20GB per GPU
|
|
|
|
This model was trained using: <br>
|
|
[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
|
|
|
|
<details><summary>See Phase 1 axolotl config</summary>
|
|
|
|
axolotl version: `0.12.2`
|
|
```yaml
|
|
# Base model configuration
|
|
base_model: Qwen/Qwen2.5-7B
|
|
trust_remote_code: true
|
|
|
|
# Custom Sanskrit tokenizer configuration
|
|
tokenizer_config: sanskrit_tokenizer
|
|
tokenizer_type: AutoTokenizer
|
|
tokenizer_use_fast: true
|
|
tokenizer_legacy: false
|
|
tokenizer_save_jinja_files: true
|
|
|
|
# Resize embeddings for new tokens if needed
|
|
resize_token_embeddings_to_32x: true
|
|
mean_resizing_embeddings: false
|
|
shrink_embeddings: false
|
|
|
|
# Dataset configuration for text completion
|
|
datasets:
|
|
- path: sanskrit_clean_dataset.jsonl
|
|
type: completion
|
|
field: text
|
|
|
|
# Dataset preparation
|
|
dataset_prepared_path:
|
|
val_set_size: 0.1
|
|
output_dir: ./outputs/sanskrt-qwen-lora
|
|
|
|
# Sequence configuration
|
|
sequence_len: 1024
|
|
sample_packing: true
|
|
eval_sample_packing: true
|
|
|
|
# LoRA configuration
|
|
adapter: lora
|
|
lora_r: 32
|
|
lora_alpha: 64
|
|
lora_dropout: 0.05
|
|
lora_target_linear: true
|
|
lora_target_modules: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
|
|
|
|
gradient_accumulation_steps: 2
|
|
micro_batch_size: 2
|
|
num_epochs: 3
|
|
optimizer: adamw_bnb_8bit
|
|
lr_scheduler: cosine
|
|
learning_rate: 0.0003
|
|
|
|
# Precision configuration
|
|
bf16: auto
|
|
tf32: true
|
|
|
|
# Memory optimization
|
|
gradient_checkpointing: true
|
|
gradient_checkpointing_kwargs:
|
|
use_reentrant: false
|
|
flash_attention: true
|
|
|
|
# Training schedule
|
|
warmup_ratio: 0.1
|
|
evals_per_epoch: 4
|
|
saves_per_epoch: 1
|
|
weight_decay: 0.0
|
|
|
|
hub_model_id: diabolic6045/sanskrit-qwen2.5-7b-base-lora
|
|
```
|
|
|
|
</details>
|
|
|
|
<details><summary>See Phase 2 axolotl config</summary>
|
|
|
|
axolotl version: `0.12.2`
|
|
```yaml
|
|
# Post-Training Configuration for Sanskrit Translation & Transliteration Enhancement
|
|
base_model: diabolic6045/Sanskrit-Qwen2.5-7B-base
|
|
|
|
chat_template: qwen_25
|
|
datasets:
|
|
- path: sanskrit_train_set.jsonl
|
|
type: chat_template
|
|
field_messages: messages
|
|
message_property_mappings:
|
|
role: role
|
|
content: content
|
|
roles:
|
|
system:
|
|
- system
|
|
user:
|
|
- user
|
|
assistant:
|
|
- assistant
|
|
|
|
val_set_size: 0.01
|
|
output_dir: ./outputs/sanskrit-chat
|
|
|
|
adapter: lora
|
|
sequence_len: 1024
|
|
pad_to_sequence_len: false
|
|
|
|
lora_r: 32
|
|
lora_alpha: 16
|
|
lora_dropout: 0.05
|
|
lora_target_modules: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
|
|
|
|
gradient_accumulation_steps: 4
|
|
micro_batch_size: 2
|
|
num_epochs: 3
|
|
optimizer: adamw_bnb_8bit
|
|
lr_scheduler: cosine
|
|
learning_rate: 0.0002
|
|
|
|
bf16: true
|
|
tf32: true
|
|
|
|
gradient_checkpointing: true
|
|
logging_steps: 1
|
|
flash_attention: true
|
|
|
|
warmup_ratio: 0.1
|
|
evals_per_epoch: 1
|
|
saves_per_epoch: 1
|
|
weight_decay: 0.0
|
|
|
|
hub_model_id: diabolic6045/Sanskrit-Qwen2.5-7B-chat-lora
|
|
```
|
|
|
|
</details><br>
|
|
|
|
### Framework Versions
|
|
|
|
- PEFT 0.17.0
|
|
- Transformers 4.55.2
|
|
- PyTorch 2.7.1+cu128
|
|
- Datasets 4.0.0
|
|
- Tokenizers 0.21.2
|
|
- Axolotl 0.12.2
|
|
|
|
## Limitations and Future Work
|
|
|
|
### Current Limitations
|
|
- **Translation Accuracy**: Exact match accuracy is low, indicating room for improvement in precise translation
|
|
- **Context Understanding**: Limited ability to handle complex contextual nuances in translation
|
|
- **Domain Specificity**: Performance may vary across different Sanskrit literature genres
|
|
|
|
### Future Improvements
|
|
- **Larger Training Dataset**: Expand with more diverse Sanskrit texts and translations
|
|
- **Advanced Training Techniques**: Implement DPO (Direct Preference Optimization) for better alignment
|
|
- **Multimodal Capabilities**: Add vision capabilities for Sanskrit manuscript processing
|
|
- **Specialized Fine-tuning**: Domain-specific fine-tuning for different Sanskrit literature types
|
|
|
|
## Citation
|
|
|
|
If you use this model in your research, please cite:
|
|
|
|
```bibtex
|
|
@misc{sanskrit-qwen2.5-7b-chat,
|
|
title={Sanskrit Qwen2.5-7B Chat Model: A Specialized Language Model for Sanskrit Translation and Transliteration},
|
|
author={[Your Name]},
|
|
year={2024},
|
|
url={https://huggingface.co/diabolic6045/Sanskrit-Qwen2.5-7B-chat}
|
|
}
|
|
```
|
|
|
|
## License
|
|
|
|
This model is released under the Apache 2.0 License. See the [LICENSE](LICENSE) file for more details.
|
|
|
|
## Acknowledgments
|
|
|
|
- **Base Model**: [Qwen/Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B) by Alibaba Cloud
|
|
- **Training Framework**: [Axolotl](https://github.com/axolotl-ai-cloud/axolotl) by OpenAccess AI Collective
|
|
- **Sanskrit Datasets**: Various Sanskrit literature sources and translation corpora
|
|
- **Community**: Sanskrit NLP research community for datasets and feedback |