SKT-ST-X-0-3B/README.md

---
license: apache-2.0
language:
- en
- hi
tags:
- moe
- slm
- skt-ai-labs
- 3b-model
- mixture-of-experts
- bilingual
library_name: pytorch
pipeline_tag: text-generation
datasets:
- SKT-NRS/SKT-OMNI-CORPUS-2T
model-index:
- name: SKT-ST-X-0-3B-V1
  results:
  - task: {type: Classification}
    dataset: {type: mteb/mtop_domain, name: MTEB MTOPDomainClassification (en)}
    metrics: [{type: accuracy, value: 70.95}]
  - task: {type: Classification}
    dataset: {type: mteb/amazon_polarity, name: MTEB AmazonPolarityClassification}
    metrics: [{type: accuracy, value: 46.88}]
  - task: {type: STS}
    dataset: {type: mteb/biosses-sts, name: MTEB BIOSSES}
    metrics: [{type: cos_sim_pearson, value: 47.19}]
  - task: {type: Reranking}
    dataset: {type: mteb/scidocs-reranking, name: MTEB SciDocsRR}
    metrics: [{type: mrr, value: 28.33}]
  - task: {type: Classification}
    dataset: {type: mteb/tweet_sentiment, name: Tweet Sentiment}
    metrics: [{type: f1, value: 26.51}]
  - task: {type: Clustering}
    dataset: {type: mteb/stackexchange_clustering, name: StackExchange Clustering}
    metrics: [{type: v_measure, value: 35.55}]
---

<div style="max-width: 900px; margin: 0 auto; font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;">

### <span style="color:#B8860B; font-size:24px; font-weight:bold;">SKT</span> <span style="color:#333; font-size:24px; font-weight:bold;">AI LABS </span>

<div style="background: linear-gradient(90deg, #FFD700 0%, #FFAA00 50%, #B8860B 100%); height: 8px; margin: 10px 0 20px 0; border-radius: 4px;"></div>

# <span style="color:#000; font-size:48px; font-weight:800; letter-spacing:-1px;">SKT-ST-X-0-3B-V1</span>

<!-- ==================== EPIC GOLDEN BANNER ==================== -->
<div style="background: linear-gradient(135deg, #FFD700 0%, #FFCC00 35%, #B8860B 70%, #FFD700 100%); 
            padding: 40px 20px; 
            border-radius: 20px; 
            text-align: center; 
            margin: 30px 0;             border: 4px solid #FFFFFF; 
            box-shadow: 0 15px 30px rgba(184, 134, 11, 0.4);">

  <h2 style="color:#000; font-size:36px; font-weight:800; margin:0 0 15px 0; text-transform: uppercase; letter-spacing: 1px;">
   COMPACT MOE POWERHOUSE
  </h2>
  
  <p style="color:#000; font-size:22px; font-weight:600; margin:15px 0;">
    3B Total Params • 1.1B Active • English & Hindi
  </p>
  
  <p style="color:#222; font-size:18px; max-width:800px; margin:0 auto; line-height:1.6;">
    A highly efficient Small Language Model (SLM) built on Mixtral MoE architecture for stability. 
    Delivers intelligent responses with a tiny footprint.
  </p>
</div>

<p align="center">
  <img src="Webp/img.png" alt="SKT AI LABS Model Card" width="100%" style="max-width:600px; border-radius: 16px; border: 4px solid #FFD700; box-shadow: 0 10px 25px rgba(0,0,0,0.2);">
</p>

<!-- Badges -->
<p align="center">
  <a href="https://huggingface.co/sKT-Ai-Labs"><img src="https://img.shields.io/badge/🤗--HuggingFace-yellow?style=for-the-badge&logo=huggingface" alt="HF"></a>
  <a href="https://www.sktailabs.in"><img src="https://img.shields.io/badge/SKT_AI_LABS-green?style=for-the-badge" alt="Website"></a>
  <a href="https://opensource.org/licenses/Apache-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-blue?style=for-the-badge" alt="License"></a>
</p>

<!-- Main Content -->
<div style="background: linear-gradient(135deg, #FFF9E6 0%, #FFF0C2 100%); padding: 30px; border-radius: 16px; border-left: 6px solid #B8860B; margin: 25px 0;">
  <h2 style="color:#B8860B; font-size:32px; margin-bottom:15px;">🏗️ Model Architecture</h2>
  
  <table style="width:100%; border-collapse: collapse; color:#333;">
    <tr style="border-bottom: 1px solid #E6C200;"><td style="padding:10px;"><strong>Total Parameters</strong></td><td style="padding:10px;">~3 Billion</td></tr>
    <tr style="border-bottom: 1px solid #E6C200;"><td style="padding:10px;"><strong>Active Parameters</strong></td><td style="padding:10px;">~1.1 Billion (2 Experts/Token)</td></tr>
    <tr style="border-bottom: 1px solid #E6C200;"><td style="padding:10px;"><strong>Architecture</strong></td><td style="padding:10px;">Mixture of Experts (MoE)</td></tr>
    <tr style="border-bottom: 1px solid #E6C200;"><td style="padding:10px;"><strong>Number of Experts</strong></td><td style="padding:10px;">4</td></tr>
    <tr style="border-bottom: 1px solid #E6C200;"><td style="padding:10px;"><strong>Context Length</strong></td><td style="padding:10px;">8K Tokens</td></tr>
    <tr><td style="padding:10px;"><strong>Training Data</strong></td><td style="padding:10px;">40B Tokens (SKT-OMNI-CORPUS-2T)</td></tr>
  </table>
</div>

<div style="background: #fff; padding: 25px; border-radius: 16px; border: 2px solid #FFD700; margin: 20px 0;">
  <h3 style="color:#B8860B; font-size:24px; margin-top:0;">✨ Key Capabilities</h3>
  <ul style="color:#333; font-size:16px; line-height:2.0;">
    <li><strong>Bilingual Mastery:</strong> Fluent in both English and Hindi.</li>
    <li><strong>Efficient Reasoning:</strong> Logical thinking and problem solving despite small size.</li>
    <li><strong>Basic Coding:</strong> Python scripts, algorithms, and logic debugging.</li>
    <li><strong>Creative Writing:</strong> Stories, poems, and roleplay with personality.</li>
    <li><strong>Knowledge QA:</strong> Accurate general knowledge retrieval.</li>  </ul>
</div>

<hr style="border: 0; height: 1px; background: #FFD700; margin: 40px 0;">

## 🛠️ Quick Start Guide

### Installation
<div style="background: linear-gradient(135deg, #FFFDF0 0%, #FFF8DC 100%); padding: 15px; border-radius: 8px; border: 1px solid #E6C200; margin-bottom: 15px;">

```bash
pip install transformers accelerate torch peft bitsandbytes
```
</div>

### Basic Inference
<div style="background: linear-gradient(135deg, #FFFDF0 0%, #FFF8DC 100%); padding: 15px; border-radius: 8px; border: 1px solid #E6C200;">

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "sKT-Ai-Labs/SKT-ST-X-0-3B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.float16
)

prompt = "What is Quantum Physics?"
formatted = f"<|user|>\n{prompt}\n<|assistant|>\n"
inputs = tokenizer(formatted, return_tensors="pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response.split("<|assistant|>")[-1].strip())
```
</div>

### ⚡ 4-bit Quantization (Low VRAM)
<div style="background: linear-gradient(135deg, #FFFDF0 0%, #FFF8DC 100%); padding: 15px; border-radius: 8px; border: 1px solid #E6C200;">

```python
from transformers import BitsAndBytesConfig

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quant_config,
    device_map="auto"
)
```
</div>

---

## 📊 MTEB Benchmark Highlights

| Task | Dataset | Metric | Score |
| :--- | :--- | :--- | :--- |
| **Classification** | MTOP Domain (en) | Accuracy | **70.95** |
| **Classification** | Amazon Polarity | Accuracy | **46.88** |
| **STS** | BIOSSES | Cosine Pearson | **47.19** |
| **Reranking** | SciDocs RR | MRR | **28.33** |
| **Classification** | Tweet Sentiment | F1 | **26.51** |
| **Clustering** | StackExchange | V-Measure | **35.55** |

*Full benchmark results available in the model metadata.*

---
## ❤️ Support Our Mission

<div style="background: linear-gradient(135deg, #FFF9E6 0%, #FFF0C2 100%); padding: 30px; border-radius: 16px; border: 2px solid #FFD700; text-align: center; margin: 40px 0; box-shadow: 0 5px 15px rgba(0,0,0,0.1);">
  <h3 style="color:#B8860B; font-size:26px; margin-bottom:15px;">🙏 Drop a Heart ❤ For Our Hard Work!</h3>
  <p style="color:#333; font-size:18px; margin-bottom:20px;">
    If you believe in the vision of Sovereign Indian AI, please show your support by dropping a heart below. Your encouragement fuels our journey!
  </p>
  
  <div style="display: flex; justify-content: center; gap: 20px; flex-wrap: wrap;">
    <a href="https://huggingface.co/sKT-Ai-Labs" target="_blank" style="background: linear-gradient(#FFD700, #FFAA00); color: #000; padding: 12px 25px; border-radius: 50px; text-decoration: none; font-weight: bold; font-size: 16px; box-shadow: 0 4px 10px rgba(0,0,0,0.2); transition: transform 0.2s;">
      ❤️ Like This Dataset
    </a>
    <a href="https://huggingface.co/sKT-Ai-Labs" target="_blank" style="background: linear-gradient(#B8860B, #8B4513); color: #fff; padding: 12px 25px; border-radius: 50px; text-decoration: none; font-weight: bold; font-size: 16px; box-shadow: 0 4px 10px rgba(0,0,0,0.2); transition: transform 0.2s;">
      ➕ Follow SKT AI LABS    </a>
  </div>
  
  <p style="color:#666; font-size:15px; margin-top:20px; font-style: italic;">
    Kindly follow us for more updates and contribute to our open-source journey!
  </p>
</div>



## 📜 License & Citation

This model is released under the **Apache-2.0 License**.

-   [View Full License](Index/USED.MD)
-   [Third Party Notices](Index/THIRD_PARTY_NOTICES.MD)

```bibtex
@misc{SKT-ST-X-0-3B,
  author = {SKT AI LABS, India},
  title = {SKT-ST-X-0-3B: A Compact Mixture of Experts Model},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/sKT-Ai-Labs/SKT-ST-X-0-3B}
}
```

<div style="text-align:center; margin-top:50px; padding: 20px; background: #FFF9E6; border-radius: 10px;">
  <h3 style="color:#B8860B; margin:0;">Made with ❤️ by SKT AI LABS</h3>
  <p style="color:#555; margin-top:5px;">Support: support@sktailabs.in</p>
</div>

</div>