---
language:
- ps
- en
- ur
license: apache-2.0
library_name: transformers
tags:
- pashto
- peshawari
- pakistani-pashto
- causal-lm
- qwen2
- sft
- cpt
- unsloth
- trl
base_model: Qwen/Qwen2.5-7B
pipeline_tag: text-generation
---

# ☕ Qehwa — Pashto's First LLM

**The first and best Pakistani Pashto large language model — specifically trained on Peshawari dialect.**

Built by a solo developer as a free and open resource for 60+ million Pashto speakers worldwide.

> ⚠️ This model performs best on Pakistani/Peshawari Pashto. Performance may be lower on Afghan Pashto dialect.

---

## 🌟 Model Description

**Qehwa** is a fully instruction-tuned Pashto language model built on top of Qwen2.5-7B. It is the result of two-stage training:

1. **Continued Pre-Training (CPT)** on 3.4 million clean Pakistani Pashto documents
2. **Supervised Fine-Tuning (SFT)** on 126,519 high-quality Peshawari Pashto instruction-response pairs 

This is the **first dedicated Pakistani Pashto LLM** — no comparable model exists publicly. It specifically targets the **Peshawari/KPK dialect** rather than generic or Afghan Pashto.

This repo contains the **fully merged model** — ready to use with standard transformers, no additional libraries required.

---

## ✨ Capabilities

- ✅ Answers questions in pure Peshawari Pashto
- ✅ Responds to English instructions in Pashto
- ✅ Responds to Urdu instructions in Pashto
- ✅ Natural Pashto conversation
- ✅ Pashto creative writing and poetry
- ✅ Islamic topics in Pashto
- ✅ KPK history, culture, and geography
- ✅ Pashtunwali traditions and ethics
- ✅ Pashto grammar correction
- ✅ English to Pashto translation
- ✅ Correct Pashto-specific characters: ښ ږ ټ ډ ړ ځ

---

## 📊 Evaluation Results

Qehwa was evaluated on a custom benchmark of **150 tests across 15 categories** — the first ever comprehensive Pashto LLM benchmark. Since no standard Pashto benchmark exists publicly, this evaluation was designed specifically for Pakistani Pashto.

### Top Performing Categories

| Category | Score |
|---|---|
| English → Pashto | **90%** 🔥🔥 |
| Urdu → Pashto | **84%** 🔥🔥 |
| Health & Daily Life in Pashto | **90%** 🔥🔥 |
| Culture & History | **90%** 🔥 |
| Geography & Nature | **90%** 🔥 |

> **Overall Average Accuracy across all 15 benchmark categories: 85.3%**

### Evaluation Methodology
- 150 custom Pashto prompts across 15 categories
- Evaluated on A100 40GB GPU
- Human reviewed outputs for fluency, accuracy and dialect correctness
- No existing Pashto benchmark was available — this is the first Pashto LLM benchmark

---

## 💻 Installation
```bash
pip install transformers accelerate torch
```

For faster inference:
```bash
pip install unsloth
```

For running locally on CPU or small GPU:
```bash
pip install transformers accelerate bitsandbytes
```

---

## 🚀 How to Use

### ✅ Method 1 — Transformers (Recommended)

Best for: Research, production, standard usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "junaid008/qehwa-pashto-llm"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model     = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype = torch.bfloat16,
    device_map  = "auto",
)

ALPACA_TEMPLATE = """Below is an instruction in Pashto or English. Write a detailed response in Pashto.

### Instruction:
{}

### Response:
{}"""

def generate(prompt):
    inputs = tokenizer(
        ALPACA_TEMPLATE.format(prompt, ""),
        return_tensors = "pt",
    ).to("cuda")

    outputs = model.generate(
        **inputs,
        max_new_tokens     = 500,
        temperature        = 0.7,
        do_sample          = True,
        repetition_penalty = 1.1,
        pad_token_id       = tokenizer.eos_token_id,
    )

    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response.split("### Response:")[-1].strip()

# Pashto input
print(generate("د پیښور تاریخ راته ووایه"))

# English input
print(generate("Tell me about Pashtunwali"))

# Urdu input
print(generate("پشاور کے بارے میں بتاؤ"))
```

---

### ✅ Method 2 — 4-bit Quantization (Low VRAM)

Best for: GPUs with 8GB VRAM or less
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_name = "junaid008/qehwa-pashto-llm"

bnb_config = BitsAndBytesConfig(
    load_in_4bit              = True,
    bnb_4bit_quant_type       = "nf4",
    bnb_4bit_compute_dtype    = torch.bfloat16,
    bnb_4bit_use_double_quant = True,
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
model     = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config = bnb_config,
    device_map          = "auto",
)

ALPACA_TEMPLATE = """Below is an instruction in Pashto or English. Write a detailed response in Pashto.

### Instruction:
{}

### Response:
{}"""

def generate(prompt):
    inputs = tokenizer(
        ALPACA_TEMPLATE.format(prompt, ""),
        return_tensors = "pt",
    ).to("cuda")

    outputs = model.generate(
        **inputs,
        max_new_tokens     = 500,
        temperature        = 0.7,
        do_sample          = True,
        repetition_penalty = 1.1,
        pad_token_id       = tokenizer.eos_token_id,
    )

    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response.split("### Response:")[-1].strip()

print(generate("پښتونولي تشریح کړه"))
```

---

### ✅ Method 3 — Unsloth (2x Faster Inference)

Best for: Speed-optimized usage, Colab, A100/H100
```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name     = "junaid008/qehwa-pashto-llm",
    max_seq_length = 2048,
    dtype          = None,
    load_in_4bit   = False,
)
FastLanguageModel.for_inference(model)

ALPACA_TEMPLATE = """Below is an instruction in Pashto or English. Write a detailed response in Pashto.

### Instruction:
{}

### Response:
{}"""

import torch
inputs = tokenizer(
    ALPACA_TEMPLATE.format("د پیښور تاریخ راته ووایه", ""),
    return_tensors = "pt",
).to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens     = 500,
    temperature        = 0.7,
    do_sample          = True,
    repetition_penalty = 1.1,
    pad_token_id       = tokenizer.pad_token_id,
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response.split("### Response:")[-1].strip())
```

---

### ✅ Method 4 — CPU Only (No GPU)

Best for: Testing on laptop, no GPU available (slow but works)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "junaid008/qehwa-pashto-llm"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model     = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype = torch.float32,  # float32 for CPU
    device_map  = "cpu",
)

ALPACA_TEMPLATE = """Below is an instruction in Pashto or English. Write a detailed response in Pashto.

### Instruction:
{}

### Response:
{}"""

inputs = tokenizer(
    ALPACA_TEMPLATE.format("پښتو ژبه د چا ده؟", ""),
    return_tensors = "pt",
)

outputs = model.generate(
    **inputs,
    max_new_tokens = 200,
    do_sample      = False,   # greedy for CPU speed
    pad_token_id   = tokenizer.eos_token_id,
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response.split("### Response:")[-1].strip())
```

---

### ✅ Method 5 — Google Colab (Free)

Best for: Trying without any local setup

Open in Colab and run:
```python
# Install
!pip install transformers accelerate -q

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("junaid008/qehwa-pashto-llm")
model     = AutoModelForCausalLM.from_pretrained(
    "junaid008/qehwa-pashto-llm",
    torch_dtype = torch.bfloat16,
    device_map  = "auto",
)

ALPACA_TEMPLATE = """Below is an instruction in Pashto or English. Write a detailed response in Pashto.

### Instruction:
{}

### Response:
{}"""

def generate(prompt):
    inputs  = tokenizer(ALPACA_TEMPLATE.format(prompt, ""), return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=500, temperature=0.7,
                              do_sample=True, pad_token_id=tokenizer.eos_token_id)
    return tokenizer.decode(outputs[0], skip_special_tokens=True).split("### Response:")[-1].strip()

print(generate("Tell me about Peshawar"))
print(generate("پښتونولي تشریح کړه"))
print(generate("پشاور کا مشہور کھانا کیا ہے؟"))
```

---

## ⚙️ Hardware Requirements

| Method | VRAM | Speed |
|---|---|---|
| bfloat16 full | 16GB+ | ✅ Fast |
| 4-bit quantized | 8GB+ | ✅ Good |
| Unsloth | 16GB+ | 🔥 2x Faster |
| CPU only | No GPU | ⚠️ Slow |

---

## 📊 Training Details

### Stage 1 — Continued Pre-Training (CPT)

| Parameter | Value |
|---|---|
| Base model | Qwen/Qwen2.5-7B |
| Hardware | NVIDIA A100-SXM4-40GB |
| Training steps | 5,000 |
| Final CPT loss | ~1.8 |
| Dataset size | 3,400,000 documents |
| Sequence length | 2,048 tokens |
| Precision | bfloat16 |
| LoRA rank | 64 |
| Learning rate | 5e-5 |
| Effective batch size | 32 |

### Stage 2 — Supervised Fine-Tuning (SFT)

| Parameter | Value |
|---|---|
| Base model | junaid008/pashto-qwen2.5-7b-v3 (CPT) |
| Hardware | NVIDIA A100-SXM4-40GB |
| Training steps | 7,908 |
| Final SFT loss | 0.455 |
| Dataset size | 126,519 pairs |
| Epochs | 2 |
| Sequence length | 2,048 tokens |
| Precision | bfloat16 |
| LoRA rank | 64 |
| Learning rate | 5e-5 |
| Effective batch size | 32 |

---

## 📚 Dataset

### CPT Dataset
- 3.4 million Pakistani Pashto documents
- Sources: news, books, religious texts, Wikipedia, web crawl
- Custom cleaned with Pashto-specific Unicode normalization

### SFT Dataset — 126,519 pairs:

| Type | Description |
|---|---|
| Pashto → Pashto | Questions and answers in pure Peshawari Pashto |
| English → Pashto | English instructions with Pashto responses |
| Urdu → Pashto | Urdu instructions with Pashto responses |
| Conversation | Natural dialogue in Peshawari dialect |
| Islamic topics | Religious knowledge in Pashto |
| Creative writing | Pashto poetry, stories, descriptions |
| Grammar | Pashto language correction and explanation |
| Translation | English to Pashto translation pairs |
| Cultural | Pashtunwali, traditions, history, geography |

---

## 🗣️ Example Outputs

**Pashto input:**
```
Prompt : د پیښور تاریخ راته ووایه
Output : پیښور د نړۍ یو له تر ټولو پخوانیو ښارونو څخه دی چې تاریخ یې زرګونه
         کاله پخوا ته رسیږي. د دې ښار پخوانی نوم پوشکلاوتي او بیا پروشاپور و.
         د پیښور د بالاحصار کلا د مغلو او سکهانو د وخت یادګار دی...
```

**English input:**
```
Prompt : Tell me about Pashtunwali
Output : پښتونولي د پښتنو د ژوند هغه لرغونی او سپېڅلی ضابطه اخلاق ده چې د دوی
         ټولنیز ژوند یې تنظیم کړی دی. په دې کې مېلمه پالنه، غیرت، او د کمزورو
         ساتنه تر ټولو لوی ارزښتونه ګڼل کېږي...
```

**Urdu input:**
```
Prompt : پشتونوالی کیا ہے؟
Output : پښتونولي د پښتنو د ژوند هغه لرغونی او زرین ضابطه اخلاق ده...
```

---

## 👤 Author

**Junaid Aslam**
- HuggingFace: [junaid008](https://huggingface.co/junaid008)
- Built independently as a contribution to Pashto NLP

---

## 📜 License

Apache 2.0 — free to use, modify, and distribute with attribution.

---

## 🤝 Citation
```bibtex
@misc{qehwa-pashto-llm,
  author    = {Junaid Aslam},
  title     = {Qehwa — Pashto's First LLM},
  year      = {2026},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/junaid008/qehwa-pashto-llm}
}
```