---
library_name: transformers
license: apache-2.0
language:
- en
base_model: distilbert/distilgpt2
tags:
- text-generation
- causal-lm
- arabic
- fine-tuned
- generated_from_trainer
model-index:
- name: my_awesome_eli5_clm-model
  results: []
---

# my_awesome_eli5_clm-model

A Causal Language Model fine-tuned on Arabic text, based on [distilbert/distilgpt2](https://huggingface.co/distilbert/distilgpt2).  
Fine-tuned by **EhabSuliman** as part of an LLM course project.

## Model Description

- **Model type:** Causal Language Model (CLM)
- **Base model:** [distilbert/distilgpt2](https://huggingface.co/distilbert/distilgpt2)
- **Language:** English 
- **Fine-tuned by:** [EhabSuliman](https://huggingface.co/EhabSuliman)
- **License:** Apache 2.0

## Intended Uses & Limitations

**Intended uses:**
- Arabic text generation
- Language modeling research
- Educational/learning purposes

**Limitations:**
- Trained on a relatively small dataset
- Loss is still relatively high (3.8027), meaning the model may generate inaccurate or repetitive text
- Not recommended for production use without further fine-tuning

## How to Use

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("EhabSuliman/my_awesome_eli5_clm-model")
model = AutoModelForCausalLM.from_pretrained("EhabSuliman/my_awesome_eli5_clm-model")

prompt = "Somatic hypermutation allows the immune system to"
inputs = tokenizer(prompt, return_tensors="pt").input_ids
outputs = model.generate(
    inputs,
    max_new_tokens=100,
    do_sample=True,
    top_k=50,
    top_p=0.95
)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
```

## Training and Evaluation Data

Fine-tuned on an Arabic text dataset using the ELI5 (Explain Like I'm 5) format.

## Training Procedure

### Training Hyperparameters

The following hyperparameters were used during training:

- **Learning rate:** 2e-05
- **Train batch size:** 8
- **Eval batch size:** 8
- **Seed:** 42
- **Optimizer:** AdamW (fused) with betas=(0.9, 0.999), epsilon=1e-08
- **LR scheduler:** Linear
- **Epochs:** 3

### Training Results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 3.8556        | 1.0   | 1327 | 3.8101          |
| 3.7851        | 2.0   | 2654 | 3.8035          |
| 3.7514        | 3.0   | 3981 | 3.8027          |

> The model shows steady improvement across epochs with validation loss decreasing from 3.8101 → 3.8027.

### Framework Versions

- Transformers 5.9.0
- PyTorch 2.11.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2

## Author

**Ehab Suliman** — Machine Learning Engineer  
🔗 [HuggingFace Profile](https://huggingface.co/EhabSuliman)