101 lines
2.7 KiB
Markdown
101 lines
2.7 KiB
Markdown
---
|
|
library_name: transformers
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
base_model: distilbert/distilgpt2
|
|
tags:
|
|
- text-generation
|
|
- causal-lm
|
|
- arabic
|
|
- fine-tuned
|
|
- generated_from_trainer
|
|
model-index:
|
|
- name: my_awesome_eli5_clm-model
|
|
results: []
|
|
---
|
|
|
|
# my_awesome_eli5_clm-model
|
|
|
|
A Causal Language Model fine-tuned on Arabic text, based on [distilbert/distilgpt2](https://huggingface.co/distilbert/distilgpt2).
|
|
Fine-tuned by **EhabSuliman** as part of an LLM course project.
|
|
|
|
## Model Description
|
|
|
|
- **Model type:** Causal Language Model (CLM)
|
|
- **Base model:** [distilbert/distilgpt2](https://huggingface.co/distilbert/distilgpt2)
|
|
- **Language:** English
|
|
- **Fine-tuned by:** [EhabSuliman](https://huggingface.co/EhabSuliman)
|
|
- **License:** Apache 2.0
|
|
|
|
## Intended Uses & Limitations
|
|
|
|
**Intended uses:**
|
|
- Arabic text generation
|
|
- Language modeling research
|
|
- Educational/learning purposes
|
|
|
|
**Limitations:**
|
|
- Trained on a relatively small dataset
|
|
- Loss is still relatively high (3.8027), meaning the model may generate inaccurate or repetitive text
|
|
- Not recommended for production use without further fine-tuning
|
|
|
|
## How to Use
|
|
|
|
```python
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("EhabSuliman/my_awesome_eli5_clm-model")
|
|
model = AutoModelForCausalLM.from_pretrained("EhabSuliman/my_awesome_eli5_clm-model")
|
|
|
|
prompt = "Somatic hypermutation allows the immune system to"
|
|
inputs = tokenizer(prompt, return_tensors="pt").input_ids
|
|
outputs = model.generate(
|
|
inputs,
|
|
max_new_tokens=100,
|
|
do_sample=True,
|
|
top_k=50,
|
|
top_p=0.95
|
|
)
|
|
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
|
|
```
|
|
|
|
## Training and Evaluation Data
|
|
|
|
Fine-tuned on an Arabic text dataset using the ELI5 (Explain Like I'm 5) format.
|
|
|
|
## Training Procedure
|
|
|
|
### Training Hyperparameters
|
|
|
|
The following hyperparameters were used during training:
|
|
|
|
- **Learning rate:** 2e-05
|
|
- **Train batch size:** 8
|
|
- **Eval batch size:** 8
|
|
- **Seed:** 42
|
|
- **Optimizer:** AdamW (fused) with betas=(0.9, 0.999), epsilon=1e-08
|
|
- **LR scheduler:** Linear
|
|
- **Epochs:** 3
|
|
|
|
### Training Results
|
|
|
|
| Training Loss | Epoch | Step | Validation Loss |
|
|
|:-------------:|:-----:|:----:|:---------------:|
|
|
| 3.8556 | 1.0 | 1327 | 3.8101 |
|
|
| 3.7851 | 2.0 | 2654 | 3.8035 |
|
|
| 3.7514 | 3.0 | 3981 | 3.8027 |
|
|
|
|
> The model shows steady improvement across epochs with validation loss decreasing from 3.8101 → 3.8027.
|
|
|
|
### Framework Versions
|
|
|
|
- Transformers 5.9.0
|
|
- PyTorch 2.11.0+cu128
|
|
- Datasets 4.0.0
|
|
- Tokenizers 0.22.2
|
|
|
|
## Author
|
|
|
|
**Ehab Suliman** — Machine Learning Engineer
|
|
🔗 [HuggingFace Profile](https://huggingface.co/EhabSuliman) |