182 lines
7.0 KiB
Markdown
182 lines
7.0 KiB
Markdown
---
|
|
---
|
|
license: mit # Or your chosen license: apache-2.0, cc-by-4.0, etc.
|
|
language:
|
|
- ps # Pashto
|
|
library_name: transformers
|
|
tags:
|
|
- text-generation
|
|
- pashto
|
|
- bloom
|
|
- zamai-bloom
|
|
datasets:
|
|
|
|
**Note on Dataset Identifiers:**
|
|
The `datasets` field in the metadata of this model card might list `tasal9/pashto_base_bloom`. This identifier may refer to an earlier version or a different collection of Pashto data. The specific training run culminating in this model update (June 2025) exclusively used the locally processed `datasets/base_pashto_clean` as described above.
|
|
|
|
|
|
- tasal9/pashto_base_bloom
|
|
pipeline_tag: text-generation
|
|
widget:
|
|
- text: "پښتو ژبه"
|
|
---
|
|
|
|
# ZamAI Bloom Pashto - checkpoint5207 (and Final Model)
|
|
|
|
This model card is for `checkpoint5207` and the final fine-tuned version of a Bloom model for Pashto text generation, developed under the ZamAI Bloom project.
|
|
|
|
## Model Description
|
|
|
|
This model is a fine-tuned version of [bigscience/bloom-560m](https://huggingface.co/bigscience/bloom-560m) on a Pashto text corpus. The goal of this project was to create a language model proficient in generating coherent and contextually relevant Pashto text.
|
|
|
|
**Base Model:** `bigscience/bloom-560m`
|
|
**Fine-tuning Checkpoint:** `checkpoint5207`
|
|
**Final Model:** [tasal9/zamai-bloom-ps-final]
|
|
|
|
## Intended Uses & Limitations
|
|
|
|
### Intended Uses
|
|
|
|
This model is intended for:
|
|
* Generating Pashto text.
|
|
* Assisting with Pashto language content creation.
|
|
* Research in Pashto NLP.
|
|
* Educational purposes for Pashto language learning.
|
|
|
|
### Limitations and Bias
|
|
|
|
* The model's performance is dependent on the quality and diversity of the training data. It may generate text that reflects biases present in the data.
|
|
* It might produce factually incorrect or nonsensical text, especially for complex topics or out-of-domain prompts.
|
|
* The model may not be suitable for critical applications without further evaluation and mitigation of potential harms.
|
|
* Performance on specific Pashto dialects might vary depending on their representation in the training data.
|
|
|
|
## How to use
|
|
|
|
You can use this model with the Hugging Face `transformers` library for text generation.
|
|
|
|
First, install the library:
|
|
```bash
|
|
pip install transformers torch
|
|
```
|
|
|
|
Then, you can use the model in Python:
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_name = "tasal9/zamai-bloom-ps-final" # Or the specific checkpoint identifier if using a checkpoint directly
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
model = AutoModelForCausalLM.from_pretrained(model_name)
|
|
|
|
prompt = "په پښتو ژبه کې یو شعر ولیکئ د پسرلي په اړه" # Example prompt: "Write a poem in Pashto about spring"
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
|
|
|
# Generate text
|
|
# Adjust generation parameters as needed (max_length, num_beams, do_sample, top_k, top_p, etc.)
|
|
outputs = model.generate(**inputs, max_length=100, num_beams=5, early_stopping=True)
|
|
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
|
|
|
print(generated_text)
|
|
```
|
|
|
|
## Training Data
|
|
|
|
Describe the dataset(s) used for fine-tuning.
|
|
* **Source:** [e.g., Web scraped data, specific Pashto corpora, data from `datasets/base_pashto/`]
|
|
* **Size:** [e.g., Number of documents, tokens, GBs]
|
|
* **Preprocessing:** [e.g., Cleaning steps, tokenization details]
|
|
* **Language Variety:** [e.g., Predominant dialects, formal/informal text]
|
|
|
|
If your dataset is on the Hugging Face Hub, link to it.
|
|
|
|
## Training Procedure
|
|
|
|
### Preprocessing
|
|
|
|
The texts were tokenized using the `AutoTokenizer` associated with the base Bloom model.
|
|
[Add any other specific preprocessing steps you took.]
|
|
|
|
### Fine-tuning
|
|
|
|
The model was fine-tuned using the Hugging Face `transformers` library with PyTorch.
|
|
* **Training script:** [Link to your `train_base_model.py` if applicable]
|
|
* **Hyperparameters:**
|
|
* Learning rate: 2e-5
|
|
* Batch size: 4 # Adjust based on your GPU memory (e.g., 8, 16)
|
|
* Number of epochs: 3 # Adjust based on convergence and overfitting
|
|
* Optimizer: AdamW
|
|
* Weight decay: 0.01
|
|
* Warmup steps: 500 # Or warmup_ratio, e.g., 0.1
|
|
* Gradient accumulation steps: 1 # Increase if actual batch size is limited by memory
|
|
* Seed: 42 # For reproducibility
|
|
* **Infrastructure:**
|
|
* Hardware: [e.g., 1x NVIDIA A100 40GB, or specify your hardware]
|
|
* Training time: [e.g., X hours]
|
|
|
|
This specific model card refers to `checkpoint5207`, which was saved at step 5207 of the training process. The final model represents the model after the completion of all training epochs/steps.
|
|
|
|
## Evaluation Results
|
|
|
|
Provide quantitative results if available (e.g., perplexity, BLEU scores on a held-out test set).
|
|
* **Test set:** [Describe your test set]
|
|
* **Metrics:** [e.g., Perplexity, BLEU, ROUGE]
|
|
* **Results for checkpoint5207:**
|
|
* [Metric 1]: [Value]
|
|
* [Metric 2]: [Value]
|
|
* **Results for final model:**
|
|
* [Metric 1]: [Value]
|
|
* [Metric 2]: [Value]
|
|
|
|
Qualitative observations can also be included.
|
|
|
|
## Model Card Contact
|
|
|
|
**Author:** Yaqoob Tasal
|
|
**Username:** tasal9
|
|
**Organization:** ZamAI
|
|
[GitHub: https://github.com/tasal9](https://github.com/tasal9)
|
|
|
|
## Citation
|
|
|
|
If you use this model or its checkpoints, please consider citing:
|
|
|
|
```bibtex
|
|
@misc{zamai_bloom_pashto_2025,
|
|
author = {Yaqoob Tasal},
|
|
title = {ZamAI Bloom Pashto - Fine-tuned Language Model},
|
|
year = {2025},
|
|
publisher = {Hugging Face},
|
|
journal = {Hugging Face Model Hub},
|
|
howpublished = {\url{https://huggingface.co/tasal9/zamai-bloom-ps-final}}
|
|
}
|
|
```
|
|
|
|
And the original Bloom model:
|
|
```bibtex
|
|
@article{scao2022bloom,
|
|
title={BLOOM: A 176B-Parameter Open-Access Multilingual Language Model},
|
|
author={Scao, Teven Le and Fan, Angela and Akiki, Christopher and Baran, Efrat and Ben Cheikh, Rim and Coavoux, Maxime and Davison, Thomas and de Vargas, Niklas Deckers and Delangue, C{\'e}line and Demeusy, Thibault and others},
|
|
journal={arXiv preprint arXiv:2211.05100},
|
|
year={2022}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
Remember to replace placeholders like dataset details, hyperparameters, and evaluation results with your actual project details. Save this as a `README.md` file in your model repository on the Hugging Face Hub.
|
|
|
|
## Training Details (Cleaned Base Model - June 2025)
|
|
|
|
This model version was trained from `bigscience/bloom-560m` using the `train_base_model.py` script.
|
|
|
|
- **Training Data:** The model was trained on a locally prepared dataset located at `datasets/base_pashto_clean`. This dataset was created using `prepare_base_dataset.py` and is derived from `pashto_data/base_model/cleaned_base_data.txt`, which primarily contains Pashto text from a bilingual Pashto-English glossary.
|
|
- **Training Objective:** To establish a foundational Pashto language model with improved coherence and reduced issues (e.g., repetition, off-language generation) compared to any prior versions trained on noisier data.
|
|
- **Output Directory (during training):** `models/pashto-bloom-base-clean-colab`
|
|
- **Key Training Hyperparameters:**
|
|
- Epochs: 3
|
|
- Per Device Batch Size: 2
|
|
- Gradient Accumulation Steps: 4
|
|
- Learning Rate: 5e-5
|
|
- FP16 (Mixed Precision): True
|
|
- Optimizer: AdamW
|