初始化项目，由ModelHub XC社区提供模型
Model: tasal9/pashto-base-bloom Source: Original Platform
This commit is contained in:
ModelHub XC
40
.gitattributes
vendored
Normal file
40
.gitattributes
vendored
Normal file
@@ -0,0 +1,40 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
checkpoint-2829/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
checkpoint-5658/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
checkpoint-8487/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
final_model/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
181
ModelCard.md
Normal file
181
ModelCard.md
Normal file
@@ -0,0 +1,181 @@
|
||||
---
|
||||
---
|
||||
license: mit # Or your chosen license: apache-2.0, cc-by-4.0, etc.
|
||||
language:
|
||||
- ps # Pashto
|
||||
library_name: transformers
|
||||
tags:
|
||||
- text-generation
|
||||
- pashto
|
||||
- bloom
|
||||
- zamai-bloom
|
||||
datasets:
|
||||
|
||||
**Note on Dataset Identifiers:**
|
||||
The `datasets` field in the metadata of this model card might list `tasal9/pashto_base_bloom`. This identifier may refer to an earlier version or a different collection of Pashto data. The specific training run culminating in this model update (June 2025) exclusively used the locally processed `datasets/base_pashto_clean` as described above.
|
||||
|
||||
|
||||
- tasal9/pashto_base_bloom
|
||||
pipeline_tag: text-generation
|
||||
widget:
|
||||
- text: "پښتو ژبه"
|
||||
---
|
||||
|
||||
# ZamAI Bloom Pashto - checkpoint5207 (and Final Model)
|
||||
|
||||
This model card is for `checkpoint5207` and the final fine-tuned version of a Bloom model for Pashto text generation, developed under the ZamAI Bloom project.
|
||||
|
||||
## Model Description
|
||||
|
||||
This model is a fine-tuned version of [bigscience/bloom-560m](https://huggingface.co/bigscience/bloom-560m) on a Pashto text corpus. The goal of this project was to create a language model proficient in generating coherent and contextually relevant Pashto text.
|
||||
|
||||
**Base Model:** `bigscience/bloom-560m`
|
||||
**Fine-tuning Checkpoint:** `checkpoint5207`
|
||||
**Final Model:** [tasal9/zamai-bloom-ps-final]
|
||||
|
||||
## Intended Uses & Limitations
|
||||
|
||||
### Intended Uses
|
||||
|
||||
This model is intended for:
|
||||
* Generating Pashto text.
|
||||
* Assisting with Pashto language content creation.
|
||||
* Research in Pashto NLP.
|
||||
* Educational purposes for Pashto language learning.
|
||||
|
||||
### Limitations and Bias
|
||||
|
||||
* The model's performance is dependent on the quality and diversity of the training data. It may generate text that reflects biases present in the data.
|
||||
* It might produce factually incorrect or nonsensical text, especially for complex topics or out-of-domain prompts.
|
||||
* The model may not be suitable for critical applications without further evaluation and mitigation of potential harms.
|
||||
* Performance on specific Pashto dialects might vary depending on their representation in the training data.
|
||||
|
||||
## How to use
|
||||
|
||||
You can use this model with the Hugging Face `transformers` library for text generation.
|
||||
|
||||
First, install the library:
|
||||
```bash
|
||||
pip install transformers torch
|
||||
```
|
||||
|
||||
Then, you can use the model in Python:
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_name = "tasal9/zamai-bloom-ps-final" # Or the specific checkpoint identifier if using a checkpoint directly
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_name)
|
||||
|
||||
prompt = "په پښتو ژبه کې یو شعر ولیکئ د پسرلي په اړه" # Example prompt: "Write a poem in Pashto about spring"
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
|
||||
# Generate text
|
||||
# Adjust generation parameters as needed (max_length, num_beams, do_sample, top_k, top_p, etc.)
|
||||
outputs = model.generate(**inputs, max_length=100, num_beams=5, early_stopping=True)
|
||||
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||||
|
||||
print(generated_text)
|
||||
```
|
||||
|
||||
## Training Data
|
||||
|
||||
Describe the dataset(s) used for fine-tuning.
|
||||
* **Source:** [e.g., Web scraped data, specific Pashto corpora, data from `datasets/base_pashto/`]
|
||||
* **Size:** [e.g., Number of documents, tokens, GBs]
|
||||
* **Preprocessing:** [e.g., Cleaning steps, tokenization details]
|
||||
* **Language Variety:** [e.g., Predominant dialects, formal/informal text]
|
||||
|
||||
If your dataset is on the Hugging Face Hub, link to it.
|
||||
|
||||
## Training Procedure
|
||||
|
||||
### Preprocessing
|
||||
|
||||
The texts were tokenized using the `AutoTokenizer` associated with the base Bloom model.
|
||||
[Add any other specific preprocessing steps you took.]
|
||||
|
||||
### Fine-tuning
|
||||
|
||||
The model was fine-tuned using the Hugging Face `transformers` library with PyTorch.
|
||||
* **Training script:** [Link to your `train_base_model.py` if applicable]
|
||||
* **Hyperparameters:**
|
||||
* Learning rate: 2e-5
|
||||
* Batch size: 4 # Adjust based on your GPU memory (e.g., 8, 16)
|
||||
* Number of epochs: 3 # Adjust based on convergence and overfitting
|
||||
* Optimizer: AdamW
|
||||
* Weight decay: 0.01
|
||||
* Warmup steps: 500 # Or warmup_ratio, e.g., 0.1
|
||||
* Gradient accumulation steps: 1 # Increase if actual batch size is limited by memory
|
||||
* Seed: 42 # For reproducibility
|
||||
* **Infrastructure:**
|
||||
* Hardware: [e.g., 1x NVIDIA A100 40GB, or specify your hardware]
|
||||
* Training time: [e.g., X hours]
|
||||
|
||||
This specific model card refers to `checkpoint5207`, which was saved at step 5207 of the training process. The final model represents the model after the completion of all training epochs/steps.
|
||||
|
||||
## Evaluation Results
|
||||
|
||||
Provide quantitative results if available (e.g., perplexity, BLEU scores on a held-out test set).
|
||||
* **Test set:** [Describe your test set]
|
||||
* **Metrics:** [e.g., Perplexity, BLEU, ROUGE]
|
||||
* **Results for checkpoint5207:**
|
||||
* [Metric 1]: [Value]
|
||||
* [Metric 2]: [Value]
|
||||
* **Results for final model:**
|
||||
* [Metric 1]: [Value]
|
||||
* [Metric 2]: [Value]
|
||||
|
||||
Qualitative observations can also be included.
|
||||
|
||||
## Model Card Contact
|
||||
|
||||
**Author:** Yaqoob Tasal
|
||||
**Username:** tasal9
|
||||
**Organization:** ZamAI
|
||||
[GitHub: https://github.com/tasal9](https://github.com/tasal9)
|
||||
|
||||
## Citation
|
||||
|
||||
If you use this model or its checkpoints, please consider citing:
|
||||
|
||||
```bibtex
|
||||
@misc{zamai_bloom_pashto_2025,
|
||||
author = {Yaqoob Tasal},
|
||||
title = {ZamAI Bloom Pashto - Fine-tuned Language Model},
|
||||
year = {2025},
|
||||
publisher = {Hugging Face},
|
||||
journal = {Hugging Face Model Hub},
|
||||
howpublished = {\url{https://huggingface.co/tasal9/zamai-bloom-ps-final}}
|
||||
}
|
||||
```
|
||||
|
||||
And the original Bloom model:
|
||||
```bibtex
|
||||
@article{scao2022bloom,
|
||||
title={BLOOM: A 176B-Parameter Open-Access Multilingual Language Model},
|
||||
author={Scao, Teven Le and Fan, Angela and Akiki, Christopher and Baran, Efrat and Ben Cheikh, Rim and Coavoux, Maxime and Davison, Thomas and de Vargas, Niklas Deckers and Delangue, C{\'e}line and Demeusy, Thibault and others},
|
||||
journal={arXiv preprint arXiv:2211.05100},
|
||||
year={2022}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Remember to replace placeholders like dataset details, hyperparameters, and evaluation results with your actual project details. Save this as a `README.md` file in your model repository on the Hugging Face Hub.
|
||||
|
||||
## Training Details (Cleaned Base Model - June 2025)
|
||||
|
||||
This model version was trained from `bigscience/bloom-560m` using the `train_base_model.py` script.
|
||||
|
||||
- **Training Data:** The model was trained on a locally prepared dataset located at `datasets/base_pashto_clean`. This dataset was created using `prepare_base_dataset.py` and is derived from `pashto_data/base_model/cleaned_base_data.txt`, which primarily contains Pashto text from a bilingual Pashto-English glossary.
|
||||
- **Training Objective:** To establish a foundational Pashto language model with improved coherence and reduced issues (e.g., repetition, off-language generation) compared to any prior versions trained on noisier data.
|
||||
- **Output Directory (during training):** `models/pashto-bloom-base-clean-colab`
|
||||
- **Key Training Hyperparameters:**
|
||||
- Epochs: 3
|
||||
- Per Device Batch Size: 2
|
||||
- Gradient Accumulation Steps: 4
|
||||
- Learning Rate: 5e-5
|
||||
- FP16 (Mixed Precision): True
|
||||
- Optimizer: AdamW
|
||||
93
README.md
Normal file
93
README.md
Normal file
@@ -0,0 +1,93 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
- ps
|
||||
license: mit
|
||||
library_name: transformers
|
||||
tags:
|
||||
- bloom
|
||||
- pashto
|
||||
- lightweight
|
||||
- language-model
|
||||
base_model: bigscience/bloomz-560m
|
||||
pipeline_tag: text-generation
|
||||
datasets:
|
||||
- tasal9/Pashto-Dataset-Creating-Dataset
|
||||
---
|
||||
|
||||
# pashto-base-bloom
|
||||
|
||||
BLOOM-based model fine-tuned for Pashto language tasks
|
||||
|
||||
## 🌟 Model Overview
|
||||
|
||||
This model is part of the **ZamAI Pro Models Strategy** - a comprehensive AI platform designed for multilingual applications with specialized focus on Pashto language support.
|
||||
|
||||
### Key Features
|
||||
- 🧠 **Advanced AI**: Based on bigscience/bloomz-560m architecture
|
||||
- 🌐 **Multilingual**: Optimized for Pashto and English
|
||||
- ⚡ **High Performance**: Optimized for production deployment
|
||||
- 🔒 **Secure**: Enterprise-grade security and privacy
|
||||
|
||||
## 📚 Usage
|
||||
|
||||
### Basic Usage with Transformers
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModel
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("tasal9/pashto-base-bloom")
|
||||
model = AutoModel.from_pretrained("tasal9/pashto-base-bloom")
|
||||
|
||||
# Example usage
|
||||
text = "Your input text here"
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
outputs = model(**inputs)
|
||||
```
|
||||
|
||||
### Usage with Hugging Face Inference API
|
||||
|
||||
```python
|
||||
from huggingface_hub import InferenceClient
|
||||
|
||||
client = InferenceClient(token="your_hf_token")
|
||||
|
||||
response = client.text_generation(
|
||||
model="tasal9/pashto-base-bloom",
|
||||
prompt="Your prompt here",
|
||||
max_new_tokens=200
|
||||
)
|
||||
```
|
||||
|
||||
## 🔧 Technical Details
|
||||
|
||||
- **Model Type**: text-generation
|
||||
- **Base Model**: bigscience/bloomz-560m
|
||||
- **Languages**: Pashto (ps), English (en)
|
||||
- **License**: MIT
|
||||
- **Training**: Fine-tuned on Pashto educational and cultural content
|
||||
|
||||
## 🚀 Applications
|
||||
|
||||
This model powers:
|
||||
- **ZamAI Educational Platform**: Pashto language tutoring
|
||||
- **Business Automation**: Document processing and analysis
|
||||
- **Voice Assistants**: Natural language understanding
|
||||
- **Cultural Preservation**: Supporting Pashto language technology
|
||||
|
||||
## 📞 Support
|
||||
|
||||
For support and integration assistance:
|
||||
- 📧 **Email**: support@zamai.dev
|
||||
- 🌐 **Website**: [zamai.dev](https://zamai.dev)
|
||||
- 💬 **Community**: [ZamAI Community](https://zamai.dev)
|
||||
|
||||
## 📄 License
|
||||
|
||||
Licensed under the MIT License.
|
||||
|
||||
---
|
||||
|
||||
**Part of the ZamAI Pro Models Strategy - Transforming AI for Multilingual Applications** 🌟
|
||||
|
||||
*Updated: 2025-07-05 21:29:16 UTC*
|
||||
31
config.json
Normal file
31
config.json
Normal file
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"apply_residual_connection_post_layernorm": false,
|
||||
"architectures": [
|
||||
"BloomForCausalLM"
|
||||
],
|
||||
"attention_dropout": 0.0,
|
||||
"attention_softmax_in_fp32": true,
|
||||
"bias_dropout_fusion": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"hidden_dropout": 0.0,
|
||||
"hidden_size": 1024,
|
||||
"initializer_range": 0.02,
|
||||
"layer_norm_epsilon": 1e-05,
|
||||
"masked_softmax_fusion": true,
|
||||
"model_type": "bloom",
|
||||
"n_head": 16,
|
||||
"n_inner": null,
|
||||
"n_layer": 24,
|
||||
"offset_alibi": 100,
|
||||
"pad_token_id": 3,
|
||||
"pretraining_tp": 1,
|
||||
"skip_bias_add": true,
|
||||
"skip_bias_add_qkv": false,
|
||||
"slow_but_exact": false,
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": "4.52.4",
|
||||
"unk_token_id": 0,
|
||||
"use_cache": true,
|
||||
"vocab_size": 250680
|
||||
}
|
||||
7
generation_config.json
Normal file
7
generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"pad_token_id": 3,
|
||||
"transformers_version": "4.52.4"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:c4bd4ca3159b4e8807d2c68cb2f5f8663a4c1617e4fd74edd7c7e51dab6817d5
|
||||
size 2236073104
|
||||
30
special_tokens_map.json
Normal file
30
special_tokens_map.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d963066d6adae5034a1dc114c3ac444512de09928cf14ed4562ba94d9a440e66
|
||||
size 21763085
|
||||
48
tokenizer_config.json
Normal file
48
tokenizer_config.json
Normal file
@@ -0,0 +1,48 @@
|
||||
{
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"3": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"bos_token": "<s>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "</s>",
|
||||
"extra_special_tokens": {},
|
||||
"merges_file": null,
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"pad_token": "<pad>",
|
||||
"padding_side": "left",
|
||||
"tokenizer_class": "BloomTokenizer",
|
||||
"unk_token": "<unk>",
|
||||
"vocab_file": null
|
||||
}
|
||||
Reference in New Issue
Block a user