初始化项目，由ModelHub XC社区提供模型
Model: riotu-lab/ArabianGPT-01B Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
47
LICENSE
Normal file
47
LICENSE
Normal file
@@ -0,0 +1,47 @@
|
||||
==========================================
|
||||
SOFTWARE LICENSE AGREEMENT - ArabianGPT
|
||||
==========================================
|
||||
|
||||
* NAME: ArabianGPT: Advanced Language Model for Arabic Text Generation and Understanding
|
||||
|
||||
* ACKNOWLEDGMENTS
|
||||
|
||||
This [software] was developed by the team at [Prince Sultan University, Riyadh, Saudi Arabia] (“Owners”).
|
||||
The statements made herein are solely the responsibility of the author[s].
|
||||
|
||||
* LICENSE
|
||||
This software is being provided to you, the LICENSEE, by the Owners under the following license.
|
||||
By obtaining, using, and/or copying this software, you agree that you have read, understood, and will comply with these terms and conditions.
|
||||
Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee or royalty
|
||||
is hereby granted, provided that you agree to comply with the following copyright notice and statements,
|
||||
including the disclaimer, and that the same appear on ALL copies of the software, documentation,
|
||||
and modifications that you make for internal use or for distribution.
|
||||
|
||||
> ArabianGPT Copyright 2023 by Prince Sultan University, Riyadh, Saudi Arabia. All rights reserved.
|
||||
|
||||
If you remix, transform, or build upon the material, you must distribute your contributions under the same license as this one.
|
||||
You may not apply legal terms or technological measures that legally restrict others from doing anything this license permits.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED "AS IS" AND THE OWNERS MAKE NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED.
|
||||
BY WAY OF EXAMPLE, BUT NOT LIMITATION, THE OWNERS MAKE NO REPRESENTATIONS OR WARRANTIES OF MERCHANT-ABILITY OR FITNESS
|
||||
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE LICENSED SOFTWARE WILL NOT INFRINGE ANY THIRD PARTY PATENTS, COPYRIGHTS,
|
||||
TRADEMARKS OR OTHER RIGHTS.
|
||||
|
||||
The name of the Owners may not be used in advertising or publicity pertaining to the distribution of the software.
|
||||
Title to copyright in this software and any associated documentation shall at all times remain with
|
||||
the Owners and LICENSEE agrees to preserve the same.
|
||||
|
||||
|
||||
* CITATION
|
||||
The use of ArabianGPT should be cited as follows:
|
||||
|
||||
@misc{ArabianGPT, 2023,
|
||||
title={ArabianGPT: A GPT-2 Based Language Model for Arabic},
|
||||
author={Najar, Omar and Sibaee, Serry and Ghouti, Lahouari and Koubaa, Anis},
|
||||
affiliation={Prince Sultan University, Riyadh, Saudi Arabia},
|
||||
year={2023},
|
||||
}
|
||||
|
||||
ArabianGPT Copyright 2023 by Prince Sultan University, Riyadh, Saudi Arabia. All rights reserved.
|
||||
|
||||
===============
|
||||
82
README.md
Normal file
82
README.md
Normal file
@@ -0,0 +1,82 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- ar
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- 'arabic '
|
||||
- text-generation
|
||||
widget:
|
||||
- text: "أعلنت وزارة الحج في المملكة العربية السعودية"
|
||||
example_title: "مثال ١"
|
||||
- text: "يبدو اليوم جميلا، سأقوم بتحضير"
|
||||
example_title: "مثال ٢"
|
||||
- text: "إن التقنيات الحديثة"
|
||||
example_title: "مثال ٣"
|
||||
---
|
||||
# ArabianGPT Model Overview
|
||||
|
||||
## Disclaimer for the Use of Large Language Models (LLMs) for Text Generation
|
||||
|
||||
<p style="color: red;">We disclaim all responsibility for any harm, inaccuracies, or inappropriate content generated by ArabianGPT-0.1B, and users engage with and apply the model's outputs at their own risk.</p>
|
||||
|
||||
> **Important Note:** Currently, we offer a raw pre-trained model. Our team is actively working on releasing instruction-based LLMs that are fine-tuned and augmented with LRHF. The first set of pre-trained models has been made available for community exploration. While we do have models fine-tuned for specific tasks such as summarization and sentiment analysis, they are still in the development phase.
|
||||
|
||||
|
||||
## How you can use this Pre-Trained?
|
||||
You are invited to utilize this pre-trained, native Arabic language model as an experimental tool to assess its capabilities, aid in its fine-tuning, and evaluate its performance across a variety of downstream tasks. We encourage you to review our technical report for a comprehensive understanding of the model's performance metrics and the specific downstream tasks it has been tested on. This will provide valuable insights into its applicability and effectiveness in diverse applications.
|
||||
|
||||
|
||||
## Introduction
|
||||
ArabianGPT-0.1B, developed under the ArabianLLM initiatives, is a specialized GPT-2 model optimized for Arabic language modeling.
|
||||
It's a product of the collaborative efforts at Prince Sultan University's Robotics and Internet of Things Lab, focusing on enhancing natural language modeling and generation in Arabic.
|
||||
This model represents a significant stride in LLM research, specifically addressing the linguistic complexities and nuances of the Arabic language.
|
||||
|
||||
## Key Features
|
||||
- **Architecture**: GPT-2
|
||||
- **Model Size**: 134 million parameters
|
||||
- **Layers**: 12
|
||||
- **Model Attention Layers (MAL)**: 12
|
||||
- **Context Window Size**: 768 tokens
|
||||
|
||||
## Training
|
||||
- **Dataset**: Scraped Arabic newspaper articles
|
||||
- **Data Size**: 15.5 GB
|
||||
- **Words**: 237.8 million
|
||||
- **Tokenizer**: Aranizer 64K
|
||||
- **Tokens**: Over 1.75 billion
|
||||
- **Hardware**: 2 NDIVIA A100 GPUs
|
||||
- **Training Scale**: 7.5 million examples
|
||||
- **Training Duration**: 3 days
|
||||
- **Performance**: Final loss of 3.97
|
||||
|
||||
|
||||
## Role in ArabianLLM Initiatives
|
||||
ArabianGPT-0.1B (Base Model) is crucial for advancing Arabic language processing, addressing challenges unique to Arabic morphology and dialects.
|
||||
|
||||
## Usage
|
||||
Suitable for Arabic text generation tasks. Example usage with Transformers Pipeline:
|
||||
```python
|
||||
from transformers import pipeline
|
||||
|
||||
pipe = pipeline("text-generation", model="riotu-lab/ArabianGPT-01B", max_new_tokens=512)
|
||||
text = ''
|
||||
pipe.predict(text)
|
||||
```
|
||||
|
||||
## Limitations and Ethical Considerations
|
||||
|
||||
- The model may have context understanding or text generation limitations in certain scenarios.
|
||||
- Emphasis on ethical use to prevent misinformation or harmful content propagation.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Special thanks to Prince Sultan University, particularly the Robotics and Internet of Things Lab.
|
||||
|
||||
## Contact Information
|
||||
|
||||
For inquiries: [riotu@psu.edu.sa](mailto:riotu@psu.edu.sa).
|
||||
|
||||
## Disclaimer for the Use of Large Language Models (LLMs) for Text Generation
|
||||
|
||||
<p style="color: red;">We disclaim all responsibility for any harm, inaccuracies, or inappropriate content generated by ArabianGPT-0.1B, and users engage with and apply the model's outputs at their own risk.</p>
|
||||
39
config.json
Normal file
39
config.json
Normal file
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"_name_or_path": "gpt2",
|
||||
"activation_function": "gelu_new",
|
||||
"architectures": [
|
||||
"GPT2LMHeadModel"
|
||||
],
|
||||
"attn_pdrop": 0.1,
|
||||
"bos_token_id": null,
|
||||
"embd_pdrop": 0.1,
|
||||
"eos_token_id": 64000,
|
||||
"initializer_range": 0.02,
|
||||
"layer_norm_epsilon": 1e-05,
|
||||
"model_type": "gpt2",
|
||||
"n_ctx": 768,
|
||||
"n_embd": 768,
|
||||
"n_head": 12,
|
||||
"n_inner": null,
|
||||
"n_layer": 12,
|
||||
"n_positions": 768,
|
||||
"reorder_and_upcast_attn": false,
|
||||
"resid_pdrop": 0.1,
|
||||
"scale_attn_by_inverse_layer_idx": false,
|
||||
"scale_attn_weights": true,
|
||||
"summary_activation": null,
|
||||
"summary_first_dropout": 0.1,
|
||||
"summary_proj_to_labels": true,
|
||||
"summary_type": "cls_index",
|
||||
"summary_use_proj": true,
|
||||
"task_specific_params": {
|
||||
"text-generation": {
|
||||
"do_sample": true,
|
||||
"max_length": 50
|
||||
}
|
||||
},
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": "4.26.1",
|
||||
"use_cache": true,
|
||||
"vocab_size": 64002
|
||||
}
|
||||
5
generation_config.json
Normal file
5
generation_config.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"eos_token_id": 64000,
|
||||
"transformers_version": "4.26.1"
|
||||
}
|
||||
3
optimizer.pt
Normal file
3
optimizer.pt
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:97931ddea8c30d77b7811b2c971760d064ea29a0e9b4b4ed3a88d3297eecabcf
|
||||
size 1078481861
|
||||
3
pytorch_model.bin
Normal file
3
pytorch_model.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e56b94b4242a617239256e00722f0f56924dd2eafa0c0007b93f178623b7ca06
|
||||
size 546331197
|
||||
3
rng_state.pth
Normal file
3
rng_state.pth
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:84290731c7ca60bd4ac2da53d4ef86e0612c7fad522f3966b7fe0bf8ded9924c
|
||||
size 15597
|
||||
3
scaler.pt
Normal file
3
scaler.pt
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d011f2ad242133664d5c210e4430195460701c9035092826b29ba59ca03962aa
|
||||
size 557
|
||||
3
scheduler.pt
Normal file
3
scheduler.pt
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b4caf1d864fcff137a7c1c21f3b039dd6bb72190348ec6c92fc6584595b6fa57
|
||||
size 627
|
||||
4
special_tokens_map.json
Normal file
4
special_tokens_map.json
Normal file
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"eos_token": "<EOS>",
|
||||
"pad_token": "<EOS>"
|
||||
}
|
||||
127886
tokenizer.json
Normal file
127886
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
4
tokenizer_config.json
Normal file
4
tokenizer_config.json
Normal file
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"tokenizer_class": "PreTrainedTokenizerFast"
|
||||
}
|
||||
18826
trainer_state.json
Normal file
18826
trainer_state.json
Normal file
File diff suppressed because it is too large
Load Diff
3
training_args.bin
Normal file
3
training_args.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:f44d489779f957d1098d4de5722bf51c6a086f0f6e57f842819279422d6d5405
|
||||
size 3451
|
||||
Reference in New Issue
Block a user