73 lines
1.7 KiB
Markdown
73 lines
1.7 KiB
Markdown
|
---
|
||
|
|
library_name: transformers
|
||
|
|
license: cc-by-nc-4.0
|
||
|
|
base_model: lelapa/InkubaLM-0.4B
|
||
|
|
tags:
|
||
|
|
- generated_from_trainer
|
||
|
|
model-index:
|
||
|
|
- name: Caracal_LM
|
||
|
|
results: []
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
- sw
|
||
|
|
- yo
|
||
|
|
- so
|
||
|
|
- ln
|
||
|
|
- ki
|
||
|
|
- rw
|
||
|
|
- xh
|
||
|
|
- ha
|
||
|
|
- luo
|
||
|
|
- kam
|
||
|
|
- lug
|
||
|
|
- mas
|
||
|
|
---
|
||
|
|
|
||
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
||
|
|
should probably proofread and complete it, then remove this comment. -->
|
||
|
|
|
||
|
|
# Caracal_GPT
|
||
|
|
|
||
|
|
This model is a Continuous Pre-trained (CPT) model, adapted from [lelapa/InkubaLM-0.4B](https://huggingface.co/lelapa/InkubaLM-0.4B) on the custom dataset.
|
||
|
|
In this new model we add different languages from Kenya, Uganda, South Africa, Western Africa, Somalia, Ethiopia; sub-saharan Africa.
|
||
|
|
|
||
|
|
## Model description
|
||
|
|
|
||
|
|
Caracal GPT is a small causal model that can be used for fine-tuning tasks. It's goal is to be used by the represented language speaker for fine-tuning to a certain language task.
|
||
|
|
|
||
|
|
The Pre-trained LM was trained on 20+ African languages, introducing Kenyan lanugages and other languages not widely available in many datasets - Luo (luo), Kamba (kam)
|
||
|
|
, Maasai (mas), Somalia (som), etc.
|
||
|
|
|
||
|
|
## Intended uses & limitations
|
||
|
|
|
||
|
|
The model is to be used for fine-tuning on instruction set data for the given languages.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
### Training hyperparameters
|
||
|
|
|
||
|
|
The following hyperparameters were used during training:
|
||
|
|
- learning_rate: 0.0002
|
||
|
|
- train_batch_size: 32
|
||
|
|
- eval_batch_size: 8
|
||
|
|
- seed: 42
|
||
|
|
- gradient_accumulation_steps: 4
|
||
|
|
- total_train_batch_size: 128
|
||
|
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
|
||
|
|
- lr_scheduler_type: cosine
|
||
|
|
- lr_scheduler_warmup_steps: 500
|
||
|
|
- num_epochs: 4
|
||
|
|
- loss: 1.8
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
### Framework versions
|
||
|
|
|
||
|
|
- Transformers 4.45.2
|
||
|
|
- Pytorch 2.11.0+cu128
|
||
|
|
- Datasets 2.21.0
|
||
|
|
- Tokenizers 0.20.3
|