---
library_name: transformers
license: cc-by-nc-4.0
base_model: lelapa/InkubaLM-0.4B
tags:
- generated_from_trainer
model-index:
- name: Caracal_LM
  results: []
language:
- en
- sw
- yo
- so
- ln
- ki
- rw
- xh
- ha
- luo
- kam
- lug
- mas
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Caracal_GPT

This model is a Continuous Pre-trained (CPT) model, adapted from [lelapa/InkubaLM-0.4B](https://huggingface.co/lelapa/InkubaLM-0.4B) on the custom dataset.
In this new model we add different languages from Kenya, Uganda, South Africa, Western Africa, Somalia, Ethiopia; sub-saharan Africa.

## Model description

Caracal GPT is a small causal model that can be used for fine-tuning tasks. It's goal is to be used by the represented language speaker for fine-tuning to a certain language task.

The Pre-trained LM was trained on 20+ African languages, introducing Kenyan lanugages and other languages not widely available in many datasets - Luo (luo), Kamba (kam)
, Maasai (mas), Somalia (som), etc.

## Intended uses & limitations

The model is to be used for fine-tuning on instruction set data for the given languages.





### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 500
- num_epochs: 4
- loss: 1.8





### Framework versions

- Transformers 4.45.2
- Pytorch 2.11.0+cu128
- Datasets 2.21.0
- Tokenizers 0.20.3