137 lines
4.5 KiB
Markdown
137 lines
4.5 KiB
Markdown
---
|
|
base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
|
|
library_name: peft
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- base_model:adapter:unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
|
|
- lora
|
|
- transformers
|
|
- unsloth
|
|
---
|
|
|
|
# Model Card for Digita
|
|
|
|
Digita has been fine-tuned (qlora) on a proprietary dataset so as to respond
|
|
similarly to the Belgian State Archive customer support. It was essentially
|
|
trained using data from the "digit" mailbox from the DiVa section.
|
|
|
|
|
|
## Model Details
|
|
|
|
### Model Description
|
|
|
|
Digita has been trained in the context of the
|
|
[Arkey project](https://www.uclouvain.be/fr/instituts-recherche/ilc/miil/arkey)
|
|
conjointly held by the Belgian State Archive and UCLouvain. The purpose of this
|
|
project is to ease access to the archives held at both institutions for the
|
|
general public.
|
|
|
|
In that context, Digita has been trained as an experiment to tune a chatbot
|
|
that would be able to talk and reply to user request in a way that feels
|
|
similar to the actual people dealing with the matter at the BSA.
|
|
|
|
|
|
- **Developed by:** Xavier GILLARD
|
|
- **Funded by BELSPO:** [Arkey project](https://www.uclouvain.be/fr/instituts-recherche/ilc/miil/arkey)
|
|
- **Model type:** Conversational (Fine Tuned from Llama-3.1-8b-Instruct-bnb-4bit)
|
|
- **Language(s) (NLP):** Dutch, French, German, English
|
|
- **License:** MIT License
|
|
- **Finetuned from model Llama-3.1-8b-Instruct:** [unsloth/meta-llama-3.1-8b-instruct-bnb-4bit](https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit)
|
|
|
|
### Model Sources
|
|
|
|
All scripts, notebooks, etc.. (except the dataset data) that have been used
|
|
to prepare the dataset and fine tune the model are [available on github](https://github.com/xgillard/corpus_digit)
|
|
|
|
|
|
**Repository:** [xgillard/corpus_digit](https://github.com/xgillard/corpus_digit)
|
|
|
|
## Uses
|
|
|
|
Prototype a chatbot to help the users of the Belgian State Archive in the
|
|
context of the Arkey project.
|
|
|
|
## Limitations
|
|
|
|
This model has been fine tuned on data from the 2022-2024 period. Which means
|
|
the majority of the data predates the introduction of the AGATHA platform.
|
|
This means that, while some of the info provided by this model can be useful,
|
|
it is essentially going to be stale at this point.
|
|
|
|
The second major limitation I envision for this model stems from the sheer size
|
|
of the model: 8b is probably not going to cut it for a majority of cases.
|
|
Especially if one envisions to use the quantized versions.
|
|
However, I think that this model is a good proof of concept and that it shows
|
|
what could be achieved if resources were allocated to perform the same task
|
|
on a more up-to-date dataset (or limited to 2024) using a larger model.
|
|
This was however not feasible on my personal machine.
|
|
|
|
### Recommendations
|
|
|
|
If anyone is serious about using this model as a basis to help the BSA personnel
|
|
of to deploy an user facing chatbot at the BSA, I would recommend to completely
|
|
redo all the finetuning using a larger model and an updated (more recent)
|
|
version of the dataset.
|
|
|
|
## How to Get Started with the Model
|
|
|
|
I do recommend that you use the QLoRA adapter directly using unsloth as this is
|
|
the asset that seemed to yield the best result while maintaining a similar
|
|
VRAM footprint (about 7G). Here is how you get started:
|
|
|
|
```
|
|
from unsloth import (
|
|
FastLanguageModel,
|
|
train_on_responses_only,
|
|
)
|
|
from unsloth.chat_templates import (
|
|
get_chat_template,
|
|
)
|
|
from transformers import TextStreamer
|
|
|
|
model, tok = FastLanguageModel.from_pretrained("xaviergillard/digita", load_in_4bit=True)
|
|
model = FastLanguageModel.for_inference(model)
|
|
```
|
|
|
|
## Training Details
|
|
|
|
### Training Data
|
|
|
|
Original training data consists of a curated dataset of response emails sent
|
|
by the DiVa cusomer service. The details of this dataset will not be disclosed.
|
|
|
|
### Training Procedure
|
|
|
|
#### Preprocessing
|
|
|
|
1. Cleaning up the encoding
|
|
2. Conversation restructuration
|
|
3. Conversation classification
|
|
4. Data Augmentation
|
|
5. Filtering
|
|
|
|
#### Training Hyperparameters
|
|
|
|
- **Training regime:** QLoRA adapter based on a 4bit quantization of Llama-3.1-8b-Instruct by unsloth.
|
|
- **LoRA rank:** `rank = 16`
|
|
- **LoRA alpha:** `alpha = 16`
|
|
- **Base model quantization:** `4 bits`
|
|
|
|
All training/eval metrics can be consulted [here](https://www.comet.com/xaviergillard/digita/view/new/panels).
|
|
|
|
|
|
## Evaluation
|
|
|
|
So far, evaluation has not been very thorough: it mostly consisted of
|
|
* comparing the eval loss to the training loss during training
|
|
* ensuring the curves seemed ok
|
|
* peforming a few manual tests on the trained & quantized models.
|
|
|
|
## Model Card Authors
|
|
|
|
Xavier Gillard
|
|
|
|
### Framework versions
|
|
|
|
- PEFT 0.18.0
|