32 lines
1.1 KiB
Markdown
32 lines
1.1 KiB
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
license: apache-2.0
|
||
|
|
tags:
|
||
|
|
- text-generation-inference
|
||
|
|
- transformers
|
||
|
|
- unsloth
|
||
|
|
- llama
|
||
|
|
- trl
|
||
|
|
- sft
|
||
|
|
---
|
||
|
|
|
||
|
|
# Llama 3.1 Mini - LoRA Finetuned
|
||
|
|
|
||
|
|
## Model Description
|
||
|
|
This model is a LoRA-finetuned version of the Llama 3.1 Mini model, which is a pruned variant of the Llama 3.1 8B model. The original Llama 3.1 Mini was created by pruning the larger model to approximately 3 billion parameters, and this version has been further adapted using Low-Rank Adaptation (LoRA) to enhance its capabilities.
|
||
|
|
|
||
|
|
## Limitations
|
||
|
|
|
||
|
|
Please note that this model, like its base version, may exhibit biases present in its training data and should be used with appropriate care and consideration.
|
||
|
|
|
||
|
|
## Training Data
|
||
|
|
|
||
|
|
The base model (Llama 3.1 Mini) was trained on my personal Claude 3 Opus and Claude 3.5 Sonnet dataset, with some synthetic pairs added on with Gemma 2 9B it being the user, and Llama 3 70B through Groq being the assistant. I have also used Guanaco alongside everything else.
|
||
|
|
|
||
|
|
|
||
|
|
## License
|
||
|
|
Llama 3.1
|
||
|
|
|
||
|
|
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|