71 lines
2.2 KiB
Markdown
71 lines
2.2 KiB
Markdown
|
---
|
||
|
|
base_model: unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit
|
||
|
|
tags:
|
||
|
|
- text-generation-inference
|
||
|
|
- transformers
|
||
|
|
- llama
|
||
|
|
- finance
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
---
|
||
|
|
|
||
|
|
# Llama-3.1-8B Financial Sentiment — QLoRA
|
||
|
|
|
||
|
|
LoRA adapters fine-tuned on [FinGPT/fingpt-sentiment-train](https://huggingface.co/datasets/FinGPT/fingpt-sentiment-train)
|
||
|
|
with QLoRA: 4-bit NF4 frozen base + bf16 LoRA adapters (r=16, alpha=32, dropout=0.05).
|
||
|
|
|
||
|
|
**Task:** 3-class financial sentiment — `negative` / `neutral` / `positive`
|
||
|
|
|
||
|
|
## Evaluation Results
|
||
|
|
|
||
|
|
| Dataset | Base Accuracy | FT Accuracy | Base Macro-F1 | FT Macro-F1 |
|
||
|
|
|----------------|---------------|---------------|---------------|---------------|
|
||
|
|
| FPB in-domain | 0.8908 | 0.9748 | 0.8765 | 0.9725 |
|
||
|
|
| FiQA-SA OOD | 0.8120 | 0.9402 | 0.6705 | 0.8335 |
|
||
|
|
|
||
|
|
> Baseline = zero-shot Meta-Llama-3.1-8B-Instruct with the same prompt template.
|
||
|
|
|
||
|
|
## Quick Start
|
||
|
|
|
||
|
|
```python
|
||
|
|
from peft import PeftModel
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
base = AutoModelForCausalLM.from_pretrained(
|
||
|
|
"meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||
|
|
load_in_4bit=True,
|
||
|
|
device_map="auto",
|
||
|
|
)
|
||
|
|
model = PeftModel.from_pretrained(base, "jhon53/Llama3_1_8B_Finance_QLoRA")
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained("jhon53/Llama3_1_8B_Finance_QLoRA")
|
||
|
|
```
|
||
|
|
|
||
|
|
## Training Details
|
||
|
|
|
||
|
|
| Param | Value |
|
||
|
|
|-------|-------|
|
||
|
|
| Base model | meta-llama/Meta-Llama-3.1-8B-Instruct |
|
||
|
|
| Method | QLoRA (4-bit NF4 + LoRA bf16) |
|
||
|
|
| LoRA rank (r) | 16 |
|
||
|
|
| LoRA alpha | 32 |
|
||
|
|
| LoRA dropout | 0.05 |
|
||
|
|
| Target modules | q, k, v, o, gate, up, down projections |
|
||
|
|
| Training data | FinGPT/fingpt-sentiment-train (~76k) |
|
||
|
|
| Optimizer | AdamW 8-bit |
|
||
|
|
| Learning rate | 2e-4 (cosine schedule) |
|
||
|
|
| Epochs | 3 (early stopping, patience=3) |
|
||
|
|
| Loss | Completion-only cross-entropy |
|
||
|
|
|
||
|
|
## Related Repos
|
||
|
|
|
||
|
|
- GGUF q4_k_m: [jhon53/Llama3_1_8B_Finance_QLoRA-GGUF](https://huggingface.co/jhon53/Llama3_1_8B_Finance_QLoRA-GGUF)
|
||
|
|
- model: [jhon53/Llama3_1_8B_Finance_QLoRA](https://huggingface.co/jhon53/Llama3_1_8B_Finance_QLoRA)
|
||
|
|
|
||
|
|
|
||
|
|
# Uploaded finetuned model
|
||
|
|
|
||
|
|
- **Developed by:** jhon53
|
||
|
|
- **License:** apache-2.0
|
||
|
|
- **Finetuned from model :** unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit
|