---
license: apache-2.0
language:
- en
base_model: meta-llama/Meta-Llama-3-8B
library_name: unsloth
tags:
- finance
- information-extraction
- structured-data
- json
- merged
- llama-3
datasets:
- manuelaschrittwieser/invoice-extraction-dataset-v2
metrics:
- train_loss
model-index:
- name: Llama-3-8B Invoice Extractor (Merged)
  results:
  - task:
      type: text-generation
      name: Information Extraction
    dataset:
      name: invoice-extraction-dataset-v2
      type: manuelaschrittwieser/invoice-extraction-dataset-v2
    metrics:
    - name: Final Train Loss
      type: loss
      value: 0.2555
pipeline_tag: text-generation
finetuned_from: meta-llama/Meta-Llama-3-8B
---

# 📑 Llama-3-8B Invoice Extractor (Merged)

- **License:** apache-2.0
- **Finetuned from model :** unsloth/llama-3-8b-bnb-4bit

This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.

[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)

## 1. Model Description & Intended Use
This model is a fine-tuned version of **Meta's Llama-3-8B**, specifically optimized for **Structured Information Extraction**.
It is designed to act as a "Parser Agent" that transforms messy, unstructured financial text—such as invoice descriptions, receipt notes, and purchase logs—into machine-readable, valid JSON objects.

**Primary Intended Use:**
- Automated accounting and bookkeeping workflows.
- Extracting data from OCR-processed receipts.
- Building serverless financial data pipelines.

**Target Schema:**
```json
{
  "item": "string",
  "quantity": "integer",
  "date": "string",
  "vendor": "string",
  "total": "float",
  "currency": "string"
}
```

## 2. Training Data Information

The model was trained on the `manuelaschrittwieser/invoice-extraction-dataset-v2`, which consists of **601 high-quality examples**.

* **Methodology:** The dataset uses a synthetic generation pipeline with 7 distinct sentence templates to ensure robustness against different linguistic structures.
* **Diversity:** Includes 20+ vendors, 20+ item categories, and 5 currency types (USD, EUR, GBP, CAD, JPY).
* **Format:** The data was prepared in an instruction-input-output format to reinforce strict adherence to the JSON schema.

## 3. Training Procedure & Hyperparameters

The training was conducted using **Parameter-Efficient Fine-Tuning (PEFT)** with the **QLoRA** method via the **Unsloth** library,
which significantly reduced VRAM usage while maintaining performance.

### Hyperparameters:

* **Epochs:** Custom step-based (120 global steps)
* **Learning Rate:** 2e-4
* **Batch Size:** 2 (with Gradient Accumulation Steps = 4)
* **Optimizer:** AdamW (8-bit)
* **Learning Rate Scheduler:** Linear
* **LoRA Rank (r):** 16
* **LoRA Alpha:** 16
* **Precision:** 4-bit Quantization (merged into 16-bit for this standalone version)

## 4. Evaluation Results

### Baseline (Llama-3-8B) vs. Fine-Tuned
| **Feature**  | **Baseline Model** | **Fine-Tuned Model (v2)** |
| ------------- |:-------------:| :-------------:|
| **Output Format** | Conversational / Explanatory | **Strict JSON Only** |
| **JSON Validity** | Often fails (includes extra text) | **100% Valid JSON** in test runs |
| **Entity Recognition** | High accuracy, but low precision | **High Precision** (mapped to schema) |
| **Instruction Following** | Moderate | **High** (Stays within Response block) |

**Tracking:** Training performance was monitored via **Weights & Biases (W&B)**, showing a consistent reduction in loss over 120 steps without signs of overfitting.

## 5. Limitations & Known Issues

* **Language:** Optimized primarily for English. Performance on other languages is not guaranteed.
* **Hallucination:** While the model is highly structured, it can occasionally misinterpret dates if they are provided in ambiguous formats (e.g., 01/02/03).
* **Context Length:** Best performance is achieved with input lengths under 512 tokens.
* **Verification:** Users should implement a JSON validation layer in production to handle the rare cases of malformed output.

## 6. Code Example: Loading and Usage

Since this is a merged model, it can be used with standard `transformers` or `vLLM`.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "manuelaschrittwieser/llama-3-invoice-extractor-merged"

# Load Model and Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    device_map="auto", 
    torch_dtype=torch.float16
)

# Inference Example
prompt = """### Instruction:
Extract invoice details into JSON.

### Input:
Bought 2 monitors at Dell for 900 USD on Jan 15 2024.

### Response:
"""

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split("### Response:\n")[-1])

```
---

Developed by: Manuela Schrittwieser,
Project: Structured Data Extractor