qwen25-1.5b-invoice-extraction-v3/README.md

# qwen25-1.5b-invoice-extraction-v3

A QLoRA fine-tune of **Qwen2.5-1.5B-Instruct** that extracts structured financial terms from SEC
contract / commitment-letter text and returns them as a fixed-schema JSON object.

## Model Description

- **Base model:** `Qwen/Qwen2.5-1.5B-Instruct`
- **Fine-tuning method:** QLoRA (4-bit NF4), LoRA r=64, alpha=64, applied to all attention and MLP
  projection layers, trained with [Unsloth](https://github.com/unslothai/unsloth)
- **Task:** Given a contract excerpt, extract every financial value present and label it with a
  `term_type` from a fixed vocabulary, returned as strict JSON.
- **Trained on:** `TheTokenFactory/sec-contracts-financial-extraction-instructions` (3,430 examples),
  using a two-stage curriculum (core financial fields first, then the full dataset).

## Output Format

```json
{
  "financial_values": [
    {"value": "US$3,050,000,000", "term_type": "credit_facility"},
    {"value": "8.5%", "term_type": "interest_rate"}
  ]
}
```

`term_type` is restricted to: `interest_rate`, `equity_grant`, `bonus`, `credit_facility`,
`salary`, `threshold`, `fee`, `loan_amount`, `compensation`, `retirement_benefit`, `other`,
`purchase_price`, `severance`.

## Intended Use

Drafting / first-pass extraction of financial terms from contracts, commitment letters, and
similar legal-financial documents, as an aid for a human reviewer — **not** as a standalone
source of truth for financial or legal decisions.

## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "dhanushmekaka/qwen25-1.5b-invoice-extraction-v3"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "system", "content": "You are a strict legal extraction AI... Output strictly as JSON."},
    {"role": "user", "content": "<contract excerpt here>"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Evaluation

Measured on a 50-example held-out split of the training dataset:

| Metric | Score |
|---|---|
| Exact Match Accuracy (full JSON object) | 16.0% |
| JSON Output Validity | 94.0% |
| Field-Level Extraction Accuracy | 44.9% |
| Avg. Inference Latency | 7.07 s |
| Avg. Throughput | 17.4 tok/s |

| `term_type` | Gold count | Precision | Recall |
|---|---|---|---|
| bonus | 9 | 66.7% | 44.4% |
| compensation | 4 | 0.0% | 0.0% |
| credit_facility | 1 | n/a | 0.0% |
| equity_grant | 3 | 40.0% | 66.7% |
| fee | 7 | 100.0% | 42.9% |
| interest_rate | 6 | 75.0% | 50.0% |
| loan_amount | 2 | 66.7% | 100.0% |
| purchase_price | 2 | 100.0% | 50.0% |
| retirement_benefit | 1 | n/a | 0.0% |
| salary | 7 | 50.0% | 57.1% |
| threshold | 7 | 42.9% | 42.9% |

## Limitations

This is a **v3, still-converging checkpoint** — training was stopped by a wall-clock budget at
90.8% of the planned steps (epoch 5.45 of 6), with eval loss still decreasing. The numbers above
should be read as a snapshot of an in-progress model, not a final result:

- **Exact Match (16%)** is a strict all-or-nothing metric over the whole JSON object; a single
  wrong value anywhere in a multi-field extraction counts as a full miss. Field-Level Accuracy
  (44.9%) is a better read on per-value correctness.
- **Rare `term_type`s** (`retirement_benefit`, `purchase_price`, `severance`, `compensation`,
  `other`) have limited training coverage (under 250 combined examples vs. 850 for
  `interest_rate` alone), and their precision/recall above is computed on single-digit eval
  counts — treat those figures as directional, not reliable.
- **JSON validity (94%)** means roughly 1 in 17 outputs needs a parsing fallback or retry in
  production.
- The 50-example eval set is small; metrics may shift with a larger held-out set.

**Recommended before production use:** complete the remaining training epoch, expand the eval set,
and add human review for `term_type`s with fewer than ~100 training examples.

## Training Data

`TheTokenFactory/sec-contracts-financial-extraction-instructions` — SEC contract excerpts
(commitment letters, credit agreements, employment/compensation terms) paired with gold JSON
financial-term annotations.

## License

Apache 2.0 (inherits from the Qwen2.5-1.5B-Instruct base model — verify against the base model's
license terms before commercial use).---
base_model: unsloth/qwen2.5-1.5b-instruct-unsloth-bnb-4bit
tags:
- text-generation-inference
- transformers
- unsloth
- qwen2
license: apache-2.0
language:
- en
---

# Uploaded finetuned  model

- **Developed by:** dhanushmekaka
- **License:** apache-2.0
- **Finetuned from model :** unsloth/qwen2.5-1.5b-instruct-unsloth-bnb-4bit

This qwen2 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.

[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)