50 lines
1.5 KiB
Markdown
50 lines
1.5 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags:
|
||
|
|
- llama
|
||
|
|
- fine-tuned
|
||
|
|
- legal
|
||
|
|
- finance
|
||
|
|
- qa
|
||
|
|
base_model: thesreedath/slm-125m-base
|
||
|
|
---
|
||
|
|
|
||
|
|
# slm-125m-finetuned-qa
|
||
|
|
|
||
|
|
A 125.8M-parameter LLaMA-architecture model, fine-tuned from
|
||
|
|
[thesreedath/slm-125m-base](https://huggingface.co/thesreedath/slm-125m-base) on a
|
||
|
|
grounded Q&A instruction dataset covering legal, financial, and general text.
|
||
|
|
|
||
|
|
## Training data
|
||
|
|
|
||
|
|
7,521 chat-format (system/user/assistant) examples generated from a legal/SEC/web
|
||
|
|
corpus using Gemini (`gemini-3.1-flash-lite`) as teacher, filtered by an LLM-as-judge
|
||
|
|
pass for grounding/faithfulness, deduplicated by embedding similarity, covering four
|
||
|
|
task types: grounded QA, summarization, extraction, and rewriting.
|
||
|
|
|
||
|
|
## Training
|
||
|
|
|
||
|
|
Full fine-tune (not LoRA), 3 epochs, 1x H100, dynamic per-batch padding, loss masked to
|
||
|
|
the assistant response only. 354 steps, ~1.6M real tokens processed, 69 seconds total.
|
||
|
|
Validation perplexity 9.13 on the Q&A task.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
Prompts use the same special-token format as the base model:
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
tok = AutoTokenizer.from_pretrained("sahuabinash/slm-125m-finetuned-qa")
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("sahuabinash/slm-125m-finetuned-qa")
|
||
|
|
|
||
|
|
prompt = "<|bos|><|system|>Answer the question using only the given passage.<|user|>What is a plaintiff?<|assistant|>"
|
||
|
|
inputs = tok(prompt, return_tensors="pt")
|
||
|
|
out = model.generate(**inputs, max_new_tokens=80)
|
||
|
|
print(tok.decode(out[0], skip_special_tokens=True))
|
||
|
|
```
|