59 lines
1.9 KiB
Markdown
59 lines
1.9 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags:
|
||
|
|
- legal
|
||
|
|
- finance
|
||
|
|
- llama
|
||
|
|
- pretraining
|
||
|
|
- small-language-model
|
||
|
|
datasets:
|
||
|
|
- HFforLegal/case-law
|
||
|
|
- PleIAs/SEC
|
||
|
|
- HuggingFaceFW/fineweb-edu
|
||
|
|
---
|
||
|
|
|
||
|
|
# legal-slm-125m-base
|
||
|
|
|
||
|
|
A 125M-parameter Llama-architecture small language model pretrained **from
|
||
|
|
scratch** on a legal/financial corpus. Base (not instruction-tuned) model.
|
||
|
|
|
||
|
|
## Results
|
||
|
|
- **Validation perplexity: 10.67** (val_loss 2.3674), 1 epoch over 1.97B tokens.
|
||
|
|
|
||
|
|
## Model
|
||
|
|
- 125,847,552 params — 12 layers / 768 hidden / 12 heads, context 1024,
|
||
|
|
vocab 16,384 (custom byte-level BPE), tied embeddings, SwiGLU, RoPE.
|
||
|
|
|
||
|
|
## Training data (~76% legal)
|
||
|
|
- `HFforLegal/case-law` (US court opinions) — ~36%
|
||
|
|
- `PleIAs/SEC` (SEC filings) — ~40%
|
||
|
|
- `HuggingFaceFW/fineweb-edu` (educational web) — ~24%
|
||
|
|
|
||
|
|
Pipeline: deterministic cleaning (OCR gate, boilerplate/repetition/language
|
||
|
|
filters) → MinHash near-dup + exact-dup removal → 13-gram decontamination
|
||
|
|
against CaseHOLD/LexGLUE (so eval is uncontaminated) → 16K byte-level BPE →
|
||
|
|
packed 1024-token windows.
|
||
|
|
|
||
|
|
## Training
|
||
|
|
- 1 epoch, 3,753 steps, global batch 524,288 tokens, 2x H100 (DDP, bf16,
|
||
|
|
torch.compile). AdamW (0.9/0.95, wd 0.1 on 2D params), grad-clip 1.0.
|
||
|
|
- LR: warmup to 6e-4 over 200M tokens, cosine to 6e-5 annealed over 1 epoch.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
tok = AutoTokenizer.from_pretrained("VigneshwarKandhaiya/legal-slm-125m-base")
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("VigneshwarKandhaiya/legal-slm-125m-base")
|
||
|
|
ids = tok("The plaintiff shall bear the burden of", return_tensors="pt").input_ids
|
||
|
|
print(tok.decode(model.generate(ids, max_new_tokens=40)[0], skip_special_tokens=True))
|
||
|
|
```
|
||
|
|
|
||
|
|
## Limitations
|
||
|
|
Base model, English only, 125M params — expect fluent domain-styled text, not
|
||
|
|
reliable facts or instruction-following. Not legal or financial advice.
|