legal-slm-125m-base/README.md

license, language, library_name, pipeline_tag, tags, datasets

license	language	library_name	pipeline_tag	tags	datasets
apache-2.0	en	transformers	text-generation	legal finance llama pretraining small-language-model	HFforLegal/case-law PleIAs/SEC HuggingFaceFW/fineweb-edu

legal-slm-125m-base

A 125M-parameter Llama-architecture small language model pretrained from scratch on a legal/financial corpus. Base (not instruction-tuned) model.

Results

• Validation perplexity: 10.67 (val_loss 2.3674), 1 epoch over 1.97B tokens.

Model

• 125,847,552 params — 12 layers / 768 hidden / 12 heads, context 1024, vocab 16,384 (custom byte-level BPE), tied embeddings, SwiGLU, RoPE.

Training data (~76% legal)

• HFforLegal/case-law (US court opinions) — ~36%
• PleIAs/SEC (SEC filings) — ~40%
• HuggingFaceFW/fineweb-edu (educational web) — ~24%

Pipeline: deterministic cleaning (OCR gate, boilerplate/repetition/language filters) → MinHash near-dup + exact-dup removal → 13-gram decontamination against CaseHOLD/LexGLUE (so eval is uncontaminated) → 16K byte-level BPE → packed 1024-token windows.

Training

• 1 epoch, 3,753 steps, global batch 524,288 tokens, 2x H100 (DDP, bf16, torch.compile). AdamW (0.9/0.95, wd 0.1 on 2D params), grad-clip 1.0.
• LR: warmup to 6e-4 over 200M tokens, cosine to 6e-5 annealed over 1 epoch.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VigneshwarKandhaiya/legal-slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("VigneshwarKandhaiya/legal-slm-125m-base")
ids = tok("The plaintiff shall bear the burden of", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=40)[0], skip_special_tokens=True))

Limitations

Base model, English only, 125M params — expect fluent domain-styled text, not reliable facts or instruction-following. Not legal or financial advice.