---
license: apache-2.0
language:
- en
tags:
- legal
- finance
- llama
pipeline_tag: text-generation
---

# slm-125m-base

A 126M-parameter Llama-architecture base model pretrained **from
scratch** on US case law, SEC filings, and educational web text. Trained on
8xH100 for ~28 minutes at a total project cost of roughly $18.

This is a **base model**. It has had no instruction tuning, no RLHF, and no
safety alignment. It completes text; it does not follow instructions.

## Architecture

| | |
|---|---|
| Parameters | 125,847,552 |
| Layers / hidden / heads | 12 / 768 / 12 |
| Context length | 1024 |
| Vocab | 16384 (byte-level BPE trained on this corpus) |
| Tied embeddings | True |

## Training data

1.92B tokens, ~76% legal:

| Source | Tokens | Share |
|---|---|---|
| `PleIAs/SEC` | 819M | 43% |
| `HFforLegal/case-law` | 639M | 33% |
| `HuggingFaceFW/fineweb-edu` | 463M | 24% |

Pipeline: deterministic rule-based cleaning (line filters, boilerplate
stripping, repetition and language gates, plus a dictionary-based OCR gate on
case law) then MinHash near-dedup, exact dedup, and 13-gram decontamination
against the CaseHOLD evaluation set.

## Training

2 epochs (3.84B tokens seen), AdamW, cosine schedule to a 6e-05 cooldown,
bf16, sequence length 1024, global batch 524,288 tokens.

## Results

| Metric | Value |
|---|---|
| Val perplexity (held-out 1%) | 9.07 |
| CaseHOLD (zero-shot, 5-way) | 19.5% |

CaseHOLD is measured on data excluded from training by 13-gram
decontamination, so it is not inflated by leakage. Expect near-chance
zero-shot performance: a 125M base model with no task tuning is not
competitive on multiple-choice legal reasoning, and this number is a baseline
for fine-tuning, not a capability claim.

## Limitations

Small, base-only, English-only, 1024-token context. Case-law inputs contain
OCR noise that survives filtering. **Not legal or financial advice** and not
suitable for any use where an incorrect statement of law carries consequences.