69 lines
1.9 KiB
Markdown
69 lines
1.9 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
tags:
|
||
|
|
- legal
|
||
|
|
- finance
|
||
|
|
- llama
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
# slm-125m-base
|
||
|
|
|
||
|
|
A 126M-parameter Llama-architecture base model pretrained **from
|
||
|
|
scratch** on US case law, SEC filings, and educational web text. Trained on
|
||
|
|
8xH100 for ~28 minutes at a total project cost of roughly $18.
|
||
|
|
|
||
|
|
This is a **base model**. It has had no instruction tuning, no RLHF, and no
|
||
|
|
safety alignment. It completes text; it does not follow instructions.
|
||
|
|
|
||
|
|
## Architecture
|
||
|
|
|
||
|
|
| | |
|
||
|
|
|---|---|
|
||
|
|
| Parameters | 125,847,552 |
|
||
|
|
| Layers / hidden / heads | 12 / 768 / 12 |
|
||
|
|
| Context length | 1024 |
|
||
|
|
| Vocab | 16384 (byte-level BPE trained on this corpus) |
|
||
|
|
| Tied embeddings | True |
|
||
|
|
|
||
|
|
## Training data
|
||
|
|
|
||
|
|
1.92B tokens, ~76% legal:
|
||
|
|
|
||
|
|
| Source | Tokens | Share |
|
||
|
|
|---|---|---|
|
||
|
|
| `PleIAs/SEC` | 819M | 43% |
|
||
|
|
| `HFforLegal/case-law` | 639M | 33% |
|
||
|
|
| `HuggingFaceFW/fineweb-edu` | 463M | 24% |
|
||
|
|
|
||
|
|
Pipeline: deterministic rule-based cleaning (line filters, boilerplate
|
||
|
|
stripping, repetition and language gates, plus a dictionary-based OCR gate on
|
||
|
|
case law) then MinHash near-dedup, exact dedup, and 13-gram decontamination
|
||
|
|
against the CaseHOLD evaluation set.
|
||
|
|
|
||
|
|
## Training
|
||
|
|
|
||
|
|
2 epochs (3.84B tokens seen), AdamW, cosine schedule to a 6e-05 cooldown,
|
||
|
|
bf16, sequence length 1024, global batch 524,288 tokens.
|
||
|
|
|
||
|
|
## Results
|
||
|
|
|
||
|
|
| Metric | Value |
|
||
|
|
|---|---|
|
||
|
|
| Val perplexity (held-out 1%) | 9.07 |
|
||
|
|
| CaseHOLD (zero-shot, 5-way) | 19.5% |
|
||
|
|
|
||
|
|
CaseHOLD is measured on data excluded from training by 13-gram
|
||
|
|
decontamination, so it is not inflated by leakage. Expect near-chance
|
||
|
|
zero-shot performance: a 125M base model with no task tuning is not
|
||
|
|
competitive on multiple-choice legal reasoning, and this number is a baseline
|
||
|
|
for fine-tuning, not a capability claim.
|
||
|
|
|
||
|
|
## Limitations
|
||
|
|
|
||
|
|
Small, base-only, English-only, 1024-token context. Case-law inputs contain
|
||
|
|
OCR noise that survives filtering. **Not legal or financial advice** and not
|
||
|
|
suitable for any use where an incorrect statement of law carries consequences.
|