81 lines
2.6 KiB
Markdown
81 lines
2.6 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- llama
|
|||
|
|
- legal
|
|||
|
|
- financial
|
|||
|
|
- small-language-model
|
|||
|
|
- pretrained
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# slm-125m-base
|
|||
|
|
|
|||
|
|
A 125M-parameter Llama-architecture small language model pretrained from scratch on a
|
|||
|
|
legal/financial corpus. This is a **base (pretrained) model** — it is not instruction-
|
|||
|
|
or chat-tuned.
|
|||
|
|
|
|||
|
|
## Model details
|
|||
|
|
|
|||
|
|
| | |
|
|||
|
|
|---|---|
|
|||
|
|
| Architecture | Llama (via `transformers.LlamaForCausalLM`) |
|
|||
|
|
| Parameters | 125.8M (tied embeddings) |
|
|||
|
|
| Vocab | 16,384 (byte-level BPE trained on this corpus) |
|
|||
|
|
| Layers / hidden / heads | 12 / 768 / 12 (head dim 64, MHA) |
|
|||
|
|
| Context length | 1,024 |
|
|||
|
|
| Positional | RoPE (θ=10,000) |
|
|||
|
|
| Norm / activation | RMSNorm (1e-5) / SwiGLU (silu) |
|
|||
|
|
| Precision | bf16 training, weights saved fp32 |
|
|||
|
|
|
|||
|
|
## Training data (~2.04B tokens, "legal-first" mix)
|
|||
|
|
|
|||
|
|
Streamed, cleaned, deduplicated (MinHash LSH + exact), and decontaminated against
|
|||
|
|
CaseHOLD / LexGLUE before tokenization. Realized token mix:
|
|||
|
|
|
|||
|
|
| Source | Share | HF dataset |
|
|||
|
|
|--------|-------|------------|
|
|||
|
|
| US case law | ~35% | `HFforLegal/case-law` (split `us`) |
|
|||
|
|
| SEC filings | ~42% | `PleIAs/SEC` |
|
|||
|
|
| Educational web | ~23% | `HuggingFaceFW/fineweb-edu` (`sample-10BT`) |
|
|||
|
|
|
|||
|
|
## Training
|
|||
|
|
|
|||
|
|
- **2 epochs** (7,778 steps) on 8×H100 (DDP, `torch.compile`, SDPA/flash attention)
|
|||
|
|
- Global batch 524,288 tokens; AdamW (β=0.9/0.95, wd 0.1, clip 1.0)
|
|||
|
|
- LR 6e-4 → 6e-5 cosine, 200M-token linear warmup; seed 1337
|
|||
|
|
- Throughput ~3.19M tok/s @ ~30% MFU
|
|||
|
|
|
|||
|
|
## Evaluation
|
|||
|
|
|
|||
|
|
**Held-out validation perplexity: 9.13** (loss 2.211, full 1% held-out split,
|
|||
|
|
20,581,737 tokens / 20,119 packed 1024-token windows).
|
|||
|
|
|
|||
|
|
Validation loss over training (subset eval): 2.796 → 2.232 (steps 1000 → 7778).
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
import torch
|
|||
|
|
|
|||
|
|
tok = AutoTokenizer.from_pretrained("s2211252/slm-125m-base")
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained("s2211252/slm-125m-base", torch_dtype=torch.bfloat16)
|
|||
|
|
|
|||
|
|
prompt = "Pursuant to the terms of this Agreement, the parties"
|
|||
|
|
ids = tok(prompt, return_tensors="pt").input_ids
|
|||
|
|
out = model.generate(ids, max_new_tokens=120, do_sample=True, top_k=50, top_p=0.95, temperature=0.8)
|
|||
|
|
print(tok.decode(out[0], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
Small base model, English only, 1,024-token context. Trained on legal/financial +
|
|||
|
|
web text; it is **not** instruction-tuned and can produce inaccurate or fabricated
|
|||
|
|
legal/financial statements. Not for legal advice or production decisions without
|
|||
|
|
review. Domain contamination against CaseHOLD/LexGLUE was filtered, but standard
|
|||
|
|
LM caveats apply.
|