63 lines
1.5 KiB
Markdown
63 lines
1.5 KiB
Markdown
---
|
|
language: en
|
|
license: apache-2.0
|
|
tags:
|
|
- causal-lm
|
|
- llama
|
|
- legal
|
|
- financial
|
|
- from-scratch
|
|
library_name: transformers
|
|
---
|
|
|
|
# slm-125m
|
|
|
|
125M-parameter causal language model trained from scratch on a legal/financial corpus.
|
|
|
|
## Model
|
|
|
|
| | |
|
|
|---|---|
|
|
| Architecture | Llama-style (SwiGLU, RoPE, RMSNorm) |
|
|
| Parameters | ~125.8M |
|
|
| Layers / dim / heads | 12 / 768 / 12 |
|
|
| Context | 1024 tokens |
|
|
| Vocab | 16384 (byte-level BPE) |
|
|
|
|
## Training data
|
|
|
|
| Source | Role |
|
|
|--------|------|
|
|
| HFforLegal/case-law | US case law |
|
|
| PleIAs/SEC | SEC filings |
|
|
| HuggingFaceFW/fineweb-edu | General educational web text |
|
|
|
|
Packed train tokens: **unknown** | val tokens: **unknown**
|
|
|
|
Eval benchmarks (LexGLUE, CaseHOLD) were held out during corpus construction.
|
|
|
|
## Training run
|
|
|
|
| Hyperparameter | Value |
|
|
|----------------|-------|
|
|
| Global batch (tokens) | 524,288 |
|
|
| LR / min LR | 0.0006 / 6e-05 |
|
|
| Warmup tokens | 200M |
|
|
| Weight decay | 0.1 |
|
|
|
|
Last logged train loss: **n/a** | val loss: **n/a**
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
tok = AutoTokenizer.from_pretrained("ppanja/slm-125m-base")
|
|
model = AutoModelForCausalLM.from_pretrained("ppanja/slm-125m-base")
|
|
inputs = tok("The plaintiff shall bear the burden of proof", return_tensors="pt")
|
|
out = model.generate(**inputs, max_new_tokens=64)
|
|
print(tok.decode(out[0]))
|
|
```
|
|
|
|
Chat special tokens (`<|user|>`, `<|assistant|>`, `<|system|>`) are in the vocabulary for future instruction tuning.
|