---
license: apache-2.0
base_model: thesreedath/slm-125m-base
pipeline_tag: text-generation
---

# slm-125m-instruct

Instruction-tuned from [`thesreedath/slm-125m-base`](thesreedath/slm-125m-base) on 9,071 legal/financial
Q&A pairs generated from a case-law + SEC-filing corpus.

## Training

- **Base**: 125.8M params (12L / 768d / 16,384 vocab)
- **Data**: 9,071 Q&A pairs, LLM-judged (90.7% pass rate) from 19,775 raw
- **Method**: full SFT, prompt tokens masked from the loss (answer-only training)
- **Hyperparameters**: lr 2e-5, cosine decay, 3 epochs, bf16, 1xA100
- **Best val loss**: 1.914

## Chat format

```
<|bos|><|system|>{system}<|user|>{question}<|assistant|>{answer}<|eos|>
```

## Note on token ids

The base repo's `config.json` declares `eos_token_id: 2`, but id 2 is `<|pad|>`
in the tokenizer -- `<|eos|>` is id 1. This model ships the **corrected** ids
(`bos=0, eos=1, pad=2`), so generation terminates properly.

## Limitations

125M parameters trained on ~9k pairs. It answers in the right form and stops
cleanly, but factual accuracy is limited -- verify anything load-bearing.