1.1 KiB
1.1 KiB
license, base_model, pipeline_tag
| license | base_model | pipeline_tag |
|---|---|---|
| apache-2.0 | thesreedath/slm-125m-base | text-generation |
slm-125m-instruct
Instruction-tuned from thesreedath/slm-125m-base on 9,071 legal/financial
Q&A pairs generated from a case-law + SEC-filing corpus.
Training
- Base: 125.8M params (12L / 768d / 16,384 vocab)
- Data: 9,071 Q&A pairs, LLM-judged (90.7% pass rate) from 19,775 raw
- Method: full SFT, prompt tokens masked from the loss (answer-only training)
- Hyperparameters: lr 2e-5, cosine decay, 3 epochs, bf16, 1xA100
- Best val loss: 1.914
Chat format
<|bos|><|system|>{system}<|user|>{question}<|assistant|>{answer}<|eos|>
Note on token ids
The base repo's config.json declares eos_token_id: 2, but id 2 is <|pad|>
in the tokenizer -- <|eos|> is id 1. This model ships the corrected ids
(bos=0, eos=1, pad=2), so generation terminates properly.
Limitations
125M parameters trained on ~9k pairs. It answers in the right form and stops cleanly, but factual accuracy is limited -- verify anything load-bearing.