36 lines
1.1 KiB
Markdown
36 lines
1.1 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
base_model: thesreedath/slm-125m-base
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
# slm-125m-instruct
|
||
|
|
|
||
|
|
Instruction-tuned from [`thesreedath/slm-125m-base`](thesreedath/slm-125m-base) on 9,071 legal/financial
|
||
|
|
Q&A pairs generated from a case-law + SEC-filing corpus.
|
||
|
|
|
||
|
|
## Training
|
||
|
|
|
||
|
|
- **Base**: 125.8M params (12L / 768d / 16,384 vocab)
|
||
|
|
- **Data**: 9,071 Q&A pairs, LLM-judged (90.7% pass rate) from 19,775 raw
|
||
|
|
- **Method**: full SFT, prompt tokens masked from the loss (answer-only training)
|
||
|
|
- **Hyperparameters**: lr 2e-5, cosine decay, 3 epochs, bf16, 1xA100
|
||
|
|
- **Best val loss**: 1.914
|
||
|
|
|
||
|
|
## Chat format
|
||
|
|
|
||
|
|
```
|
||
|
|
<|bos|><|system|>{system}<|user|>{question}<|assistant|>{answer}<|eos|>
|
||
|
|
```
|
||
|
|
|
||
|
|
## Note on token ids
|
||
|
|
|
||
|
|
The base repo's `config.json` declares `eos_token_id: 2`, but id 2 is `<|pad|>`
|
||
|
|
in the tokenizer -- `<|eos|>` is id 1. This model ships the **corrected** ids
|
||
|
|
(`bos=0, eos=1, pad=2`), so generation terminates properly.
|
||
|
|
|
||
|
|
## Limitations
|
||
|
|
|
||
|
|
125M parameters trained on ~9k pairs. It answers in the right form and stops
|
||
|
|
cleanly, but factual accuracy is limited -- verify anything load-bearing.
|