初始化项目,由ModelHub XC社区提供模型
Model: sudhisrk1982/slm-125m-instruct Source: Original Platform
This commit is contained in:
35
README.md
Normal file
35
README.md
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
base_model: thesreedath/slm-125m-base
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# slm-125m-instruct
|
||||
|
||||
Instruction-tuned from [`thesreedath/slm-125m-base`](thesreedath/slm-125m-base) on 9,071 legal/financial
|
||||
Q&A pairs generated from a case-law + SEC-filing corpus.
|
||||
|
||||
## Training
|
||||
|
||||
- **Base**: 125.8M params (12L / 768d / 16,384 vocab)
|
||||
- **Data**: 9,071 Q&A pairs, LLM-judged (90.7% pass rate) from 19,775 raw
|
||||
- **Method**: full SFT, prompt tokens masked from the loss (answer-only training)
|
||||
- **Hyperparameters**: lr 2e-5, cosine decay, 3 epochs, bf16, 1xA100
|
||||
- **Best val loss**: 1.914
|
||||
|
||||
## Chat format
|
||||
|
||||
```
|
||||
<|bos|><|system|>{system}<|user|>{question}<|assistant|>{answer}<|eos|>
|
||||
```
|
||||
|
||||
## Note on token ids
|
||||
|
||||
The base repo's `config.json` declares `eos_token_id: 2`, but id 2 is `<|pad|>`
|
||||
in the tokenizer -- `<|eos|>` is id 1. This model ships the **corrected** ids
|
||||
(`bos=0, eos=1, pad=2`), so generation terminates properly.
|
||||
|
||||
## Limitations
|
||||
|
||||
125M parameters trained on ~9k pairs. It answers in the right form and stops
|
||||
cleanly, but factual accuracy is limited -- verify anything load-bearing.
|
||||
Reference in New Issue
Block a user