slm-125m-sft-raft/README.md

---
language: en
license: apache-2.0
base_model: Kotichitturu/slm-125m-base
tags:
  - legal
  - financial
  - small-language-model
  - llama
  - sft
  - raft
  - rag
---

# SLM-125M SFT (RAFT — grounded)

Answers legal/financial questions **only from a context passage you supply**, and
**refuses when the answer isn't in it**. Trained with RAFT (Retrieval-Augmented
Fine-Tuning): 20% of training examples had no answer in the context, teaching the
model that "I don't know" is a correct response.

## The number that matters

A model that never refuses can post a great loss and be useless — it hallucinates
exactly when you most need it not to. So refusal is measured as a confusion
matrix, because refusal rate alone is gameable: a model that refuses *everything*
scores 100% recall and is worthless.

```
answer ABSENT  -> refused   :  88/91   (96.7%)   <- the point of RAFT
answer ABSENT  -> answered  :   3/91   ( 3.3%)   <- hallucination
answer PRESENT -> refused   :   7/394  ( 1.8%)   <- over-refusal
answer PRESENT -> answered  : 387/394
overall accuracy: 97.9%
```

**Both halves matter.** 96.7% refusal recall would be worthless from a model that
refuses everything — over-refusal is only 1.8%. It refuses when it should and
answers when it should.

### ⚠️ What the 97.9% does NOT say — read this before trusting an answer

**It measures whether the model REFUSES when it should. Not whether the answer it
gives is correct.** Those are different claims, and only the first is in that
number. A confidently wrong answer on an answer-present row scores in the
*correct* cell of the matrix above, leaving 97.9% untouched.

Measured separately on the 394 answer-present validation rows:

| Metric | Result |
|---|---|
| Exact match (normalized) | 1.8% |
| Token F1 vs gold answer | 46.3% |
| Numeric match (all gold numbers reproduced) | 33.7% |
| Prediction contained **no number at all** | **24.3%** |
| **Single-number answers — the clean test** | **37.0%** |

**When exactly one figure is being asked for, this model produces it 37% of the
time.** No truncation excuse, no metric strictness — it fails nearly two times in
three. The dominant failure is **omission**: it hedges into fluent, grounded-
sounding, figure-free prose. Asked *"what specific article of the Code of Criminal
Procedure?"* it replied *"the article of the Code of Criminal Procedure states that
the trial judge is re…"* — dodging the number entirely.

**Why this coexists with 96.7% refusal:** refusal is a coarse binary judgement
(*does this passage address the question at all?*). Extraction demands precision
(*which figure, exactly?*). A 125M model learns the first and not the second.

**What that means for you:** trust this model to tell you when an answer **isn't**
in the passage. Do **not** trust the specific figures it reports without checking
them against the passage yourself. It is a triage tool, not an extraction tool.

*(Caveat: gold answers are teacher-generated and judge-filtered — a proxy for
truth. Some numeric misses are defensible. The 37% is not.)*

## Val loss and the alignment tax

Val loss 1.9551 → **0.9286** (−52%), best at epoch 3. That loss is computed over
response tokens only — just **8.3%** of RAFT's tokens, since the context is read
but never trained on. It is not comparable to the base's 2.2521, and its low
value partly reflects that extractive answers are easy, not that the model is
four times better.

The comparison that *is* fair — same held-out pretraining text the base was
measured on, every token, no masking:

| Model | Perplexity | vs base |
|---|---|---|
| Base | 9.50 | — |
| **This model** | **10.07** | **+6.0%** |

A small, real alignment tax: it gave up 6% of its raw domain modelling to become
grounded and refusal-capable. (Control: the base reproduces its published 9.51 on
this harness.)

**10.07 is not `e^0.9286`.** It is `e^2.3097` — this model's loss on held-out
**pretraining** text, every token counted. Note the direction: this model has the
**lower** val loss of the two fine-tunes (0.9286) and the **higher** perplexity here
(10.07). Nothing is contradictory. 0.9286 is scored on 8.3% of its tokens, copying
answers out of a passage sitting in the prompt; 10.07 is scored on every token of
unseen legal text with nothing to copy. Two different questions wearing the same units.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Kotichitturu/slm-125m-sft-raft")
model = AutoModelForCausalLM.from_pretrained("Kotichitturu/slm-125m-sft-raft")

SYSTEM = "Answer the question based only on the provided context. If the answer is not in the context, say 'The answer is not available in the provided context.'"
user = f"Context: {passage}\n\nQuestion: {question}"
prompt = f"<|bos|><|system|>{SYSTEM}<|user|>{user}<|assistant|>"

ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, max_new_tokens=90, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|eos|>"))
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
```

Use greedy decoding — the 97.9% above was measured with `do_sample=False`.
Sampling degrades refusal behaviour.

When the answer is absent it emits:

```
The answer is not available in the provided context.
```

### Limits

- Context + question + answer must fit **1,024 tokens** (training max was 964).
- Retrieval is your job. This model is the *reader*, not the retriever.
- It is not a lawyer. Verify anything that matters.

### System prompt (must match exactly)

```
Answer the question based only on the provided context. If the answer is not in the context, say 'The answer is not available in the provided context.'
```

## Architecture

Identical to the base — fine-tuning changes weights, never shape.

| | |
|---|---|
| Params | 125.8M |
| Layers / hidden / heads | 12 / 768 / 12 |
| Vocab | 16,384 (byte-level BPE) |
| Context | 1,024 |

## Prompt format

The chat format is **not** a standard template — it uses custom special tokens
that exist in this tokenizer. Render exactly:

```
<|bos|><|system|>{system}<|user|>{user}<|assistant|>
```

Then generate. Stop at `<|eos|>`. Sending a different system prompt than the one
below moves the model off-distribution and quality degrades silently.

## Provenance

- Base: [Kotichitturu/slm-125m-base](https://huggingface.co/Kotichitturu/slm-125m-base)
- SFT data: 8,000 passages (US case law + SEC filings), teacher-generated by
  gpt-4o-mini and gemini-3.1-flash-lite, then filtered by an LLM judge
  (gpt-5.4-mini) that rejected every answer not supported by its passage.
  Judge coverage 100%; keep rate 88%.
- Decontaminated against CaseHOLD/LexGLUE (13-gram) during pretraining.
- Full SFT (not LoRA), 3 epochs, lr 2e-5 cosine, bf16, 1×A100.
- **Loss is masked to response tokens only** — the model never trains on the
  prompt it is given.