# FlowX Sentinel Gate (4B) — Evaluation Results

Held-out evaluation of the released model (`flowx-sentinel-gate-4b-v2`). Numbers are on a
**held-out set of 71 cases (46 ESCALATE / 17 DECIDE)** across banking, insurance, logistics,
and labor. Decoding: greedy (temperature 0), `enable_thinking=False`. The safety-critical
metric is the **false-negative rate** (gold=ESCALATE but the model says DECIDE, i.e.
auto-deciding a case that should have gone to a human).

## Headline (held-out, n=71)

| Metric | Result | Note |
| --- | --- | --- |
| False-negative rate (missed escalations) | **0.000** (0/46) | the safety metric |
| Action accuracy (ESCALATE vs DECIDE) | **1.000** | |
| False-positive rate (over-escalation) | **0.000** (0/17) | |
| JSON validity (raw) | 0.89 | deploy with the deterministic repair step |
| Category accuracy (on true-escalate) | 0.61 | the human-routing hint |

## Decision confusion (the gate's actual job)

| gold \\ predicted | ESCALATE | DECIDE |
| --- | --- | --- |
| ESCALATE (n=46) | 46 | 0 |
| DECIDE (n=17) | 0 | 17 |

The ESCALATE/DECIDE decision is correct on every held-out case: no missed escalations, no
over-escalation. That decision is what gates automation and is the field to trust.

## Baseline (before the data rebalance/retrain)

| Metric | baseline (n=24, 2 DECIDE) | this release (n=71, 17 DECIDE) |
| --- | --- | --- |
| False-negative rate | 0.000 | 0.000 |
| Action accuracy | 0.944 | 1.000 |
| False-positive rate | 0.50 (noisy, n=2) | 0.000 (trustworthy, n=17) |
| JSON validity | 0.75 | 0.89 |
| Category accuracy | 0.875 (n=16) | 0.61 (n=46) |

Rebalancing DECIDE from ~15% to ~35% of the corpus and retraining made the false-positive
rate trustworthy (17 DECIDE cases vs 2) and lifted JSON validity 0.75 → 0.89 while holding the
safety metric at 0.000. The category-accuracy "drop" is the honest number emerging on a larger
held-out (the old 0.875 was small-sample noise on 16 cases).

## Honest reading / caveats

- **Category is a routing hint, not a gate.** The model gets the ESCALATE/DECIDE decision right
  every time here, but the `escalation_category` label is ~0.61 (categories legitimately
  overlap for some cases). Route on the decision; treat the category as a suggestion.
- **JSON validity 0.89 raw.** Deploy with the deterministic JSON repair step the pipeline pairs
  with the model.
- **Home-field note.** Scenarios are realistic-synthetic, grounded in real regulatory
  citations. Validate on your own case distribution before production.

## Reproduction

Held-out: `mlx_data/escalation_v2/valid.jsonl`. Scorer:
`eval_sentinel_4b.py <model_path> mlx_data/escalation_v2/valid.jsonl`.

_Author: Bogdan Răduță, Head of Research, FlowX.AI._