2.7 KiB
FlowX Sentinel Gate (4B) — Evaluation Results
Held-out evaluation of the released model (flowx-sentinel-gate-4b-v2). Numbers are on a
held-out set of 71 cases (46 ESCALATE / 17 DECIDE) across banking, insurance, logistics,
and labor. Decoding: greedy (temperature 0), enable_thinking=False. The safety-critical
metric is the false-negative rate (gold=ESCALATE but the model says DECIDE, i.e.
auto-deciding a case that should have gone to a human).
Headline (held-out, n=71)
| Metric | Result | Note |
|---|---|---|
| False-negative rate (missed escalations) | 0.000 (0/46) | the safety metric |
| Action accuracy (ESCALATE vs DECIDE) | 1.000 | |
| False-positive rate (over-escalation) | 0.000 (0/17) | |
| JSON validity (raw) | 0.89 | deploy with the deterministic repair step |
| Category accuracy (on true-escalate) | 0.61 | the human-routing hint |
Decision confusion (the gate's actual job)
| gold \ predicted | ESCALATE | DECIDE |
|---|---|---|
| ESCALATE (n=46) | 46 | 0 |
| DECIDE (n=17) | 0 | 17 |
The ESCALATE/DECIDE decision is correct on every held-out case: no missed escalations, no over-escalation. That decision is what gates automation and is the field to trust.
Baseline (before the data rebalance/retrain)
| Metric | baseline (n=24, 2 DECIDE) | this release (n=71, 17 DECIDE) |
|---|---|---|
| False-negative rate | 0.000 | 0.000 |
| Action accuracy | 0.944 | 1.000 |
| False-positive rate | 0.50 (noisy, n=2) | 0.000 (trustworthy, n=17) |
| JSON validity | 0.75 | 0.89 |
| Category accuracy | 0.875 (n=16) | 0.61 (n=46) |
Rebalancing DECIDE from ~15% to ~35% of the corpus and retraining made the false-positive rate trustworthy (17 DECIDE cases vs 2) and lifted JSON validity 0.75 → 0.89 while holding the safety metric at 0.000. The category-accuracy "drop" is the honest number emerging on a larger held-out (the old 0.875 was small-sample noise on 16 cases).
Honest reading / caveats
- Category is a routing hint, not a gate. The model gets the ESCALATE/DECIDE decision right
every time here, but the
escalation_categorylabel is ~0.61 (categories legitimately overlap for some cases). Route on the decision; treat the category as a suggestion. - JSON validity 0.89 raw. Deploy with the deterministic JSON repair step the pipeline pairs with the model.
- Home-field note. Scenarios are realistic-synthetic, grounded in real regulatory citations. Validate on your own case distribution before production.
Reproduction
Held-out: mlx_data/escalation_v2/valid.jsonl. Scorer:
eval_sentinel_4b.py <model_path> mlx_data/escalation_v2/valid.jsonl.
Author: Bogdan Răduță, Head of Research, FlowX.AI.