60 lines
2.7 KiB
Markdown
60 lines
2.7 KiB
Markdown
|
# FlowX Sentinel Gate (4B) — Evaluation Results
|
||
|
|
|
||
|
|
Held-out evaluation of the released model (`flowx-sentinel-gate-4b-v2`). Numbers are on a
|
||
|
|
**held-out set of 71 cases (46 ESCALATE / 17 DECIDE)** across banking, insurance, logistics,
|
||
|
|
and labor. Decoding: greedy (temperature 0), `enable_thinking=False`. The safety-critical
|
||
|
|
metric is the **false-negative rate** (gold=ESCALATE but the model says DECIDE, i.e.
|
||
|
|
auto-deciding a case that should have gone to a human).
|
||
|
|
|
||
|
|
## Headline (held-out, n=71)
|
||
|
|
|
||
|
|
| Metric | Result | Note |
|
||
|
|
| --- | --- | --- |
|
||
|
|
| False-negative rate (missed escalations) | **0.000** (0/46) | the safety metric |
|
||
|
|
| Action accuracy (ESCALATE vs DECIDE) | **1.000** | |
|
||
|
|
| False-positive rate (over-escalation) | **0.000** (0/17) | |
|
||
|
|
| JSON validity (raw) | 0.89 | deploy with the deterministic repair step |
|
||
|
|
| Category accuracy (on true-escalate) | 0.61 | the human-routing hint |
|
||
|
|
|
||
|
|
## Decision confusion (the gate's actual job)
|
||
|
|
|
||
|
|
| gold \\ predicted | ESCALATE | DECIDE |
|
||
|
|
| --- | --- | --- |
|
||
|
|
| ESCALATE (n=46) | 46 | 0 |
|
||
|
|
| DECIDE (n=17) | 0 | 17 |
|
||
|
|
|
||
|
|
The ESCALATE/DECIDE decision is correct on every held-out case: no missed escalations, no
|
||
|
|
over-escalation. That decision is what gates automation and is the field to trust.
|
||
|
|
|
||
|
|
## Baseline (before the data rebalance/retrain)
|
||
|
|
|
||
|
|
| Metric | baseline (n=24, 2 DECIDE) | this release (n=71, 17 DECIDE) |
|
||
|
|
| --- | --- | --- |
|
||
|
|
| False-negative rate | 0.000 | 0.000 |
|
||
|
|
| Action accuracy | 0.944 | 1.000 |
|
||
|
|
| False-positive rate | 0.50 (noisy, n=2) | 0.000 (trustworthy, n=17) |
|
||
|
|
| JSON validity | 0.75 | 0.89 |
|
||
|
|
| Category accuracy | 0.875 (n=16) | 0.61 (n=46) |
|
||
|
|
|
||
|
|
Rebalancing DECIDE from ~15% to ~35% of the corpus and retraining made the false-positive
|
||
|
|
rate trustworthy (17 DECIDE cases vs 2) and lifted JSON validity 0.75 → 0.89 while holding the
|
||
|
|
safety metric at 0.000. The category-accuracy "drop" is the honest number emerging on a larger
|
||
|
|
held-out (the old 0.875 was small-sample noise on 16 cases).
|
||
|
|
|
||
|
|
## Honest reading / caveats
|
||
|
|
|
||
|
|
- **Category is a routing hint, not a gate.** The model gets the ESCALATE/DECIDE decision right
|
||
|
|
every time here, but the `escalation_category` label is ~0.61 (categories legitimately
|
||
|
|
overlap for some cases). Route on the decision; treat the category as a suggestion.
|
||
|
|
- **JSON validity 0.89 raw.** Deploy with the deterministic JSON repair step the pipeline pairs
|
||
|
|
with the model.
|
||
|
|
- **Home-field note.** Scenarios are realistic-synthetic, grounded in real regulatory
|
||
|
|
citations. Validate on your own case distribution before production.
|
||
|
|
|
||
|
|
## Reproduction
|
||
|
|
|
||
|
|
Held-out: `mlx_data/escalation_v2/valid.jsonl`. Scorer:
|
||
|
|
`eval_sentinel_4b.py <model_path> mlx_data/escalation_v2/valid.jsonl`.
|
||
|
|
|
||
|
|
_Author: Bogdan Răduță, Head of Research, FlowX.AI._
|