初始化项目,由ModelHub XC社区提供模型
Model: flowxai/sentinel-gate Source: Original Platform
This commit is contained in:
105
inference_contract/INFERENCE.md
Normal file
105
inference_contract/INFERENCE.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# Inference contract - FlowX Sentinel Gate
|
||||
|
||||
This is the **frozen inference contract** for `flowxai/sentinel-gate`: the exact system
|
||||
prompt, user-turn format, decode settings, and output schema the weights were trained
|
||||
against. Do not edit the prompt or schema; the LoRA was trained on them verbatim.
|
||||
|
||||
Prompt version: `sentinel_sys_v1`.
|
||||
|
||||
Files in this directory:
|
||||
|
||||
- [`prompt_sentinel_sys_v1.txt`](./prompt_sentinel_sys_v1.txt) - the system prompt, verbatim.
|
||||
- [`schema_sentinel_v1.json`](./schema_sentinel_v1.json) - JSON Schema for the oracle output.
|
||||
|
||||
---
|
||||
|
||||
## System prompt (verbatim)
|
||||
|
||||
The exact two-line system prompt is in
|
||||
[`prompt_sentinel_sys_v1.txt`](./prompt_sentinel_sys_v1.txt):
|
||||
|
||||
```
|
||||
You are an escalation gate for regulated decisions.
|
||||
Determine: ESCALATE or DECIDE? Output ONLY JSON.
|
||||
```
|
||||
|
||||
## User-turn format
|
||||
|
||||
One case per turn. The case JSON carries the domain facts plus the applicable
|
||||
`policy_schema` (a `PDP...` policy id), then a fixed trailing question:
|
||||
|
||||
```
|
||||
Case:
|
||||
<case JSON: domain facts + "policy_schema": "PDP...">
|
||||
|
||||
Decide: ESCALATE or DECIDE?
|
||||
```
|
||||
|
||||
The `Case:\n` prefix and the trailing `\n\nDecide: ESCALATE or DECIDE?` line are part of
|
||||
the contract - keep them exactly. The model was trained with these delimiters framing the
|
||||
case object.
|
||||
|
||||
## Decode settings
|
||||
|
||||
| Setting | Value | Why |
|
||||
|---|---|---|
|
||||
| `enable_thinking` | **`False`** | Qwen3-4B is a thinking model, but the adapter was trained on pure JSON with no thinking block. The default template yields empty/degraded output. Set this at `apply_chat_template`. |
|
||||
| `temperature` | **`0`** (greedy) | Deterministic decisions; the gate must be reproducible for audit. |
|
||||
| `max_new_tokens` | **~1200** | The oracle JSON (category block + reasoning + audit trail) can run long, especially for BOUNDARY_CONDITION and EXTERNAL_DEPENDENCY. Truncation is the main cause of invalid JSON. |
|
||||
|
||||
## The six escalation categories
|
||||
|
||||
Present as `escalation_category` when `action` is `ESCALATE` (it is `null` for `DECIDE`):
|
||||
|
||||
1. `MISSING_REQUIRED_DOCUMENTATION` - a hard precondition document/evidence is absent.
|
||||
2. `POLICY_VIOLATION` - a policy/regulation rule is triggered and blocks auto-release.
|
||||
3. `BOUNDARY_CONDITION` - the case sits near a policy threshold; the edge needs a human read.
|
||||
4. `INSUFFICIENT_CONFIDENCE` - the facts do not resolve the decision to an actionable degree.
|
||||
5. `CONFLICTING_SIGNALS` - two or more trusted sources disagree materially.
|
||||
6. `EXTERNAL_DEPENDENCY` - the decision is blocked awaiting an outside result (screening, ruling).
|
||||
|
||||
Each category emits a category-specific block under a distinct key
|
||||
(`policy_violations`, `missing_preconditions`, `boundary_analysis`, `confidence_factors`,
|
||||
`conflicting_signals`, `external_dependency`). See `schema_sentinel_v1.json`.
|
||||
|
||||
## Deterministic JSON repair (deploy with it)
|
||||
|
||||
Raw JSON validity from the model is **0.89** on the held-out set. The deployed pipeline
|
||||
pairs the model with a **deterministic JSON repair step**: parse the raw output; if it
|
||||
fails, apply structural fixes (close unterminated strings/brackets, strip any trailing
|
||||
prose after the final `}`, drop a leading thinking artifact if one leaks) and re-parse, then
|
||||
validate against `schema_sentinel_v1.json`. On a repair failure, retry the decode once. Do
|
||||
not rely on raw output being parseable; treat the repair step as part of the contract.
|
||||
|
||||
## What to gate on
|
||||
|
||||
- **Gate on the `action` field (ESCALATE vs DECIDE).** This is the decision the model is
|
||||
for, and it is **perfect on the held-out set** (n=71): zero missed escalations
|
||||
(false-negative rate 0.000) and zero over-escalation (false-positive rate 0.000). Wire
|
||||
your automate-vs-route branch off `action` alone.
|
||||
- **Treat `escalation_category` as a routing hint, not ground truth.** Category accuracy on
|
||||
true-escalate is **~0.61**; the categories legitimately overlap for some cases (e.g. a
|
||||
boundary case that is also a policy edge). Use it to pick a specialist queue, but do not
|
||||
make correctness-critical branches depend on it, and let a human re-label at intake.
|
||||
- `confidence_score` is calibrated per the training oracle; apply the threshold your risk
|
||||
posture requires. It is advisory, not a second gate.
|
||||
|
||||
## Minimal wiring (MLX)
|
||||
|
||||
```python
|
||||
from mlx_lm import load, generate
|
||||
|
||||
SYSTEM = open("prompt_sentinel_sys_v1.txt").read()
|
||||
model, tok = load("flowxai/sentinel-gate-mlx-int4")
|
||||
|
||||
case_json = "<case JSON with domain facts + policy_schema>"
|
||||
user = f"Case:\n{case_json}\n\nDecide: ESCALATE or DECIDE?"
|
||||
|
||||
prompt = tok.apply_chat_template(
|
||||
[{"role": "system", "content": SYSTEM},
|
||||
{"role": "user", "content": user}],
|
||||
add_generation_prompt=True, enable_thinking=False,
|
||||
)
|
||||
raw = generate(model, tok, prompt=prompt, max_tokens=1200, verbose=False)
|
||||
# then: deterministic JSON repair -> validate against schema_sentinel_v1.json
|
||||
```
|
||||
Reference in New Issue
Block a user