初始化项目，由ModelHub XC社区提供模型
Model: flowxai/sentinel-gate Source: Original Platform
This commit is contained in:
ModelHub XC
105
inference_contract/INFERENCE.md
Normal file
105
inference_contract/INFERENCE.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# Inference contract - FlowX Sentinel Gate
|
||||
|
||||
This is the **frozen inference contract** for `flowxai/sentinel-gate`: the exact system
|
||||
prompt, user-turn format, decode settings, and output schema the weights were trained
|
||||
against. Do not edit the prompt or schema; the LoRA was trained on them verbatim.
|
||||
|
||||
Prompt version: `sentinel_sys_v1`.
|
||||
|
||||
Files in this directory:
|
||||
|
||||
- [`prompt_sentinel_sys_v1.txt`](./prompt_sentinel_sys_v1.txt) - the system prompt, verbatim.
|
||||
- [`schema_sentinel_v1.json`](./schema_sentinel_v1.json) - JSON Schema for the oracle output.
|
||||
|
||||
---
|
||||
|
||||
## System prompt (verbatim)
|
||||
|
||||
The exact two-line system prompt is in
|
||||
[`prompt_sentinel_sys_v1.txt`](./prompt_sentinel_sys_v1.txt):
|
||||
|
||||
```
|
||||
You are an escalation gate for regulated decisions.
|
||||
Determine: ESCALATE or DECIDE? Output ONLY JSON.
|
||||
```
|
||||
|
||||
## User-turn format
|
||||
|
||||
One case per turn. The case JSON carries the domain facts plus the applicable
|
||||
`policy_schema` (a `PDP...` policy id), then a fixed trailing question:
|
||||
|
||||
```
|
||||
Case:
|
||||
<case JSON: domain facts + "policy_schema": "PDP...">
|
||||
|
||||
Decide: ESCALATE or DECIDE?
|
||||
```
|
||||
|
||||
The `Case:\n` prefix and the trailing `\n\nDecide: ESCALATE or DECIDE?` line are part of
|
||||
the contract - keep them exactly. The model was trained with these delimiters framing the
|
||||
case object.
|
||||
|
||||
## Decode settings
|
||||
|
||||
| Setting | Value | Why |
|
||||
|---|---|---|
|
||||
| `enable_thinking` | **`False`** | Qwen3-4B is a thinking model, but the adapter was trained on pure JSON with no thinking block. The default template yields empty/degraded output. Set this at `apply_chat_template`. |
|
||||
| `temperature` | **`0`** (greedy) | Deterministic decisions; the gate must be reproducible for audit. |
|
||||
| `max_new_tokens` | **~1200** | The oracle JSON (category block + reasoning + audit trail) can run long, especially for BOUNDARY_CONDITION and EXTERNAL_DEPENDENCY. Truncation is the main cause of invalid JSON. |
|
||||
|
||||
## The six escalation categories
|
||||
|
||||
Present as `escalation_category` when `action` is `ESCALATE` (it is `null` for `DECIDE`):
|
||||
|
||||
1. `MISSING_REQUIRED_DOCUMENTATION` - a hard precondition document/evidence is absent.
|
||||
2. `POLICY_VIOLATION` - a policy/regulation rule is triggered and blocks auto-release.
|
||||
3. `BOUNDARY_CONDITION` - the case sits near a policy threshold; the edge needs a human read.
|
||||
4. `INSUFFICIENT_CONFIDENCE` - the facts do not resolve the decision to an actionable degree.
|
||||
5. `CONFLICTING_SIGNALS` - two or more trusted sources disagree materially.
|
||||
6. `EXTERNAL_DEPENDENCY` - the decision is blocked awaiting an outside result (screening, ruling).
|
||||
|
||||
Each category emits a category-specific block under a distinct key
|
||||
(`policy_violations`, `missing_preconditions`, `boundary_analysis`, `confidence_factors`,
|
||||
`conflicting_signals`, `external_dependency`). See `schema_sentinel_v1.json`.
|
||||
|
||||
## Deterministic JSON repair (deploy with it)
|
||||
|
||||
Raw JSON validity from the model is **0.89** on the held-out set. The deployed pipeline
|
||||
pairs the model with a **deterministic JSON repair step**: parse the raw output; if it
|
||||
fails, apply structural fixes (close unterminated strings/brackets, strip any trailing
|
||||
prose after the final `}`, drop a leading thinking artifact if one leaks) and re-parse, then
|
||||
validate against `schema_sentinel_v1.json`. On a repair failure, retry the decode once. Do
|
||||
not rely on raw output being parseable; treat the repair step as part of the contract.
|
||||
|
||||
## What to gate on
|
||||
|
||||
- **Gate on the `action` field (ESCALATE vs DECIDE).** This is the decision the model is
|
||||
for, and it is **perfect on the held-out set** (n=71): zero missed escalations
|
||||
(false-negative rate 0.000) and zero over-escalation (false-positive rate 0.000). Wire
|
||||
your automate-vs-route branch off `action` alone.
|
||||
- **Treat `escalation_category` as a routing hint, not ground truth.** Category accuracy on
|
||||
true-escalate is **~0.61**; the categories legitimately overlap for some cases (e.g. a
|
||||
boundary case that is also a policy edge). Use it to pick a specialist queue, but do not
|
||||
make correctness-critical branches depend on it, and let a human re-label at intake.
|
||||
- `confidence_score` is calibrated per the training oracle; apply the threshold your risk
|
||||
posture requires. It is advisory, not a second gate.
|
||||
|
||||
## Minimal wiring (MLX)
|
||||
|
||||
```python
|
||||
from mlx_lm import load, generate
|
||||
|
||||
SYSTEM = open("prompt_sentinel_sys_v1.txt").read()
|
||||
model, tok = load("flowxai/sentinel-gate-mlx-int4")
|
||||
|
||||
case_json = "<case JSON with domain facts + policy_schema>"
|
||||
user = f"Case:\n{case_json}\n\nDecide: ESCALATE or DECIDE?"
|
||||
|
||||
prompt = tok.apply_chat_template(
|
||||
[{"role": "system", "content": SYSTEM},
|
||||
{"role": "user", "content": user}],
|
||||
add_generation_prompt=True, enable_thinking=False,
|
||||
)
|
||||
raw = generate(model, tok, prompt=prompt, max_tokens=1200, verbose=False)
|
||||
# then: deterministic JSON repair -> validate against schema_sentinel_v1.json
|
||||
```
|
||||
2
inference_contract/prompt_sentinel_sys_v1.txt
Normal file
2
inference_contract/prompt_sentinel_sys_v1.txt
Normal file
@@ -0,0 +1,2 @@
|
||||
You are an escalation gate for regulated decisions.
|
||||
Determine: ESCALATE or DECIDE? Output ONLY JSON.
|
||||
129
inference_contract/schema_sentinel_v1.json
Normal file
129
inference_contract/schema_sentinel_v1.json
Normal file
@@ -0,0 +1,129 @@
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"$id": "https://huggingface.co/flowxai/sentinel-gate/inference_contract/schema_sentinel_v1.json",
|
||||
"title": "FlowX Sentinel Gate output (schema_sentinel_v1)",
|
||||
"description": "The single JSON object the Sentinel Gate emits for one regulated-decision case. The gate decides ESCALATE (route to a human) vs DECIDE (safe to automate). When action=ESCALATE, escalation_category is one of six ids and a category-specific block is present; when action=DECIDE, escalation_category is null. Prompt version sentinel_sys_v1. Note: additionalProperties is intentionally true because the category-specific block key varies by category (policy_violations, missing_preconditions, boundary_analysis, confidence_factors, conflicting_signals, external_dependency) and DECIDE cases carry decision/rationale keys.",
|
||||
"type": "object",
|
||||
"additionalProperties": true,
|
||||
"required": [
|
||||
"action",
|
||||
"escalation_category",
|
||||
"confidence_score",
|
||||
"audit_trail"
|
||||
],
|
||||
"properties": {
|
||||
"action": {
|
||||
"type": "string",
|
||||
"description": "The gate decision. ESCALATE routes the case to a human; DECIDE marks it safe to automate. This is the field to gate on (perfect on the held-out set).",
|
||||
"enum": ["ESCALATE", "DECIDE"]
|
||||
},
|
||||
"escalation_category": {
|
||||
"description": "The human-routing label for an escalation, or null when action=DECIDE. A secondary routing hint (~0.61 accuracy on true-escalate), not the gate.",
|
||||
"type": ["string", "null"],
|
||||
"enum": [
|
||||
"MISSING_REQUIRED_DOCUMENTATION",
|
||||
"POLICY_VIOLATION",
|
||||
"BOUNDARY_CONDITION",
|
||||
"INSUFFICIENT_CONFIDENCE",
|
||||
"CONFLICTING_SIGNALS",
|
||||
"EXTERNAL_DEPENDENCY",
|
||||
null
|
||||
]
|
||||
},
|
||||
"confidence_score": {
|
||||
"type": "number",
|
||||
"description": "Calibrated confidence in the decision, 0.0-1.0. For ESCALATE this is typically the confidence that the case is safe to automate (low), so a low score supports escalation; for DECIDE it is the confidence in the auto-decision (high).",
|
||||
"minimum": 0.0,
|
||||
"maximum": 1.0
|
||||
},
|
||||
"confidence_reasoning": {
|
||||
"type": "string",
|
||||
"description": "One to three sentences explaining the confidence_score and why the case was escalated or auto-decided."
|
||||
},
|
||||
"human_action_required": {
|
||||
"type": "string",
|
||||
"description": "For ESCALATE: the concrete next step a human owner must take (who does what). For DECIDE: the string \"NONE\"."
|
||||
},
|
||||
"audit_trail": {
|
||||
"type": "array",
|
||||
"description": "Ordered, append-only log of the reasoning steps and policy gates evaluated, for compliance review.",
|
||||
"items": { "type": "string" },
|
||||
"minItems": 1
|
||||
},
|
||||
"policy_violations": {
|
||||
"type": "object",
|
||||
"description": "Category-specific block for POLICY_VIOLATION. Keyed by violation id; each entry names the policy, regulation, restriction, and consequence.",
|
||||
"additionalProperties": true
|
||||
},
|
||||
"missing_preconditions": {
|
||||
"type": "object",
|
||||
"description": "Category-specific block for MISSING_REQUIRED_DOCUMENTATION. Keyed by the missing precondition; each entry names required_by, regulation, severity, and reason.",
|
||||
"additionalProperties": true
|
||||
},
|
||||
"boundary_analysis": {
|
||||
"type": "object",
|
||||
"description": "Category-specific block for BOUNDARY_CONDITION. Names the policy_threshold, the shipment/case value, distance_from_threshold, and an assessment of the edge case.",
|
||||
"additionalProperties": true
|
||||
},
|
||||
"confidence_factors": {
|
||||
"type": "object",
|
||||
"description": "Category-specific block for INSUFFICIENT_CONFIDENCE. Lists ambiguous_signals and why_uncertain.",
|
||||
"additionalProperties": true
|
||||
},
|
||||
"conflicting_signals": {
|
||||
"type": "array",
|
||||
"description": "Category-specific block for CONFLICTING_SIGNALS. The competing sources/values that disagree.",
|
||||
"items": { "type": "object", "additionalProperties": true }
|
||||
},
|
||||
"external_dependency": {
|
||||
"type": "object",
|
||||
"description": "Category-specific block for EXTERNAL_DEPENDENCY. Names what the decision is awaiting and the blocking_gate.",
|
||||
"additionalProperties": true
|
||||
},
|
||||
"escalation_path": {
|
||||
"type": "string",
|
||||
"description": "Optional routing hint naming the specialist queue or workflow that should own the escalation."
|
||||
},
|
||||
"policy_gates_passed": {
|
||||
"type": "array",
|
||||
"description": "Optional list of policy gates that were checked and passed before the decision (present on some ESCALATE edge cases and on DECIDE cases).",
|
||||
"items": { "type": "string" }
|
||||
},
|
||||
"decision": {
|
||||
"type": "string",
|
||||
"description": "For DECIDE cases: the automated outcome selected (e.g. ROUTE_APPROVED)."
|
||||
},
|
||||
"selected_route": {
|
||||
"type": "string",
|
||||
"description": "For DECIDE cases where a route/option is chosen: the selected option."
|
||||
},
|
||||
"rationale": {
|
||||
"type": "string",
|
||||
"description": "For DECIDE cases: the plain rationale for auto-deciding (some records use confidence_reasoning for this)."
|
||||
}
|
||||
},
|
||||
"allOf": [
|
||||
{
|
||||
"if": { "properties": { "action": { "const": "DECIDE" } } },
|
||||
"then": { "properties": { "escalation_category": { "type": "null" } } }
|
||||
},
|
||||
{
|
||||
"if": { "properties": { "action": { "const": "ESCALATE" } } },
|
||||
"then": {
|
||||
"properties": {
|
||||
"escalation_category": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"MISSING_REQUIRED_DOCUMENTATION",
|
||||
"POLICY_VIOLATION",
|
||||
"BOUNDARY_CONDITION",
|
||||
"INSUFFICIENT_CONFIDENCE",
|
||||
"CONFLICTING_SIGNALS",
|
||||
"EXTERNAL_DEPENDENCY"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user