128 lines
6.4 KiB
Markdown
128 lines
6.4 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
base_model: mistralai/Mistral-7B-Instruct-v0.3
|
|||
|
|
datasets:
|
|||
|
|
- hvss/dispatch-7b-data
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- agent
|
|||
|
|
- router
|
|||
|
|
- orchestration
|
|||
|
|
- tool-use
|
|||
|
|
- planning
|
|||
|
|
- unsloth
|
|||
|
|
- mistral
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
# The Dispatcher: Agent Ops 🚦
|
|||
|
|
|
|||
|
|
> **Dispatch-7B, an agent orchestrator in 7B parameters — the tiny AI manager that runs a team of expensive AI workers.**
|
|||
|
|
|
|||
|
|
🎬 **[Watch it route work live →](https://huggingface.co/spaces/hvss/dispatch-7b-demo)**
|
|||
|
|
|
|||
|
|
Dispatch-7B is a fine-tuned **Mistral 7B Instruct v0.3** that acts as the **orchestrator** in an agentic system. It reads a user request plus a catalog of available tools and outputs a structured execution plan: which tool handles each step, in what order (as a dependency DAG), and which steps genuinely deserve escalation to a frontier model.
|
|||
|
|
|
|||
|
|
**Small model as the boss. Big models as the workers.**
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
Most AI products send *every* request — even trivial ones — to the biggest, most expensive model. That's like having your highest-paid executive answer the front desk phone. Dispatch-7B is the dispatcher: it plans in milliseconds on cheap hardware, routes each step to the cheapest adequate worker, and escalates only when a step actually requires frontier-level reasoning.
|
|||
|
|
|
|||
|
|
## Evaluation
|
|||
|
|
|
|||
|
|
Measured on 114 held-out tasks (unseen requests *and* unseen tool-catalog combinations):
|
|||
|
|
|
|||
|
|
| Metric | Score |
|
|||
|
|
|---|---|
|
|||
|
|
| Valid plan rate (parseable JSON, schema-correct, acyclic DAG, catalog-consistent) | **97.4%** |
|
|||
|
|
| Tool selection F1 (vs. teacher plans) | **0.92** |
|
|||
|
|
| Escalation precision / recall | **0.92 / 0.75** |
|
|||
|
|
| Est. cost vs. "frontier model does everything" | **−86.7%** |
|
|||
|
|
|
|||
|
|
Robustness breakdowns:
|
|||
|
|
|
|||
|
|
| Slice | Valid plan rate |
|
|||
|
|
|---|---|
|
|||
|
|
| Distractor-heavy catalogs (relevant tools buried among irrelevant ones) | 93% |
|
|||
|
|
| Gap catalogs (a needed tool deliberately missing) | 94% |
|
|||
|
|
| Hard tasks requiring escalation judgment | 89% |
|
|||
|
|
|
|||
|
|
Notes: routing labels are distilled from a teacher model (gpt-5-mini), so accuracy metrics measure agreement with the teacher. The cost figure is a transparent cost *model* over the eval plans (frontier $3/$15 per MTok, mini-tier $0.15/$0.60, self-hosted 7B router; invalid router plans charged as full frontier fallback), not a measured bill — assumptions are in the [eval scripts](https://huggingface.co/datasets/hvss/dispatch-7b-data).
|
|||
|
|
|
|||
|
|
The model deliberately **under-escalates** slightly (11.7% of plans vs. the teacher's 15.8%): it errs on the side of cheap execution, which is usually the right default for a router.
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
The model expects Mistral's `[INST]` format with the dispatcher system prompt and your tool catalog:
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained("hvss/Dispatch-7B", torch_dtype="auto", device_map="auto")
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained("hvss/Dispatch-7B")
|
|||
|
|
|
|||
|
|
SYSTEM = '''You are a dispatcher. Read the user request and the tool catalog, then output ONLY a JSON execution plan routing each step to the cheapest adequate worker: "tool:<name>" (catalog tools only), "small_model" (simple generation), or "escalate:frontier" (genuinely hard reasoning only). Steps form a DAG via "depends_on". Output format: {"goal": ..., "steps": [{"id", "task", "assignee", "depends_on"}], "escalation_reason": ...}
|
|||
|
|
|
|||
|
|
Available tools:
|
|||
|
|
- weather_lookup: Get the weather forecast for a location.
|
|||
|
|
- calendar_create_event: Create a calendar event with title, time, and attendees.
|
|||
|
|
- email_send: Send an email to one or more recipients.'''
|
|||
|
|
|
|||
|
|
request = "Set up a team picnic next Friday at noon if the weather looks good, and email the team an invite."
|
|||
|
|
|
|||
|
|
prompt = f"[INST] {SYSTEM}\n\n{request} [/INST]"
|
|||
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|||
|
|
out = model.generate(**inputs, max_new_tokens=768, do_sample=False)
|
|||
|
|
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Output:
|
|||
|
|
|
|||
|
|
```json
|
|||
|
|
{"goal": "...", "steps": [
|
|||
|
|
{"id": 1, "task": "Check the weather forecast for next Friday.", "assignee": "tool:weather_lookup", "depends_on": []},
|
|||
|
|
{"id": 2, "task": "Create the picnic calendar event if the forecast is good.", "assignee": "tool:calendar_create_event", "depends_on": [1]},
|
|||
|
|
{"id": 3, "task": "Email the team an invite with a short fun blurb.", "assignee": "tool:email_send", "depends_on": [2]}
|
|||
|
|
], "escalation_reason": null}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Three assignee classes:
|
|||
|
|
|
|||
|
|
| Assignee | Meaning | Cost |
|
|||
|
|
|---|---|---|
|
|||
|
|
| `tool:<name>` | Direct tool/API call from your catalog | ~free |
|
|||
|
|
| `small_model` | Simple generation a small LLM handles fine | cheap |
|
|||
|
|
| `escalate:frontier` | Genuinely hard reasoning — send to the big model | expensive, used sparingly |
|
|||
|
|
|
|||
|
|
Swap in **your own catalog** — the model was trained on randomized catalogs (with distractors and deliberate gaps), so it plans with whatever tools you give it rather than assuming a fixed set.
|
|||
|
|
|
|||
|
|
## Variants
|
|||
|
|
|
|||
|
|
| Repo | Use case |
|
|||
|
|
|---|---|
|
|||
|
|
| [hvss/Dispatch-7B](https://huggingface.co/hvss/Dispatch-7B) | merged fp16 — vLLM / transformers |
|
|||
|
|
| [hvss/Dispatch-7B-GGUF](https://huggingface.co/hvss/Dispatch-7B-GGUF) | Q4_K_M / Q8_0 — llama.cpp, Ollama, local |
|
|||
|
|
| [hvss/Dispatch-7B-LoRA](https://huggingface.co/hvss/Dispatch-7B-LoRA) | adapter only — further fine-tuning |
|
|||
|
|
|
|||
|
|
## Training
|
|||
|
|
|
|||
|
|
- **Base**: Mistral 7B Instruct v0.3 (Apache 2.0)
|
|||
|
|
- **Method**: QLoRA (4-bit, r=16 on all attention + MLP projections) via Unsloth, on a single free Colab T4
|
|||
|
|
- **Data**: 2,158 (request, gold plan) pairs distilled from a teacher model over a structured taxonomy: 8 domains × 4 difficulty tiers × 3 catalog conditions (full / distractor-heavy / deliberately-missing-tool), ~16% escalation rate to teach restraint
|
|||
|
|
- **Loss**: response-only (the model learns to write plans, not to regenerate catalogs)
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
- English-only training data.
|
|||
|
|
- Routing quality is bounded by the teacher's judgment; gold labels are synthetic.
|
|||
|
|
- The tool catalogs are synthetic (40 tools across 8 domains); very different domains or tool-description styles may need a light further fine-tune (start from the [LoRA repo](https://huggingface.co/hvss/Dispatch-7B-LoRA)).
|
|||
|
|
- The model plans; it does not execute. Your harness owns tool execution, worker calls, and validation (schema-check the output — 2.6% of plans fail validation and should fall back to your default model).
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Apache 2.0 — free for commercial use.
|