128 lines
3.5 KiB
Markdown
128 lines
3.5 KiB
Markdown
---
|
|
license: apache-2.0
|
|
tags:
|
|
- tool-calling
|
|
- edge-inference
|
|
library_name: transformers
|
|
---
|
|
|
|
# qwen3-0.6b-tool-router
|
|
|
|
**A low-latency, schema-strict tool/function calling model optimized for edge-device inference.**
|
|
|
|
|
|
## Overview
|
|
|
|
`qwen3-0.6b-tool-router` is a **verticalized Small Language Model (SLM)** derived from **Qwen3-0.6B**, purpose-built for **tool and function routing** under strict JSON schemas.
|
|
|
|
Unlike general-purpose chat or instruction-following models, this model is optimized to run as a **deterministic router** in agentic systems, especially in **resource-constrained edge environments** (e.g., CPUs, embedded GPUs, mobile accelerators).
|
|
|
|
Its sole responsibility is to reliably map **natural language queries → structured tool calls**, with **minimal latency** and **zero tolerance for hallucinated tools**.
|
|
|
|
|
|
### Key Properties
|
|
|
|
- **Model Size:** 0.6B parameters
|
|
- **No Chain-of-Thought:** Disabled to reduce token count and parsing cost
|
|
- **Strict JSON Output:** Designed for direct machine consumption
|
|
- **Low Memory Footprint:** QLoRA fine-tuning, edge-friendly quantization
|
|
- **Fast Cold Start:** Ideal for on-device or near-device inference
|
|
|
|
This makes it well-suited for:
|
|
- On-device assistants
|
|
- Local agent routers
|
|
- Offline-capable systems
|
|
- Privacy-sensitive deployments
|
|
|
|
### BFCL Results
|
|
|
|
| Category | Score |
|
|
|---------------------------|-------|
|
|
| **Non-Live Parallel AST** | **83.50%** |
|
|
| **Multi-Turn Base** | **90.42%** |
|
|
| **Live Simple AST** | **62.86%** |
|
|
| **Live Parallel AST** | **52.00%** |
|
|
| **Relevance Detection** | **90.89%** |
|
|
|
|
```python
|
|
import json
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
MODEL_PATH = "AryanNsc/qwen3-0.6b-tool-router"
|
|
|
|
# Load tokenizer & model
|
|
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)
|
|
tokenizer.padding_side = "left"
|
|
tokenizer.pad_token = tokenizer.eos_token
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
MODEL_PATH,
|
|
device_map="auto",
|
|
torch_dtype="auto",
|
|
trust_remote_code=True
|
|
)
|
|
|
|
# Define a tool
|
|
tools = [{
|
|
"name": "get_weather",
|
|
"description": "Get weather for a city",
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"city": {"type": "string"}
|
|
},
|
|
"required": ["city"]
|
|
}
|
|
}]
|
|
|
|
# Build system prompt with tools
|
|
system_prompt = (
|
|
"You may call one or more functions.\n\n"
|
|
"<tools>\n"
|
|
+ "\n".join(json.dumps(t) for t in tools)
|
|
+ "\n</tools>\n\n"
|
|
"Return the function call inside <tool_call></tool_call> tags."
|
|
)
|
|
|
|
messages = [
|
|
{"role": "system", "content": system_prompt},
|
|
{"role": "user", "content": "What's the weather in Tokyo?"}
|
|
]
|
|
|
|
# Apply chat template
|
|
prompt = tokenizer.apply_chat_template(
|
|
messages,
|
|
tokenize=False,
|
|
add_generation_prompt=True,
|
|
enable_thinking=False
|
|
)
|
|
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|
|
|
# Generate
|
|
with torch.no_grad():
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=256,
|
|
do_sample=False,
|
|
pad_token_id=tokenizer.pad_token_id
|
|
)
|
|
|
|
# Decode only the generated tokens
|
|
generated = outputs[:, inputs.input_ids.shape[1]:]
|
|
text = tokenizer.decode(generated[0], skip_special_tokens=True)
|
|
|
|
print(text)
|
|
```
|
|
|
|
## Why This Model for Edge Inference?
|
|
|
|
Edge environments demand:
|
|
- Small model size
|
|
- Predictable latency
|
|
- Deterministic outputs
|
|
- Minimal parsing overhead
|
|
|
|
This model was explicitly trained to satisfy those constraints.
|