qwen2.5-32b-agentic-orchestrator/README.md

---
library_name: transformers
tags:
- agentic
- tool-calling
- orchestrator
- gguf
- lora
- qwen2
- m1300x
license: apache-2.0
datasets:
- WaltonFuture/agentic-sft-new
language:
- en
base_model:
- Qwen/Qwen2.5-32B-Instruct
---

# Qwen2.5-32B Agentic Orchestrator

**A 32B parameter model fine-tuned for agentic tool-calling workflows — trained on AMD Instinct MI300X (192GB VRAM).**

Fine-tuned from [Qwen/Qwen2.5-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) using LoRA on 22,000 multi-turn agentic conversations covering tool call chaining, structured function calling, and orchestrator decision-making.

---

## Why This Model Exists

Most LLMs are trained to answer questions. This model is trained to **act**.

The gap this fills: base Qwen2.5-32B-Instruct is a strong reasoner, but it wasn't specifically trained on the behavioral patterns that make agents actually work in production — when to call a tool vs respond directly, how to chain tool calls across multiple turns, how to handle tool failures and escalate correctly, and how to format structured tool calls consistently.

This fine-tune addresses exactly that. Trained on 22,000 real multi-turn agentic trajectories, the model learns:

- **Tool call decision-making** — when to act vs when to respond
- **Structured tool call formatting** — consistent `<tool_call>` / `<tool_response>` patterns
- **Multi-turn chaining** — maintaining context across 9+ turn agentic conversations
- **Escalation and handoff patterns** — knowing when to transfer to a human or a different agent
- **Constraint awareness** — following system-level policies while serving user goals

The MI300X was the enabling hardware. Running a 32B model at full BF16 precision with LoRA adapters requires ~80GB+ VRAM minimum. 192GB gave full headroom without quantization tricks during training — meaning the fine-tune captures the full expressiveness of the 32B model.

---

## Files in This Repository

| File | Description |
|------|-------------|
| `model-q4_k_m.gguf` | **Start here.** Q4_K_M quantized — runs on 24GB RAM, Mac M2 Pro+, single A100 |
| `model-00001-of-00002.safetensors` | Full BF16 merged model shard 1 |
| `model-00002-of-00002.safetensors` | Full BF16 merged model shard 2 |
| `tokenizer.json` | Tokenizer |
| `config.json` | Model config |

---

## Quickstart

### Ollama (easiest)
```bash
ollama run hf.co/MohitML10/qwen2.5-32b-agentic-orchestrator
```

### llama.cpp
```bash
wget https://huggingface.co/MohitML10/qwen2.5-32b-agentic-orchestrator/resolve/main/model-q4_k_m.gguf

./llama-cli -m model-q4_k_m.gguf \
  --chat-template qwen2 \
  -p "You are an agentic orchestrator. You have access to tools and decide when to use them."
```

### Transformers
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("MohitML10/qwen2.5-32b-agentic-orchestrator")
model = AutoModelForCausalLM.from_pretrained(
    "MohitML10/qwen2.5-32b-agentic-orchestrator",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {
        "role": "system",
        "content": "You are an agentic orchestrator with access to tools. Decide when to call a tool and when to respond directly."
    },
    {
        "role": "user",
        "content": "Search for the latest news on LLM benchmarks and summarize the top 3 findings."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
    add_generation_prompt=True
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))
```

### vLLM (for serving)
```bash
vllm serve MohitML10/qwen2.5-32b-agentic-orchestrator \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

---

## Training Details

| Parameter | Value |
|-----------|-------|
| Base model | Qwen/Qwen2.5-32B-Instruct |
| Hardware | AMD Instinct MI300X (192GB VRAM) |
| Training method | Supervised Fine-Tuning (SFT) with LoRA |
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Target modules | q_proj, k_proj, v_proj, o_proj |
| Trainable parameters | 33.5M / 32.8B (0.10%) |
| Dataset | WaltonFuture/agentic-sft-new |
| Training samples | 22,000 |
| Epochs | 1 |
| Batch size | 1 (gradient accumulation 16, effective batch 16) |
| Learning rate | 2e-4 |
| LR scheduler | Cosine with warmup |
| Precision | BF16 |
| Max sequence length | 2048 tokens |
| Framework | HuggingFace TRL + PEFT |

---

## Dataset

Trained on a filtered subset of [WaltonFuture/agentic-sft-new](https://huggingface.co/datasets/WaltonFuture/agentic-sft-new) — specifically the multi-turn agentic conversation subset (None split label), which contains real customer service and workflow agent trajectories with structured tool calls.

Conversation format uses `<tool_call>` and `<tool_response>` blocks in standard chat format, directly compatible with Qwen2.5's chat template.

---

## Intended Use

- **Agentic orchestrators** — the decision layer that decides what tool to call next
- **Multi-agent systems** — as the planning layer above specialized worker agents  
- **Tool-calling pipelines** — structured JSON tool call generation
- **AI infrastructure research** — studying agentic behavior at 32B scale

## Out of Scope

- General question answering (use base Qwen2.5-32B-Instruct)
- Image or multimodal tasks (text only)
- Tasks requiring >32k context

---

## Hardware Requirements

| Format | Minimum | Recommended |
|--------|---------|-------------|
| GGUF Q4_K_M | 24 GB RAM | 32 GB RAM |
| BF16 safetensors | 64 GB VRAM | 80 GB VRAM (A100/H100) |

Runs locally on Mac M2 Pro / M3 Pro or better using the GGUF file.

---

## Limitations

- Trained for 1 epoch on 22k samples — robust on tool-calling patterns but may benefit from further fine-tuning on domain-specific agentic data
- No benchmark evaluation yet — before/after comparison on Berkeley Function Calling Leaderboard is a planned next step
- Training data is customer service heavy — performance on other agentic domains (code, research, SWE) may vary

---

## Developed By

[MohitML10](https://huggingface.co/MohitML10)

Trained on AMD Developer Cloud — AMD Instinct MI300X (192GB VRAM) instance.