qwen2.5-32b-agentic-orchestrator/README.md

library_name, tags, license, datasets, language, base_model

library_name	tags	license	datasets	language	base_model
transformers	agentic tool-calling orchestrator gguf lora qwen2 m1300x	apache-2.0	WaltonFuture/agentic-sft-new	en	Qwen/Qwen2.5-32B-Instruct

Qwen2.5-32B Agentic Orchestrator

A 32B parameter model fine-tuned for agentic tool-calling workflows — trained on AMD Instinct MI300X (192GB VRAM).

Fine-tuned from Qwen/Qwen2.5-32B-Instruct using LoRA on 22,000 multi-turn agentic conversations covering tool call chaining, structured function calling, and orchestrator decision-making.

---

Why This Model Exists

Most LLMs are trained to answer questions. This model is trained to act.

The gap this fills: base Qwen2.5-32B-Instruct is a strong reasoner, but it wasn't specifically trained on the behavioral patterns that make agents actually work in production — when to call a tool vs respond directly, how to chain tool calls across multiple turns, how to handle tool failures and escalate correctly, and how to format structured tool calls consistently.

This fine-tune addresses exactly that. Trained on 22,000 real multi-turn agentic trajectories, the model learns:

• Tool call decision-making — when to act vs when to respond
• Structured tool call formatting — consistent <tool_call> / <tool_response> patterns
• Multi-turn chaining — maintaining context across 9+ turn agentic conversations
• Escalation and handoff patterns — knowing when to transfer to a human or a different agent
• Constraint awareness — following system-level policies while serving user goals

The MI300X was the enabling hardware. Running a 32B model at full BF16 precision with LoRA adapters requires ~80GB+ VRAM minimum. 192GB gave full headroom without quantization tricks during training — meaning the fine-tune captures the full expressiveness of the 32B model.

---

Files in This Repository

File	Description
model-q4_k_m.gguf	Start here. Q4_K_M quantized — runs on 24GB RAM, Mac M2 Pro+, single A100
model-00001-of-00002.safetensors	Full BF16 merged model shard 1
model-00002-of-00002.safetensors	Full BF16 merged model shard 2
tokenizer.json	Tokenizer
config.json	Model config

---

Quickstart

Ollama (easiest)

ollama run hf.co/MohitML10/qwen2.5-32b-agentic-orchestrator

llama.cpp

wget https://huggingface.co/MohitML10/qwen2.5-32b-agentic-orchestrator/resolve/main/model-q4_k_m.gguf

./llama-cli -m model-q4_k_m.gguf \
  --chat-template qwen2 \
  -p "You are an agentic orchestrator. You have access to tools and decide when to use them."

Transformers

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("MohitML10/qwen2.5-32b-agentic-orchestrator")
model = AutoModelForCausalLM.from_pretrained(
    "MohitML10/qwen2.5-32b-agentic-orchestrator",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {
        "role": "system",
        "content": "You are an agentic orchestrator with access to tools. Decide when to call a tool and when to respond directly."
    },
    {
        "role": "user",
        "content": "Search for the latest news on LLM benchmarks and summarize the top 3 findings."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
    add_generation_prompt=True
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

vLLM (for serving)

vllm serve MohitML10/qwen2.5-32b-agentic-orchestrator \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

---

Training Details

Parameter	Value
Base model	Qwen/Qwen2.5-32B-Instruct
Hardware	AMD Instinct MI300X (192GB VRAM)
Training method	Supervised Fine-Tuning (SFT) with LoRA
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj
Trainable parameters	33.5M / 32.8B (0.10%)
Dataset	WaltonFuture/agentic-sft-new
Training samples	22,000
Epochs	1
Batch size	1 (gradient accumulation 16, effective batch 16)
Learning rate	2e-4
LR scheduler	Cosine with warmup
Precision	BF16
Max sequence length	2048 tokens
Framework	HuggingFace TRL + PEFT

---

Dataset

Trained on a filtered subset of WaltonFuture/agentic-sft-new — specifically the multi-turn agentic conversation subset (None split label), which contains real customer service and workflow agent trajectories with structured tool calls.

Conversation format uses <tool_call> and <tool_response> blocks in standard chat format, directly compatible with Qwen2.5's chat template.

---

Intended Use

• Agentic orchestrators — the decision layer that decides what tool to call next
• Multi-agent systems — as the planning layer above specialized worker agents
• Tool-calling pipelines — structured JSON tool call generation
• AI infrastructure research — studying agentic behavior at 32B scale

Out of Scope

• General question answering (use base Qwen2.5-32B-Instruct)
• Image or multimodal tasks (text only)
• Tasks requiring >32k context

---

Hardware Requirements

Format	Minimum	Recommended
GGUF Q4_K_M	24 GB RAM	32 GB RAM
BF16 safetensors	64 GB VRAM	80 GB VRAM (A100/H100)

Runs locally on Mac M2 Pro / M3 Pro or better using the GGUF file.

---

Limitations

• Trained for 1 epoch on 22k samples — robust on tool-calling patterns but may benefit from further fine-tuning on domain-specific agentic data
• No benchmark evaluation yet — before/after comparison on Berkeley Function Calling Leaderboard is a planned next step
• Training data is customer service heavy — performance on other agentic domains (code, research, SWE) may vary

---

Developed By

MohitML10

Trained on AMD Developer Cloud — AMD Instinct MI300X (192GB VRAM) instance.