--- library_name: transformers tags: - agentic - tool-calling - orchestrator - gguf - lora - qwen2 - m1300x license: apache-2.0 datasets: - WaltonFuture/agentic-sft-new language: - en base_model: - Qwen/Qwen2.5-32B-Instruct --- # Qwen2.5-32B Agentic Orchestrator **A 32B parameter model fine-tuned for agentic tool-calling workflows — trained on AMD Instinct MI300X (192GB VRAM).** Fine-tuned from [Qwen/Qwen2.5-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) using LoRA on 22,000 multi-turn agentic conversations covering tool call chaining, structured function calling, and orchestrator decision-making. --- ## Why This Model Exists Most LLMs are trained to answer questions. This model is trained to **act**. The gap this fills: base Qwen2.5-32B-Instruct is a strong reasoner, but it wasn't specifically trained on the behavioral patterns that make agents actually work in production — when to call a tool vs respond directly, how to chain tool calls across multiple turns, how to handle tool failures and escalate correctly, and how to format structured tool calls consistently. This fine-tune addresses exactly that. Trained on 22,000 real multi-turn agentic trajectories, the model learns: - **Tool call decision-making** — when to act vs when to respond - **Structured tool call formatting** — consistent `` / `` patterns - **Multi-turn chaining** — maintaining context across 9+ turn agentic conversations - **Escalation and handoff patterns** — knowing when to transfer to a human or a different agent - **Constraint awareness** — following system-level policies while serving user goals The MI300X was the enabling hardware. Running a 32B model at full BF16 precision with LoRA adapters requires ~80GB+ VRAM minimum. 192GB gave full headroom without quantization tricks during training — meaning the fine-tune captures the full expressiveness of the 32B model. --- ## Files in This Repository | File | Description | |------|-------------| | `model-q4_k_m.gguf` | **Start here.** Q4_K_M quantized — runs on 24GB RAM, Mac M2 Pro+, single A100 | | `model-00001-of-00002.safetensors` | Full BF16 merged model shard 1 | | `model-00002-of-00002.safetensors` | Full BF16 merged model shard 2 | | `tokenizer.json` | Tokenizer | | `config.json` | Model config | --- ## Quickstart ### Ollama (easiest) ```bash ollama run hf.co/MohitML10/qwen2.5-32b-agentic-orchestrator ``` ### llama.cpp ```bash wget https://huggingface.co/MohitML10/qwen2.5-32b-agentic-orchestrator/resolve/main/model-q4_k_m.gguf ./llama-cli -m model-q4_k_m.gguf \ --chat-template qwen2 \ -p "You are an agentic orchestrator. You have access to tools and decide when to use them." ``` ### Transformers ```python from transformers import AutoTokenizer, AutoModelForCausalLM import torch tokenizer = AutoTokenizer.from_pretrained("MohitML10/qwen2.5-32b-agentic-orchestrator") model = AutoModelForCausalLM.from_pretrained( "MohitML10/qwen2.5-32b-agentic-orchestrator", torch_dtype=torch.bfloat16, device_map="auto" ) messages = [ { "role": "system", "content": "You are an agentic orchestrator with access to tools. Decide when to call a tool and when to respond directly." }, { "role": "user", "content": "Search for the latest news on LLM benchmarks and summarize the top 3 findings." } ] inputs = tokenizer.apply_chat_template( messages, return_tensors="pt", add_generation_prompt=True ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) ``` ### vLLM (for serving) ```bash vllm serve MohitML10/qwen2.5-32b-agentic-orchestrator \ --dtype bfloat16 \ --max-model-len 32768 \ --enable-auto-tool-choice \ --tool-call-parser hermes ``` --- ## Training Details | Parameter | Value | |-----------|-------| | Base model | Qwen/Qwen2.5-32B-Instruct | | Hardware | AMD Instinct MI300X (192GB VRAM) | | Training method | Supervised Fine-Tuning (SFT) with LoRA | | LoRA rank | 16 | | LoRA alpha | 32 | | LoRA dropout | 0.05 | | Target modules | q_proj, k_proj, v_proj, o_proj | | Trainable parameters | 33.5M / 32.8B (0.10%) | | Dataset | WaltonFuture/agentic-sft-new | | Training samples | 22,000 | | Epochs | 1 | | Batch size | 1 (gradient accumulation 16, effective batch 16) | | Learning rate | 2e-4 | | LR scheduler | Cosine with warmup | | Precision | BF16 | | Max sequence length | 2048 tokens | | Framework | HuggingFace TRL + PEFT | --- ## Dataset Trained on a filtered subset of [WaltonFuture/agentic-sft-new](https://huggingface.co/datasets/WaltonFuture/agentic-sft-new) — specifically the multi-turn agentic conversation subset (None split label), which contains real customer service and workflow agent trajectories with structured tool calls. Conversation format uses `` and `` blocks in standard chat format, directly compatible with Qwen2.5's chat template. --- ## Intended Use - **Agentic orchestrators** — the decision layer that decides what tool to call next - **Multi-agent systems** — as the planning layer above specialized worker agents - **Tool-calling pipelines** — structured JSON tool call generation - **AI infrastructure research** — studying agentic behavior at 32B scale ## Out of Scope - General question answering (use base Qwen2.5-32B-Instruct) - Image or multimodal tasks (text only) - Tasks requiring >32k context --- ## Hardware Requirements | Format | Minimum | Recommended | |--------|---------|-------------| | GGUF Q4_K_M | 24 GB RAM | 32 GB RAM | | BF16 safetensors | 64 GB VRAM | 80 GB VRAM (A100/H100) | Runs locally on Mac M2 Pro / M3 Pro or better using the GGUF file. --- ## Limitations - Trained for 1 epoch on 22k samples — robust on tool-calling patterns but may benefit from further fine-tuning on domain-specific agentic data - No benchmark evaluation yet — before/after comparison on Berkeley Function Calling Leaderboard is a planned next step - Training data is customer service heavy — performance on other agentic domains (code, research, SWE) may vary --- ## Developed By [MohitML10](https://huggingface.co/MohitML10) Trained on AMD Developer Cloud — AMD Instinct MI300X (192GB VRAM) instance.