203 lines
6.4 KiB
Markdown
203 lines
6.4 KiB
Markdown
---
|
|
license: mpl-2.0
|
|
base_model: Igriscodes/qwen3-4b-tool
|
|
tags:
|
|
- tool-use
|
|
- function-calling
|
|
- reinforcement-learning
|
|
- mcp
|
|
- gguf
|
|
- quantized
|
|
pipeline_tag: text-generation
|
|
language:
|
|
- en
|
|
---
|
|
|
|
# Qwen3-4B-Agentic-MCP-RL - GGUF
|
|
|
|
This repository contains the GGUF quantization files for [Igriscodes/qwen-tool](https://huggingface.co/Igriscodes/qwen-tool), a fine-tuned `Qwen/Qwen3-1.7B` model optimized for multi-step tool use and structured payload delivery via the **Model Context Protocol (MCP)**.
|
|
|
|
The base model was aligned using **Proximal Policy Optimization (PPO)** on strict JSON validation, execution tracking, and tool-error recovery loops. These GGUF files allow for low-latency, low-memory local inference on edge devices, CPU-only systems, and Apple Silicon.
|
|
|
|
## Available Quantizations
|
|
|
|
* **Q2_K**: Maximum compression. Significant loss in logic, not recommended for complex tool-use but fits on ultra-low-memory devices.
|
|
* **Q3_K_M**: Balanced 3-bit compression. Better logic than Q2, suitable for highly constrained memory footprints.
|
|
* **Q4_0**: Standard legacy 4-bit quantization. Faster on certain older hardware architectures but slightly lower quality than K-quants.
|
|
* **Q4_K_M**: **Recommended.** Optimal balance of reasoning performance, generation speed, and VRAM savings.
|
|
* **Q5_0**: Standard legacy 5-bit quantization. Good middle ground, but outpaced by K-quants.
|
|
* **Q5_K_M**: High quality 5-bit compression. Retains nearly all unquantized capabilities while saving substantial VRAM.
|
|
* **Q6_K**: 6-bit quantization. Near-zero degradation from F16 while shaving off a decent chunk of file size.
|
|
* **Q8_0**: Maximum 8-bit fidelity. Extremely close to native F16 performance, ideal for strict syntax and reliable tool-calling.
|
|
* **F16**: Unquantized baseline. High fidelity, near-native performance for systems with more memory overhead.
|
|
|
|
## Local Deployment Quickstart
|
|
|
|
### Using Ollama
|
|
Ollama supports running models directly from Hugging Face via the `hf.co` registry prefix. You can pull and run your preferred precision instantly:
|
|
|
|
```bash
|
|
# Q2_K (Extreme compression)
|
|
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q2_K
|
|
|
|
# Q3_K_M (Medium 3-bit)
|
|
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q3_K_M
|
|
|
|
# Q4_0 (Legacy 4-bit)
|
|
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q4_0
|
|
|
|
# Q4_K_M (Recommended balanced version)
|
|
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q4_K_M
|
|
|
|
# Q5_0 (Legacy 5-bit)
|
|
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q5_0
|
|
|
|
# Q5_K_M (High-fidelity 5-bit)
|
|
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q5_K_M
|
|
|
|
# Q6_K (Deep 6-bit)
|
|
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q6_K
|
|
|
|
# Q8_0 (Near-lossless 8-bit)
|
|
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:Q8_0
|
|
|
|
# F16 (High-fidelity unquantized float version)
|
|
ollama run hf.co/Igriscodes/qwen3-4b-tool-gguf:F16
|
|
|
|
```
|
|
|
|
**or**
|
|
|
|
## Ollama Setup Guide
|
|
|
|
To run this model locally with full tool-calling (function calling) and thinking capabilities, you can easily package it into an **Ollama** model using the provided template configuration.
|
|
|
|
### 1. Create the Modelfile
|
|
|
|
Save the configuration block below exactly as a file named `Modelfile` in the same directory where your downloaded GGUF file is located.
|
|
|
|
> 💡 **Note:** If you are using a different quantization format than the `q4_k_m` example below, make sure to update the `FROM` line to match your exact `.gguf` filename.
|
|
|
|
```text
|
|
# Point to your quantized GGUF file
|
|
FROM ./qwen3-4b-tool-q4_k_m.gguf
|
|
|
|
# Custom template optimizing tool-use syntax and thought blocks
|
|
TEMPLATE """{{- $lastUserIdx := -1 -}}
|
|
{{- range $idx, $msg := .Messages -}}
|
|
{{- if eq $msg.Role "user" }}{{ $lastUserIdx = $idx }}{{ end -}}
|
|
{{- end }}
|
|
{{- if or .System .Tools }}<|im_start|>system
|
|
{{ if .System }}{{ .System }}
|
|
|
|
{{ end }}
|
|
{{- if .Tools }}# Tools
|
|
|
|
You may call one or more functions to assist with the user query.
|
|
|
|
You are provided with function signatures within <tools></tools> XML tags:
|
|
<tools>
|
|
{{- range .Tools }}
|
|
{"type": "function", "function": {{ .Function }}}
|
|
{{- end }}
|
|
</tools>
|
|
|
|
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
|
|
<tool_call>
|
|
{"name": <function-name>, "arguments": <args-json-object>}
|
|
</tool_call>
|
|
{{- end -}}
|
|
<|im_end|>
|
|
{{ end }}
|
|
{{- range $i, $_ := .Messages }}
|
|
{{- $last := eq (len (slice $.Messages $i)) 1 -}}
|
|
{{- if eq .Role "user" }}<|im_start|>user
|
|
{{ .Content }}<|im_end|>
|
|
{{ else if eq .Role "assistant" }}<|im_start|>assistant
|
|
{{ if (and $.IsThinkSet (and .Thinking (or $last (gt $i $lastUserIdx)))) -}}
|
|
<think>{{ .Thinking }}</think>
|
|
{{ end -}}
|
|
{{ if .Content }}{{ .Content }}{{ end }}
|
|
{{- if .ToolCalls }}
|
|
{{- range .ToolCalls }}
|
|
<tool_call>
|
|
{"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}}
|
|
</tool_call>
|
|
{{- end }}
|
|
{{- end }}{{ if not $last }}<|im_end|>
|
|
{{ end }}
|
|
{{- else if eq .Role "tool" }}<|im_start|>user
|
|
<tool_response>
|
|
{{ .Content }}
|
|
</tool_response><|im_end|>
|
|
{{ end }}
|
|
{{- if and (ne .Role "assistant") $last }}<|im_start|>assistant
|
|
<think>
|
|
{{ end }}
|
|
{{- end }}"""
|
|
|
|
# Inference parameters optimized for structured reasoning
|
|
PARAMETER temperature 0.6
|
|
PARAMETER num_ctx 8192
|
|
PARAMETER num_gpu -1
|
|
PARAMETER top_k 20
|
|
PARAMETER top_p 0.95
|
|
PARAMETER repeat_penalty 1
|
|
PARAMETER stop <|im_start|>
|
|
PARAMETER stop <|im_end|>
|
|
|
|
```
|
|
|
|
### 2. Build and Run the Model
|
|
|
|
Open your terminal, navigate to the directory containing your `Modelfile` and your `.gguf` file, and execute the build command:
|
|
|
|
```bash
|
|
ollama create qwen3-4b-tool --file Modelfile
|
|
|
|
```
|
|
|
|
Once the build process completes, you can launch and interact with your new custom model natively via Ollama:
|
|
|
|
```bash
|
|
ollama run qwen3-4b-tool
|
|
|
|
```
|
|
|
|
### Using Python (`llama-cpp-python`)
|
|
|
|
First, ensure you have the library installed:
|
|
|
|
```bash
|
|
pip install llama-cpp-python
|
|
```
|
|
|
|
Depending on your hardware constraints, you can load either the uncompressed precision or the quantized version using the snippets below:
|
|
|
|
#### Option 1: High Fidelity (F16 Precision)
|
|
|
|
```python
|
|
from llama_cpp import Llama
|
|
|
|
llm = Llama.from_pretrained(
|
|
repo_id="Igriscodes/qwen3-4b-tool-gguf",
|
|
filename="qwen3-4b-tool-f16.gguf",
|
|
n_ctx=2048,
|
|
n_gpu_layers=-1 # Use -1 to offload all layers to GPU (Metal/CUDA)
|
|
)
|
|
|
|
```
|
|
|
|
#### Option 2: Low Resource (Q4 Quantization)
|
|
|
|
```python
|
|
from llama_cpp import Llama
|
|
|
|
llm = Llama.from_pretrained(
|
|
repo_id="Igriscodes/qwen3-4b-tool-gguf",
|
|
filename="qwen3-4b-tool-q4.gguf",
|
|
n_ctx=2048,
|
|
n_gpu_layers=-1 # Optimized for CPU execution or limited VRAM
|
|
)
|
|
|
|
``` |