79 lines
3.5 KiB
Markdown
79 lines
3.5 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
|
||
tags:
|
||
- code
|
||
- function-calling
|
||
- tool-use
|
||
- agent
|
||
- small-language-model
|
||
datasets:
|
||
- NousResearch/hermes-function-calling-v1
|
||
language:
|
||
- en
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# smolcode-coder-1.5b-tools
|
||
|
||
A LoRA fine-tune of **Qwen2.5-Coder-1.5B-Instruct** that teaches the model to emit
|
||
**native `<tool_call>` function calls**, so a 1.5B *coder* model can actually drive an
|
||
agentic write → run → fix → verify loop.
|
||
|
||
Built for [**smolcode**](https://gitea.poyner.ai/sean/smolcode) — an SLM-optimized
|
||
agentic coding assistant — for the Hugging Face **Build Small** hackathon.
|
||
|
||
## Why
|
||
Out of the box, small Qwen-Coder models describe tool calls as plain-text/```json
|
||
instead of emitting the native `<tool_call>` token (id 151657) that runtimes (Ollama,
|
||
llama.cpp) parse into OpenAI-style `tool_calls` — which breaks agentic loops. This
|
||
fine-tune closes that gap on a tiny (1.5B) model: **100% native `<tool_call>` emission**
|
||
in free generation on held-out prompts (base model: 0%).
|
||
|
||
## Results
|
||
- **Native tool-call rate:** 100% (16/16 held-out prompts) — the release gate.
|
||
- **Agentic bench (smolcode pass@1, 10 tasks):** 9/10 as the entry tier of a
|
||
1.5B→8B→30B ladder, solving **7/10 entirely on its own** (2–16s each). For
|
||
comparison the all-Granite ladder (3B entry) scores 10/10 — the 1.5B carries the
|
||
same standalone load as a 2×-larger 3B.
|
||
- **Train loss:** 0.138 (3 epochs, assistant-only loss).
|
||
|
||
## Training
|
||
- **Base:** Qwen/Qwen2.5-Coder-1.5B-Instruct
|
||
- **Method:** bf16 LoRA (r=16, α=32) on attention + MLP projections, **plus full
|
||
training of `embed_tokens` + `lm_head`** (`modules_to_save`) — required so the model
|
||
can *output* the `<tool_call>` special token, which LoRA on attention/MLP alone
|
||
cannot. **Assistant-only loss** (loss on tool calls + final answers only).
|
||
- **Data:** NousResearch/hermes-function-calling-v1 (breadth) + synthetic smolcode
|
||
tool-use trajectories (sharpness), all rendered through the *same*
|
||
`apply_chat_template(tools=...)` used at inference — training target is byte-identical
|
||
to the served prompt (fixes the v1 train/inference template mismatch).
|
||
- **Schedule:** 3 epochs, full 2048 sequence length. Trained on Modal (A100).
|
||
|
||
## Serving — read this, two non-obvious requirements
|
||
1. **Serve via the GGUF, not the safetensors directly.** Ollama's bf16-safetensors
|
||
auto-import produces garbage (`??????`) for this model. Use the included
|
||
`smolcode-1.5b-q4_k_m.gguf` (converted with llama.cpp `convert_hf_to_gguf.py`):
|
||
```bash
|
||
ollama create smolcode-coder-1.5b:tools -f Modelfile # Modelfile is in this repo
|
||
```
|
||
2. **`repeat_penalty` / `repetition_penalty` MUST be 1.0.** The tool system prompt
|
||
literally contains the `<tool_call>` token, so any penalty > 1 suppresses the model
|
||
from emitting it (you'll see a stray token + bare JSON instead). The included
|
||
`Modelfile` sets `PARAMETER repeat_penalty 1.0`. For raw `transformers.generate`,
|
||
pass `repetition_penalty=1.0`.
|
||
|
||
With those, Ollama's `/v1/chat/completions` returns proper native `tool_calls`.
|
||
|
||
## Use (transformers)
|
||
Standard Qwen2.5 chat template with `tools=`; greedy, `repetition_penalty=1.0`. The
|
||
model responds with `<tool_call>{"name": ..., "arguments": ...}</tool_call>`.
|
||
|
||
## Files
|
||
- `model.safetensors` + tokenizer/config — the merged model (lm_head untied).
|
||
- `smolcode-1.5b-q4_k_m.gguf` — quantized GGUF for serving.
|
||
- `Modelfile` — Ollama import recipe (template + `repeat_penalty 1.0`).
|
||
|
||
## License
|
||
Apache-2.0 (inherits from the base model).
|