79 lines
3.5 KiB
Markdown
79 lines
3.5 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
|
|||
|
|
tags:
|
|||
|
|
- code
|
|||
|
|
- function-calling
|
|||
|
|
- tool-use
|
|||
|
|
- agent
|
|||
|
|
- small-language-model
|
|||
|
|
datasets:
|
|||
|
|
- NousResearch/hermes-function-calling-v1
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# smolcode-coder-1.5b-tools
|
|||
|
|
|
|||
|
|
A LoRA fine-tune of **Qwen2.5-Coder-1.5B-Instruct** that teaches the model to emit
|
|||
|
|
**native `<tool_call>` function calls**, so a 1.5B *coder* model can actually drive an
|
|||
|
|
agentic write → run → fix → verify loop.
|
|||
|
|
|
|||
|
|
Built for [**smolcode**](https://gitea.poyner.ai/sean/smolcode) — an SLM-optimized
|
|||
|
|
agentic coding assistant — for the Hugging Face **Build Small** hackathon.
|
|||
|
|
|
|||
|
|
## Why
|
|||
|
|
Out of the box, small Qwen-Coder models describe tool calls as plain-text/```json
|
|||
|
|
instead of emitting the native `<tool_call>` token (id 151657) that runtimes (Ollama,
|
|||
|
|
llama.cpp) parse into OpenAI-style `tool_calls` — which breaks agentic loops. This
|
|||
|
|
fine-tune closes that gap on a tiny (1.5B) model: **100% native `<tool_call>` emission**
|
|||
|
|
in free generation on held-out prompts (base model: 0%).
|
|||
|
|
|
|||
|
|
## Results
|
|||
|
|
- **Native tool-call rate:** 100% (16/16 held-out prompts) — the release gate.
|
|||
|
|
- **Agentic bench (smolcode pass@1, 10 tasks):** 9/10 as the entry tier of a
|
|||
|
|
1.5B→8B→30B ladder, solving **7/10 entirely on its own** (2–16s each). For
|
|||
|
|
comparison the all-Granite ladder (3B entry) scores 10/10 — the 1.5B carries the
|
|||
|
|
same standalone load as a 2×-larger 3B.
|
|||
|
|
- **Train loss:** 0.138 (3 epochs, assistant-only loss).
|
|||
|
|
|
|||
|
|
## Training
|
|||
|
|
- **Base:** Qwen/Qwen2.5-Coder-1.5B-Instruct
|
|||
|
|
- **Method:** bf16 LoRA (r=16, α=32) on attention + MLP projections, **plus full
|
|||
|
|
training of `embed_tokens` + `lm_head`** (`modules_to_save`) — required so the model
|
|||
|
|
can *output* the `<tool_call>` special token, which LoRA on attention/MLP alone
|
|||
|
|
cannot. **Assistant-only loss** (loss on tool calls + final answers only).
|
|||
|
|
- **Data:** NousResearch/hermes-function-calling-v1 (breadth) + synthetic smolcode
|
|||
|
|
tool-use trajectories (sharpness), all rendered through the *same*
|
|||
|
|
`apply_chat_template(tools=...)` used at inference — training target is byte-identical
|
|||
|
|
to the served prompt (fixes the v1 train/inference template mismatch).
|
|||
|
|
- **Schedule:** 3 epochs, full 2048 sequence length. Trained on Modal (A100).
|
|||
|
|
|
|||
|
|
## Serving — read this, two non-obvious requirements
|
|||
|
|
1. **Serve via the GGUF, not the safetensors directly.** Ollama's bf16-safetensors
|
|||
|
|
auto-import produces garbage (`??????`) for this model. Use the included
|
|||
|
|
`smolcode-1.5b-q4_k_m.gguf` (converted with llama.cpp `convert_hf_to_gguf.py`):
|
|||
|
|
```bash
|
|||
|
|
ollama create smolcode-coder-1.5b:tools -f Modelfile # Modelfile is in this repo
|
|||
|
|
```
|
|||
|
|
2. **`repeat_penalty` / `repetition_penalty` MUST be 1.0.** The tool system prompt
|
|||
|
|
literally contains the `<tool_call>` token, so any penalty > 1 suppresses the model
|
|||
|
|
from emitting it (you'll see a stray token + bare JSON instead). The included
|
|||
|
|
`Modelfile` sets `PARAMETER repeat_penalty 1.0`. For raw `transformers.generate`,
|
|||
|
|
pass `repetition_penalty=1.0`.
|
|||
|
|
|
|||
|
|
With those, Ollama's `/v1/chat/completions` returns proper native `tool_calls`.
|
|||
|
|
|
|||
|
|
## Use (transformers)
|
|||
|
|
Standard Qwen2.5 chat template with `tools=`; greedy, `repetition_penalty=1.0`. The
|
|||
|
|
model responds with `<tool_call>{"name": ..., "arguments": ...}</tool_call>`.
|
|||
|
|
|
|||
|
|
## Files
|
|||
|
|
- `model.safetensors` + tokenizer/config — the merged model (lm_head untied).
|
|||
|
|
- `smolcode-1.5b-q4_k_m.gguf` — quantized GGUF for serving.
|
|||
|
|
- `Modelfile` — Ollama import recipe (template + `repeat_penalty 1.0`).
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
Apache-2.0 (inherits from the base model).
|