61 lines
2.4 KiB
Markdown
61 lines
2.4 KiB
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
license: apache-2.0
|
||
|
|
base_model:
|
||
|
|
- unsloth/Qwen2.5-1.5B-Instruct
|
||
|
|
datasets:
|
||
|
|
- r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation
|
||
|
|
tags:
|
||
|
|
- qwen2.5
|
||
|
|
- qwen
|
||
|
|
- tool-calling
|
||
|
|
- function-calling
|
||
|
|
- agents
|
||
|
|
- sft
|
||
|
|
- lora
|
||
|
|
library_name: transformers
|
||
|
|
---
|
||
|
|
|
||
|
|
# Qwen2.5-1.5B-tools
|
||
|
|
|
||
|
|
Merged BF16 weights of the LoRA tool-calling fine-tune of
|
||
|
|
`unsloth/Qwen2.5-1.5B-Instruct`. Built by
|
||
|
|
`scripts/build_tools_gguf.sh Qwen2.5-1.5B-tools` (merge_and_unload of the
|
||
|
|
latest checkpoint in `checkpoints/Qwen2.5-1.5B-tools/`).
|
||
|
|
|
||
|
|
## Training
|
||
|
|
|
||
|
|
- Recipe: `TrainFinetuneRecipeForNextTokenPrediction` (NeMo AutoModel 0.5.0)
|
||
|
|
- Config: `configs/qwen25_1.5b_tools.yaml`
|
||
|
|
- LoRA: dim 32, alpha 32, dropout 0.05, target_modules `*.proj`
|
||
|
|
- Data: `sft_tools` split of `r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation`
|
||
|
|
- Learning rate: 1e-5 (cosine decay), 501 steps, global batch 64, seq length 4096
|
||
|
|
|
||
|
|
## Sibling artifacts
|
||
|
|
|
||
|
|
- GGUF quantizations: `models/Qwen2.5-1.5B-tools-GGUF/`
|
||
|
|
- NVFP4 checkpoint: `models/Qwen2.5-1.5B-tools-NVFP4/`
|
||
|
|
|
||
|
|
<!-- VALIDATION:BEGIN (auto-generated, do not edit) -->
|
||
|
|
## Validation matrix
|
||
|
|
|
||
|
|
Tool-calling validation on the `sft_tools` validation split (greedy decoding, 384 max new tokens). Throughput is single-stream greedy decode, not serving throughput.
|
||
|
|
|
||
|
|
Pretrained base (unsloth/Qwen2.5-1.5B-Instruct): 8.8% exact-args match (24/274).
|
||
|
|
Fine-tuned (BF16): 0.7% exact-args match (2/274) (-8pp vs base).
|
||
|
|
- GGUF-BF16: 2/274 (0.7%) exact, 88.6 tok/s — 100% of BF16.
|
||
|
|
- GGUF-Q4_K_M: 3/274 (1.1%) exact, 142.9 tok/s — 150% of BF16.
|
||
|
|
- GGUF-Q5_K_M: 6/274 (2.2%) exact, 131.4 tok/s — 300% of BF16.
|
||
|
|
- GGUF-Q8_0: 2/274 (0.7%) exact, 115.3 tok/s — 100% of BF16.
|
||
|
|
|
||
|
|
| Model | Quant | n | Tool call emitted | Names match | Exact args match | Δ exact vs BASE | tok/s |
|
||
|
|
|---|---|---|---|---|---|---|---|
|
||
|
|
| Qwen2.5-1.5B-tools | BASE (unsloth/Qwen2.5-1.5B-Instruct) | 274 | 272/274 (99.3%) | 59/274 (21.5%) | 24/274 (8.8%) | — | 40.6 |
|
||
|
|
| Qwen2.5-1.5B-tools | BF16 | 274 | 258/274 (94.2%) | 216/274 (78.8%) | 2/274 (0.7%) | -8pp | 38.0 |
|
||
|
|
| Qwen2.5-1.5B-tools | GGUF-BF16 | 274 | 272/274 (99.3%) | 167/274 (60.9%) | 2/274 (0.7%) | -8pp | 88.6 |
|
||
|
|
| Qwen2.5-1.5B-tools | GGUF-Q4_K_M | 274 | 248/274 (90.5%) | 49/274 (17.9%) | 3/274 (1.1%) | -7.7pp | 142.9 |
|
||
|
|
| Qwen2.5-1.5B-tools | GGUF-Q5_K_M | 274 | 260/274 (94.9%) | 79/274 (28.8%) | 6/274 (2.2%) | -6.6pp | 131.4 |
|
||
|
|
| Qwen2.5-1.5B-tools | GGUF-Q8_0 | 274 | 271/274 (98.9%) | 187/274 (68.2%) | 2/274 (0.7%) | -8pp | 115.3 |
|
||
|
|
<!-- VALIDATION:END -->
|