106 lines
3.8 KiB
Markdown
106 lines
3.8 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
|
|
tags:
|
|
- mlx
|
|
- lora
|
|
- text-generation
|
|
- shell
|
|
- command-line
|
|
- code
|
|
language:
|
|
- en
|
|
pipeline_tag: text-generation
|
|
library_name: mlx
|
|
---
|
|
|
|
# Qwen2.5-Coder-1.5B-Shellsmith
|
|
|
|
A small, fast model that turns **plain-English instructions into a single shell
|
|
command** for macOS/Linux. LoRA fine-tune of
|
|
[`Qwen/Qwen2.5-Coder-1.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct),
|
|
trained and quantized end-to-end on an Apple Silicon Mac with
|
|
[MLX](https://github.com/ml-explore/mlx).
|
|
|
|
> "list files by size, biggest first" → `ls -lS`
|
|
> "find files larger than 100 megabytes" → `find . -type f -size +100M`
|
|
> "create a gzip tar archive of src named src.tar.gz" → `tar -czf src.tar.gz src`
|
|
|
|
## Results
|
|
|
|
Evaluated on a held-out test split the model never saw during training.
|
|
Metrics are structural (no command execution) and conservative — see the
|
|
[eval rubric](https://github.com/saiajay1/shellsmith/blob/main/eval/rubric.md).
|
|
|
|
| Model | exact-match | command-match | flag-F1 |
|
|
| --- | :---: | :---: | :---: |
|
|
| Base (Qwen2.5-Coder-1.5B-Instruct) | 71% | 83.9% | 89.2% |
|
|
| **This model (LoRA)** | **100%** | **100%** | **100%** |
|
|
|
|
*command-match = correct program **and** option-flag F1 ≥ 0.8.*
|
|
|
|
> **What the 100% means (and doesn't):** the test split holds out unseen
|
|
> *phrasings*, but the underlying task distribution (84 canonical tasks) overlaps
|
|
> with training. So this measures reliable **in-distribution generalization across
|
|
> wording** — the model consistently emits the canonical, idiomatic command
|
|
> (`git add -A`, `ls -lS`, `git log --oneline -5`) where the base model drifts to
|
|
> looser variants (`git add .`, `ls -lh | sort -rh`, `git log -5`). It is **not**
|
|
> evidence of generalization to entirely novel tasks; broadening the task set is
|
|
> the obvious next step.
|
|
|
|
## Usage
|
|
|
|
### MLX (Apple Silicon)
|
|
|
|
```bash
|
|
pip install mlx-lm
|
|
mlx_lm.generate --model ajayk007/Qwen2.5-Coder-1.5B-Shellsmith \
|
|
--prompt "compress the logs folder into logs.tar.gz"
|
|
```
|
|
|
|
```python
|
|
from mlx_lm import load, generate
|
|
model, tok = load("ajayk007/Qwen2.5-Coder-1.5B-Shellsmith")
|
|
messages = [
|
|
{"role": "system", "content": "You are a shell command generator for macOS/Linux. "
|
|
"Given a task in plain English, reply with a single safe shell command. "
|
|
"Output only the command on one line, no explanation, no markdown."},
|
|
{"role": "user", "content": "find all python files modified today"},
|
|
]
|
|
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
|
|
print(generate(model, tok, prompt=prompt, max_tokens=64))
|
|
```
|
|
|
|
### GGUF (llama.cpp / Ollama / LM Studio)
|
|
|
|
A `shellsmith-1.5b-f16.gguf` file is included in this repo for use with llama.cpp-based runtimes.
|
|
|
|
## Prompt format
|
|
|
|
Uses the Qwen chat template with the system prompt shown above. Keep the system
|
|
prompt for best results.
|
|
|
|
## Training
|
|
|
|
- **Method:** LoRA (rank 16), 16 layers, 400 iterations, batch size 4, lr 1e-4
|
|
- **Hardware:** Apple M5 Pro (48 GB), MLX
|
|
- **Data:** [`ajayk007/shellsmith-commands`](https://huggingface.co/datasets/ajayk007/shellsmith-commands) —
|
|
curated (instruction, command) pairs with paraphrase augmentation, 80/10/10 split.
|
|
|
|
## Limitations & safety
|
|
|
|
- Generates commands across common categories (files, find/grep, archives, git,
|
|
processes, networking). Outside this scope it falls back to base behavior.
|
|
- **Always read a generated command before running it.** It can produce
|
|
destructive commands (`rm`, `kill`, `chmod`) if you ask for them. There is no
|
|
sandbox or confirmation step.
|
|
- Single-command only; it does not write multi-step scripts.
|
|
|
|
## Related
|
|
Part of a series of focused "English → developer DSL" fine-tunes:
|
|
- [Qwen2.5-Coder-7B-Querysmith](https://huggingface.co/ajayk007/Qwen2.5-Coder-7B-Querysmith) — schema-grounded text-to-SQL.
|
|
|
|
## License
|
|
|
|
Apache-2.0, inheriting from the base model.
|