113 lines
5.4 KiB
Markdown
113 lines
5.4 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
pipeline_tag: text-generation
|
||
base_model: Qwen/Qwen3-0.6B
|
||
tags:
|
||
- jq
|
||
- code-generation
|
||
- json
|
||
- lora
|
||
datasets:
|
||
- gauthierpiarrette/nl2jq
|
||
metrics:
|
||
- execution-accuracy
|
||
---
|
||
|
||
# nl2jq-qwen3-0.6b
|
||
|
||
*Part of the **nl2jq** project — [all artifacts](https://huggingface.co/collections/gauthierpiarrette/nl2jq-natural-language-to-jq-locally-6a5299203df3832415275223) · [code + CLI](https://github.com/gauthierpiarrette/nl2jq) · [live demo](https://huggingface.co/spaces/gauthierpiarrette/nl2jq) · [benchmark](https://huggingface.co/datasets/gauthierpiarrette/nl2jq-bench)*
|
||
|
||
A LoRA fine-tune of [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) that translates a
|
||
natural-language request plus a JSON sample into a [jq](https://jqlang.github.io/jq/)
|
||
program. It is the **practical baseline** companion to the from-scratch
|
||
[`nl2jq-40m`](https://huggingface.co/gauthierpiarrette/nl2jq-40m): larger, but its
|
||
pretrained language priors make it markedly more reliable at **copying novel field names**
|
||
from your JSON — which is what most real CLI use needs.
|
||
|
||
The LoRA adapter has been merged into the base weights, so it loads as an ordinary model.
|
||
|
||
**This is the recommended default backend of the `jqgen` CLI** — the best
|
||
latency/accuracy balance for local use (~1–3s per query on laptop CPU).
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
cat data.json | jqgen --backend qwen "total spend per customer, paid orders only"
|
||
```
|
||
|
||
Or directly:
|
||
```python
|
||
import re
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
tok = AutoTokenizer.from_pretrained("gauthierpiarrette/nl2jq-qwen3-0.6b")
|
||
model = AutoModelForCausalLM.from_pretrained("gauthierpiarrette/nl2jq-qwen3-0.6b")
|
||
msgs = [{"role": "system", "content": "You translate a natural-language request plus a "
|
||
"sample of JSON into a single jq program. Output only the jq program, nothing else."},
|
||
{"role": "user", "content": "Request: highest paid person\nJSON sample: [...]"}]
|
||
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
|
||
ids = enc["input_ids"] # newer transformers return a BatchEncoding here, not a tensor
|
||
out = tok.decode(model.generate(ids, max_new_tokens=128)[0][ids.shape[1]:],
|
||
skip_special_tokens=True)
|
||
# Qwen3 emits an (empty) reasoning block before the program — strip it:
|
||
print(re.sub(r"(?s)^\s*<think>.*?</think>\s*", "", out).strip())
|
||
```
|
||
|
||
## Results
|
||
|
||
Execution accuracy on the frozen 400-item
|
||
[`nl2jq-bench v1.0.0`](https://huggingface.co/datasets/gauthierpiarrette/nl2jq-bench)
|
||
(0% field overlap with training, novel domains, one-shot evaluation). **These weights are
|
||
the v6 fine-tune** — trained on data with per-example-unique field names and coverage of
|
||
`reduce`/`foreach`/`walk`/`paths`/`if` constructs; the superseded v5 row is kept for the
|
||
record:
|
||
|
||
| Model | frozen pass@1 | valid | T1 | T2 | T3 | T4 | T5-gen |
|
||
|---|---|---|---|---|---|---|---|
|
||
| **this model (v6)** | **0.40** | 0.73 | 0.65 | 0.52 | 0.36 | 0.28 | **0.20** |
|
||
| v5 fine-tune (superseded) | 0.24 | 0.67 | 0.45 | 0.38 | 0.21 | 0.12 | 0.02 |
|
||
| Qwen3-0.6B base (zero-shot) | 0.01 | 0.06 | 0.02 | 0.00 | 0.00 | 0.00 | 0.02 |
|
||
| [nl2jq-qwen3.5-2b](https://huggingface.co/gauthierpiarrette/nl2jq-qwen3.5-2b) (bigger sibling, CLI config) | 0.48 | 0.82 | 0.75 | 0.61 | 0.47 | 0.35 | 0.18 |
|
||
| Claude Opus 4.8 (zero-shot, context row) | 0.96 | 0.98 | 1.00 | 0.96 | 0.96 | 0.96 | 0.90 |
|
||
|
||
Three honest readings:
|
||
|
||
- **Fine-tuning does all the work at this scale**: the base model zero-shot scores 0.01
|
||
(it emits pseudo-code, not jq); this LoRA takes it to 0.40 on held-out fields.
|
||
- **The v6 data fix mattered more than anything else**: same base, same LoRA recipe,
|
||
+67% pass@1 and 10× T5 purely from making field-copying the only winning strategy in
|
||
training.
|
||
- **This is the fast backend, not the most accurate one**: it answers in ~1–3s on laptop
|
||
CPU; the 2B sibling is +8 points at ~3× the latency. Both remain far from the frontier's
|
||
0.96 — that gap is disclosed, not hidden.
|
||
|
||
## Training
|
||
|
||
- **Base**: Qwen3-0.6B. **Method**: LoRA (r=32, α=64, all-linear targets), 1 epoch over
|
||
150k execution-verified examples, cosine schedule, bf16, gradient checkpointing.
|
||
- **Data**: the v6 generation of the synthetic pipeline — per-example-unique field names
|
||
(so copying from the input, not vocabulary recall, is the only winning strategy) plus
|
||
generators for `reduce`/`foreach`/`walk`/`paths`/`if`-banding constructs. 100% synthetic
|
||
and execution-verified — see the
|
||
[nl2jq dataset](https://huggingface.co/datasets/gauthierpiarrette/nl2jq). No web text, no
|
||
scraped code. Every program was run under jq 1.7.1 and kept only if it produced
|
||
non-degenerate output.
|
||
- Frontier-model outputs are **never** used as training targets; all natural-language
|
||
phrasings are template/grammar-generated by the author.
|
||
|
||
## Limitations
|
||
|
||
- jq 1.7.1 only; no streaming, modules, or user-defined functions.
|
||
- Ambiguous requests may yield a valid program that answers a different reading.
|
||
- **Inspect before trusting the output** — the `jqgen` CLI prints the program to stderr.
|
||
jq runs locally with no network access, but an incorrect program can still mislead.
|
||
- Not a general assistant — it emits jq and nothing else.
|
||
|
||
## License
|
||
|
||
Apache-2.0 (inherits the base model's license). Fine-tuned on CC BY 4.0 synthetic data.
|
||
|
||
Code, benchmark tooling, and training pipeline: [github.com/gauthierpiarrette/nl2jq](https://github.com/gauthierpiarrette/nl2jq)
|