nl2jq-qwen3-0.6b/README.md

license, language, pipeline_tag, base_model, tags, datasets, metrics

license	language	pipeline_tag	base_model	tags	datasets	metrics
apache-2.0	en	text-generation	Qwen/Qwen3-0.6B	jq code-generation json lora	gauthierpiarrette/nl2jq	execution-accuracy

nl2jq-qwen3-0.6b

Part of the nl2jq project — all artifacts · code + CLI · live demo · benchmark

A LoRA fine-tune of Qwen3-0.6B that translates a natural-language request plus a JSON sample into a jq program. It is the practical baseline companion to the from-scratch nl2jq-40m: larger, but its pretrained language priors make it markedly more reliable at copying novel field names from your JSON — which is what most real CLI use needs.

The LoRA adapter has been merged into the base weights, so it loads as an ordinary model.

This is the recommended default backend of the jqgen CLI — the best latency/accuracy balance for local use (~1–3s per query on laptop CPU).

Usage

cat data.json | jqgen --backend qwen "total spend per customer, paid orders only"

Or directly:

import re
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gauthierpiarrette/nl2jq-qwen3-0.6b")
model = AutoModelForCausalLM.from_pretrained("gauthierpiarrette/nl2jq-qwen3-0.6b")
msgs = [{"role": "system", "content": "You translate a natural-language request plus a "
         "sample of JSON into a single jq program. Output only the jq program, nothing else."},
        {"role": "user", "content": "Request: highest paid person\nJSON sample: [...]"}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
ids = enc["input_ids"]  # newer transformers return a BatchEncoding here, not a tensor
out = tok.decode(model.generate(ids, max_new_tokens=128)[0][ids.shape[1]:],
                 skip_special_tokens=True)
# Qwen3 emits an (empty) reasoning block before the program — strip it:
print(re.sub(r"(?s)^\s*<think>.*?</think>\s*", "", out).strip())

Results

Execution accuracy on the frozen 400-item nl2jq-bench v1.0.0 (0% field overlap with training, novel domains, one-shot evaluation). These weights are the v6 fine-tune — trained on data with per-example-unique field names and coverage of reduce/foreach/walk/paths/if constructs; the superseded v5 row is kept for the record:

Model	frozen pass@1	valid	T1	T2	T3	T4	T5-gen
this model (v6)	0.40	0.73	0.65	0.52	0.36	0.28	0.20
v5 fine-tune (superseded)	0.24	0.67	0.45	0.38	0.21	0.12	0.02
Qwen3-0.6B base (zero-shot)	0.01	0.06	0.02	0.00	0.00	0.00	0.02
nl2jq-qwen3.5-2b (bigger sibling, CLI config)	0.48	0.82	0.75	0.61	0.47	0.35	0.18
Claude Opus 4.8 (zero-shot, context row)	0.96	0.98	1.00	0.96	0.96	0.96	0.90

Three honest readings:

• Fine-tuning does all the work at this scale: the base model zero-shot scores 0.01 (it emits pseudo-code, not jq); this LoRA takes it to 0.40 on held-out fields.
• The v6 data fix mattered more than anything else: same base, same LoRA recipe, +67% pass@1 and 10× T5 purely from making field-copying the only winning strategy in training.
• This is the fast backend, not the most accurate one: it answers in ~1–3s on laptop CPU; the 2B sibling is +8 points at ~3× the latency. Both remain far from the frontier's 0.96 — that gap is disclosed, not hidden.

Training

• Base: Qwen3-0.6B. Method: LoRA (r=32, α=64, all-linear targets), 1 epoch over 150k execution-verified examples, cosine schedule, bf16, gradient checkpointing.
• Data: the v6 generation of the synthetic pipeline — per-example-unique field names (so copying from the input, not vocabulary recall, is the only winning strategy) plus generators for reduce/foreach/walk/paths/if-banding constructs. 100% synthetic and execution-verified — see the nl2jq dataset. No web text, no scraped code. Every program was run under jq 1.7.1 and kept only if it produced non-degenerate output.
• Frontier-model outputs are never used as training targets; all natural-language phrasings are template/grammar-generated by the author.

Limitations

• jq 1.7.1 only; no streaming, modules, or user-defined functions.
• Ambiguous requests may yield a valid program that answers a different reading.
• Inspect before trusting the output — the jqgen CLI prints the program to stderr. jq runs locally with no network access, but an incorrect program can still mislead.
• Not a general assistant — it emits jq and nothing else.

License

Apache-2.0 (inherits the base model's license). Fine-tuned on CC BY 4.0 synthetic data.

Code, benchmark tooling, and training pipeline: github.com/gauthierpiarrette/nl2jq