初始化项目,由ModelHub XC社区提供模型
Model: gauthierpiarrette/nl2jq-qwen3-0.6b Source: Original Platform
This commit is contained in:
112
README.md
Normal file
112
README.md
Normal file
@@ -0,0 +1,112 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
pipeline_tag: text-generation
|
||||
base_model: Qwen/Qwen3-0.6B
|
||||
tags:
|
||||
- jq
|
||||
- code-generation
|
||||
- json
|
||||
- lora
|
||||
datasets:
|
||||
- gauthierpiarrette/nl2jq
|
||||
metrics:
|
||||
- execution-accuracy
|
||||
---
|
||||
|
||||
# nl2jq-qwen3-0.6b
|
||||
|
||||
*Part of the **nl2jq** project — [all artifacts](https://huggingface.co/collections/gauthierpiarrette/nl2jq-natural-language-to-jq-locally-6a5299203df3832415275223) · [code + CLI](https://github.com/gauthierpiarrette/nl2jq) · [live demo](https://huggingface.co/spaces/gauthierpiarrette/nl2jq) · [benchmark](https://huggingface.co/datasets/gauthierpiarrette/nl2jq-bench)*
|
||||
|
||||
A LoRA fine-tune of [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) that translates a
|
||||
natural-language request plus a JSON sample into a [jq](https://jqlang.github.io/jq/)
|
||||
program. It is the **practical baseline** companion to the from-scratch
|
||||
[`nl2jq-40m`](https://huggingface.co/gauthierpiarrette/nl2jq-40m): larger, but its
|
||||
pretrained language priors make it markedly more reliable at **copying novel field names**
|
||||
from your JSON — which is what most real CLI use needs.
|
||||
|
||||
The LoRA adapter has been merged into the base weights, so it loads as an ordinary model.
|
||||
|
||||
**This is the recommended default backend of the `jqgen` CLI** — the best
|
||||
latency/accuracy balance for local use (~1–3s per query on laptop CPU).
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
cat data.json | jqgen --backend qwen "total spend per customer, paid orders only"
|
||||
```
|
||||
|
||||
Or directly:
|
||||
```python
|
||||
import re
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
tok = AutoTokenizer.from_pretrained("gauthierpiarrette/nl2jq-qwen3-0.6b")
|
||||
model = AutoModelForCausalLM.from_pretrained("gauthierpiarrette/nl2jq-qwen3-0.6b")
|
||||
msgs = [{"role": "system", "content": "You translate a natural-language request plus a "
|
||||
"sample of JSON into a single jq program. Output only the jq program, nothing else."},
|
||||
{"role": "user", "content": "Request: highest paid person\nJSON sample: [...]"}]
|
||||
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
|
||||
ids = enc["input_ids"] # newer transformers return a BatchEncoding here, not a tensor
|
||||
out = tok.decode(model.generate(ids, max_new_tokens=128)[0][ids.shape[1]:],
|
||||
skip_special_tokens=True)
|
||||
# Qwen3 emits an (empty) reasoning block before the program — strip it:
|
||||
print(re.sub(r"(?s)^\s*<think>.*?</think>\s*", "", out).strip())
|
||||
```
|
||||
|
||||
## Results
|
||||
|
||||
Execution accuracy on the frozen 400-item
|
||||
[`nl2jq-bench v1.0.0`](https://huggingface.co/datasets/gauthierpiarrette/nl2jq-bench)
|
||||
(0% field overlap with training, novel domains, one-shot evaluation). **These weights are
|
||||
the v6 fine-tune** — trained on data with per-example-unique field names and coverage of
|
||||
`reduce`/`foreach`/`walk`/`paths`/`if` constructs; the superseded v5 row is kept for the
|
||||
record:
|
||||
|
||||
| Model | frozen pass@1 | valid | T1 | T2 | T3 | T4 | T5-gen |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **this model (v6)** | **0.40** | 0.73 | 0.65 | 0.52 | 0.36 | 0.28 | **0.20** |
|
||||
| v5 fine-tune (superseded) | 0.24 | 0.67 | 0.45 | 0.38 | 0.21 | 0.12 | 0.02 |
|
||||
| Qwen3-0.6B base (zero-shot) | 0.01 | 0.06 | 0.02 | 0.00 | 0.00 | 0.00 | 0.02 |
|
||||
| [nl2jq-qwen3.5-2b](https://huggingface.co/gauthierpiarrette/nl2jq-qwen3.5-2b) (bigger sibling, CLI config) | 0.48 | 0.82 | 0.75 | 0.61 | 0.47 | 0.35 | 0.18 |
|
||||
| Claude Opus 4.8 (zero-shot, context row) | 0.96 | 0.98 | 1.00 | 0.96 | 0.96 | 0.96 | 0.90 |
|
||||
|
||||
Three honest readings:
|
||||
|
||||
- **Fine-tuning does all the work at this scale**: the base model zero-shot scores 0.01
|
||||
(it emits pseudo-code, not jq); this LoRA takes it to 0.40 on held-out fields.
|
||||
- **The v6 data fix mattered more than anything else**: same base, same LoRA recipe,
|
||||
+67% pass@1 and 10× T5 purely from making field-copying the only winning strategy in
|
||||
training.
|
||||
- **This is the fast backend, not the most accurate one**: it answers in ~1–3s on laptop
|
||||
CPU; the 2B sibling is +8 points at ~3× the latency. Both remain far from the frontier's
|
||||
0.96 — that gap is disclosed, not hidden.
|
||||
|
||||
## Training
|
||||
|
||||
- **Base**: Qwen3-0.6B. **Method**: LoRA (r=32, α=64, all-linear targets), 1 epoch over
|
||||
150k execution-verified examples, cosine schedule, bf16, gradient checkpointing.
|
||||
- **Data**: the v6 generation of the synthetic pipeline — per-example-unique field names
|
||||
(so copying from the input, not vocabulary recall, is the only winning strategy) plus
|
||||
generators for `reduce`/`foreach`/`walk`/`paths`/`if`-banding constructs. 100% synthetic
|
||||
and execution-verified — see the
|
||||
[nl2jq dataset](https://huggingface.co/datasets/gauthierpiarrette/nl2jq). No web text, no
|
||||
scraped code. Every program was run under jq 1.7.1 and kept only if it produced
|
||||
non-degenerate output.
|
||||
- Frontier-model outputs are **never** used as training targets; all natural-language
|
||||
phrasings are template/grammar-generated by the author.
|
||||
|
||||
## Limitations
|
||||
|
||||
- jq 1.7.1 only; no streaming, modules, or user-defined functions.
|
||||
- Ambiguous requests may yield a valid program that answers a different reading.
|
||||
- **Inspect before trusting the output** — the `jqgen` CLI prints the program to stderr.
|
||||
jq runs locally with no network access, but an incorrect program can still mislead.
|
||||
- Not a general assistant — it emits jq and nothing else.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0 (inherits the base model's license). Fine-tuned on CC BY 4.0 synthetic data.
|
||||
|
||||
Code, benchmark tooling, and training pipeline: [github.com/gauthierpiarrette/nl2jq](https://github.com/gauthierpiarrette/nl2jq)
|
||||
Reference in New Issue
Block a user