qwen3-1.7b-json/README.md

license, base_model, pipeline_tag, library_name, tags, datasets, language

license	base_model	pipeline_tag	library_name	tags	datasets	language
apache-2.0	Qwen/Qwen3-1.7B	text-generation	transformers	json json-schema structured-output qlora function-calling	ujwal00/jsonschema-sft	en

Qwen3-1.7B-JSON — native JSON-Schema compliance fine-tune

Fine-tune of Qwen/Qwen3-1.7B that emits JSON conforming to a requested JSON Schema natively — no grammar-constrained decoding — with a large gain on the hard case: nested / $ref / $defs schemas.

• Base model: Qwen/Qwen3-1.7B (Apache-2.0)
• Method: QLoRA SFT (Unsloth), r=16, α=32, 2 epochs, NEFTune α=5
• Data: ~4k schema→valid-instance pairs generated from JSONSchemaBench train-split schemas (72% nested), every target validated with a Draft-2020-12 validator; val/test schemas hash-excluded (decontaminated)
• Eval: native (unconstrained, greedy) schema-compliance, scored with the Draft-2020-12 validator + full format checker

Results (held-out, measured identically base vs fine-tuned)

Measured on 150–200 held-out schemas the model never trained on. Fine-tuned result independently cross-checked via the merged fp16 model (74% nested).

Metric	Base Qwen3-1.7B	Fine-tuned	Δ
Nested / $ref schemas	58%	72–74%	+14–16
Flat schemas	87%	92%	+5
Overall	72.5%	82–83%	+10
Parse-fail rate	4%	2.5–4%	~flat

Official results — JSONSchemaBench test split (n=360, native/unconstrained)

Base Qwen3-1.7B vs fine-tuned, identical prompt + Draft-2020-12 validator, greedy.

Dataset	Base	Fine-tuned	Gain
Github_trivial	49%	82%	+33%
Github_easy	78%	93%	+16%
Github_medium	42%	56%	+13%
Github_hard	11%	24%	+13%
Glaiveai2K	98%	96%	-2%
Kubernetes	53%	67%	+13%
Snowplow	33%	73%	+40%
JsonSchemaStore	31%	42%	+11%
Overall	49%	67%	+17%

McNemar paired test: 76 schemas fixed, 14 broke, p=1.78e-11 (highly significant).

Our base measurement aligns with the published Llama-3.2-1B native numbers (confirming comparability); the fine-tune beats published small-model native compliance on nearly every dataset.

Why this matters

Small models are widely deployed for narrow, high-volume, structured tasks — but their published native schema-compliance collapses on complex schemas (JSONSchemaBench reports ~13–38% for small models on the hardest datasets). This fine-tune lifts native compliance on exactly that hard regime, so the model is more usable without attaching a constrained-decoding engine (and better as a base with one).

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
import json, torch

m = AutoModelForCausalLM.from_pretrained("ujwal00/qwen3-1.7b-json", dtype=torch.float16, device_map="auto")
tok = AutoTokenizer.from_pretrained("ujwal00/qwen3-1.7b-json")

SYSTEM = ("You are a precise JSON generator. Given a JSON Schema, output ONE JSON value "
          "that strictly validates against it. Output ONLY the JSON value.")
schema = {"type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}},"required":["name","age"]}
msgs = [{"role":"system","content":SYSTEM},
        {"role":"user","content":f"Generate a JSON value that validates against this JSON Schema:\n\n{json.dumps(schema)}\n\nReturn only the JSON value."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to(m.device)
out = m.generate(ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Low-VRAM machines (e.g. a 6 GB laptop GPU): add low_cpu_mem_usage=True to from_pretrained (and optionally max_memory={0: "5GiB", "cpu": "6GiB"}) so loading doesn't exhaust memory. On a normal 8 GB+ GPU the snippet above works as-is.

Verified live output

Schema	Model output	Valid?
{name:str, age:int} (required both)	{"name": "sample_value", "age": 50}	✅
nested: user{id,email:email-format, roles:enum[]}	{"user": {"id": 50, "email": "user@example.com", "roles": ["user"]}}	✅

Limitations (stated honestly)

• Numbers are native / unconstrained; constrained decoding scores differently and is out of scope.
• Trained on JSONSchemaBench-style schemas; may generalize less to very different schema dialects.
• Ultra-large schemas (>4000 chars) were excluded from training data.
• Reported gains are on held-out gold; the official test-split table is the definitive comparison.

Reproducibility

Data recipe, generator, validator, training + eval code, and manifest are in the project repo (jsonc/, kaggle/). Seed=42; prompt hashed; validator format self-tested. Base and fine-tuned measured with identical prompt/validator/greedy.