117 lines
5.0 KiB
Markdown
117 lines
5.0 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model: Qwen/Qwen3-1.7B
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
library_name: transformers
|
|||
|
|
tags:
|
|||
|
|
- json
|
|||
|
|
- json-schema
|
|||
|
|
- structured-output
|
|||
|
|
- qlora
|
|||
|
|
- function-calling
|
|||
|
|
datasets:
|
|||
|
|
- ujwal00/jsonschema-sft
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Qwen3-1.7B-JSON — native JSON-Schema compliance fine-tune
|
|||
|
|
|
|||
|
|
Fine-tune of **Qwen/Qwen3-1.7B** that emits JSON conforming to a requested
|
|||
|
|
**JSON Schema** *natively* — no grammar-constrained decoding — with a large
|
|||
|
|
gain on the hard case: **nested / `$ref` / `$defs` schemas**.
|
|||
|
|
|
|||
|
|
- **Base model:** Qwen/Qwen3-1.7B (Apache-2.0)
|
|||
|
|
- **Method:** QLoRA SFT (Unsloth), r=16, α=32, 2 epochs, NEFTune α=5
|
|||
|
|
- **Data:** ~4k schema→valid-instance pairs generated from JSONSchemaBench
|
|||
|
|
*train-split* schemas (72% nested), every target validated with a
|
|||
|
|
Draft-2020-12 validator; val/test schemas hash-excluded (decontaminated)
|
|||
|
|
- **Eval:** native (unconstrained, greedy) schema-compliance, scored with the
|
|||
|
|
Draft-2020-12 validator + full format checker
|
|||
|
|
|
|||
|
|
## Results (held-out, measured identically base vs fine-tuned)
|
|||
|
|
|
|||
|
|
Measured on 150–200 held-out schemas the model never trained on. Fine-tuned
|
|||
|
|
result independently cross-checked via the merged fp16 model (74% nested).
|
|||
|
|
|
|||
|
|
| Metric | Base Qwen3-1.7B | **Fine-tuned** | Δ |
|
|||
|
|
|---|---|---|---|
|
|||
|
|
| Nested / `$ref` schemas | 58% | **72–74%** | **+14–16** |
|
|||
|
|
| Flat schemas | 87% | **92%** | +5 |
|
|||
|
|
| Overall | 72.5% | **82–83%** | +10 |
|
|||
|
|
| Parse-fail rate | 4% | **2.5–4%** | ~flat |
|
|||
|
|
|
|||
|
|
## Official results — JSONSchemaBench test split (n=360, native/unconstrained)
|
|||
|
|
|
|||
|
|
Base Qwen3-1.7B vs fine-tuned, identical prompt + Draft-2020-12 validator, greedy.
|
|||
|
|
|
|||
|
|
| Dataset | Base | Fine-tuned | Gain |
|
|||
|
|
|---|---|---|---|
|
|||
|
|
| Github_trivial | 49% | **82%** | +33% |
|
|||
|
|
| Github_easy | 78% | **93%** | +16% |
|
|||
|
|
| Github_medium | 42% | **56%** | +13% |
|
|||
|
|
| Github_hard | 11% | **24%** | +13% |
|
|||
|
|
| Glaiveai2K | 98% | **96%** | -2% |
|
|||
|
|
| Kubernetes | 53% | **67%** | +13% |
|
|||
|
|
| Snowplow | 33% | **73%** | +40% |
|
|||
|
|
| JsonSchemaStore | 31% | **42%** | +11% |
|
|||
|
|
| **Overall** | 49% | **67%** | +17% |
|
|||
|
|
|
|||
|
|
**McNemar paired test:** 76 schemas fixed, 14 broke, p=1.78e-11 (highly significant).
|
|||
|
|
|
|||
|
|
Our base measurement aligns with the published Llama-3.2-1B native numbers
|
|||
|
|
(confirming comparability); the fine-tune beats published small-model native
|
|||
|
|
compliance on nearly every dataset.
|
|||
|
|
|
|||
|
|
## Why this matters
|
|||
|
|
|
|||
|
|
Small models are widely deployed for narrow, high-volume, structured tasks —
|
|||
|
|
but their published *native* schema-compliance collapses on complex schemas
|
|||
|
|
(JSONSchemaBench reports ~13–38% for small models on the hardest datasets).
|
|||
|
|
This fine-tune lifts native compliance on exactly that hard regime, so the
|
|||
|
|
model is more usable **without** attaching a constrained-decoding engine
|
|||
|
|
(and better as a base *with* one).
|
|||
|
|
|
|||
|
|
## How to use
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
import json, torch
|
|||
|
|
|
|||
|
|
m = AutoModelForCausalLM.from_pretrained("ujwal00/qwen3-1.7b-json", dtype=torch.float16, device_map="auto")
|
|||
|
|
tok = AutoTokenizer.from_pretrained("ujwal00/qwen3-1.7b-json")
|
|||
|
|
|
|||
|
|
SYSTEM = ("You are a precise JSON generator. Given a JSON Schema, output ONE JSON value "
|
|||
|
|
"that strictly validates against it. Output ONLY the JSON value.")
|
|||
|
|
schema = {"type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}},"required":["name","age"]}
|
|||
|
|
msgs = [{"role":"system","content":SYSTEM},
|
|||
|
|
{"role":"user","content":f"Generate a JSON value that validates against this JSON Schema:\n\n{json.dumps(schema)}\n\nReturn only the JSON value."}]
|
|||
|
|
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to(m.device)
|
|||
|
|
out = m.generate(ids, max_new_tokens=512, do_sample=False)
|
|||
|
|
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Low-VRAM machines** (e.g. a 6 GB laptop GPU): add `low_cpu_mem_usage=True` to
|
|||
|
|
`from_pretrained` (and optionally `max_memory={0: "5GiB", "cpu": "6GiB"}`) so
|
|||
|
|
loading doesn't exhaust memory. On a normal 8 GB+ GPU the snippet above works as-is.
|
|||
|
|
|
|||
|
|
### Verified live output
|
|||
|
|
|
|||
|
|
| Schema | Model output | Valid? |
|
|||
|
|
|---|---|---|
|
|||
|
|
| `{name:str, age:int}` (required both) | `{"name": "sample_value", "age": 50}` | ✅ |
|
|||
|
|
| nested: user{id,email:email-format, roles:enum[]} | `{"user": {"id": 50, "email": "user@example.com", "roles": ["user"]}}` | ✅ |
|
|||
|
|
|
|||
|
|
## Limitations (stated honestly)
|
|||
|
|
|
|||
|
|
- Numbers are **native / unconstrained**; constrained decoding scores differently and is out of scope.
|
|||
|
|
- Trained on JSONSchemaBench-style schemas; may generalize less to very different schema dialects.
|
|||
|
|
- Ultra-large schemas (>4000 chars) were excluded from training data.
|
|||
|
|
- Reported gains are on held-out gold; the official test-split table is the definitive comparison.
|
|||
|
|
|
|||
|
|
## Reproducibility
|
|||
|
|
|
|||
|
|
Data recipe, generator, validator, training + eval code, and manifest are in the
|
|||
|
|
project repo (`jsonc/`, `kaggle/`). Seed=42; prompt hashed; validator format
|
|||
|
|
self-tested. Base and fine-tuned measured with identical prompt/validator/greedy.
|