117 lines
5.0 KiB
Markdown
117 lines
5.0 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: Qwen/Qwen3-1.7B
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
tags:
|
||
- json
|
||
- json-schema
|
||
- structured-output
|
||
- qlora
|
||
- function-calling
|
||
datasets:
|
||
- ujwal00/jsonschema-sft
|
||
language:
|
||
- en
|
||
---
|
||
|
||
# Qwen3-1.7B-JSON — native JSON-Schema compliance fine-tune
|
||
|
||
Fine-tune of **Qwen/Qwen3-1.7B** that emits JSON conforming to a requested
|
||
**JSON Schema** *natively* — no grammar-constrained decoding — with a large
|
||
gain on the hard case: **nested / `$ref` / `$defs` schemas**.
|
||
|
||
- **Base model:** Qwen/Qwen3-1.7B (Apache-2.0)
|
||
- **Method:** QLoRA SFT (Unsloth), r=16, α=32, 2 epochs, NEFTune α=5
|
||
- **Data:** ~4k schema→valid-instance pairs generated from JSONSchemaBench
|
||
*train-split* schemas (72% nested), every target validated with a
|
||
Draft-2020-12 validator; val/test schemas hash-excluded (decontaminated)
|
||
- **Eval:** native (unconstrained, greedy) schema-compliance, scored with the
|
||
Draft-2020-12 validator + full format checker
|
||
|
||
## Results (held-out, measured identically base vs fine-tuned)
|
||
|
||
Measured on 150–200 held-out schemas the model never trained on. Fine-tuned
|
||
result independently cross-checked via the merged fp16 model (74% nested).
|
||
|
||
| Metric | Base Qwen3-1.7B | **Fine-tuned** | Δ |
|
||
|---|---|---|---|
|
||
| Nested / `$ref` schemas | 58% | **72–74%** | **+14–16** |
|
||
| Flat schemas | 87% | **92%** | +5 |
|
||
| Overall | 72.5% | **82–83%** | +10 |
|
||
| Parse-fail rate | 4% | **2.5–4%** | ~flat |
|
||
|
||
## Official results — JSONSchemaBench test split (n=360, native/unconstrained)
|
||
|
||
Base Qwen3-1.7B vs fine-tuned, identical prompt + Draft-2020-12 validator, greedy.
|
||
|
||
| Dataset | Base | Fine-tuned | Gain |
|
||
|---|---|---|---|
|
||
| Github_trivial | 49% | **82%** | +33% |
|
||
| Github_easy | 78% | **93%** | +16% |
|
||
| Github_medium | 42% | **56%** | +13% |
|
||
| Github_hard | 11% | **24%** | +13% |
|
||
| Glaiveai2K | 98% | **96%** | -2% |
|
||
| Kubernetes | 53% | **67%** | +13% |
|
||
| Snowplow | 33% | **73%** | +40% |
|
||
| JsonSchemaStore | 31% | **42%** | +11% |
|
||
| **Overall** | 49% | **67%** | +17% |
|
||
|
||
**McNemar paired test:** 76 schemas fixed, 14 broke, p=1.78e-11 (highly significant).
|
||
|
||
Our base measurement aligns with the published Llama-3.2-1B native numbers
|
||
(confirming comparability); the fine-tune beats published small-model native
|
||
compliance on nearly every dataset.
|
||
|
||
## Why this matters
|
||
|
||
Small models are widely deployed for narrow, high-volume, structured tasks —
|
||
but their published *native* schema-compliance collapses on complex schemas
|
||
(JSONSchemaBench reports ~13–38% for small models on the hardest datasets).
|
||
This fine-tune lifts native compliance on exactly that hard regime, so the
|
||
model is more usable **without** attaching a constrained-decoding engine
|
||
(and better as a base *with* one).
|
||
|
||
## How to use
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import json, torch
|
||
|
||
m = AutoModelForCausalLM.from_pretrained("ujwal00/qwen3-1.7b-json", dtype=torch.float16, device_map="auto")
|
||
tok = AutoTokenizer.from_pretrained("ujwal00/qwen3-1.7b-json")
|
||
|
||
SYSTEM = ("You are a precise JSON generator. Given a JSON Schema, output ONE JSON value "
|
||
"that strictly validates against it. Output ONLY the JSON value.")
|
||
schema = {"type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}},"required":["name","age"]}
|
||
msgs = [{"role":"system","content":SYSTEM},
|
||
{"role":"user","content":f"Generate a JSON value that validates against this JSON Schema:\n\n{json.dumps(schema)}\n\nReturn only the JSON value."}]
|
||
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to(m.device)
|
||
out = m.generate(ids, max_new_tokens=512, do_sample=False)
|
||
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
**Low-VRAM machines** (e.g. a 6 GB laptop GPU): add `low_cpu_mem_usage=True` to
|
||
`from_pretrained` (and optionally `max_memory={0: "5GiB", "cpu": "6GiB"}`) so
|
||
loading doesn't exhaust memory. On a normal 8 GB+ GPU the snippet above works as-is.
|
||
|
||
### Verified live output
|
||
|
||
| Schema | Model output | Valid? |
|
||
|---|---|---|
|
||
| `{name:str, age:int}` (required both) | `{"name": "sample_value", "age": 50}` | ✅ |
|
||
| nested: user{id,email:email-format, roles:enum[]} | `{"user": {"id": 50, "email": "user@example.com", "roles": ["user"]}}` | ✅ |
|
||
|
||
## Limitations (stated honestly)
|
||
|
||
- Numbers are **native / unconstrained**; constrained decoding scores differently and is out of scope.
|
||
- Trained on JSONSchemaBench-style schemas; may generalize less to very different schema dialects.
|
||
- Ultra-large schemas (>4000 chars) were excluded from training data.
|
||
- Reported gains are on held-out gold; the official test-split table is the definitive comparison.
|
||
|
||
## Reproducibility
|
||
|
||
Data recipe, generator, validator, training + eval code, and manifest are in the
|
||
project repo (`jsonc/`, `kaggle/`). Seed=42; prompt hashed; validator format
|
||
self-tested. Base and fine-tuned measured with identical prompt/validator/greedy.
|