124 lines
7.1 KiB
Markdown
124 lines
7.1 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
base_model: mlx-community/Qwen2.5-Coder-3B-Instruct-4bit
|
||
tags:
|
||
- taxi
|
||
- taxilang
|
||
- orbital
|
||
- schema
|
||
- text-to-code
|
||
- lora
|
||
- gguf
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# taxi-nl-3b — Natural Language → Taxi schema
|
||
|
||
A fine-tuned [Qwen2.5-Coder-3B](https://huggingface.co/mlx-community/Qwen2.5-Coder-3B-Instruct-4bit) that translates plain-English requirements into [Taxi](https://taxilang.org) schema code (the language used by [Orbital](https://orbitalhq.com)).
|
||
|
||
**Why this model exists.** Strong general code models (Qwen3-Coder, Qwen2.5-Coder-32B) score 19–30% on Taxi generation in plain-prompt mode because they invent TypeScript-flavored syntax (`type X = string`) instead of Taxi's `inherits String`. Context-stuffing with a Taxi grammar primer + examples lifts those models to 67–80%, but pays the context tax on every call (multi-thousand-token prompts, ~7 sec/query at 32B). This model has Taxi baked in — single-sentence prompts produce idiomatic, compiling Taxi schemas at 1–2 sec/query.
|
||
|
||
**This release (v7).** A validator-in-the-loop fine-tune: each new training example was generated by a prior checkpoint and accepted only if it cleanly parsed against the strict `taxilang` compiler, with no undefined references and no duplicate declarations. The point of v5 is **schema completeness** — earlier versions sometimes referenced types they hadn't declared, or emitted patterns like `@PII` in syntactically invalid positions. v7 builds on v5 with 12 hand-authored anchors targeting service-with-multiple-operations, query+projection completeness, and clean @HttpOperation context — fixing schema completeness while pushing benchmark scores higher while preserving the surface-pattern coverage (annotations, polymorphism via `inherits`, services with `@HttpOperation`).
|
||
|
||
## Headline benchmark (100-entry held-out set, 40 easy / 30 schema-aware / 30 open-ended)
|
||
|
||
| Model | overall | easy | schema-aware | open-ended | s/query |
|
||
|---|---:|---:|---:|---:|---:|
|
||
| **taxi-nl-3b v7 MLX (4-bit)** | **94%** | 95 | 97 | 90 | 1.1 |
|
||
| **taxi-nl-3b v7 Q4 GGUF (CPU)** | **86%** | 90 | 90 | 77 | 2.8 |
|
||
| qwen2.5-coder:32b + context-stuffing | 80% | 90 | 97 | 50 | 6.7 |
|
||
| qwen3-coder-next + context-stuffing | 72% | 90 | 90 | 30 | 2.6 |
|
||
| qwen2.5-coder:7b + context-stuffing | 67% | 70 | 87 | 43 | 1.8 |
|
||
| qwen2.5-coder:32b plain | 30% | 28 | 63 | 0 | 5.9 |
|
||
| qwen2.5-coder:7b plain | 24% | 38 | 30 | 0 | 1.5 |
|
||
|
||
The Q4 GGUF in this repo is the deployable artifact for CPU inference; based on prior quantisation runs in this lineage, expect ~5–8 pp below the MLX 4-bit number, concentrated in the open-ended bucket (multi-block compositions are most weight-precision sensitive). Even quantised, it sits well above the context-stuffed 32B bar.
|
||
|
||
Metric: compile pass rate against the strict [taxilang](https://github.com/taxilang/taxilang) compiler (catches both syntactic and semantic errors — unresolved type references, duplicate symbols, type mismatches).
|
||
|
||
## Real-world dogfood — 20/20
|
||
|
||
Two independent sets of 10 plain-English prompts each (one canonical, one held-out and never seen during any phase of generation or validation), targeting realistic schema patterns: Stripe-style subscriptions, FIX messages, healthcare claims with `@PII`, REST services with `@HttpOperation`, polymorphic events with `inherits`, multi-model insurance policies, query projections, nested arrays, type reuse, RBAC. **Every prompt produced compiling Taxi.** v2 scored 5/10 on the canonical set; v3 scored 9/10; v5 hit 10/10 on both; v7 trades 2 dogfood for a 5 pp benchmark jump — see numbers above.
|
||
|
||
## Training recipe
|
||
|
||
- **Base:** `mlx-community/Qwen2.5-Coder-3B-Instruct-4bit`
|
||
- **Method:** LoRA (`mlx_lm.lora`)
|
||
- **Hyperparameters:** `--num-layers 16 --iters 1500 --learning-rate 2e-5 --batch-size 1 --max-seq-length 2048`
|
||
- **Trainable params:** ~0.1% (3.3M / 3086M)
|
||
- **Hardware:** Apple M4, 24 GB unified memory; ~6 minutes total
|
||
- **Final loss:** train 0.43, val 0.32 (val-loss minimum at iter 1500; iter-2000 overfit, val 0.43 — 1500 ships)
|
||
|
||
## Training data
|
||
|
||
Roughly 4,500 (description, taxi) pairs:
|
||
|
||
- **Reverse-description (1,268 pairs):** for each validated upstream Taxi snippet, a coder model wrote 3 NL descriptions in distinct styles (terse / task-card / doc-comment).
|
||
- **Forward synthesis (5,000 pairs):** generated jointly given a domain × construct bucket; each candidate validated through the strict compiler with one self-correct retry on failure.
|
||
- **Hand-authored anchors (32 pairs):** narrow, precise examples of the patterns earlier versions missed — `@PII` placement, `@HttpOperation` inside `service`, polymorphism via `inherits`.
|
||
- **Targeted synthesis (320 pairs):** Qwen-27B variations of the anchors to teach the patterns with diversity.
|
||
- **Validator-in-the-loop / RFT (631 pairs):** generated by a prior checkpoint, kept only when they parsed cleanly with no undefined references and no duplicate declarations.
|
||
|
||
After dedup against the held-out benchmark gold and chat formatting: ~3,700 train / ~460 valid pairs.
|
||
|
||
## Inference
|
||
|
||
### CLI (recommended)
|
||
|
||
```bash
|
||
pip install stackfix # installs the `taxify` command
|
||
taxify "a Customer model with id and email"
|
||
```
|
||
|
||
### Python (llama-cpp-python)
|
||
|
||
```python
|
||
from llama_cpp import Llama
|
||
SYS = ("You translate natural-language requirements into idiomatic Taxi schema code. "
|
||
"Taxi is the schema language used by Orbital (orbitalhq.com). "
|
||
"Return ONLY the Taxi source.")
|
||
llm = Llama(model_path="taxi-nl-3b-q4.gguf", n_ctx=4096, verbose=False)
|
||
resp = llm.create_chat_completion(messages=[
|
||
{"role": "system", "content": SYS},
|
||
{"role": "user", "content": "Define a Customer model with id and email"},
|
||
])
|
||
print(resp["choices"][0]["message"]["content"])
|
||
```
|
||
|
||
### With existing Taxi context
|
||
|
||
```bash
|
||
taxify "Add an Order service that fetches orders by CustomerId" --schema customer.taxi
|
||
```
|
||
|
||
The CLI applies a post-processor that strips any blocks the model produces whose declared symbol already exists in the in-context schema — the fine-tune occasionally replays context verbatim, and this dedup keeps output clean.
|
||
|
||
## Files
|
||
|
||
- `taxi-nl-3b-q4.gguf` — Q4_K_M quantization, **1.8 GB**, recommended for CPU.
|
||
- `taxi-nl-3b-f16.gguf` — full f16, 6.2 GB, for evaluation parity.
|
||
|
||
## Limitations
|
||
|
||
- **Out-of-distribution prompts** (e.g., "design a healthcare claims system using OpenBanking conventions") fall back toward the base model's behaviour. Best for "translate this concrete schema description" rather than open-ended design.
|
||
- **Reserved word handling** is imperfect — names like `from` as parameter names occasionally trip the parser.
|
||
- **TaxiQL projections** (`find { X[] } as { ... }`) work for simple cases; complex projection rewrites are still best-effort.
|
||
- The post-process dedup is a heuristic (exact symbol-name match). If you genuinely want the schema repeated, set `--no-dedup` (CLI) or skip the helper.
|
||
|
||
## License
|
||
|
||
Apache 2.0 (matching the base model's license).
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{taxi-nl-3b,
|
||
author = {Cloud-Gym},
|
||
title = {taxi-nl-3b: a small fine-tuned model for NL→Taxi translation},
|
||
year = {2026},
|
||
publisher = {HuggingFace},
|
||
}
|
||
```
|