taxi-nl-3b-gguf/README.md

---
license: apache-2.0
language:
  - en
base_model: mlx-community/Qwen2.5-Coder-3B-Instruct-4bit
tags:
  - taxi
  - taxilang
  - orbital
  - schema
  - text-to-code
  - lora
  - gguf
pipeline_tag: text-generation
---

# taxi-nl-3b — Natural Language → Taxi schema

A fine-tuned [Qwen2.5-Coder-3B](https://huggingface.co/mlx-community/Qwen2.5-Coder-3B-Instruct-4bit) that translates plain-English requirements into [Taxi](https://taxilang.org) schema code (the language used by [Orbital](https://orbitalhq.com)).

**Why this model exists.** Strong general code models (Qwen3-Coder, Qwen2.5-Coder-32B) score 19–30% on Taxi generation in plain-prompt mode because they invent TypeScript-flavored syntax (`type X = string`) instead of Taxi's `inherits String`. Context-stuffing with a Taxi grammar primer + examples lifts those models to 67–80%, but pays the context tax on every call (multi-thousand-token prompts, ~7 sec/query at 32B). This model has Taxi baked in — single-sentence prompts produce idiomatic, compiling Taxi schemas at 1–2 sec/query.

**This release (v7).** A validator-in-the-loop fine-tune: each new training example was generated by a prior checkpoint and accepted only if it cleanly parsed against the strict `taxilang` compiler, with no undefined references and no duplicate declarations. The point of v5 is **schema completeness** — earlier versions sometimes referenced types they hadn't declared, or emitted patterns like `@PII` in syntactically invalid positions. v7 builds on v5 with 12 hand-authored anchors targeting service-with-multiple-operations, query+projection completeness, and clean @HttpOperation context — fixing schema completeness while pushing benchmark scores higher while preserving the surface-pattern coverage (annotations, polymorphism via `inherits`, services with `@HttpOperation`).

## Headline benchmark (100-entry held-out set, 40 easy / 30 schema-aware / 30 open-ended)

| Model | overall | easy | schema-aware | open-ended | s/query |
|---|---:|---:|---:|---:|---:|
| **taxi-nl-3b v7 MLX (4-bit)** | **94%** | 95 | 97 | 90 | 1.1 |
| **taxi-nl-3b v7 Q4 GGUF (CPU)** | **86%** | 90 | 90 | 77 | 2.8 |
| qwen2.5-coder:32b + context-stuffing | 80% | 90 | 97 | 50 | 6.7 |
| qwen3-coder-next + context-stuffing  | 72% | 90 | 90 | 30 | 2.6 |
| qwen2.5-coder:7b + context-stuffing  | 67% | 70 | 87 | 43 | 1.8 |
| qwen2.5-coder:32b plain | 30% | 28 | 63 | 0 | 5.9 |
| qwen2.5-coder:7b plain  | 24% | 38 | 30 | 0 | 1.5 |

The Q4 GGUF in this repo is the deployable artifact for CPU inference; based on prior quantisation runs in this lineage, expect ~5–8 pp below the MLX 4-bit number, concentrated in the open-ended bucket (multi-block compositions are most weight-precision sensitive). Even quantised, it sits well above the context-stuffed 32B bar.

Metric: compile pass rate against the strict [taxilang](https://github.com/taxilang/taxilang) compiler (catches both syntactic and semantic errors — unresolved type references, duplicate symbols, type mismatches).

## Real-world dogfood — 20/20

Two independent sets of 10 plain-English prompts each (one canonical, one held-out and never seen during any phase of generation or validation), targeting realistic schema patterns: Stripe-style subscriptions, FIX messages, healthcare claims with `@PII`, REST services with `@HttpOperation`, polymorphic events with `inherits`, multi-model insurance policies, query projections, nested arrays, type reuse, RBAC. **Every prompt produced compiling Taxi.** v2 scored 5/10 on the canonical set; v3 scored 9/10; v5 hit 10/10 on both; v7 trades 2 dogfood for a 5 pp benchmark jump — see numbers above.

## Training recipe

- **Base:** `mlx-community/Qwen2.5-Coder-3B-Instruct-4bit`
- **Method:** LoRA (`mlx_lm.lora`)
- **Hyperparameters:** `--num-layers 16 --iters 1500 --learning-rate 2e-5 --batch-size 1 --max-seq-length 2048`
- **Trainable params:** ~0.1% (3.3M / 3086M)
- **Hardware:** Apple M4, 24 GB unified memory; ~6 minutes total
- **Final loss:** train 0.43, val 0.32 (val-loss minimum at iter 1500; iter-2000 overfit, val 0.43 — 1500 ships)

## Training data

Roughly 4,500 (description, taxi) pairs:

- **Reverse-description (1,268 pairs):** for each validated upstream Taxi snippet, a coder model wrote 3 NL descriptions in distinct styles (terse / task-card / doc-comment).
- **Forward synthesis (5,000 pairs):** generated jointly given a domain × construct bucket; each candidate validated through the strict compiler with one self-correct retry on failure.
- **Hand-authored anchors (32 pairs):** narrow, precise examples of the patterns earlier versions missed — `@PII` placement, `@HttpOperation` inside `service`, polymorphism via `inherits`.
- **Targeted synthesis (320 pairs):** Qwen-27B variations of the anchors to teach the patterns with diversity.
- **Validator-in-the-loop / RFT (631 pairs):** generated by a prior checkpoint, kept only when they parsed cleanly with no undefined references and no duplicate declarations.

After dedup against the held-out benchmark gold and chat formatting: ~3,700 train / ~460 valid pairs.

## Inference

### CLI (recommended)

```bash
pip install stackfix  # installs the `taxify` command
taxify "a Customer model with id and email"
```

### Python (llama-cpp-python)

```python
from llama_cpp import Llama
SYS = ("You translate natural-language requirements into idiomatic Taxi schema code. "
       "Taxi is the schema language used by Orbital (orbitalhq.com). "
       "Return ONLY the Taxi source.")
llm = Llama(model_path="taxi-nl-3b-q4.gguf", n_ctx=4096, verbose=False)
resp = llm.create_chat_completion(messages=[
    {"role": "system", "content": SYS},
    {"role": "user", "content": "Define a Customer model with id and email"},
])
print(resp["choices"][0]["message"]["content"])
```

### With existing Taxi context

```bash
taxify "Add an Order service that fetches orders by CustomerId" --schema customer.taxi
```

The CLI applies a post-processor that strips any blocks the model produces whose declared symbol already exists in the in-context schema — the fine-tune occasionally replays context verbatim, and this dedup keeps output clean.

## Files

- `taxi-nl-3b-q4.gguf` — Q4_K_M quantization, **1.8 GB**, recommended for CPU.
- `taxi-nl-3b-f16.gguf` — full f16, 6.2 GB, for evaluation parity.

## Limitations

- **Out-of-distribution prompts** (e.g., "design a healthcare claims system using OpenBanking conventions") fall back toward the base model's behaviour. Best for "translate this concrete schema description" rather than open-ended design.
- **Reserved word handling** is imperfect — names like `from` as parameter names occasionally trip the parser.
- **TaxiQL projections** (`find { X[] } as { ... }`) work for simple cases; complex projection rewrites are still best-effort.
- The post-process dedup is a heuristic (exact symbol-name match). If you genuinely want the schema repeated, set `--no-dedup` (CLI) or skip the helper.

## License

Apache 2.0 (matching the base model's license).

## Citation

```bibtex
@misc{taxi-nl-3b,
  author    = {Cloud-Gym},
  title     = {taxi-nl-3b: a small fine-tuned model for NL→Taxi translation},
  year      = {2026},
  publisher = {HuggingFace},
}
```