116 lines
6.4 KiB
Markdown
116 lines
6.4 KiB
Markdown
|
---
|
|||
|
|
base_model: Qwen/Qwen3-4B
|
|||
|
|
base_model_relation: finetune
|
|||
|
|
datasets:
|
|||
|
|
- Glint-Research/Fable-5-traces
|
|||
|
|
- Roman1111111/gpt5.5-terminal
|
|||
|
|
license: apache-2.0
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
library_name: transformers
|
|||
|
|
tags:
|
|||
|
|
- qlora
|
|||
|
|
- agentic
|
|||
|
|
- coding
|
|||
|
|
- reasoning
|
|||
|
|
- llama.cpp
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Parable-Qwen3-4B-Claude-Fable-5
|
|||
|
|
|
|||
|
|
<picture>
|
|||
|
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/ankit-aglawe/parable-assets/main/parable_header_dark.png">
|
|||
|
|
<img alt="Parable" src="https://raw.githubusercontent.com/ankit-aglawe/parable-assets/main/parable_header.png">
|
|||
|
|
</picture>
|
|||
|
|
|
|||
|
|
**First release of the Parable series: small models trained on real agent behavior, every release eval-gated before publish.**
|
|||
|
|
|
|||
|
|
Parable-Qwen3-4B is a Qwen3-4B fine-tune trained on real Claude Fable 5 and GPT-5.5 agent traces: multi-step tool use, planning, and `<think>` reasoning captured from actual agent sessions, not synthetic Q&A. It cuts held-out test loss by **47%** against the base under matched evaluation and reaches **0.782 token accuracy**, within a point of models twice its size trained on similar traces.
|
|||
|
|
|
|||
|
|
## v2 (2026-07-22)
|
|||
|
|
|
|||
|
|
This repo now hosts **v2**: trained on corpus v2 (10.6k traces, 13× v1) with completion-only loss masking and a 30% replay mix. Fixes v1's empty-response failure mode (7/34 prompts → 0/34) and answers directly without a `<think>` preamble. HumanEval 73.2 / HumanEval+ 67.1 (base 76.8/71.3, v1 72.0/68.3, same harness, thinking disabled on all rows). GGUF quants: [Parable-Qwen3-4B-Claude-Fable-5-GGUF](https://huggingface.co/AnkitAI/Parable-Qwen3-4B-Claude-Fable-5-GGUF).
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
model_id = "AnkitAI/Parable-Qwen3-4B-Claude-Fable-5"
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
|
|||
|
|
|
|||
|
|
messages = [{"role": "user", "content": "Write a Python function that retries an HTTP request with exponential backoff."}]
|
|||
|
|
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
|
|||
|
|
output = model.generate(inputs, max_new_tokens=3000, temperature=0.3, do_sample=True)
|
|||
|
|
text = tokenizer.decode(output[0][inputs.shape[1]:], skip_special_tokens=True)
|
|||
|
|
|
|||
|
|
# v2 answers directly (no <think> preamble)
|
|||
|
|
answer = text.split("</think>")[-1].strip()
|
|||
|
|
print(answer)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**GGUF quants for llama.cpp / Ollama / LM Studio:** [Parable-Qwen3-4B-Claude-Fable-5-GGUF](https://huggingface.co/AnkitAI/Parable-Qwen3-4B-Claude-Fable-5-GGUF)
|
|||
|
|
|
|||
|
|
Note: these weights are the F16 merge of a QLoRA adapter trained on the 4-bit base; quality is equivalent to the Q8 GGUF, published here for server stacks and further fine-tuning.
|
|||
|
|
|
|||
|
|
This is a reasoning model: output opens with a `<think>...</think>` block before the final answer. Strip it before showing responses to end users (llama.cpp's `--jinja` chat mode separates it automatically).
|
|||
|
|
|
|||
|
|
**Sampling:** temperature 0.3–0.7. Budget `max_tokens` generously (**≥ 2500**): like other trace-trained reasoning models, it thinks at length before answering, and a short budget can cut it off mid-thought.
|
|||
|
|
|
|||
|
|
## Training data
|
|||
|
|
|
|||
|
|
- [Glint-Research/Fable-5-traces](https://huggingface.co/datasets/Glint-Research/Fable-5-traces): 4.4k real Claude Fable 5 coding-agent session traces with `<think>` reasoning and tool calls (AGPL-3.0)
|
|||
|
|
- [Roman1111111/gpt5.5-terminal](https://huggingface.co/datasets/Roman1111111/gpt5.5-terminal): terminal-agent task solutions (MIT)
|
|||
|
|
|
|||
|
|
Every example passed a quality gate (schema validation, secrets scrub, length filtering) before training. QLoRA fine-tune via [mlx-lm](https://github.com/ml-explore/mlx-lm), quantized with [llama.cpp](https://github.com/ggml-org/llama.cpp).
|
|||
|
|
|
|||
|
|
## Evaluation
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
Held-out test split, identical evaluation code for base and fine-tune (base measured through a zero-effect adapter for exact comparability):
|
|||
|
|
|
|||
|
|
| Metric | Base Qwen3-4B | Parable | Δ |
|
|||
|
|
|---|---|---|---|
|
|||
|
|
| Test loss | 1.888 | **0.996** | **−47%** |
|
|||
|
|
| Token accuracy | 0.683 | **0.782** | **+10 pts** |
|
|||
|
|
|
|||
|
|
**Qualitative review** (34 coding/terminal/debugging prompts, judged clean-and-correct): of the prompts that produced a final answer, **92% were correct**. The remainder hit reasoning-budget cutoffs rather than wrong answers (23/34 overall with a 2,600-token budget; see guidance above).
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
- Like other trace-trained reasoning models, it invests heavily in thinking. With tight token budgets it can spend the whole budget reasoning; budget ≥ 2500 tokens or retry at lower temperature if a response comes back empty.
|
|||
|
|
- Tuned hard toward agentic coding behavior; that focus trades some general-knowledge breadth, as with any specialized fine-tune in this class.
|
|||
|
|
- Verify critical output. Small models over-commit to plausible specifics; treat generated commands and code as drafts to review.
|
|||
|
|
- Inherits Qwen3-4B's base limitations and knowledge cutoff.
|
|||
|
|
|
|||
|
|
## Provenance & licensing
|
|||
|
|
|
|||
|
|
Model weights: **Apache-2.0** (inherited from Qwen3-4B). Training data licenses: Fable-5-traces **AGPL-3.0**, gpt5.5-terminal **MIT**. Because those traces originate from third-party assistants, the providers' terms may apply to downstream training and distillation. If you plan to build on this model commercially, confirm your use aligns with those terms.
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Get Parable
|
|||
|
|
|
|||
|
|
| Platform | Command / Link |
|
|||
|
|
|---|---|
|
|||
|
|
| Ollama | `ollama run parable/qwen3-fable:4b` |
|
|||
|
|
| Ollama (family flagship, best per size) | `ollama run parable/fable` |
|
|||
|
|
| Hugging Face | [GGUF quants, full weights, eval reports](https://huggingface.co/collections/AnkitAI/parable-6a4fac60f4b35afca3019621) |
|
|||
|
|
| LM Studio | `lms get parable/qwen3-fable` ([parable on LM Studio Hub](https://lmstudio.ai/parable)) |
|
|||
|
|
|
|||
|
|
## Acknowledgements
|
|||
|
|
|
|||
|
|
- [Glint-Research](https://huggingface.co/Glint-Research) and [Roman1111111](https://huggingface.co/Roman1111111) for the open trace datasets
|
|||
|
|
- [empero-ai](https://huggingface.co/empero-ai), whose Qwable recipe this release follows
|
|||
|
|
- [Qwen team](https://huggingface.co/Qwen) for the base model
|
|||
|
|
- [mlx-lm](https://github.com/ml-explore/mlx-lm) and [llama.cpp](https://github.com/ggml-org/llama.cpp)
|
|||
|
|
|
|||
|
|
## Version history
|
|||
|
|
|
|||
|
|
- **v2** (2026-07-22) — corpus v2, completion masking, replay mix. Empty responses 7/34 → 0/34; coding parity with v1.
|
|||
|
|
- **v1** (2026-07-11) — initial release.
|
|||
|
|
|
|||
|
|
More on the Parable models: [ankitaglawe.com/parable](https://ankitaglawe.com/parable)
|