132 lines
6.9 KiB
Markdown
132 lines
6.9 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- merge
|
||
- model-merging
|
||
- crane
|
||
- qwen3
|
||
- moe
|
||
- code
|
||
- reasoning
|
||
base_model:
|
||
- Qwen/Qwen3-30B-A3B-Instruct-2507
|
||
- Qwen/Qwen3-30B-A3B-Thinking-2507
|
||
---
|
||
|
||
# crane-30b
|
||
|
||
**crane-30b** is a CRANE merge: it is produced by *merging* two **Qwen3-30B-A3B** checkpoints (an Instruct base and a Thinking donor) with the **CRANE** method — it is **not** trained or fine-tuned from scratch. CRANE (*Constrained Reasoning Injection for Code Agents via Nullspace Editing*) injects reasoning ability from the Thinking donor into the tool-disciplined Instruct / code base while preserving the base model's output format and tool-calling behavior.
|
||
|
||
**Project page:** **[https://rpi-nsl.github.io/CRANE/](https://rpi-nsl.github.io/CRANE/)** · Code: [github.com/rpi-nsl/CRANE](https://github.com/rpi-nsl/CRANE)
|
||
|
||
> Note: this is the CRANE *weight-merging* method for code agents. It is unrelated to the similarly-named "CRANE: Reasoning with constrained LLM generation" (arXiv 2502.09061), despite the shared acronym.
|
||
|
||
## How it was made (CRANE)
|
||
|
||
CRANE is a **training-free, parameter-editing weight merge** that injects reasoning ability from a "Thinking" donor into a tool-disciplined Instruct / code base, while constraining the edit so the base model's output format and tool-calling behavior are preserved. It treats the **Thinking − Instruct delta** \\(\delta = \theta_{\text{think}} - \theta_{\text{inst}}\\) as a pool of candidate reasoning edits, and applies three composable stages per layer \\(l\\) and parameter component \\(c\\):
|
||
|
||
$$
|
||
\theta_{\text{merged}}^{(l,c)} = \theta_{\text{inst}}^{(l,c)} + \underbrace{\Pi_{\tau,\,q(l,c)}^{\text{GSP}}}_{\text{Stage 3}}\!\Big( \alpha \cdot \underbrace{S_{\text{CTG}}(c,l)}_{\text{Stage 2}} \cdot \underbrace{T\big(\delta^{(l,c)}\big)}_{\text{Stage 1}} \Big)
|
||
$$
|
||
|
||
|
||
|
||
Three small calibration sets drive the stages — \\(\mathcal{D}_R\\) (reasoning transfer), \\(\mathcal{D}_A\\) (agent-behavior / tool-use preservation), and \\(\mathcal{D}_F\\) (format preservation):
|
||
|
||
- **Stage 1 — Magnitude thresholding \\(T(\delta)\\).** A deterministic median-magnitude threshold keeps only the larger (top-half) delta coordinates and rescales them by 2, discarding low-confidence noise.
|
||
- **Stage 2 — Conservative Taylor Gate \\(S_{\text{CTG}}\\).** From a signed, direction-aware score \\(s_K(j) = -g_{K,j}\,\delta_j\\) per calibration loss, CTG keeps the positive part of the per-coordinate minimum over the reasoning and agent-behavior objectives, \\(p_j = [\min\{s_R(j), s_A(j)\}]_+\\) — rewarding a coordinate only when the edit helps *both*. These aggregate into the per-component, per-layer coefficient \\(S_{\text{CTG}}(c,l)\\), scaled by the single global merge strength \\(\alpha\\).
|
||
- **Stage 3 — Graduated Sigmoidal Projection (GSP).** From the SVD of format-critical Instruct activations \\(H_q = U_q\Sigma_q V_q^{\top}\\), a smooth sigmoidal weight \\(\mathbf{w}_q\\) (set by singular amplitude and threshold \\(\tau\\)) gives the projector \\(\Pi_{\tau,q}^{\text{GSP}}(\Delta_q) = \Delta_q - \Delta_q V_q \operatorname{diag}(\mathbf{w}_q) V_q^{\top}\\), attenuating high-amplitude format directions so reasoning is injected without perturbing chat-template tokens, tool-call delimiters, or JSON/schema structure.
|
||
|
||
The result is a merge that gains planning / reflection / recovery reasoning while keeping the base agent's compact, tool-call-disciplined behavior — the entire merge is a closed-form edit of the Instruct weights, with no fine-tuning.
|
||
|
||
### This checkpoint's recipe
|
||
|
||
This checkpoint merges **`Qwen/Qwen3-30B-A3B-Instruct-2507`** (base) and **`Qwen/Qwen3-30B-A3B-Thinking-2507`** (donor) with:
|
||
|
||
- **Global injection strength** — \\(\alpha = 0.25\\), multiplied by the per-component CTG coefficients, so the Thinking delta is added at roughly a quarter strength.
|
||
- **Per-layer / per-component gating** — attention, expert (FFN), norm, and router components each get their own \\(S_{\text{CTG}}(c,l)\\) coefficient, varying by layer index rather than a single flat scalar.
|
||
- **GSP projector** — a freshly rebuilt Qwen3-30B graduated-sigmoidal projector (sigmoid threshold \\(\tau = 0.03\\)) protects the format / tool-call subspace before injection.
|
||
|
||
## Architecture
|
||
|
||
The merge preserves the standard **Qwen3-30B-A3B** (MoE) topology unchanged:
|
||
|
||
| Property | Value |
|
||
| --- | --- |
|
||
| model_type | qwen3_moe |
|
||
| Architecture class | Qwen3MoeForCausalLM |
|
||
| Total params | ~30B |
|
||
| Active params | ~3B |
|
||
| hidden_size | 2048 |
|
||
| num_hidden_layers | 48 |
|
||
| num_experts | 128 |
|
||
| num_experts_per_tok | 8 |
|
||
| num_attention_heads | 32 |
|
||
| num_key_value_heads | 4 |
|
||
| head_dim | 128 |
|
||
| moe_intermediate_size | 768 |
|
||
| max_position_embeddings | 262144 |
|
||
| vocab_size | 151936 |
|
||
| dtype | bfloat16 |
|
||
| rope_theta | 10000000 |
|
||
|
||
A `config_1m.json` is also included for the extended long-context variant: it keeps the same `rope_scaling` (null) and `max_position_embeddings` (262144), but adds a `dual_chunk_attention_config` (dual chunk attention, `original_max_position_embeddings = 131072` + sparse-attention settings) for longer-context inference.
|
||
|
||
## Usage
|
||
|
||
```python
|
||
import torch
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model_id = "zmzfpc/crane-30b"
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_id,
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
)
|
||
|
||
messages = [
|
||
{"role": "user", "content": "Write a Python function that returns the nth Fibonacci number."},
|
||
]
|
||
|
||
inputs = tokenizer.apply_chat_template(
|
||
messages,
|
||
add_generation_prompt=True,
|
||
return_tensors="pt",
|
||
).to(model.device)
|
||
|
||
outputs = model.generate(inputs, max_new_tokens=512)
|
||
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
> Requires a recent `transformers` with Qwen3-MoE support (`transformers >= 4.51`).
|
||
|
||
## Citation / attribution
|
||
|
||
If you use this model or the CRANE method, please cite:
|
||
|
||
```bibtex
|
||
@misc{zhu2026crane,
|
||
title = {CRANE: Constrained Reasoning Injection for Code Agents via Nullspace Editing},
|
||
author = {Zhu, Mingzhi and Merler, Michele and Pavuluri, Raju and Patterson, Stacy},
|
||
year = {2026},
|
||
eprint = {2605.14084},
|
||
archivePrefix= {arXiv},
|
||
primaryClass = {cs.SE},
|
||
url = {https://arxiv.org/abs/2605.14084}
|
||
}
|
||
```
|
||
|
||
**Project page:** [https://rpi-nsl.github.io/CRANE/](https://rpi-nsl.github.io/CRANE/) · **Code:** [github.com/rpi-nsl/CRANE](https://github.com/rpi-nsl/CRANE)
|
||
|
||
**Base models** — built from two Apache-2.0 checkpoints:
|
||
|
||
- [Qwen/Qwen3-30B-A3B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507) (base / backbone)
|
||
- [Qwen/Qwen3-30B-A3B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507) (reasoning donor)
|
||
|
||
**License:** Apache-2.0 (consistent with both base models and the CRANE code).
|