初始化项目，由ModelHub XC社区提供模型
Model: zmzfpc/crane-30b Source: Original Platform
This commit is contained in:
ModelHub XC
131
README.md
Normal file
131
README.md
Normal file
@@ -0,0 +1,131 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- merge
|
||||
- model-merging
|
||||
- crane
|
||||
- qwen3
|
||||
- moe
|
||||
- code
|
||||
- reasoning
|
||||
base_model:
|
||||
- Qwen/Qwen3-30B-A3B-Instruct-2507
|
||||
- Qwen/Qwen3-30B-A3B-Thinking-2507
|
||||
---
|
||||
|
||||
# crane-30b
|
||||
|
||||
**crane-30b** is a CRANE merge: it is produced by *merging* two **Qwen3-30B-A3B** checkpoints (an Instruct base and a Thinking donor) with the **CRANE** method — it is **not** trained or fine-tuned from scratch. CRANE (*Constrained Reasoning Injection for Code Agents via Nullspace Editing*) injects reasoning ability from the Thinking donor into the tool-disciplined Instruct / code base while preserving the base model's output format and tool-calling behavior.
|
||||
|
||||
**Project page:** **[https://rpi-nsl.github.io/CRANE/](https://rpi-nsl.github.io/CRANE/)** · Code: [github.com/rpi-nsl/CRANE](https://github.com/rpi-nsl/CRANE)
|
||||
|
||||
> Note: this is the CRANE *weight-merging* method for code agents. It is unrelated to the similarly-named "CRANE: Reasoning with constrained LLM generation" (arXiv 2502.09061), despite the shared acronym.
|
||||
|
||||
## How it was made (CRANE)
|
||||
|
||||
CRANE is a **training-free, parameter-editing weight merge** that injects reasoning ability from a "Thinking" donor into a tool-disciplined Instruct / code base, while constraining the edit so the base model's output format and tool-calling behavior are preserved. It treats the **Thinking − Instruct delta** \\(\delta = \theta_{\text{think}} - \theta_{\text{inst}}\\) as a pool of candidate reasoning edits, and applies three composable stages per layer \\(l\\) and parameter component \\(c\\):
|
||||
|
||||
$$
|
||||
\theta_{\text{merged}}^{(l,c)} = \theta_{\text{inst}}^{(l,c)} + \underbrace{\Pi_{\tau,\,q(l,c)}^{\text{GSP}}}_{\text{Stage 3}}\!\Big( \alpha \cdot \underbrace{S_{\text{CTG}}(c,l)}_{\text{Stage 2}} \cdot \underbrace{T\big(\delta^{(l,c)}\big)}_{\text{Stage 1}} \Big)
|
||||
$$
|
||||
|
||||
|
||||
|
||||
Three small calibration sets drive the stages — \\(\mathcal{D}_R\\) (reasoning transfer), \\(\mathcal{D}_A\\) (agent-behavior / tool-use preservation), and \\(\mathcal{D}_F\\) (format preservation):
|
||||
|
||||
- **Stage 1 — Magnitude thresholding \\(T(\delta)\\).** A deterministic median-magnitude threshold keeps only the larger (top-half) delta coordinates and rescales them by 2, discarding low-confidence noise.
|
||||
- **Stage 2 — Conservative Taylor Gate \\(S_{\text{CTG}}\\).** From a signed, direction-aware score \\(s_K(j) = -g_{K,j}\,\delta_j\\) per calibration loss, CTG keeps the positive part of the per-coordinate minimum over the reasoning and agent-behavior objectives, \\(p_j = [\min\{s_R(j), s_A(j)\}]_+\\) — rewarding a coordinate only when the edit helps *both*. These aggregate into the per-component, per-layer coefficient \\(S_{\text{CTG}}(c,l)\\), scaled by the single global merge strength \\(\alpha\\).
|
||||
- **Stage 3 — Graduated Sigmoidal Projection (GSP).** From the SVD of format-critical Instruct activations \\(H_q = U_q\Sigma_q V_q^{\top}\\), a smooth sigmoidal weight \\(\mathbf{w}_q\\) (set by singular amplitude and threshold \\(\tau\\)) gives the projector \\(\Pi_{\tau,q}^{\text{GSP}}(\Delta_q) = \Delta_q - \Delta_q V_q \operatorname{diag}(\mathbf{w}_q) V_q^{\top}\\), attenuating high-amplitude format directions so reasoning is injected without perturbing chat-template tokens, tool-call delimiters, or JSON/schema structure.
|
||||
|
||||
The result is a merge that gains planning / reflection / recovery reasoning while keeping the base agent's compact, tool-call-disciplined behavior — the entire merge is a closed-form edit of the Instruct weights, with no fine-tuning.
|
||||
|
||||
### This checkpoint's recipe
|
||||
|
||||
This checkpoint merges **`Qwen/Qwen3-30B-A3B-Instruct-2507`** (base) and **`Qwen/Qwen3-30B-A3B-Thinking-2507`** (donor) with:
|
||||
|
||||
- **Global injection strength** — \\(\alpha = 0.25\\), multiplied by the per-component CTG coefficients, so the Thinking delta is added at roughly a quarter strength.
|
||||
- **Per-layer / per-component gating** — attention, expert (FFN), norm, and router components each get their own \\(S_{\text{CTG}}(c,l)\\) coefficient, varying by layer index rather than a single flat scalar.
|
||||
- **GSP projector** — a freshly rebuilt Qwen3-30B graduated-sigmoidal projector (sigmoid threshold \\(\tau = 0.03\\)) protects the format / tool-call subspace before injection.
|
||||
|
||||
## Architecture
|
||||
|
||||
The merge preserves the standard **Qwen3-30B-A3B** (MoE) topology unchanged:
|
||||
|
||||
| Property | Value |
|
||||
| --- | --- |
|
||||
| model_type | qwen3_moe |
|
||||
| Architecture class | Qwen3MoeForCausalLM |
|
||||
| Total params | ~30B |
|
||||
| Active params | ~3B |
|
||||
| hidden_size | 2048 |
|
||||
| num_hidden_layers | 48 |
|
||||
| num_experts | 128 |
|
||||
| num_experts_per_tok | 8 |
|
||||
| num_attention_heads | 32 |
|
||||
| num_key_value_heads | 4 |
|
||||
| head_dim | 128 |
|
||||
| moe_intermediate_size | 768 |
|
||||
| max_position_embeddings | 262144 |
|
||||
| vocab_size | 151936 |
|
||||
| dtype | bfloat16 |
|
||||
| rope_theta | 10000000 |
|
||||
|
||||
A `config_1m.json` is also included for the extended long-context variant: it keeps the same `rope_scaling` (null) and `max_position_embeddings` (262144), but adds a `dual_chunk_attention_config` (dual chunk attention, `original_max_position_embeddings = 131072` + sparse-attention settings) for longer-context inference.
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "zmzfpc/crane-30b"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "user", "content": "Write a Python function that returns the nth Fibonacci number."},
|
||||
]
|
||||
|
||||
inputs = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
add_generation_prompt=True,
|
||||
return_tensors="pt",
|
||||
).to(model.device)
|
||||
|
||||
outputs = model.generate(inputs, max_new_tokens=512)
|
||||
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
> Requires a recent `transformers` with Qwen3-MoE support (`transformers >= 4.51`).
|
||||
|
||||
## Citation / attribution
|
||||
|
||||
If you use this model or the CRANE method, please cite:
|
||||
|
||||
```bibtex
|
||||
@misc{zhu2026crane,
|
||||
title = {CRANE: Constrained Reasoning Injection for Code Agents via Nullspace Editing},
|
||||
author = {Zhu, Mingzhi and Merler, Michele and Pavuluri, Raju and Patterson, Stacy},
|
||||
year = {2026},
|
||||
eprint = {2605.14084},
|
||||
archivePrefix= {arXiv},
|
||||
primaryClass = {cs.SE},
|
||||
url = {https://arxiv.org/abs/2605.14084}
|
||||
}
|
||||
```
|
||||
|
||||
**Project page:** [https://rpi-nsl.github.io/CRANE/](https://rpi-nsl.github.io/CRANE/) · **Code:** [github.com/rpi-nsl/CRANE](https://github.com/rpi-nsl/CRANE)
|
||||
|
||||
**Base models** — built from two Apache-2.0 checkpoints:
|
||||
|
||||
- [Qwen/Qwen3-30B-A3B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507) (base / backbone)
|
||||
- [Qwen/Qwen3-30B-A3B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507) (reasoning donor)
|
||||
|
||||
**License:** Apache-2.0 (consistent with both base models and the CRANE code).
|
||||
Reference in New Issue
Block a user