license, base_model, library_name, tags

license	base_model	library_name	tags
apache-2.0	Qwen/Qwen2.5-1.5B-Instruct	transformers	interpretability natural-language-autoencoder activation-reconstructor qwen2.5

Qwen2.5-1.5B-Instruct NLA L18 — AR (Activation Reconstructor)

The AR half of a Natural Language Autoencoder trained on the layer-18 residual stream of Qwen/Qwen2.5-1.5B-Instruct. A copy of the base model truncated to layers 0…17 (num_hidden_layers: 18), plus a Linear(1536, 1536) value head, trained with MSE to reconstruct the L2-normalised layer-18 activation from the AV's description text.

This repo mirrors the layout of kitft/Llama-3.3-70B-NLA-L53-ar: full merged truncated weights at the root (LoRA already folded in, standard model.* keys), value_head.safetensors at the root, and an nla_meta.yaml sidecar (schema v2) at the root. The AV half lives in the sibling repo dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-av. The original LoRA-adapter release (both halves in one repo, av/ + ar/ subdirs) is dormantx/Qwen2.5-1.5B-Instruct-NLA-L18; weights here are numerically identical to that release (merged-vs-adapter reconstruction cosine 0.999999).

Notes for loaders expecting the 70B layout:

• value_head.safetensors holds two tensors, weight [1536, 1536] and bias [1536], both float32 (the 70B AR head is a bias-free bf16 weight only). The head was trained with a bias — drop it and reconstructions shift.
• There is no AR wrapper template: prompt_templates.ar is "{explanation}". The AR consumes the raw explanation text, no prefix/suffix, so critic_suffix_ids: null.
• Pooling: the reconstruction input is hidden_states[18] at the last non-pad token, fed to the value head in float32. Targets were L2-normalised to unit norm (mse_scale: 1.0), so compare reconstructions by direction (cosine), not magnitude.
• Qwen2.5-1.5B ties embeddings, so there is no lm_head.weight key; the checkpoint also keeps model.norm.weight (unused by the pre-norm extraction path, harmless to load).

Usage

import torch, torch.nn.functional as F
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModel, AutoTokenizer

repo = "dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-ar"
tok = AutoTokenizer.from_pretrained(repo)
ar = AutoModel.from_pretrained(repo, torch_dtype=torch.bfloat16).cuda().eval()
head = load_file(hf_hub_download(repo, "value_head.safetensors"))
W, b = head["weight"].cuda(), head["bias"].cuda()

if tok.pad_token is None:
    tok.pad_token = tok.eos_token
enc = tok(["the outcome of a football game"], return_tensors="pt", padding=True).to("cuda")
with torch.no_grad():
    h = ar(**enc, output_hidden_states=True).hidden_states[18]      # [B, T, 1536]
    last = enc["attention_mask"].sum(1) - 1
    pooled = h[torch.arange(h.shape[0]), last].to(torch.float32)    # last non-pad token
    pred = pooled @ W.T.float() + b.float()                         # reconstructed direction
    pred = F.normalize(pred, dim=-1)

Provenance

Pipeline and honesty-check methodology build on kitft/natural_language_autoencoders, kameshkanna/nla-train, and SolshineCode/nla-gemma-4-e2b.