Qwen2.5-1.5B-Instruct-NLA-L18-ar/README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: transformers
tags:
  - interpretability
  - natural-language-autoencoder
  - activation-reconstructor
  - qwen2.5
---

# Qwen2.5-1.5B-Instruct NLA L18 — AR (Activation Reconstructor)

The AR half of a Natural Language Autoencoder trained on the layer-18 residual stream of
`Qwen/Qwen2.5-1.5B-Instruct`. A copy of the base model truncated to layers
0…17 (`num_hidden_layers: 18`), plus a `Linear(1536, 1536)` value head, trained with MSE to
reconstruct the L2-normalised layer-18 activation from the AV's description text.

This repo mirrors the layout of
[kitft/Llama-3.3-70B-NLA-L53-ar](https://huggingface.co/kitft/Llama-3.3-70B-NLA-L53-ar):
full merged truncated weights at the root (LoRA already folded in, standard `model.*` keys),
`value_head.safetensors` at the root, and an `nla_meta.yaml` sidecar (schema v2) at the root.
The AV half lives in the sibling repo
[dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-av](https://huggingface.co/dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-av).
The original LoRA-adapter release (both halves in one repo, `av/` + `ar/` subdirs) is
[dormantx/Qwen2.5-1.5B-Instruct-NLA-L18](https://huggingface.co/dormantx/Qwen2.5-1.5B-Instruct-NLA-L18);
weights here are numerically identical to that release
(merged-vs-adapter reconstruction cosine 0.999999).

Notes for loaders expecting the 70B layout:

- `value_head.safetensors` holds two tensors, `weight` `[1536, 1536]` and `bias` `[1536]`,
  both float32 (the 70B AR head is a bias-free bf16 `weight` only). The head was trained
  with a bias — drop it and reconstructions shift.
- There is no AR wrapper template: `prompt_templates.ar` is `"{explanation}"`. The AR
  consumes the raw explanation text, no prefix/suffix, so `critic_suffix_ids: null`.
- Pooling: the reconstruction input is `hidden_states[18]` at the **last non-pad token**, fed to
  the value head in float32. Targets were L2-normalised to **unit** norm (`mse_scale: 1.0`), so
  compare reconstructions by direction (cosine), not magnitude.
- Qwen2.5-1.5B ties embeddings, so there is no `lm_head.weight` key; the checkpoint also keeps
  `model.norm.weight` (unused by the pre-norm extraction path, harmless to load).

## Usage

```python
import torch, torch.nn.functional as F
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModel, AutoTokenizer

repo = "dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-ar"
tok = AutoTokenizer.from_pretrained(repo)
ar = AutoModel.from_pretrained(repo, torch_dtype=torch.bfloat16).cuda().eval()
head = load_file(hf_hub_download(repo, "value_head.safetensors"))
W, b = head["weight"].cuda(), head["bias"].cuda()

if tok.pad_token is None:
    tok.pad_token = tok.eos_token
enc = tok(["the outcome of a football game"], return_tensors="pt", padding=True).to("cuda")
with torch.no_grad():
    h = ar(**enc, output_hidden_states=True).hidden_states[18]      # [B, T, 1536]
    last = enc["attention_mask"].sum(1) - 1
    pooled = h[torch.arange(h.shape[0]), last].to(torch.float32)    # last non-pad token
    pred = pooled @ W.T.float() + b.float()                         # reconstructed direction
    pred = F.normalize(pred, dim=-1)
```

## Provenance

Pipeline and honesty-check methodology build on
[kitft/natural_language_autoencoders](https://github.com/kitft/natural_language_autoencoders),
[kameshkanna/nla-train](https://github.com/kameshkanna/nla-train), and
[SolshineCode/nla-gemma-4-e2b](https://github.com/SolshineCode/nla-gemma-4-e2b).