---
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: transformers
tags:
  - interpretability
  - natural-language-autoencoder
  - activation-verbalizer
  - qwen2.5
---

# Qwen2.5-1.5B-Instruct NLA L18 — AV (Activation Verbalizer)

The AV half of a Natural Language Autoencoder trained on the layer-18 residual stream of
`Qwen/Qwen2.5-1.5B-Instruct`. An activation is L2-normalised, scaled by
`injection_scale`, and spliced into the embedding slot of the single injection token in a fixed
prompt; the model then generates a free-text description of what the activation encodes.

This repo mirrors the layout of
[kitft/Llama-3.3-70B-NLA-L53-av](https://huggingface.co/kitft/Llama-3.3-70B-NLA-L53-av):
full merged weights at the root (LoRA already folded in — load with plain
`AutoModelForCausalLM`, no PEFT needed) and an `nla_meta.yaml` sidecar (schema v2) at the root.
The AR half lives in the sibling repo
[dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-ar](https://huggingface.co/dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-ar).
The original LoRA-adapter release (both halves in one repo, `av/` + `ar/` subdirs) is
[dormantx/Qwen2.5-1.5B-Instruct-NLA-L18](https://huggingface.co/dormantx/Qwen2.5-1.5B-Instruct-NLA-L18);
weights here are numerically identical to that release
(merged-vs-adapter greedy generations match exactly).

Notes for loaders expecting the 70B layout:

- Qwen2.5-1.5B ties embeddings (`tie_word_embeddings: true`), so there is no separate
  `lm_head.weight` key in the shards; `from_pretrained` re-ties automatically.
- The AV prompt template in `nla_meta.yaml` uses the `{injection_char}` placeholder; the
  injection token is `<|image_pad|>` (id 151655), a registered special token that tokenizes
  atomically in context.
- `mse_scale: 1.0` — AR targets in this pipeline are L2-normalised to unit norm, not
  `sqrt(d_model)`.

## Usage

```python
import torch, torch.nn.functional as F, yaml
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-av"
meta = yaml.safe_load(open(hf_hub_download(repo, "nla_meta.yaml")))
tok = AutoTokenizer.from_pretrained(repo)
av = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16).cuda().eval()

prompt = meta["prompt_templates"]["av"].format(injection_char=meta["tokens"]["injection_char"])
ids = tok(prompt, add_special_tokens=False)["input_ids"]
slot = ids.index(meta["tokens"]["injection_token_id"])
emb = av.get_input_embeddings()(torch.tensor(ids).cuda()[None]).clone()

act = ...  # a raw layer-18 residual-stream activation, i.e. hidden_states[18], shape [1536]
emb[:, slot] = F.normalize(act, dim=-1).to(emb.dtype) * meta["extraction"]["injection_scale"]
out = av.generate(inputs_embeds=emb,
                  attention_mask=torch.ones(emb.shape[:2], device=emb.device),
                  max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
```

Activations must come from `hidden_states[18]` of the base model (output of block 18, before the
final norm) and be passed raw — the injection step does the normalising and rescaling.

## Provenance

Pipeline and honesty-check methodology build on
[kitft/natural_language_autoencoders](https://github.com/kitft/natural_language_autoencoders),
[kameshkanna/nla-train](https://github.com/kameshkanna/nla-train), and
[SolshineCode/nla-gemma-4-e2b](https://github.com/SolshineCode/nla-gemma-4-e2b).