74 lines
3.5 KiB
Markdown
74 lines
3.5 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: Qwen/Qwen2.5-1.5B-Instruct
|
|
library_name: transformers
|
|
tags:
|
|
- interpretability
|
|
- natural-language-autoencoder
|
|
- activation-reconstructor
|
|
- qwen2.5
|
|
---
|
|
|
|
# Qwen2.5-1.5B-Instruct NLA L18 — AR (Activation Reconstructor)
|
|
|
|
The AR half of a Natural Language Autoencoder trained on the layer-18 residual stream of
|
|
`Qwen/Qwen2.5-1.5B-Instruct`. A copy of the base model truncated to layers
|
|
0…17 (`num_hidden_layers: 18`), plus a `Linear(1536, 1536)` value head, trained with MSE to
|
|
reconstruct the L2-normalised layer-18 activation from the AV's description text.
|
|
|
|
This repo mirrors the layout of
|
|
[kitft/Llama-3.3-70B-NLA-L53-ar](https://huggingface.co/kitft/Llama-3.3-70B-NLA-L53-ar):
|
|
full merged truncated weights at the root (LoRA already folded in, standard `model.*` keys),
|
|
`value_head.safetensors` at the root, and an `nla_meta.yaml` sidecar (schema v2) at the root.
|
|
The AV half lives in the sibling repo
|
|
[dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-av](https://huggingface.co/dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-av).
|
|
The original LoRA-adapter release (both halves in one repo, `av/` + `ar/` subdirs) is
|
|
[dormantx/Qwen2.5-1.5B-Instruct-NLA-L18](https://huggingface.co/dormantx/Qwen2.5-1.5B-Instruct-NLA-L18);
|
|
weights here are numerically identical to that release
|
|
(merged-vs-adapter reconstruction cosine 0.999999).
|
|
|
|
Notes for loaders expecting the 70B layout:
|
|
|
|
- `value_head.safetensors` holds two tensors, `weight` `[1536, 1536]` and `bias` `[1536]`,
|
|
both float32 (the 70B AR head is a bias-free bf16 `weight` only). The head was trained
|
|
with a bias — drop it and reconstructions shift.
|
|
- There is no AR wrapper template: `prompt_templates.ar` is `"{explanation}"`. The AR
|
|
consumes the raw explanation text, no prefix/suffix, so `critic_suffix_ids: null`.
|
|
- Pooling: the reconstruction input is `hidden_states[18]` at the **last non-pad token**, fed to
|
|
the value head in float32. Targets were L2-normalised to **unit** norm (`mse_scale: 1.0`), so
|
|
compare reconstructions by direction (cosine), not magnitude.
|
|
- Qwen2.5-1.5B ties embeddings, so there is no `lm_head.weight` key; the checkpoint also keeps
|
|
`model.norm.weight` (unused by the pre-norm extraction path, harmless to load).
|
|
|
|
## Usage
|
|
|
|
```python
|
|
import torch, torch.nn.functional as F
|
|
from huggingface_hub import hf_hub_download
|
|
from safetensors.torch import load_file
|
|
from transformers import AutoModel, AutoTokenizer
|
|
|
|
repo = "dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-ar"
|
|
tok = AutoTokenizer.from_pretrained(repo)
|
|
ar = AutoModel.from_pretrained(repo, torch_dtype=torch.bfloat16).cuda().eval()
|
|
head = load_file(hf_hub_download(repo, "value_head.safetensors"))
|
|
W, b = head["weight"].cuda(), head["bias"].cuda()
|
|
|
|
if tok.pad_token is None:
|
|
tok.pad_token = tok.eos_token
|
|
enc = tok(["the outcome of a football game"], return_tensors="pt", padding=True).to("cuda")
|
|
with torch.no_grad():
|
|
h = ar(**enc, output_hidden_states=True).hidden_states[18] # [B, T, 1536]
|
|
last = enc["attention_mask"].sum(1) - 1
|
|
pooled = h[torch.arange(h.shape[0]), last].to(torch.float32) # last non-pad token
|
|
pred = pooled @ W.T.float() + b.float() # reconstructed direction
|
|
pred = F.normalize(pred, dim=-1)
|
|
```
|
|
|
|
## Provenance
|
|
|
|
Pipeline and honesty-check methodology build on
|
|
[kitft/natural_language_autoencoders](https://github.com/kitft/natural_language_autoencoders),
|
|
[kameshkanna/nla-train](https://github.com/kameshkanna/nla-train), and
|
|
[SolshineCode/nla-gemma-4-e2b](https://github.com/SolshineCode/nla-gemma-4-e2b).
|