74 lines
3.4 KiB
Markdown
74 lines
3.4 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
base_model: Qwen/Qwen2.5-1.5B-Instruct
|
||
|
|
library_name: transformers
|
||
|
|
tags:
|
||
|
|
- interpretability
|
||
|
|
- natural-language-autoencoder
|
||
|
|
- activation-verbalizer
|
||
|
|
- qwen2.5
|
||
|
|
---
|
||
|
|
|
||
|
|
# Qwen2.5-1.5B-Instruct NLA L18 — AV (Activation Verbalizer)
|
||
|
|
|
||
|
|
The AV half of a Natural Language Autoencoder trained on the layer-18 residual stream of
|
||
|
|
`Qwen/Qwen2.5-1.5B-Instruct`. An activation is L2-normalised, scaled by
|
||
|
|
`injection_scale`, and spliced into the embedding slot of the single injection token in a fixed
|
||
|
|
prompt; the model then generates a free-text description of what the activation encodes.
|
||
|
|
|
||
|
|
This repo mirrors the layout of
|
||
|
|
[kitft/Llama-3.3-70B-NLA-L53-av](https://huggingface.co/kitft/Llama-3.3-70B-NLA-L53-av):
|
||
|
|
full merged weights at the root (LoRA already folded in — load with plain
|
||
|
|
`AutoModelForCausalLM`, no PEFT needed) and an `nla_meta.yaml` sidecar (schema v2) at the root.
|
||
|
|
The AR half lives in the sibling repo
|
||
|
|
[dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-ar](https://huggingface.co/dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-ar).
|
||
|
|
The original LoRA-adapter release (both halves in one repo, `av/` + `ar/` subdirs) is
|
||
|
|
[dormantx/Qwen2.5-1.5B-Instruct-NLA-L18](https://huggingface.co/dormantx/Qwen2.5-1.5B-Instruct-NLA-L18);
|
||
|
|
weights here are numerically identical to that release
|
||
|
|
(merged-vs-adapter greedy generations match exactly).
|
||
|
|
|
||
|
|
Notes for loaders expecting the 70B layout:
|
||
|
|
|
||
|
|
- Qwen2.5-1.5B ties embeddings (`tie_word_embeddings: true`), so there is no separate
|
||
|
|
`lm_head.weight` key in the shards; `from_pretrained` re-ties automatically.
|
||
|
|
- The AV prompt template in `nla_meta.yaml` uses the `{injection_char}` placeholder; the
|
||
|
|
injection token is `<|image_pad|>` (id 151655), a registered special token that tokenizes
|
||
|
|
atomically in context.
|
||
|
|
- `mse_scale: 1.0` — AR targets in this pipeline are L2-normalised to unit norm, not
|
||
|
|
`sqrt(d_model)`.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
import torch, torch.nn.functional as F, yaml
|
||
|
|
from huggingface_hub import hf_hub_download
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
repo = "dormantx/Qwen2.5-1.5B-Instruct-NLA-L18-av"
|
||
|
|
meta = yaml.safe_load(open(hf_hub_download(repo, "nla_meta.yaml")))
|
||
|
|
tok = AutoTokenizer.from_pretrained(repo)
|
||
|
|
av = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16).cuda().eval()
|
||
|
|
|
||
|
|
prompt = meta["prompt_templates"]["av"].format(injection_char=meta["tokens"]["injection_char"])
|
||
|
|
ids = tok(prompt, add_special_tokens=False)["input_ids"]
|
||
|
|
slot = ids.index(meta["tokens"]["injection_token_id"])
|
||
|
|
emb = av.get_input_embeddings()(torch.tensor(ids).cuda()[None]).clone()
|
||
|
|
|
||
|
|
act = ... # a raw layer-18 residual-stream activation, i.e. hidden_states[18], shape [1536]
|
||
|
|
emb[:, slot] = F.normalize(act, dim=-1).to(emb.dtype) * meta["extraction"]["injection_scale"]
|
||
|
|
out = av.generate(inputs_embeds=emb,
|
||
|
|
attention_mask=torch.ones(emb.shape[:2], device=emb.device),
|
||
|
|
max_new_tokens=32, do_sample=False)
|
||
|
|
print(tok.decode(out[0], skip_special_tokens=True))
|
||
|
|
```
|
||
|
|
|
||
|
|
Activations must come from `hidden_states[18]` of the base model (output of block 18, before the
|
||
|
|
final norm) and be passed raw — the injection step does the normalising and rescaling.
|
||
|
|
|
||
|
|
## Provenance
|
||
|
|
|
||
|
|
Pipeline and honesty-check methodology build on
|
||
|
|
[kitft/natural_language_autoencoders](https://github.com/kitft/natural_language_autoencoders),
|
||
|
|
[kameshkanna/nla-train](https://github.com/kameshkanna/nla-train), and
|
||
|
|
[SolshineCode/nla-gemma-4-e2b](https://github.com/SolshineCode/nla-gemma-4-e2b).
|