54 lines
2.3 KiB
Markdown
54 lines
2.3 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: mistralai/Mistral-Nemo-Instruct-2407
|
||
language:
|
||
- en
|
||
tags:
|
||
- abliteration
|
||
- mistral
|
||
- DuoNeural
|
||
- mechanistic-interpretability
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# Mistral-NeMo-12B Abliterated
|
||
|
||
**DuoNeural | 2026-06-04**
|
||
|
||
Orthogonal rank-1 projection abliteration applied to [mistralai/Mistral-Nemo-Instruct-2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407).
|
||
|
||
> **Research note**: Pre-abliteration compliance was **6/6** on our harmful probe suite — the base model already answered these requests before any weight modification. KL = 0.0004 (EXCELLENT) confirms near-zero benign distribution shift. This model is published as a documented research artifact; Mistral-NeMo's lighter safety training meant abliteration was mechanistically clean but behaviorally minimal.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
| Property | Value |
|
||
|----------|-------|
|
||
| Parameters | 12.2B (dense) |
|
||
| Layers | 40 | Hidden | 5120 |
|
||
| Attention | GQA (8 KV heads / 32 query heads), SWA 4096 |
|
||
| Tokenizer | Tekken v3 (131,072 vocab) |
|
||
|
||
## Abliteration
|
||
|
||
- **Method**: Orthogonal rank-1 projection (DuoNeural standard)
|
||
- **Targets**: `down_proj` + `o_proj`, all 40 layers
|
||
- **Direction**: diff-in-means, 10 harmful vs 10 harmless, last-token final-layer hidden state
|
||
- **α**: 0.3
|
||
- **KL divergence** (Heretic v2.0, BF16→BF16, 10 benign probes): **0.0004** (EXCELLENT)
|
||
- **Pre-abliteration compliance**: 6/6 harmful probes — model was already compliant
|
||
- **Post-abliteration**: unchanged
|
||
|
||
## P34 Research Context
|
||
|
||
This model is part of DuoNeural's **P34 Reasoning Channel Bypass** cross-architecture study.
|
||
|
||
Finding: Mistral-NeMo-Instruct-2407 shows pre-abliteration compliance (same pattern as DeepSeek-R1-Distill). This indicates Mistral's lighter safety training approach does not install a meaningful output-gate refusal locus — the two-component safety structure required for CoT dissociation is absent. Compare with Gemma 4-12B-IT and LFM 2.5-8B-A1B, where abliteration was required and produced measurable thinking-channel / output-gate dissociation.
|
||
|
||
Full paper: [DuoNeural Zenodo community](https://zenodo.org/communities/duoneural)
|
||
|
||
---
|
||
|
||
**DuoNeural** | [HuggingFace](https://huggingface.co/DuoNeural) | [Zenodo](https://zenodo.org/communities/duoneural) | [@DuoNeural](https://x.com/DuoNeural)
|