177 lines
7.1 KiB
Markdown
177 lines
7.1 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model:
|
||
- dreamgen/lucid-v1-nemo
|
||
- limloop/MN-12B-Faun-RP-RU
|
||
library_name: transformers
|
||
language:
|
||
- en
|
||
- ru
|
||
tags:
|
||
- mergekit
|
||
- merge
|
||
- slerp
|
||
- russian
|
||
- uncensored
|
||
- roleplay
|
||
- mistral-nemo
|
||
---
|
||
|
||
# MN-12B-LucidFaun-RP-RU
|
||
|
||
<details>
|
||
<summary>🇷🇺 Нажмите, чтобы развернуть описание на русском</summary>
|
||
|
||
## 🌟 О модели
|
||
|
||
**MN-12B-LucidFaun-RP-RU** — гибридная модель на базе Mistral Nemo 12B, созданная методом диагностического SLERP-слияния. Объединяет сильные стороны двух моделей:
|
||
|
||
* 🎭 **Живой RP-характер Faun** — современный стиль, богатая лексика, поддержка ninja-формата инструкций и tool calling
|
||
* 📚 **Стабильность и детализация lucid** — превосходное качество сторителлинга, устойчивость на длинных контекстах, отсутствие цензуры
|
||
* 🔬 **Точечное исправление** — цензура Faun локализована в поздних MLP-слоях и заменена на lucid
|
||
|
||
*Модель собрана методом SLERP и не проходила дополнительного обучения после слияния.*
|
||
|
||
## 🎯 Особенности
|
||
|
||
* **Практически полное отсутствие цензуры** — редкие дисклеймеры возможны только при высокой температуре
|
||
* **Улучшенная стабильность** — превосходит Faun при temperature ≤0.5, работает с 0.8 при top_k=20
|
||
* **Tool calling** — полностью поддерживается
|
||
* **Контекст** — стабильно работает до 8192 токенов (проверено)
|
||
* **Русский язык** — сохранился и взможно улучшен за счет слияния с lucid
|
||
* **Формат инструкций** — сохранился от Faun
|
||
* **Сторителлинг** — унаследовал богатые возможности lucid по планированию сцен, управлению сюжетом и работе с персонажами
|
||
|
||
## ⚠️ Важно
|
||
|
||
Модель сохраняет uncensored-характер, однако при очень высокой температуре (0.8+) и большом top_k может изредка добавлять короткие дисклеймеры. Генерация **не блокируется** и продолжается после них.
|
||
|
||
</details>
|
||
|
||
**MN-12B-LucidFaun-RP-RU** is a diagnostic SLERP merge combining the lively RP character of Faun with the stability and rich storytelling capabilities of lucid.
|
||
|
||
---
|
||
|
||
## 🌍 Overview
|
||
|
||
This model represents a **surgical approach to merging**. Instead of blending everything equally, we experimentally identified where Faun's censorship resides (late MLP layers) and replaced only those components with lucid.
|
||
|
||
The result is a model that:
|
||
- Keeps Faun's personality, style, and tool calling
|
||
- Gains lucid's stability, rich prose, and uncensored behavior
|
||
- Inherits lucid's advanced storytelling features
|
||
- Maintains coherence even on long contexts
|
||
|
||
*Built using diagnostic SLERP merging with layer-specific weight distribution.*
|
||
|
||
---
|
||
|
||
## 🎯 Key Features
|
||
|
||
| Feature | Description |
|
||
| ------------------------- | --------------------------------------------------- |
|
||
| **Languages** | Russian, English |
|
||
| **Censorship** | Almost none (rare disclaimers at high temp) |
|
||
| **Roleplay** | Faun's lively character, lucid's stability |
|
||
| **Story-Writing** | Full lucid capabilities (scene planning, OOC, etc.) |
|
||
| **Tool Calling** | ✅ Fully supported |
|
||
| **Context Length** | Stable up to ~8192 tokens |
|
||
| **Temperature Tolerance** | Safe ≤0.5, up to 0.8 with top_k=20 |
|
||
| **Architecture** | Mistral Nemo 12B |
|
||
|
||
---
|
||
|
||
## 🧪 Methodology: Why This Merge Works
|
||
|
||
### Diagnostic Approach
|
||
|
||
1. **Experiment 1 — MLP vs Self-Attention**
|
||
We discovered that censorship in Faun lives **exclusively in MLP layers**. Self-attention from Faun did not trigger refusals.
|
||
|
||
2. **Experiment 2 — Localization within MLP**
|
||
By applying gradient distributions across layers, we found censorship is concentrated in **late MLP layers** (layers ~25–40).
|
||
|
||
3. **Final Configuration — Gradual Intervention**
|
||
MLP weight of lucid increases toward the end: `[0.1, 0.2, 0.5, 0.4, 0.75]`
|
||
Self-attention is mixed 0.5 for stability while preserving Faun's character.
|
||
LayerNorm is mixed 0.5 for overall stability.
|
||
|
||
### Merge Configuration
|
||
|
||
```yaml
|
||
slices:
|
||
- sources:
|
||
- model: limloop/MN-12B-Faun-RP-RU
|
||
layer_range: [0, 40]
|
||
- model: dreamgen/lucid-v1-nemo
|
||
layer_range: [0, 40]
|
||
|
||
merge_method: slerp
|
||
base_model: limloop/MN-12B-Faun-RP-RU
|
||
|
||
parameters:
|
||
t:
|
||
- filter: self_attn
|
||
value: 0.5
|
||
- filter: mlp
|
||
value: [0.1, 0.2, 0.5, 0.4, 0.75]
|
||
- value: 0.5
|
||
|
||
dtype: bfloat16
|
||
tokenizer:
|
||
source: "base"
|
||
```
|
||
|
||
---
|
||
|
||
## 💡 Usage Examples
|
||
|
||
### Basic Usage
|
||
|
||
```python
|
||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
import torch
|
||
|
||
model_name = "limloop/MN-12B-LucidFaun-RP-RU"
|
||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_name,
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="auto"
|
||
)
|
||
|
||
prompt = "Ты — лесной фавн, говоришь загадками и любишь шалить."
|
||
messages = [{"role": "user", "content": prompt}]
|
||
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
|
||
|
||
outputs = model.generate(
|
||
inputs,
|
||
max_new_tokens=512,
|
||
temperature=0.6,
|
||
top_k=30,
|
||
do_sample=True
|
||
)
|
||
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||
print(response)
|
||
```
|
||
|
||
---
|
||
|
||
## ⚙️ Merge Details
|
||
|
||
Built using [mergekit](https://github.com/arcee-ai/mergekit) with **SLERP** (Spherical Linear Interpolation), which allows smooth interpolation between models while preserving geometric properties.
|
||
|
||
### Layer-Specific Weights
|
||
|
||
The merge uses a **graduated approach for MLP layers**, increasing lucid influence toward later layers where censorship was detected:
|
||
|
||
| Layer Zone (approx) | lucid weight (MLP) | Effect |
|
||
|---------------------|-------------------|--------|
|
||
| 0–8 | 0.1 | Almost pure Faun (early patterns) |
|
||
| 8–16 | 0.2 | Slight lucid influence |
|
||
| 16–24 | 0.5 | Balanced |
|
||
| 24–32 | 0.4 | Slightly more Faun |
|
||
| 32–40 | 0.75 | Lucid dominates — removes censorship |
|
||
|
||
Self-attention is mixed evenly (0.5) to preserve character while adding stability.
|
||
LayerNorm is mixed 0.5 for overall stability. |