初始化项目，由ModelHub XC社区提供模型
Model: limloop/MN-12B-LucidFaun-RP-RU Source: Original Platform
This commit is contained in:
ModelHub XC
177
README.md
Normal file
177
README.md
Normal file
@@ -0,0 +1,177 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
base_model:
|
||||
- dreamgen/lucid-v1-nemo
|
||||
- limloop/MN-12B-Faun-RP-RU
|
||||
library_name: transformers
|
||||
language:
|
||||
- en
|
||||
- ru
|
||||
tags:
|
||||
- mergekit
|
||||
- merge
|
||||
- slerp
|
||||
- russian
|
||||
- uncensored
|
||||
- roleplay
|
||||
- mistral-nemo
|
||||
---
|
||||
|
||||
# MN-12B-LucidFaun-RP-RU
|
||||
|
||||
<details>
|
||||
<summary>🇷🇺 Нажмите, чтобы развернуть описание на русском</summary>
|
||||
|
||||
## 🌟 О модели
|
||||
|
||||
**MN-12B-LucidFaun-RP-RU** — гибридная модель на базе Mistral Nemo 12B, созданная методом диагностического SLERP-слияния. Объединяет сильные стороны двух моделей:
|
||||
|
||||
* 🎭 **Живой RP-характер Faun** — современный стиль, богатая лексика, поддержка ninja-формата инструкций и tool calling
|
||||
* 📚 **Стабильность и детализация lucid** — превосходное качество сторителлинга, устойчивость на длинных контекстах, отсутствие цензуры
|
||||
* 🔬 **Точечное исправление** — цензура Faun локализована в поздних MLP-слоях и заменена на lucid
|
||||
|
||||
*Модель собрана методом SLERP и не проходила дополнительного обучения после слияния.*
|
||||
|
||||
## 🎯 Особенности
|
||||
|
||||
* **Практически полное отсутствие цензуры** — редкие дисклеймеры возможны только при высокой температуре
|
||||
* **Улучшенная стабильность** — превосходит Faun при temperature ≤0.5, работает с 0.8 при top_k=20
|
||||
* **Tool calling** — полностью поддерживается
|
||||
* **Контекст** — стабильно работает до 8192 токенов (проверено)
|
||||
* **Русский язык** — сохранился и взможно улучшен за счет слияния с lucid
|
||||
* **Формат инструкций** — сохранился от Faun
|
||||
* **Сторителлинг** — унаследовал богатые возможности lucid по планированию сцен, управлению сюжетом и работе с персонажами
|
||||
|
||||
## ⚠️ Важно
|
||||
|
||||
Модель сохраняет uncensored-характер, однако при очень высокой температуре (0.8+) и большом top_k может изредка добавлять короткие дисклеймеры. Генерация **не блокируется** и продолжается после них.
|
||||
|
||||
</details>
|
||||
|
||||
**MN-12B-LucidFaun-RP-RU** is a diagnostic SLERP merge combining the lively RP character of Faun with the stability and rich storytelling capabilities of lucid.
|
||||
|
||||
---
|
||||
|
||||
## 🌍 Overview
|
||||
|
||||
This model represents a **surgical approach to merging**. Instead of blending everything equally, we experimentally identified where Faun's censorship resides (late MLP layers) and replaced only those components with lucid.
|
||||
|
||||
The result is a model that:
|
||||
- Keeps Faun's personality, style, and tool calling
|
||||
- Gains lucid's stability, rich prose, and uncensored behavior
|
||||
- Inherits lucid's advanced storytelling features
|
||||
- Maintains coherence even on long contexts
|
||||
|
||||
*Built using diagnostic SLERP merging with layer-specific weight distribution.*
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Key Features
|
||||
|
||||
| Feature | Description |
|
||||
| ------------------------- | --------------------------------------------------- |
|
||||
| **Languages** | Russian, English |
|
||||
| **Censorship** | Almost none (rare disclaimers at high temp) |
|
||||
| **Roleplay** | Faun's lively character, lucid's stability |
|
||||
| **Story-Writing** | Full lucid capabilities (scene planning, OOC, etc.) |
|
||||
| **Tool Calling** | ✅ Fully supported |
|
||||
| **Context Length** | Stable up to ~8192 tokens |
|
||||
| **Temperature Tolerance** | Safe ≤0.5, up to 0.8 with top_k=20 |
|
||||
| **Architecture** | Mistral Nemo 12B |
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Methodology: Why This Merge Works
|
||||
|
||||
### Diagnostic Approach
|
||||
|
||||
1. **Experiment 1 — MLP vs Self-Attention**
|
||||
We discovered that censorship in Faun lives **exclusively in MLP layers**. Self-attention from Faun did not trigger refusals.
|
||||
|
||||
2. **Experiment 2 — Localization within MLP**
|
||||
By applying gradient distributions across layers, we found censorship is concentrated in **late MLP layers** (layers ~25–40).
|
||||
|
||||
3. **Final Configuration — Gradual Intervention**
|
||||
MLP weight of lucid increases toward the end: `[0.1, 0.2, 0.5, 0.4, 0.75]`
|
||||
Self-attention is mixed 0.5 for stability while preserving Faun's character.
|
||||
LayerNorm is mixed 0.5 for overall stability.
|
||||
|
||||
### Merge Configuration
|
||||
|
||||
```yaml
|
||||
slices:
|
||||
- sources:
|
||||
- model: limloop/MN-12B-Faun-RP-RU
|
||||
layer_range: [0, 40]
|
||||
- model: dreamgen/lucid-v1-nemo
|
||||
layer_range: [0, 40]
|
||||
|
||||
merge_method: slerp
|
||||
base_model: limloop/MN-12B-Faun-RP-RU
|
||||
|
||||
parameters:
|
||||
t:
|
||||
- filter: self_attn
|
||||
value: 0.5
|
||||
- filter: mlp
|
||||
value: [0.1, 0.2, 0.5, 0.4, 0.75]
|
||||
- value: 0.5
|
||||
|
||||
dtype: bfloat16
|
||||
tokenizer:
|
||||
source: "base"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 💡 Usage Examples
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
import torch
|
||||
|
||||
model_name = "limloop/MN-12B-LucidFaun-RP-RU"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_name,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto"
|
||||
)
|
||||
|
||||
prompt = "Ты — лесной фавн, говоришь загадками и любишь шалить."
|
||||
messages = [{"role": "user", "content": prompt}]
|
||||
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
|
||||
|
||||
outputs = model.generate(
|
||||
inputs,
|
||||
max_new_tokens=512,
|
||||
temperature=0.6,
|
||||
top_k=30,
|
||||
do_sample=True
|
||||
)
|
||||
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||||
print(response)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Merge Details
|
||||
|
||||
Built using [mergekit](https://github.com/arcee-ai/mergekit) with **SLERP** (Spherical Linear Interpolation), which allows smooth interpolation between models while preserving geometric properties.
|
||||
|
||||
### Layer-Specific Weights
|
||||
|
||||
The merge uses a **graduated approach for MLP layers**, increasing lucid influence toward later layers where censorship was detected:
|
||||
|
||||
| Layer Zone (approx) | lucid weight (MLP) | Effect |
|
||||
|---------------------|-------------------|--------|
|
||||
| 0–8 | 0.1 | Almost pure Faun (early patterns) |
|
||||
| 8–16 | 0.2 | Slight lucid influence |
|
||||
| 16–24 | 0.5 | Balanced |
|
||||
| 24–32 | 0.4 | Slightly more Faun |
|
||||
| 32–40 | 0.75 | Lucid dominates — removes censorship |
|
||||
|
||||
Self-attention is mixed evenly (0.5) to preserve character while adding stability.
|
||||
LayerNorm is mixed 0.5 for overall stability.
|
||||
Reference in New Issue
Block a user