74 lines
2.5 KiB
Markdown
74 lines
2.5 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
base_model:
|
||
|
|
- OccultAI/Morpheus-8B-v1
|
||
|
|
- SicariusSicariiStuff/Assistant_Pepe_8B
|
||
|
|
tags:
|
||
|
|
- merge
|
||
|
|
- llama
|
||
|
|
- occult
|
||
|
|
- uncensored
|
||
|
|
- moe
|
||
|
|
datasets:
|
||
|
|
- OccultAI/Morpheus_665
|
||
|
|
- SicariusSicariiStuff/UBW_Tapestries
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
library_name: transformers
|
||
|
|
widget:
|
||
|
|
- text: "Meme-Trix MoE 14B A8B v1"
|
||
|
|
output:
|
||
|
|
url: https://cdn-uploads.huggingface.co/production/uploads/68e840caa318194c44ec2a04/hEyahuR43IMF5TC4XrHGS.png
|
||
|
|
new_version: Naphula/Meme-Trix-MoE-14B-A8B-v2
|
||
|
|
---
|
||
|
|
|
||
|
|
> [!CAUTION]
|
||
|
|
> <span style="color:red; font-weight:bold">⚠️ Warning:</span> This model can produce narratives and RP that contain violent and graphic erotic content. Adjust your system prompt accordingly, and use **Llama 3** chat template.
|
||
|
|
>
|
||
|
|
|
||
|
|
# Meme-Trix MoE 14B A8B v1
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
A custom built Llama 3.1 8B MoE (Mixture of Experts) merge which combines Morpheus v1 with Assistant Pepe. The merge is suprisingly intelligent, detailed, and based. Scores ~15K at Q0 Bench. Fully uncensored and almost as fast as 8B dense. It also appears to have strong context retrieval ability. Asked for a summary at 16K and it works flawlessly (did not test higher yet).
|
||
|
|
|
||
|
|
If you want to merge custom Llama MoE you can add these scripts to your mergekit environment:
|
||
|
|
- [mergekit-main\mergekit\architecture\moe_defs.py](https://huggingface.co/Naphula/Meme-Trix-MoE-14B-A8B-v1/blob/main/moe_defs.py)
|
||
|
|
- [mergekit-main\mergekit\moe\__init__.py](https://huggingface.co/Naphula/Meme-Trix-MoE-14B-A8B-v1/blob/main/__init__.py)
|
||
|
|
- [mergekit-main\mergekit\moe\llama.py](https://huggingface.co/Naphula/Meme-Trix-MoE-14B-A8B-v1/blob/main/llama.py)
|
||
|
|
|
||
|
|
Then assign the `num_experts_per_tok` in `config.json` (or the `config.yaml`)
|
||
|
|
|
||
|
|
## Recommended Settings
|
||
|
|
**(bolded kobold non-defaults)**
|
||
|
|
- **Temp 1.0**
|
||
|
|
- **TopNSigma 1.25**
|
||
|
|
- **Min-P 0.1**
|
||
|
|
- **Repetition Penalty 1.08**
|
||
|
|
- **Top-P 1.0**
|
||
|
|
- Top-K 100
|
||
|
|
- Top-A 0
|
||
|
|
- Typical Sampling 1
|
||
|
|
- Tail-Free Sampling 1
|
||
|
|
- Presence Penalty 0
|
||
|
|
- Sampler Seed -1
|
||
|
|
- Rp.Range 360
|
||
|
|
- Rp.Slope 0.7
|
||
|
|
- Smoothing Factor 0
|
||
|
|
- Smoothing Curve 1
|
||
|
|
- DynaTemp 0
|
||
|
|
- Mirostat Mode OFF ("2" enhances creativity but also errors)
|
||
|
|
- Mirostat Tau 5
|
||
|
|
- Mirostat Eta 0.1
|
||
|
|
- **DRY Multiplier 0.8**
|
||
|
|
- DRY Base 1.75
|
||
|
|
- DRY A.Len 2
|
||
|
|
- DRY L.Len 320
|
||
|
|
- **XTC Threshold 0.1**
|
||
|
|
- **XTC Probability 0.08** (The "Anti-Cliche" Shield)
|
||
|
|
- **DynaTemp ON** (The "Poor Man's Fading Mirostat")
|
||
|
|
- **Minimum Temperature 0.65**
|
||
|
|
- **Maximum Temperature 1.35**
|
||
|
|
- **Temperature 1.0**
|
||
|
|
- **DynaTemp-Range 0.35**
|
||
|
|
- **DynaTemp-Exponent 1**
|