mixtral-megamerge-dare-8x7b-v1/README.md

license, language, pipeline_tag, inference, tags

license	language	pipeline_tag	inference	tags
apache-2.0	en	text-generation	false	dare super mario merge pytorch mixtral merge

mixtral megamerge 8x7b v1

The following models were merged with DARE using https://github.com/martyn/safetensors-merge-supermario

Mergelist

cognitivecomputations/dolphin-2.6-mixtral-8x7b
mistralai/Mixtral-8x7B-v0.1
mistralai/Mixtral-8x7B-Instruct-v0.1

Merge command

python hf_merge.py mergelist.txt mixtral-merge-1 -p 0.1 -lambda 1.95

Notes

• seems to generalize instruct styles
• MoE gates are not modified