50 lines
1.2 KiB
Markdown
50 lines
1.2 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
tags:
|
||
|
|
- merge
|
||
|
|
library_name: transformers
|
||
|
|
---
|
||
|
|
|
||
|
|
# Model Card for Model ID
|
||
|
|
|
||
|
|
This is a mixture of experts created with [mergekit](https://github.com/cg123/mergekit) and based on [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).
|
||
|
|
## Model Details
|
||
|
|
|
||
|
|
The model was created using a recipe detailed in this article:
|
||
|
|
[The Mayonnaise: Rank First on the Open LLM Leaderboard with TIES-Merging
|
||
|
|
](https://kaitchup.substack.com/p/the-mayonnaise-rank-first-on-the)
|
||
|
|
|
||
|
|
### Model Description
|
||
|
|
|
||
|
|
<!-- Provide a longer summary of what this model is. -->
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
- **Developed by:** [The Kaitchup](https://kaitchup.substack.com/)
|
||
|
|
- **Model type:** Causal
|
||
|
|
- **Language(s) (NLP):** English
|
||
|
|
- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
|
||
|
|
|
||
|
|
### Model Sources
|
||
|
|
|
||
|
|
Created with mergekit with this configuration:
|
||
|
|
```
|
||
|
|
models:
|
||
|
|
- model: mncai/mistral-7b-dpo-v5
|
||
|
|
# no parameters necessary for base model
|
||
|
|
- model: flemmingmiguel/MBX-7B
|
||
|
|
parameters:
|
||
|
|
density: 0.3
|
||
|
|
weight: 0.3
|
||
|
|
- model: BarryFutureman/NeuralTurdusVariant1-7B
|
||
|
|
parameters:
|
||
|
|
density: 0.3
|
||
|
|
weight: 0.5
|
||
|
|
merge_method: ties
|
||
|
|
base_model: mncai/mistral-7b-dpo-v5
|
||
|
|
parameters:
|
||
|
|
normalize: true
|
||
|
|
dtype: float16
|
||
|
|
```
|