230 lines
9.2 KiB
Markdown
230 lines
9.2 KiB
Markdown
---
|
|
base_model:
|
|
- mistralai/Mistral-7B-v0.1
|
|
- argilla/distilabeled-OpenHermes-2.5-Mistral-7B
|
|
- NeverSleep/Noromaid-7B-0.4-DPO
|
|
- senseable/WestLake-7B-v2
|
|
- mlabonne/AlphaMonarch-7B
|
|
library_name: transformers
|
|
tags:
|
|
- mergekit
|
|
- merge
|
|
license: cc-by-nc-4.0
|
|
model-index:
|
|
- name: WestLake_Noromaid_OpenHermes_neural-chatv0.1
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: EQ-Bench
|
|
type: eq-bench
|
|
config: EQ-Bench
|
|
split: v2.1
|
|
args:
|
|
num_few_shot: 3
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 77.19
|
|
name: self-reported
|
|
source:
|
|
url: https://github.com/EQ-bench/EQ-Bench
|
|
name: EQ-Bench v2.1
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: AI2 Reasoning Challenge (25-Shot)
|
|
type: ai2_arc
|
|
config: ARC-Challenge
|
|
split: test
|
|
args:
|
|
num_few_shot: 25
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 70.22
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=giraffe176/WestMaid_HermesMonarchv0.1
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: HellaSwag (10-Shot)
|
|
type: hellaswag
|
|
split: validation
|
|
args:
|
|
num_few_shot: 10
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 87.42
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=giraffe176/WestMaid_HermesMonarchv0.1
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MMLU (5-Shot)
|
|
type: cais/mmlu
|
|
config: all
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 64.31
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=giraffe176/WestMaid_HermesMonarchv0.1
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: TruthfulQA (0-shot)
|
|
type: truthful_qa
|
|
config: multiple_choice
|
|
split: validation
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: mc2
|
|
value: 61.99
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=giraffe176/WestMaid_HermesMonarchv0.1
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: Winogrande (5-shot)
|
|
type: winogrande
|
|
config: winogrande_xl
|
|
split: validation
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 82.16
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=giraffe176/WestMaid_HermesMonarchv0.1
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: GSM8k (5-shot)
|
|
type: gsm8k
|
|
config: main
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 69.6
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=giraffe176/WestMaid_HermesMonarchv0.1
|
|
name: Open LLM Leaderboard
|
|
---
|
|
# WestMaid_HermesMonarchv0.1
|
|
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/655a9883cbbaec115c3fd6b3/YJTMJZF80hKaKnPDu_yMV.png" alt="drawing" width="800"/>
|
|
|
|
This model benchmarks quite well compared to other 7b models, and has exceptional [MT-Bench](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge) and [EQ-Bench v2.1](https://github.com/EQ-bench/EQ-Bench) scores, ranking higher than ChatGPT-3.5-turbo and Claude-1 in both tests, and Goliath-120b, and other 70B models in the latter .
|
|
|
|
This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit)
|
|
|
|
## Merge Details
|
|
### Merge Method
|
|
|
|
This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method using [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) as a base.
|
|
Density was chosen deterministically between the models chosen for this merge. After testing many densities, I settled on 0.58 for each of the chosen models as it returned the highest EQ-Bench score. Not much testing was done with the weights, but I thought that I'd try gradients. Conceptually, Westlake and a Distilled version of Open Heremes are heavier in the initial layers (guiding understanding, and thoughts), before Noromaid and AlphaMonarch come in to guide its wants, reasoning, and conversation.
|
|
|
|
|
|
|
|
### Models Merged
|
|
|
|
The following models were included in the merge:
|
|
* [mlabonne/AlphaMonarch-7B](https://huggingface.co/mlabonne/AlphaMonarch-7B)
|
|
* [NeverSleep/Noromaid-7B-0.4-DPO](https://huggingface.co/NeverSleep/Noromaid-7B-0.4-DPO)
|
|
* [senseable/WestLake-7B-v2](https://huggingface.co/senseable/WestLake-7B-v2)
|
|
* [argilla/distilabeled-OpenHermes-2.5-Mistral-7B](https://huggingface.co/argilla/distilabeled-OpenHermes-2.5-Mistral-7B)
|
|
|
|
### Configuration
|
|
|
|
The following YAML configuration was used to produce this model:
|
|
|
|
```yaml
|
|
models:
|
|
- model: mistralai/Mistral-7B-v0.1
|
|
# No parameters necessary for base model
|
|
- model: senseable/WestLake-7B-v2
|
|
parameters:
|
|
density: 0.58
|
|
weight: [0.50, 0.40, 0.25, 0.05]
|
|
- model: NeverSleep/Noromaid-7B-0.4-DPO
|
|
parameters:
|
|
density: 0.58
|
|
weight: [0.05, 0.05, 0.25, 0.40]
|
|
- model: argilla/distilabeled-OpenHermes-2.5-Mistral-7B
|
|
parameters:
|
|
density: 0.58
|
|
weight: [0.40, 0.50, 0.25, 0.05]
|
|
- model: mlabonne/AlphaMonarch-7B
|
|
parameters:
|
|
density: 0.58
|
|
weight: [0.05, 0.05, 0.25, 0.50]
|
|
merge_method: dare_ties
|
|
base_model: mistralai/Mistral-7B-v0.1
|
|
parameters:
|
|
int8_mask: true
|
|
dtype: bfloat16
|
|
|
|
```
|
|
## Benchmark Testing
|
|
### MT-Bench
|
|
|
|
|
|
### EQ-Bench Leaderboard
|
|
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/655a9883cbbaec115c3fd6b3/0Z6AIhaqCiKREf0fQEVqr.png" alt="drawing" width="800"/>
|
|
|
|
|
|
### Table of Benchmarks
|
|
|
|
## Open LLM Leaderboard
|
|
|
|
| | Average | ARC | HellaSwag | MMLU | TruthfulQA | Winogrande | GSM8K |
|
|
|---------------------------------------------------------|---------|-------|-----------|-------|------------|------------|-------|
|
|
| giraffe176/WestMaid_HermesMonarchv0.1 | 72.62 | 70.22 | 87.42 | 64.31 | 61.99 | 82.16 | 69.6 |
|
|
| AlphaMonarch-7B | 75.99 | 73.04 | 89.18 | 64.4 | 77.91 | 84.69 | 66.72 |
|
|
| senseable/WestLake-7B-v2 | 74.68 | 73.04 | 88.65 | 64.71 | 67.06 | 86.98 | 67.63 |
|
|
| teknium/OpenHermes-2.5-Mistral-7B | 61.52 | 64.93 | 84.18 | 63.64 | 52.24 | 78.06 | 26.08 |
|
|
| NeverSleep/Noromaid-7B-0.4-DPO | 59.08 | 62.29 | 84.32 | 63.2 | 42.28 | 76.95 | 25.47 |
|
|
|
|
|
|
|
|
## Yet Another LLM Leaderboard benchmarks
|
|
|
|
| Model |AGIEval|GPT4All|TruthfulQA|Bigbench|Average|
|
|
|------------------------------------------------------------------------------------------|------:|------:|---------:|-------:|------:|
|
|
|[WestMaid_HermesMonarchv0.1](https://huggingface.co/giraffe176/WestMaid_HermesMonarchv0.1)| 45.34| 76.33| 61.99| 46.02| 57.42|
|
|
|
|
## Misc. Benchmarks
|
|
|
|
| | MT-Bench | EQ-Bench v2.1 |
|
|
|---------------------------------------------------------|---------------------------------------------|---------------------------------------------------------------------------------|
|
|
| giraffe176/WestMaid_HermesMonarchv0.1 | 8.021875 | 77.19 (3 Shot, ooba) |
|
|
| AlphaMonarch-7B | 7.928125 | 76.08 |
|
|
| senseable/WestLake-7B-v2 | | 78.7 |
|
|
| teknium/OpenHermes-2.5-Mistral-7B | | 66.89 |
|
|
| claude-v1 | 7.900000 | 76.83 |
|
|
| gpt-3.5-turbo | 7.943750 | 71.74 |
|
|
| | [(Paper)](https://arxiv.org/abs/2306.05685) | [(Paper)](https://arxiv.org/abs/2312.06281) [Leaderboard](https://eqbench.com/) |
|
|
|