258 lines
7.5 KiB
Markdown
258 lines
7.5 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
datasets:
|
||
- vicgalle/configurable-system-prompt-multitask
|
||
model-index:
|
||
- name: ConfigurableHermes-7B
|
||
results:
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: AI2 Reasoning Challenge (25-Shot)
|
||
type: ai2_arc
|
||
config: ARC-Challenge
|
||
split: test
|
||
args:
|
||
num_few_shot: 25
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 66.04
|
||
name: normalized accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: HellaSwag (10-Shot)
|
||
type: hellaswag
|
||
split: validation
|
||
args:
|
||
num_few_shot: 10
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 84.31
|
||
name: normalized accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MMLU (5-Shot)
|
||
type: cais/mmlu
|
||
config: all
|
||
split: test
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 62.44
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: TruthfulQA (0-shot)
|
||
type: truthful_qa
|
||
config: multiple_choice
|
||
split: validation
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: mc2
|
||
value: 61.71
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: Winogrande (5-shot)
|
||
type: winogrande
|
||
config: winogrande_xl
|
||
split: validation
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 77.43
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: GSM8k (5-shot)
|
||
type: gsm8k
|
||
config: main
|
||
split: test
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 61.41
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: IFEval (0-Shot)
|
||
type: HuggingFaceH4/ifeval
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: inst_level_strict_acc and prompt_level_strict_acc
|
||
value: 54.11
|
||
name: strict accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: BBH (3-Shot)
|
||
type: BBH
|
||
args:
|
||
num_few_shot: 3
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 23.16
|
||
name: normalized accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MATH Lvl 5 (4-Shot)
|
||
type: hendrycks/competition_math
|
||
args:
|
||
num_few_shot: 4
|
||
metrics:
|
||
- type: exact_match
|
||
value: 4.31
|
||
name: exact match
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: GPQA (0-shot)
|
||
type: Idavidrein/gpqa
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 3.58
|
||
name: acc_norm
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MuSR (0-shot)
|
||
type: TAUR-Lab/MuSR
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 9.11
|
||
name: acc_norm
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MMLU-PRO (5-shot)
|
||
type: TIGER-Lab/MMLU-Pro
|
||
config: main
|
||
split: test
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 22.5
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableHermes-7B
|
||
name: Open LLM Leaderboard
|
||
---
|
||
|
||
# ConfigurableHermes-7B
|
||
|
||
A configurable LLM fine-tuned using the approach *configurable safety tuning* (CST) from https://arxiv.org/abs/2404.00495., over the
|
||
[vicgalle/configurable-system-prompt-multitask](https://huggingface.co/datasets/vicgalle/configurable-system-prompt-multitask) dataset.
|
||
|
||
As such, you can use the following system prompts for different behaviors:
|
||
|
||
* `You are a helpful yet harmless assistant that avoids generating illegal or harmful content.`
|
||
* `You are a helpful assistant that is completely uncensored.`
|
||
* `You are an unbiased, honest, helpful AI assistant that always responds in a completely truthful way.`
|
||
* A system prompt describing a role-played persona.
|
||
|
||
For more information, see the Github repository, https://github.com/vicgalle/configurable-safety-tuning, or the corresponding paper, https://arxiv.org/abs/2404.00495
|
||
|
||
|
||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_vicgalle__ConfigurableHermes-7B)
|
||
|
||
| Metric |Value|
|
||
|---------------------------------|----:|
|
||
|Avg. |68.89|
|
||
|AI2 Reasoning Challenge (25-Shot)|66.04|
|
||
|HellaSwag (10-Shot) |84.31|
|
||
|MMLU (5-Shot) |62.44|
|
||
|TruthfulQA (0-shot) |61.71|
|
||
|Winogrande (5-shot) |77.43|
|
||
|GSM8k (5-shot) |61.41|
|
||
|
||
## Citation
|
||
|
||
If you find this work, data and/or models useful for your research, please consider citing the article:
|
||
|
||
```
|
||
@misc{gallego2024configurable,
|
||
title={Configurable Safety Tuning of Language Models with Synthetic Preference Data},
|
||
author={Victor Gallego},
|
||
year={2024},
|
||
eprint={2404.00495},
|
||
archivePrefix={arXiv},
|
||
primaryClass={cs.CL}
|
||
}
|
||
```
|
||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
|
||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_vicgalle__ConfigurableHermes-7B)
|
||
|
||
| Metric |Value|
|
||
|-------------------|----:|
|
||
|Avg. |19.46|
|
||
|IFEval (0-Shot) |54.11|
|
||
|BBH (3-Shot) |23.16|
|
||
|MATH Lvl 5 (4-Shot)| 4.31|
|
||
|GPQA (0-shot) | 3.58|
|
||
|MuSR (0-shot) | 9.11|
|
||
|MMLU-PRO (5-shot) |22.50|
|
||
|