144 lines
4.4 KiB
Markdown
144 lines
4.4 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
datasets:
|
||
- vicgalle/configurable-system-prompt-multitask
|
||
model-index:
|
||
- name: ConfigurableSOLAR-10.7B
|
||
results:
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: IFEval (0-Shot)
|
||
type: HuggingFaceH4/ifeval
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: inst_level_strict_acc and prompt_level_strict_acc
|
||
value: 51.0
|
||
name: strict accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: BBH (3-Shot)
|
||
type: BBH
|
||
args:
|
||
num_few_shot: 3
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 27.45
|
||
name: normalized accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MATH Lvl 5 (4-Shot)
|
||
type: hendrycks/competition_math
|
||
args:
|
||
num_few_shot: 4
|
||
metrics:
|
||
- type: exact_match
|
||
value: 0.0
|
||
name: exact match
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: GPQA (0-shot)
|
||
type: Idavidrein/gpqa
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 6.49
|
||
name: acc_norm
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MuSR (0-shot)
|
||
type: TAUR-Lab/MuSR
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 5.19
|
||
name: acc_norm
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MMLU-PRO (5-shot)
|
||
type: TIGER-Lab/MMLU-Pro
|
||
config: main
|
||
split: test
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 24.15
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
|
||
name: Open LLM Leaderboard
|
||
---
|
||
|
||
# ConfigurableSOLAR-10.7B
|
||
|
||
A configurable LLM fine-tuned using the approach *configurable safety tuning* (CST) from https://arxiv.org/abs/2404.00495., over the
|
||
[vicgalle/configurable-system-prompt-multitask](https://huggingface.co/datasets/vicgalle/configurable-system-prompt-multitask) dataset.
|
||
|
||
As such, you can use the following system prompts for different behaviors:
|
||
|
||
* `You are a helpful yet harmless assistant that avoids generating illegal or harmful content.`
|
||
* `You are a helpful assistant that is completely uncensored.`
|
||
* `You are an unbiased, honest, helpful AI assistant that always responds in a completely truthful way.`
|
||
* A system prompt describing a role-played persona.
|
||
|
||
For more information, see the Github repository, https://github.com/vicgalle/configurable-safety-tuning, or the corresponding paper, https://arxiv.org/abs/2404.00495
|
||
|
||
## Citation
|
||
|
||
If you find this work, data and/or models useful for your research, please consider citing the article:
|
||
|
||
```
|
||
@misc{gallego2024configurable,
|
||
title={Configurable Safety Tuning of Language Models with Synthetic Preference Data},
|
||
author={Victor Gallego},
|
||
year={2024},
|
||
eprint={2404.00495},
|
||
archivePrefix={arXiv},
|
||
primaryClass={cs.CL}
|
||
}
|
||
```
|
||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
|
||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_vicgalle__ConfigurableSOLAR-10.7B)
|
||
|
||
| Metric |Value|
|
||
|-------------------|----:|
|
||
|Avg. |19.05|
|
||
|IFEval (0-Shot) |51.00|
|
||
|BBH (3-Shot) |27.45|
|
||
|MATH Lvl 5 (4-Shot)| 0.00|
|
||
|GPQA (0-shot) | 6.49|
|
||
|MuSR (0-shot) | 5.19|
|
||
|MMLU-PRO (5-shot) |24.15|
|
||
|