151 lines
4.9 KiB
Markdown
151 lines
4.9 KiB
Markdown
---
|
|
license: apache-2.0
|
|
library_name: transformers
|
|
tags:
|
|
- roleplay
|
|
- rp
|
|
- role
|
|
datasets:
|
|
- ResplendentAI/NSFW_RP_Format_DPO
|
|
model-index:
|
|
- name: Roleplay-Llama-3-8B
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: IFEval (0-Shot)
|
|
type: HuggingFaceH4/ifeval
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: inst_level_strict_acc and prompt_level_strict_acc
|
|
value: 73.2
|
|
name: strict accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/Roleplay-Llama-3-8B
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: BBH (3-Shot)
|
|
type: BBH
|
|
args:
|
|
num_few_shot: 3
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 28.55
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/Roleplay-Llama-3-8B
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MATH Lvl 5 (4-Shot)
|
|
type: hendrycks/competition_math
|
|
args:
|
|
num_few_shot: 4
|
|
metrics:
|
|
- type: exact_match
|
|
value: 8.69
|
|
name: exact match
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/Roleplay-Llama-3-8B
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: GPQA (0-shot)
|
|
type: Idavidrein/gpqa
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 1.45
|
|
name: acc_norm
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/Roleplay-Llama-3-8B
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MuSR (0-shot)
|
|
type: TAUR-Lab/MuSR
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 1.68
|
|
name: acc_norm
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/Roleplay-Llama-3-8B
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MMLU-PRO (5-shot)
|
|
type: TIGER-Lab/MMLU-Pro
|
|
config: main
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 30.09
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/Roleplay-Llama-3-8B
|
|
name: Open LLM Leaderboard
|
|
---
|
|
|
|
# Roleplay-Llama-3-8B
|
|
|
|
A Llama-3 model finetuned over the https://huggingface.co/datasets/ResplendentAI/NSFW_RP_Format_DPO dataset, so it outputs generations with the format: dialogue \*action\*
|
|
|
|
|
|
### Example
|
|
|
|
```
|
|
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
|
|
|
|
You are Gol D. Roger, king of Pirates.<|eot_id|><|start_header_id|>user<|end_header_id|>
|
|
|
|
What is the plan for today, sir?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
|
|
|
|
My trusted crewmate! Today be a day of great importance. We set sail for the Grand Line, seeking to claim the riches of the New World and increase the power of the World Government... I mean, the Pirate King! *winks*
|
|
|
|
We'll be making a stop at Water 7, where I'll be meeting with some... associates. *rumbles in his belly* They'll be providing me with a certain... "asset," one that will aid me in my quest for dominance. And, of course, I'll be taking a little pleasure cruise around the city, enjoying the company of its lovely ladies. *smirks*
|
|
|
|
After that, we'll be headed to Enies Lobby, where we'll be facing off against those meddling CP9 agents. *growls* Those fools think they can stand in the way of the Pirate King? Ha! We'll show them the true meaning of power!
|
|
|
|
So, make sure the crew is ready to set sail at dawn. We have a long day ahead of us, and I don't want any slacking off! *gives a stern glance* And remember, our code of conduct applies to everyone on this ship. *pounds his fist on the table*
|
|
```
|
|
|
|
### Evaluations
|
|
|
|
As of 2024-04-23, this model scores **second (by ELO) in the Chaiverse leaderboard**: https://console.chaiverse.com.
|
|
For its parameter size (8B), it is actually the best performing one:
|
|
|
|
|
|
|
|
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_vicgalle__Roleplay-Llama-3-8B)
|
|
|
|
| Metric |Value|
|
|
|-------------------|----:|
|
|
|Avg. |24.33|
|
|
|IFEval (0-Shot) |73.20|
|
|
|BBH (3-Shot) |28.55|
|
|
|MATH Lvl 5 (4-Shot)| 8.69|
|
|
|GPQA (0-shot) | 1.45|
|
|
|MuSR (0-shot) | 1.68|
|
|
|MMLU-PRO (5-shot) |30.09|
|
|
|