StarDust-12b-v2/README.md

language, license, tags, base_model, pipeline_tag, model-index, library_name

language	license	tags	base_model	pipeline_tag	model-index	library_name
en	apache-2.0	chat mistral roleplay creative-writing	nbeerbower/mistral-nemo-bophades-12B anthracite-org/magnum-v2-12b Sao10K/MN-12B-Lyra-v3 Gryphe/Pantheon-RP-1.6-12b-Nemo	text-generation	name results StarDust-12b-v2 task dataset metrics source type name text-generation Text Generation name type args IFEval (0-Shot) HuggingFaceH4/ifeval num_few_shot 0 type value name inst_level_strict_acc and prompt_level_strict_acc 56.29 strict accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Luni/StarDust-12b-v2 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args BBH (3-Shot) BBH num_few_shot 3 type value name acc_norm 34.95 normalized accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Luni/StarDust-12b-v2 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MATH Lvl 5 (4-Shot) hendrycks/competition_math num_few_shot 4 type value name exact_match 5.97 exact match url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Luni/StarDust-12b-v2 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args GPQA (0-shot) Idavidrein/gpqa num_few_shot 0 type value name acc_norm 5.82 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Luni/StarDust-12b-v2 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MuSR (0-shot) TAUR-Lab/MuSR num_few_shot 0 type value name acc_norm 14.26 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Luni/StarDust-12b-v2 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args MMLU-PRO (5-shot) TIGER-Lab/MMLU-Pro main test num_few_shot 5 type value name acc 27.1 accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Luni/StarDust-12b-v2 Open LLM Leaderboard	transformers

StarDust-12b-v2

Quants

• GGUF: mradermacher/StarDust-12b-v2-GGUF
• weighted/imatrix GGUF: mradermacher/StarDust-12b-v2-i1-GGUF
• exl2: lucyknada/Luni_StarDust-12b-v2-exl2

Description | Usecase

• The result of this merge is in my opinion a more vibrant and less generic sonnet inspired prose, it's able to be gentle and harsh where asked.
• The v2 uses the non-kto magnum which tends to have less "claudeism" (making the story feel rather repetitive)
• Note on Non-Kto: There is a very big gap between people preferring and disliking the KTO. To make things easier, you can still use Luni/StarDust-12b-v1 which has the KTO version.
• In early testing users have reported a much better experience in longer roleplays and a abillity to add a creative touch to the stable experience.

Just like with v1:

• This model is intended to be used as a Role-playing model.
• Its direct conversational output is... I can't even say it's luck, it's just not made for it.
• Extension to Conversational output: The Model is designed for roleplay, direct instructing or general purpose is NOT recommended.
• It can be used with simple setup to be used as an "Agent" by forcing it into an "impersonation" of a character which works very well but sometimes narratives sneak in.

Initial Feedback

• Initial feedback has proven the model to be a solid "go-to" choice for creative storywriting
• The prose has been certified as "amazing" with many making it their default model.

Prompting

ChatML has proven to be the BEST choice.

Both Mistral and ChatML should work though I had better results with ChatML: ChatML Example:

"""<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
<|im_start|>assistant
"""

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using Sao10K/MN-12B-Lyra-v3 as a base.

Models Merged

The following models were included in the merge:

• nbeerbower/mistral-nemo-bophades-12B
• anthracite-org/magnum-v2-12b
• Gryphe/Pantheon-RP-1.6-12b-Nemo
• Sao10K/MN-12B-Lyra-v3

Special Thanks

Special thanks to the SillyTilly and myself for helping me find the energy to finish this.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	24.06
IFEval (0-Shot)	56.29
BBH (3-Shot)	34.95
MATH Lvl 5 (4-Shot)	5.97
GPQA (0-shot)	5.82
MuSR (0-shot)	14.26
MMLU-PRO (5-shot)	27.10