license, library_name, tags, base_model, model-index

license	library_name	tags	base_model	model-index
apache-2.0	transformers	general-purpose roleplay storywriting merge finetune	mistralai/Mistral-Nemo-Base-2407	name results Chronos-Gold-12B-1.0 task dataset metrics source type name text-generation Text Generation name type args IFEval (0-Shot) HuggingFaceH4/ifeval num_few_shot 0 type value name inst_level_strict_acc and prompt_level_strict_acc 31.66 strict accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args BBH (3-Shot) BBH num_few_shot 3 type value name acc_norm 35.91 normalized accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MATH Lvl 5 (4-Shot) hendrycks/competition_math num_few_shot 4 type value name exact_match 4.38 exact match url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args GPQA (0-shot) Idavidrein/gpqa num_few_shot 0 type value name acc_norm 9.06 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MuSR (0-shot) TAUR-Lab/MuSR num_few_shot 0 type value name acc_norm 19.42 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args MMLU-PRO (5-shot) TIGER-Lab/MMLU-Pro main test num_few_shot 5 type value name acc 27.98 accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0 Open LLM Leaderboard

Chronos Gold 12B-1.0

Chronos Gold 12B 1.0 is a very unique model that applies to domain areas such as general chatbot functionatliy, roleplay, and storywriting. The model has been observed to write up to 2250 tokens in a single sequence. The model was trained at a sequence length of 16384 (16k) and will still retain the apparent 128k context length from Mistral-Nemo, though it deteriorates over time like regular Nemo does based on the RULER Test

As a result, is recommended to keep your sequence length max at 16384, or you will experience performance degredation.

The base model is mistralai/Mistral-Nemo-Base-2407 which was heavily modified to produce a more coherent model, comparable to much larger models.

Chronos Gold 12B-1.0 re-creates the uniqueness of the original Chronos with significiantly enhanced prompt adherence (following), coherence, a modern dataset, as well as supporting a majority of "character card" formats in applications like SillyTavern.

It went through an iterative and objective merge process as my previous models and was further finetuned on a dataset curated for it.

The specifics of the model will not be disclosed at the time due to dataset ownership.

Instruct Template

This model uses ChatML - below is an example. It is a preset in many frontends.

<|im_start|>system
A system prompt describing how you'd like your bot to act.<|im_end|>
<|im_start|>user
Hello there!<|im_end|>
<|im_start|>assistant
I can assist you or we can discuss other things?<|im_end|>
<|im_start|>user
I was wondering how transformers work?<|im_end|>
<|im_start|>assistant

Quantization

LlamaCPP

GGUFs from @bartowski

Exllama2

4.5bpw by @Pyroserenus

5.5bpw by @Pyroserenus

6.5bpw by @Pyroserenus

8.0bpw by @Pyroserenus

FP8

FP8 Quant by @Pyroserenus

Sampling Settings

Nemo is a bit sensitive to high temperatures, so I use lower. Here are my settings:

Temp - 0.7 (0.9 max)
Presence Penalty - 1.0
Repetition Penalty range - 2800
Min P - 0.10

Additional Details

This model was created by elinas on discord. Thank you to @kalomaze for providing a model that made this merge possible.

This is one of multiple models to come out in the series by size and model architecture, so look forward to it!

Contact me on Discord for inquiries.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	21.40
IFEval (0-Shot)	31.66
BBH (3-Shot)	35.91
MATH Lvl 5 (4-Shot)	4.38
GPQA (0-shot)	9.06
MuSR (0-shot)	19.42
MMLU-PRO (5-shot)	27.98