192 lines
6.2 KiB
Markdown
192 lines
6.2 KiB
Markdown
---
|
|
license: apache-2.0
|
|
library_name: transformers
|
|
tags:
|
|
- general-purpose
|
|
- roleplay
|
|
- storywriting
|
|
- merge
|
|
- finetune
|
|
base_model: mistralai/Mistral-Nemo-Base-2407
|
|
model-index:
|
|
- name: Chronos-Gold-12B-1.0
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: IFEval (0-Shot)
|
|
type: HuggingFaceH4/ifeval
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: inst_level_strict_acc and prompt_level_strict_acc
|
|
value: 31.66
|
|
name: strict accuracy
|
|
source:
|
|
url: >-
|
|
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: BBH (3-Shot)
|
|
type: BBH
|
|
args:
|
|
num_few_shot: 3
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 35.91
|
|
name: normalized accuracy
|
|
source:
|
|
url: >-
|
|
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MATH Lvl 5 (4-Shot)
|
|
type: hendrycks/competition_math
|
|
args:
|
|
num_few_shot: 4
|
|
metrics:
|
|
- type: exact_match
|
|
value: 4.38
|
|
name: exact match
|
|
source:
|
|
url: >-
|
|
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: GPQA (0-shot)
|
|
type: Idavidrein/gpqa
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 9.06
|
|
name: acc_norm
|
|
source:
|
|
url: >-
|
|
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MuSR (0-shot)
|
|
type: TAUR-Lab/MuSR
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 19.42
|
|
name: acc_norm
|
|
source:
|
|
url: >-
|
|
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MMLU-PRO (5-shot)
|
|
type: TIGER-Lab/MMLU-Pro
|
|
config: main
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 27.98
|
|
name: accuracy
|
|
source:
|
|
url: >-
|
|
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
|
|
name: Open LLM Leaderboard
|
|
---
|
|
|
|
# Chronos Gold 12B-1.0
|
|
|
|
|
|
|
|
Chronos Gold 12B 1.0 is a very unique model that applies to domain areas such as
|
|
general chatbot functionatliy, *roleplay*, and storywriting. The model has been observed to write up to 2250 tokens in a single sequence. The model was trained at a
|
|
sequence length of 16384 (16k) and will still retain the *apparent* 128k context length from Mistral-Nemo, though it deteriorates over time like regular Nemo does based on
|
|
the [RULER Test](https://github.com/hsiehjackson/RULER?tab=readme-ov-file#-ruler-whats-the-real-context-size-of-your-long-context-language-models)
|
|
|
|
As a result, is recommended to keep your sequence length max at 16384, or you will experience performance degredation.
|
|
|
|
The base model is `mistralai/Mistral-Nemo-Base-2407` which was heavily modified to produce a more coherent model, comparable to much larger models.
|
|
|
|
**Chronos Gold 12B-1.0** re-creates the uniqueness of the original Chronos with significiantly enhanced prompt adherence (following), coherence, a modern dataset, as well as supporting a majority of "character card" formats in applications like SillyTavern.
|
|
|
|
It went through an iterative and objective merge process as my previous models and was further finetuned on a dataset curated for it.
|
|
|
|
The specifics of the model will not be disclosed at the time due to dataset ownership.
|
|
|
|
## Instruct Template
|
|
|
|
This model uses `ChatML` - below is an example. It is a preset in many frontends.
|
|
|
|
```
|
|
<|im_start|>system
|
|
A system prompt describing how you'd like your bot to act.<|im_end|>
|
|
<|im_start|>user
|
|
Hello there!<|im_end|>
|
|
<|im_start|>assistant
|
|
I can assist you or we can discuss other things?<|im_end|>
|
|
<|im_start|>user
|
|
I was wondering how transformers work?<|im_end|>
|
|
<|im_start|>assistant
|
|
```
|
|
|
|
## Quantization
|
|
|
|
#### LlamaCPP
|
|
[GGUFs from @bartowski](https://huggingface.co/bartowski/Chronos-Gold-12B-1.0-GGUF)
|
|
|
|
#### Exllama2
|
|
[4.5bpw by @Pyroserenus](https://huggingface.co/Pyroserenus/Chronos-Gold-12B-1.0-4.5bpw-h6-exl2)
|
|
|
|
[5.5bpw by @Pyroserenus](https://huggingface.co/Pyroserenus/Chronos-Gold-12B-1.0-5.5bpw-h6-exl2)
|
|
|
|
[6.5bpw by @Pyroserenus](https://huggingface.co/Pyroserenus/Chronos-Gold-12B-1.0-6.5bpw-h6-exl2)
|
|
|
|
[8.0bpw by @Pyroserenus](https://huggingface.co/Pyroserenus/Chronos-Gold-12B-1.0-8.0bpw-h8-exl2)
|
|
|
|
#### FP8
|
|
[FP8 Quant by @Pyroserenus](https://huggingface.co/Pyroserenus/Chronos-Gold-12B-1.0-FP8)
|
|
|
|
## Sampling Settings
|
|
Nemo is a bit sensitive to high temperatures, so I use lower. Here are my settings:
|
|
```
|
|
Temp - 0.7 (0.9 max)
|
|
Presence Penalty - 1.0
|
|
Repetition Penalty range - 2800
|
|
Min P - 0.10
|
|
```
|
|
|
|
## Additional Details
|
|
This model was created by **elinas** on discord. Thank you to [@kalomaze](https://huggingface.co/kalomaze) for providing a model that made this merge possible.
|
|
|
|
This is one of multiple models to come out in the series by size and model architecture, so look forward to it!
|
|
|
|
Contact me on Discord for inquiries.
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_elinas__Chronos-Gold-12B-1.0)
|
|
|
|
| Metric |Value|
|
|
|-------------------|----:|
|
|
|Avg. |21.40|
|
|
|IFEval (0-Shot) |31.66|
|
|
|BBH (3-Shot) |35.91|
|
|
|MATH Lvl 5 (4-Shot)| 4.38|
|
|
|GPQA (0-shot) | 9.06|
|
|
|MuSR (0-shot) |19.42|
|
|
|MMLU-PRO (5-shot) |27.98| |