nbeerbower/Lyra-Gutenberg-mistral-nemo-12B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
7460b9c5c6267aa00f7d5f4170631e5db6aa8db1
Lyra-Gutenberg-mistral-nemo…/README.md
ModelHub XC 7460b9c5c6 初始化项目，由ModelHub XC社区提供模型 
Model: nbeerbower/Lyra-Gutenberg-mistral-nemo-12B
Source: Original Platform
2026-10-18 01:20:12 +08:00

3.6 KiB
Raw Blame History

license, library_name, base_model, datasets, model-index
license library_name base_model datasets model-index
cc-by-nc-4.0 transformers
Sao10K/MN-12B-Lyra-v1
jondurbin/gutenberg-dpo-v0.1
name results
Lyra-Gutenberg-mistral-nemo-12B
task dataset metrics source
type name
text-generation Text Generation
name type args
IFEval (0-Shot) HuggingFaceH4/ifeval
num_few_shot
0
type value name
inst_level_strict_acc and prompt_level_strict_acc 34.95 strict accuracy
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Lyra-Gutenberg-mistral-nemo-12B Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type args
BBH (3-Shot) BBH
num_few_shot
3
type value name
acc_norm 36.99 normalized accuracy
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Lyra-Gutenberg-mistral-nemo-12B Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type args
MATH Lvl 5 (4-Shot) hendrycks/competition_math
num_few_shot
4
type value name
exact_match 8.31 exact match
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Lyra-Gutenberg-mistral-nemo-12B Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type args
GPQA (0-shot) Idavidrein/gpqa
num_few_shot
0
type value name
acc_norm 11.19 acc_norm
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Lyra-Gutenberg-mistral-nemo-12B Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type args
MuSR (0-shot) TAUR-Lab/MuSR
num_few_shot
0
type value name
acc_norm 14.76 acc_norm
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Lyra-Gutenberg-mistral-nemo-12B Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type config split args
MMLU-PRO (5-shot) TIGER-Lab/MMLU-Pro main test
num_few_shot
5
type value name
acc 29.2 accuracy
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Lyra-Gutenberg-mistral-nemo-12B Open LLM Leaderboard

Lyra-Gutenberg-12B

Sao10K/MN-12B-Lyra-v1 finetuned on jondurbin/gutenberg-dpo-v0.1.

Method

Finetuned using an A100 on Google Colab for 3 epochs.

Fine-tune Llama 3 with ORPO

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 22.57
IFEval (0-Shot) 34.95
BBH (3-Shot) 36.99
MATH Lvl 5 (4-Shot) 8.31
GPQA (0-shot) 11.19
MuSR (0-shot) 14.76
MMLU-PRO (5-shot) 29.20