license, library_name, base_model, datasets, model-index

license	library_name	base_model	datasets	model-index
other	transformers	meta-llama/Meta-Llama-3-8B	mlabonne/orpo-dpo-mix-40k Open-Orca/SlimOrca-Dedup jondurbin/airoboros-3.2 microsoft/orca-math-word-problems-200k m-a-p/Code-Feedback MaziyarPanahi/WizardLM_evol_instruct_V2_196k	name results llama-3-neural-chat-v1-8b task dataset metrics source type name text-generation Text Generation name type config split args AI2 Reasoning Challenge (25-Shot) ai2_arc ARC-Challenge test num_few_shot 25 type value name acc_norm 60.84 normalized accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args HellaSwag (10-Shot) hellaswag validation num_few_shot 10 type value name acc_norm 84.13 normalized accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args MMLU (5-Shot) cais/mmlu all test num_few_shot 5 type value name acc 64.69 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args TruthfulQA (0-shot) truthful_qa multiple_choice validation num_few_shot 0 type value mc2 56.34 url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args Winogrande (5-shot) winogrande winogrande_xl validation num_few_shot 5 type value name acc 78.22 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args GSM8k (5-shot) gsm8k main test num_few_shot 5 type value name acc 54.81 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b Open LLM Leaderboard

llama-3-neural-chat-v1-8b

Model Details

Model Description

I fine-tuned llama-3 8B on an approach similar to Intel's neural chat language model. I have slightly modified the data sources so it is stronger in coding, math, and writing. I use both SFT and DPO.

• Developed by: Locutusque
• Model type: Built with Meta Llama 3
• Language(s) (NLP): Many?
• License: Llama 3 license https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/LICENSE

Quants

EXL2 @bartowski

• https://huggingface.co/bartowski/llama-3-neural-chat-v1-8b-exl2

GGUF @bartowski

• https://huggingface.co/bartowski/llama-3-neural-chat-v1-8b-GGUF

Uses

This model has great performance in writing and coding.

Training Data

• Open-Orca/SlimOrca-Dedup
• jondurbin/airoboros-3.2
• microsoft/orca-math-word-problems-200k
• m-a-p/Code-Feedback
• MaziyarPanahi/WizardLM_evol_instruct_V2_196k
• mlabonne/orpo-dpo-mix-40k

Direct Use

Conversational AI.

Evaluations

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
truthfulqa_mc2	2	none	0	acc	0.5627	±	0.0154
gsm8k	3	strict-match	5	exact_match	0.5481	±	0.0137
		flexible-extract	5	exact_match	0.5557	±	0.0137
agieval_nous	N/A	none	0	acc	0.3763	±	0.0093
		none	0	acc_norm	0.3665	±	0.0093
- agieval_aqua_rat	1	none	0	acc	0.2087	±	0.0255
		none	0	acc_norm	0.2047	±	0.0254
- agieval_logiqa_en	1	none	0	acc	0.3456	±	0.0187
		none	0	acc_norm	0.3594	±	0.0188
- agieval_lsat_ar	1	none	0	acc	0.1826	±	0.0255
		none	0	acc_norm	0.1783	±	0.0253
- agieval_lsat_lr	1	none	0	acc	0.3549	±	0.0212
		none	0	acc_norm	0.3451	±	0.0211
- agieval_lsat_rc	1	none	0	acc	0.5242	±	0.0305
		none	0	acc_norm	0.5130	±	0.0305
- agieval_sat_en	1	none	0	acc	0.6650	±	0.0330
		none	0	acc_norm	0.6505	±	0.0333
- agieval_sat_en_without_passage	1	none	0	acc	0.4175	±	0.0344
		none	0	acc_norm	0.3738	±	0.0338
- agieval_sat_math	1	none	0	acc	0.4227	±	0.0334
		none	0	acc_norm	0.3682	±	0.0326

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	66.50
AI2 Reasoning Challenge (25-Shot)	60.84
HellaSwag (10-Shot)	84.13
MMLU (5-Shot)	64.69
TruthfulQA (0-shot)	56.34
Winogrande (5-shot)	78.22
GSM8k (5-shot)	54.81