Tiny-Vicuna-1B/README.md

language, license, model-index

language	license	model-index
en	apache-2.0	name results Tiny-Vicuna-1B task dataset metrics source type name text-generation Text Generation name type config split args AI2 Reasoning Challenge (25-Shot) ai2_arc ARC-Challenge test num_few_shot 25 type value name acc_norm 33.45 normalized accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Jiayi-Pan/Tiny-Vicuna-1B Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args HellaSwag (10-Shot) hellaswag validation num_few_shot 10 type value name acc_norm 55.92 normalized accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Jiayi-Pan/Tiny-Vicuna-1B Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args MMLU (5-Shot) cais/mmlu all test num_few_shot 5 type value name acc 25.45 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Jiayi-Pan/Tiny-Vicuna-1B Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args TruthfulQA (0-shot) truthful_qa multiple_choice validation num_few_shot 0 type value mc2 33.82 url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Jiayi-Pan/Tiny-Vicuna-1B Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args Winogrande (5-shot) winogrande winogrande_xl validation num_few_shot 5 type value name acc 58.41 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Jiayi-Pan/Tiny-Vicuna-1B Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args GSM8k (5-shot) gsm8k main test num_few_shot 5 type value name acc 1.52 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Jiayi-Pan/Tiny-Vicuna-1B Open LLM Leaderboard

Tiny Vicuna 1B

This model is a fine-tuned version of TinyLlama on WizardVicuna Dataset. It should be fully compatible with Vicuna-v1.5 series.

This model is easy to iterate on for early experiments!

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	34.76
AI2 Reasoning Challenge (25-Shot)	33.45
HellaSwag (10-Shot)	55.92
MMLU (5-Shot)	25.45
TruthfulQA (0-shot)	33.82
Winogrande (5-shot)	58.41
GSM8k (5-shot)	1.52