smol_llama-220M-GQA/README.md

language, license, tags, datasets, inference, widget, pipeline_tag, model-index

language	license	tags	datasets	inference	widget	pipeline_tag	model-index
en	apache-2.0	smol_llama llama2	JeanKaddour/minipile pszemraj/simple_wikipedia_LM mattymchen/refinedweb-3m BEE-spoke-data/knowledge-inoc-concat-v1	parameters max_new_tokens do_sample temperature repetition_penalty no_repeat_ngram_size eta_cutoff renormalize_logits 64 true 0.8 1.05 4 0.0006 true	text example_title My name is El Microondas the Wise, and El Microondas text example_title Kennesaw State University is a public Kennesaw State University text example_title Bungie Studios is an American video game developer. They are most famous for developing the award winning Halo series of video games. They also made Destiny. The studio was founded Bungie text example_title The Mona Lisa is a world-renowned painting created by Mona Lisa text example_title The Harry Potter series, written by J.K. Rowling, begins with the book titled Harry Potter Series text example_title Question: I have cities, but no houses. I have mountains, but no trees. I have water, but no fish. What am I? Answer: Riddle text example_title The process of photosynthesis involves the conversion of Photosynthesis text example_title Jane went to the store to buy some groceries. She picked up apples, oranges, and a loaf of bread. When she got home, she realized she forgot Story Continuation text example_title Problem 2: If a train leaves Station A at 9:00 AM and travels at 60 mph, and another train leaves Station B at 10:00 AM and travels at 80 mph, when will they meet if the distance between the stations is 300 miles? To determine Math Problem text example_title In the context of computer programming, an algorithm is Algorithm Definition	text-generation	name results smol_llama-220M-GQA task dataset metrics source type name text-generation Text Generation name type config split args AI2 Reasoning Challenge (25-Shot) ai2_arc ARC-Challenge test num_few_shot 25 type value name acc_norm 24.83 normalized accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args HellaSwag (10-Shot) hellaswag validation num_few_shot 10 type value name acc_norm 29.76 normalized accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args MMLU (5-Shot) cais/mmlu all test num_few_shot 5 type value name acc 25.85 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args TruthfulQA (0-shot) truthful_qa multiple_choice validation num_few_shot 0 type value mc2 44.55 url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args Winogrande (5-shot) winogrande winogrande_xl validation num_few_shot 5 type value name acc 50.99 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args GSM8k (5-shot) gsm8k main test num_few_shot 5 type value name acc 0.68 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args IFEval (0-Shot) HuggingFaceH4/ifeval num_few_shot 0 type value name inst_level_strict_acc and prompt_level_strict_acc 23.86 strict accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args BBH (3-Shot) BBH num_few_shot 3 type value name acc_norm 3.04 normalized accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MATH Lvl 5 (4-Shot) hendrycks/competition_math num_few_shot 4 type value name exact_match 0.0 exact match url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args GPQA (0-shot) Idavidrein/gpqa num_few_shot 0 type value name acc_norm 0.78 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MuSR (0-shot) TAUR-Lab/MuSR num_few_shot 0 type value name acc_norm 9.07 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args MMLU-PRO (5-shot) TIGER-Lab/MMLU-Pro main test num_few_shot 5 type value name acc 1.66 accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA Open LLM Leaderboard

smol_llama: 220M GQA

A small 220M param (total) decoder model. This is the first version of the model.

• 1024 hidden size, 10 layers
• GQA (32 heads, 8 key-value), context length 2048
• train-from-scratch on one GPU :)

Links

Here are some fine-tunes we did, but there are many more possibilities out there!

• instruct
  • openhermes - link
  • open-instruct - link
• code
  • python (pypi) - link
• zephyr DPO tune
  • SFT - link
  • full DPO - link

---

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	29.44
AI2 Reasoning Challenge (25-Shot)	24.83
HellaSwag (10-Shot)	29.76
MMLU (5-Shot)	25.85
TruthfulQA (0-shot)	44.55
Winogrande (5-shot)	50.99
GSM8k (5-shot)	0.68

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	6.62
IFEval (0-Shot)	23.86
BBH (3-Shot)	3.04
MATH Lvl 5 (4-Shot)	0.00
GPQA (0-shot)	0.78
MuSR (0-shot)	9.07
MMLU-PRO (5-shot)	1.66