l20-edu-135m/eval/comparison.md

lm-eval Comparison

Candidate: l20-edu-135m-deepthin

Task	Metric	l20-edu-135m-deepthin	gpt2-small	opt-125m	gpt-neo-125m	cerebras-gpt-111m	pythia-160m	smollm-135m	smollm2-135m
arc_challenge	acc_norm,none	0.2765	0.2261	0.2210	0.2321	0.2099	0.2312	0.2875	0.2969
arc_easy	acc_norm,none	0.5059	0.3973	0.3990	0.3965	0.3506	0.3641	0.5610	0.5854
hellaswag	acc_norm,none	0.3272	0.3138	0.3160	0.3055	0.2720	0.3030	0.4265	0.4301
lambada_openai	acc,none	0.2540	0.3076	0.3856	0.3765	0.1912	0.1225	0.3757	0.4289
piqa	acc_norm,none	0.6224	0.6208	0.6202	0.6213	0.5811	0.5979	0.6823	0.6839
winogrande	acc,none	0.5099	0.5067	0.5178	0.5099	0.4901	0.5075	0.5272	0.5249

Win Rates

• gpt2-small: 5/6 = 0.833
• opt-125m: 4/6 = 0.667
• gpt-neo-125m: 4/6 = 0.667
• cerebras-gpt-111m: 6/6 = 1.000
• pythia-160m: 6/6 = 1.000
• smollm-135m: 0/6 = 0.000
• smollm2-135m: 0/6 = 0.000