23 lines
1.1 KiB
Markdown
23 lines
1.1 KiB
Markdown
|
# lm-eval Comparison
|
||
|
|
|
||
|
|
Candidate: `l20-edu-135m-deepthin`
|
||
|
|
|
||
|
|
| Task | Metric | l20-edu-135m-deepthin | gpt2-small | opt-125m | gpt-neo-125m | cerebras-gpt-111m | pythia-160m | smollm-135m | smollm2-135m |
|
||
|
|
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||
|
|
| arc_challenge | acc_norm,none | 0.2765 | 0.2261 | 0.2210 | 0.2321 | 0.2099 | 0.2312 | 0.2875 | 0.2969 |
|
||
|
|
| arc_easy | acc_norm,none | 0.5059 | 0.3973 | 0.3990 | 0.3965 | 0.3506 | 0.3641 | 0.5610 | 0.5854 |
|
||
|
|
| hellaswag | acc_norm,none | 0.3272 | 0.3138 | 0.3160 | 0.3055 | 0.2720 | 0.3030 | 0.4265 | 0.4301 |
|
||
|
|
| lambada_openai | acc,none | 0.2540 | 0.3076 | 0.3856 | 0.3765 | 0.1912 | 0.1225 | 0.3757 | 0.4289 |
|
||
|
|
| piqa | acc_norm,none | 0.6224 | 0.6208 | 0.6202 | 0.6213 | 0.5811 | 0.5979 | 0.6823 | 0.6839 |
|
||
|
|
| winogrande | acc,none | 0.5099 | 0.5067 | 0.5178 | 0.5099 | 0.4901 | 0.5075 | 0.5272 | 0.5249 |
|
||
|
|
|
||
|
|
## Win Rates
|
||
|
|
|
||
|
|
- `gpt2-small`: 5/6 = 0.833
|
||
|
|
- `opt-125m`: 4/6 = 0.667
|
||
|
|
- `gpt-neo-125m`: 4/6 = 0.667
|
||
|
|
- `cerebras-gpt-111m`: 6/6 = 1.000
|
||
|
|
- `pythia-160m`: 6/6 = 1.000
|
||
|
|
- `smollm-135m`: 0/6 = 0.000
|
||
|
|
- `smollm2-135m`: 0/6 = 0.000
|