30 lines
831 B
Markdown
30 lines
831 B
Markdown
|
# SmolLM Target Benchmark
|
||
|
|
|
||
|
|
| Task | Metric | ours-stage2 | smollm-135m | smollm2-135m |
|
||
|
|
| --- | --- | --- | --- | --- |
|
||
|
|
| ARC-Challenge | acc_norm,none | 0.2807 | 0.2875 | 0.2969 |
|
||
|
|
| ARC-Easy | acc_norm,none | 0.5059 | 0.5610 | 0.5854 |
|
||
|
|
| HellaSwag | acc_norm,none | 0.3242 | 0.4265 | 0.4301 |
|
||
|
|
| LAMBADA OpenAI | acc,none | | 0.3757 | 0.4289 |
|
||
|
|
| PIQA | acc_norm,none | 0.6137 | 0.6823 | 0.6839 |
|
||
|
|
| WinoGrande | acc,none | | 0.5272 | 0.5249 |
|
||
|
|
|
||
|
|
## Means
|
||
|
|
|
||
|
|
- `ours-stage2`: 0.4311
|
||
|
|
- `smollm-135m`: 0.4767
|
||
|
|
- `smollm2-135m`: 0.4917
|
||
|
|
|
||
|
|
## Candidate Gaps
|
||
|
|
|
||
|
|
- `ours-stage2` vs `smollm-135m` mean gap: -0.0456
|
||
|
|
- HellaSwag: -0.1023
|
||
|
|
- PIQA: -0.0686
|
||
|
|
- ARC-Easy: -0.0551
|
||
|
|
- ARC-Challenge: -0.0068
|
||
|
|
- `ours-stage2` vs `smollm2-135m` mean gap: -0.0605
|
||
|
|
- HellaSwag: -0.1059
|
||
|
|
- ARC-Easy: -0.0795
|
||
|
|
- PIQA: -0.0702
|
||
|
|
- ARC-Challenge: -0.0162
|