38 lines
1.3 KiB
Markdown
38 lines
1.3 KiB
Markdown
---
|
|
datasets:
|
|
- HuggingFaceFW/fineweb-edu
|
|
- HuggingFaceTB/cosmopedia
|
|
- epfml/FineWeb2-HQ
|
|
language:
|
|
- en
|
|
---
|
|
An ultra-tiny 28 million parameter model trained on 2.6 billion tokens using a custom dataset mixture. Context length of 1024 tokens.
|
|
|
|
| Dataset | Weight |
|
|
|---|---|
|
|
| `HuggingFaceFW/fineweb-edu` | 50% |
|
|
| `epfml/FineWeb-HQ` | 30% |
|
|
| `HuggingFaceTB/cosmopedia` (stories split) | 20% |
|
|
|
|
The tokenizer is a basic bpe tokenizer that was trained on a smaller subset of 80_000 samples of this same data mixture with a vocab size of 8000.
|
|
|
|
This model has not undergone any post-training.
|
|
|
|
This base model is best suited for fine-tuning on specific tasks. On its own, it is very limited, but it is a pretty flexible foundation for applications such as toxic comment detection or sentiment analysis.
|
|
|
|
| Benchmark | Metric | Value |
|
|
|---|---|---|
|
|
| ARC-Challenge | Acc | 0.1980 |
|
|
| ARC-Challenge | Acc Norm | 0.2312 |
|
|
| ARC-Easy | Acc | 0.4091 |
|
|
| ARC-Easy | Acc Norm | 0.3859 |
|
|
| BoolQ | Acc | 0.5850 |
|
|
| HellaSwag | Acc | 0.2749 |
|
|
| HellaSwag | Acc Norm | 0.2823 |
|
|
| PIQA | Acc | 0.5963 |
|
|
| PIQA | Acc Norm | 0.5762 |
|
|
| WinoGrande | Acc | 0.4917 |
|
|
| WikiText | Word PPL | 86.6418 |
|
|
| WikiText | Byte PPL | 2.3034 |
|
|
| WikiText | Bits/Byte | 1.2037 |
|
|
| BLIMP (Avg) | Acc | 0.6975 | |