70 lines
2.6 KiB
Markdown
70 lines
2.6 KiB
Markdown
---
|
|
library_name: transformers
|
|
tags:
|
|
- mergekit
|
|
- merge
|
|
datasets:
|
|
- HuggingFaceFW/fineweb-edu
|
|
- epfml/FineWeb2-HQ
|
|
- HuggingFaceTB/cosmopedia
|
|
language:
|
|
- en
|
|
---
|
|
# michel-nano
|
|
|
|
michel-nano is an ultra-tiny ~6 million parameter base language model trained on 1.14 billion tokens. It was created by merging two intermediate training checkpoints using [mergekit](https://github.com/cg123/mergekit) to combine their strengths into a single, superior model.
|
|
|
|
## Merge Details
|
|
|
|
This model is a 50/50 SLERP (Spherical Linear Interpolation) merge of two checkpoints from the same training run:
|
|
* **Checkpoint A (Step 60,000):** Excelled at language modeling and reasoning, achieving the best WikiText perplexity and ARC-Easy scores.
|
|
* **Checkpoint B (Step 70,000):** Showed stronger grammatical understanding, achieving the best BLiMP score.
|
|
|
|
By merging them, this model inherits the best traits of both—achieving lower WikiText perplexity and higher BLiMP/ARC-E scores than either parent checkpoint individually.
|
|
|
|
## Training Details
|
|
|
|
* **Parameters:** ~6,000,000
|
|
* **Training Tokens:** 1.14 Billion
|
|
* **Context Length:** 512 tokens
|
|
* **Post-training:** None (This is a base pre-trained model)
|
|
|
|
### Dataset Mixture
|
|
|
|
| Dataset | Weight |
|
|
| :--- | :--- |
|
|
| `HuggingFaceFW/fineweb-edu` | 50% |
|
|
| `epfml/FineWeb-HQ` | 30% |
|
|
| `HuggingFaceTB/cosmopedia` (stories split) | 20% |
|
|
|
|
### Tokenizer
|
|
|
|
The tokenizer is a basic Byte-Pair Encoding (BPE) tokenizer trained from scratch on a subset of 100,000 samples drawn from the same data mixture. It features a compact vocabulary size of 6,000 plus chatml special tokens for future finetuning.
|
|
|
|
## Evaluation
|
|
|
|
Evaluated using the `lm-evaluation-harness` (0-shot).
|
|
|
|
| Task | Metric | Value |
|
|
| :--- | :--- | :--- |
|
|
| **BLiMP (Avg)** | acc | 0.6523 |
|
|
| ARC-Challenge | acc | 0.1732 |
|
|
| ARC-Challenge | acc_norm | 0.2278 |
|
|
| ARC-Easy | acc | 0.3443 |
|
|
| ARC-Easy | acc_norm | 0.3338 |
|
|
| BoolQ | acc | 0.3801 |
|
|
| HellaSwag | acc | 0.2667 |
|
|
| HellaSwag | acc_norm | 0.2708 |
|
|
| PIQA | acc | 0.5647 |
|
|
| PIQA | acc_norm | 0.5457 |
|
|
| WikiText | bits_per_byte | 1.6987 |
|
|
| WikiText | byte_perplexity | 3.2461 |
|
|
| WikiText | word_perplexity | 542.6415 |
|
|
| Winogrande | acc | 0.5012 |
|
|
|
|
## Intended Use
|
|
|
|
As a base model that has not undergone any instruction tuning or alignment, michel-nano is very limited in its raw conversational abilities. It is best suited as a lightweight foundation for fine-tuning on specific downstream tasks. Its small footprint and strong grammatical foundation make it highly adaptable for applications such as:
|
|
* Text classification (e.g., toxic comment detection, sentiment analysis)
|
|
* Lightweight named entity recognition (NER)
|
|
* Random toy projects |