---
library_name: transformers
tags:
- mergekit
- merge
datasets:
- HuggingFaceFW/fineweb-edu
- epfml/FineWeb2-HQ
- HuggingFaceTB/cosmopedia
language:
- en
---
# michel-nano

michel-nano is an ultra-tiny ~6 million parameter base language model trained on 1.14 billion tokens. It was created by merging two intermediate training checkpoints using [mergekit](https://github.com/cg123/mergekit) to combine their strengths into a single, superior model.

## Merge Details

This model is a 50/50 SLERP (Spherical Linear Interpolation) merge of two checkpoints from the same training run:
* **Checkpoint A (Step 60,000):** Excelled at language modeling and reasoning, achieving the best WikiText perplexity and ARC-Easy scores.
* **Checkpoint B (Step 70,000):** Showed stronger grammatical understanding, achieving the best BLiMP score.

By merging them, this model inherits the best traits of both—achieving lower WikiText perplexity and higher BLiMP/ARC-E scores than either parent checkpoint individually.

## Training Details

* **Parameters:** ~6,000,000
* **Training Tokens:** 1.14 Billion
* **Context Length:** 512 tokens
* **Post-training:** None (This is a base pre-trained model)

### Dataset Mixture

| Dataset | Weight |
| :--- | :--- |
| `HuggingFaceFW/fineweb-edu` | 50% |
| `epfml/FineWeb-HQ` | 30% |
| `HuggingFaceTB/cosmopedia` (stories split) | 20% |

### Tokenizer

The tokenizer is a basic Byte-Pair Encoding (BPE) tokenizer trained from scratch on a subset of 100,000 samples drawn from the same data mixture. It features a compact vocabulary size of 6,000 plus chatml special tokens for future finetuning.

## Evaluation

Evaluated using the `lm-evaluation-harness` (0-shot).

| Task | Metric | Value |
| :--- | :--- | :--- |
| **BLiMP (Avg)** | acc | 0.6523 |
| ARC-Challenge | acc | 0.1732 |
| ARC-Challenge | acc_norm | 0.2278 |
| ARC-Easy | acc | 0.3443 |
| ARC-Easy | acc_norm | 0.3338 |
| BoolQ | acc | 0.3801 |
| HellaSwag | acc | 0.2667 |
| HellaSwag | acc_norm | 0.2708 |
| PIQA | acc | 0.5647 |
| PIQA | acc_norm | 0.5457 |
| WikiText | bits_per_byte | 1.6987 |
| WikiText | byte_perplexity | 3.2461 |
| WikiText | word_perplexity | 542.6415 |
| Winogrande | acc | 0.5012 |

## Intended Use

As a base model that has not undergone any instruction tuning or alignment, michel-nano is very limited in its raw conversational abilities. It is best suited as a lightweight foundation for fine-tuning on specific downstream tasks. Its small footprint and strong grammatical foundation make it highly adaptable for applications such as:
* Text classification (e.g., toxic comment detection, sentiment analysis)
* Lightweight named entity recognition (NER)
* Random toy projects