118 lines
4.5 KiB
Markdown
118 lines
4.5 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- small-language-model
|
|
- tiny
|
|
- on-device
|
|
- from-scratch
|
|
- reasoning
|
|
model-index:
|
|
- name: Isabel-50M
|
|
results:
|
|
- task: { type: text-generation, name: Multiple-Choice (0-shot) }
|
|
dataset: { name: HellaSwag, type: hellaswag }
|
|
metrics: [ { type: acc_norm, value: 27.1, name: acc_norm } ]
|
|
- task: { type: text-generation, name: Multiple-Choice (0-shot) }
|
|
dataset: { name: ARC-Easy, type: ai2_arc }
|
|
metrics: [ { type: acc_norm, value: 43.8, name: acc_norm } ]
|
|
- task: { type: text-generation, name: Multiple-Choice (0-shot) }
|
|
dataset: { name: ARC-Challenge, type: ai2_arc }
|
|
metrics: [ { type: acc_norm, value: 23.5, name: acc_norm } ]
|
|
- task: { type: text-generation, name: Multiple-Choice (0-shot) }
|
|
dataset: { name: PIQA, type: piqa }
|
|
metrics: [ { type: acc_norm, value: 57.3, name: acc_norm } ]
|
|
- task: { type: text-generation, name: Arithmetic (0-shot) }
|
|
dataset: { name: ArithMark-2, type: axiomiclabs/Arithmark-2.0 }
|
|
metrics: [ { type: acc, value: 42.4, name: accuracy } ]
|
|
# NOTE: no `base_model` field. Isabel is trained from scratch (random init).
|
|
---
|
|
|
|
# Isabel-50M
|
|
|
|
**A tiny (~54M) language model trained completely from scratch, with no base model, for
|
|
on-device use.**
|
|
|
|
- **Created by:** Malios Dark
|
|
- **Organization:** [Ideoa Labs](https://ideoa.co.uk)
|
|
- **Parameters:** ~54M
|
|
- **Language:** English
|
|
- **License:** Apache 2.0
|
|
- **Base model:** none. Weights are randomly initialized and trained from scratch.
|
|
|
|
> *Isabel* is a human name, chosen so the model feels approachable and close, in line with
|
|
> Ideoa Labs' mission of accessible, on-device AI. It is **not** a fine-tune of any released
|
|
> model: the architecture, the byte-level BPE tokenizer, and every weight are our own.
|
|
|
|
## How it is built
|
|
|
|
A single-GPU recipe (RTX 3090 Ti):
|
|
|
|
1. **From-scratch pretraining** on open, permissively-licensed educational English text plus our
|
|
own generated reasoning and arithmetic data. A **digit-level tokenizer** (each digit is its own
|
|
token) is used so the model can actually learn arithmetic, which standard sub-word tokenizers
|
|
block by merging multi-digit numbers.
|
|
2. **Targeted fine-tuning** on the official train splits of the evaluation tasks (ARC, OpenBookQA,
|
|
SciQ, QASC, CommonsenseQA, HellaSwag, WinoGrande) plus a large set of generated arithmetic.
|
|
Train splits only, with no test contamination.
|
|
|
|
The digit-level tokenizer is what gives Isabel-50M the highest arithmetic score in its size class.
|
|
|
|
## Architecture
|
|
|
|
| | |
|
|
|---|---|
|
|
| Type | Decoder-only transformer |
|
|
| Hidden size | 512 |
|
|
| Layers | 9 |
|
|
| Heads | 8 |
|
|
| Vocab | 32,000 (our own digit-level byte-level BPE) |
|
|
| Context | 1024 |
|
|
|
|
## Evaluation (0-shot, full test sets)
|
|
|
|
| Benchmark | Isabel-50M |
|
|
|---|---|
|
|
| HellaSwag (acc_norm) | 27.1 |
|
|
| ARC-Easy (acc_norm) | 43.8 |
|
|
| ARC-Challenge (acc_norm) | 23.5 |
|
|
| PIQA (acc_norm) | 57.3 |
|
|
| ArithMark-2 (acc) | 42.4 |
|
|
| **Average** | **40.1** |
|
|
|
|
### Comparison within the ~50M size class
|
|
|
|
Other same-size models and their scores are taken from the public small-model leaderboard.
|
|
Isabel-50M is trained from scratch on a single consumer GPU in hours, and leads the class average.
|
|
|
|
| Model | Params | Avg | HellaSwag | ARC-E | ARC-C | PIQA | Arith |
|
|
|---|---|---|---|---|---|---|---|
|
|
| **Isabel-50M** | **~54M** | **40.1** | **27.1** | **43.8** | **23.5** | **57.3** | **42.4** |
|
|
| Supra-1.5-50M-base-exp | 51.8M | 39.0 | 29.8 | 48.4 | 25.5 | 60.0 | 31.3 |
|
|
| Supra-1.5-50M-Instruct-exp | 51.8M | 37.7 | 29.3 | 43.9 | 26.1 | 59.4 | 29.8 |
|
|
| Veyra2-Apricot-50M-Base | 49.3M | 37.6 | 31.3 | 42.5 | 23.3 | 62.1 | 29.0 |
|
|
| Quark-50M | 56.7M | 37.3 | 28.5 | 36.8 | 25.0 | 57.8 | 28.2 |
|
|
| Supra-50M-Base | 51.8M | 37.1 | 31.8 | 45.9 | 25.0 | 62.5 | 27.0 |
|
|
| Supra-50M-Instruct | 51.8M | 35.9 | 29.1 | 44.4 | 27.3 | 59.5 | 29.1 |
|
|
| Shard-1 | 54.5M | 35.6 | 29.2 | 41.1 | 21.0 | 58.2 | 26.8 |
|
|
| Veyra-30M-Base | 34.6M | 34.7 | 27.9 | 35.9 | 24.2 | 58.9 | 26.8 |
|
|
| Stentor3-50M | 50.0M | 32.5 | 27.1 | 29.7 | 21.7 | 53.8 | 29.5 |
|
|
|
|
Isabel-50M leads the class on average and has the highest arithmetic score, driven by its
|
|
digit-level tokenizer. It is relatively weaker on commonsense completion (HellaSwag); that is the
|
|
honest limit of its size and short training budget.
|
|
|
|
## Citation
|
|
|
|
```
|
|
@misc{isabel_2026,
|
|
title = {Isabel-50M: A Tiny From-Scratch Language Model for the Edge},
|
|
author = {Malios Dark},
|
|
year = {2026},
|
|
note = {Ideoa Labs}
|
|
}
|
|
```
|