103 lines
2.8 KiB
Markdown
103 lines
2.8 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
tags:
|
||
|
|
- small-language-model
|
||
|
|
- slm
|
||
|
|
- from-scratch
|
||
|
|
- tiny
|
||
|
|
- nexus-erebus
|
||
|
|
- llama
|
||
|
|
- arithmetic
|
||
|
|
---
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
# Nexus-Erebus-50M
|
||
|
|
|
||
|
|
**Nexus-Erebus-50M** is a compact ~54M-parameter language model trained from scratch by **Ideoa Labs**,
|
||
|
|
combining strong commonsense reasoning with genuine integer arithmetic ability at tiny scale.
|
||
|
|
|
||
|
|
## Model details
|
||
|
|
|
||
|
|
| | |
|
||
|
|
|---|---|
|
||
|
|
| Parameters | ~54.1M |
|
||
|
|
| Architecture | Llama-style decoder |
|
||
|
|
| Hidden size | 512 |
|
||
|
|
| Layers | 9 |
|
||
|
|
| Attention heads | 8 |
|
||
|
|
| Vocab size | 32,000 (custom digit-aware tokenizer) |
|
||
|
|
| Context length | 1,024 |
|
||
|
|
| Precision | bfloat16 |
|
||
|
|
|
||
|
|
The tokenizer keeps digits atomic rather than merging them into BPE units, which preserves the
|
||
|
|
positional structure that integer arithmetic depends on.
|
||
|
|
|
||
|
|
## Benchmarks
|
||
|
|
|
||
|
|
Measured with `lm-eval-harness`, 0-shot, `acc_norm`, on the **full test sets** (no subsampling).
|
||
|
|
ArithMark-2 is scored on its full 2500 items with the public evaluation script.
|
||
|
|
|
||
|
|
| Task | Items | Nexus-Erebus-50M |
|
||
|
|
|---|---:|---:|
|
||
|
|
| ARC-easy | 2,376 | 44.40 |
|
||
|
|
| ARC-challenge | 1,172 | 22.70 |
|
||
|
|
| HellaSwag | 10,042 | 27.05 |
|
||
|
|
| PIQA | 1,838 | 58.27 |
|
||
|
|
| **ArithMark-2** | 2,500 | **52.48** |
|
||
|
|
| **Average** | | **40.98** |
|
||
|
|
|
||
|
|
### Against the sub-100M field
|
||
|
|
|
||
|
|
Published Open SLM Leaderboard values, same five tasks, same full-set protocol.
|
||
|
|
|
||
|
|
| Model | Params | Average |
|
||
|
|
|---|---:|---:|
|
||
|
|
| **Nexus-Erebus-50M** | **54M** | **40.98** |
|
||
|
|
| Atom 2.7M | 3M | 40.43 |
|
||
|
|
| Supra-1.5-50M-base-exp | 52M | 39.00 |
|
||
|
|
| Isabel-50M | 54M | 38.74 |
|
||
|
|
| Supra-50M-Base | 52M | 38.45 |
|
||
|
|
| Archaea-74M-V1.1 | 74M | 37.96 |
|
||
|
|
|
||
|
|
It leads the sub-100M class on ARC-easy and PIQA, and its ArithMark-2 score of 52.48 is the second
|
||
|
|
highest in that class.
|
||
|
|
|
||
|
|
## Training
|
||
|
|
|
||
|
|
Trained from scratch with a custom digit-aware 32k tokenizer, then refined on a curated mix of
|
||
|
|
educational, science, commonsense and reading-comprehension data, plus a large synthetic integer
|
||
|
|
arithmetic set covering addition, subtraction, multiplication, exact division, mixed multi-operator
|
||
|
|
expressions and parenthesised expressions. No benchmark test items were used at any stage.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
tok = AutoTokenizer.from_pretrained("MaliosDark/Nexus-Erebus-50M")
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("MaliosDark/Nexus-Erebus-50M")
|
||
|
|
|
||
|
|
prompt = "16 + 4 * 3 ="
|
||
|
|
print(tok.decode(model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=6)[0]))
|
||
|
|
```
|
||
|
|
|
||
|
|
## Example outputs
|
||
|
|
|
||
|
|
Real, unedited outputs from this checkpoint.
|
||
|
|
|
||
|
|
| Prompt | Output |
|
||
|
|
|---|---|
|
||
|
|
| `Question: What force pulls objects toward the Earth?` | gravity. |
|
||
|
|
| `Question: What gas do humans need to breathe to survive?` | oxygen. |
|
||
|
|
| `Question: What do we call the process by which plants make food?` | photosynthesis. |
|
||
|
|
|
||
|
|
## License
|
||
|
|
|
||
|
|
Apache-2.0.
|
||
|
|
|
||
|
|
Built by Ideoa Labs.
|