tinybrain-100m-base/README.md

---
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - llama
  - text-generation
  - causal-lm
  - tinybrain
  - from-scratch
  - 100m
  - base-model
  - small-language-model
  - tiny-llm
  - english
  - pretraining
  - transformers
datasets:
  - exnivo/tinybrain-pretrain-corpus-2b
---

<p align="center">
  <img
    src="https://huggingface.co/exnivo/tinybrain-100m-base/resolve/main/assets/tinybrain-100m-base-banner.png"
    alt="TinyBrain-100M Base — Base language model for small LLMs"
    width="100%"
  />
</p>

# TinyBrain-100M Base

**A 103M parameter English causal language model trained from scratch.**

TinyBrain-100M Base is a small LLaMA-style causal language model trained from scratch on the [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) dataset.

This is a **base model**, not an instruct/chat model. It is intended for language modeling experiments, continued pretraining, supervised fine-tuning, and small-model research.

For chat or instruction-following behavior, use the instruction-tuned version:

[`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct)

## Quick Start

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "exnivo/tinybrain-100m-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "Photosynthesis is the process by which plants"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    temperature=0.8,
    top_p=0.9,
    repetition_penalty=1.08,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## At a Glance

| Item | Details |
|---|---|
| Model type | Base causal language model |
| Parameters | 103,385,856 |
| Approx. size | 103.4M |
| Architecture | LLaMA-style causal transformer |
| Language | English |
| Context length | 2048 tokens |
| Vocabulary size | 24,000 |
| Tokenizer | Custom TinyBrain tokenizer |
| Training style | From scratch pretraining |
| Pretraining dataset | [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) |
| Instruct dataset | [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k) |
| Instruct model | [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct) |

## Model Details

| Item | Value |
|---|---|
| Parameters | 103.4M |
| Architecture | `llama` / `LlamaForCausalLM` |
| Vocabulary size | 24,000 |
| Context length | 2048 tokens |
| Hidden size | 768 |
| Intermediate size | 2048 |
| Layers | 12 |
| Attention heads | 12 |
| Key/value heads | 12 |
| Activation | SiLU |
| RMS norm epsilon | `1e-05` |
| Tied embeddings | true |
| BOS token | `<|bos|>` |
| EOS token | `<|eos|>` |
| PAD token | `<|pad|>` |
| Dataset | [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) |

## Intended Use

TinyBrain-100M Base is intended for:

- small language model research
- causal language modeling experiments
- continued pretraining
- supervised fine-tuning
- instruction tuning
- tokenizer/model experiments
- educational small-model projects
- comparing base vs instruct behavior
- lightweight local model experiments

This model is best used as a **base checkpoint** for further training.

## Not Intended For

This model is not intended to be used directly as a finished assistant.

Do not rely on the base model for:

- polished chat behavior
- instruction following
- safety-critical answers
- factual authority
- medical, legal, or financial advice
- live/current information
- advanced reasoning
- production use without evaluation

For assistant-style behavior, use [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct) instead.

## Training Data

TinyBrain-100M Base was trained on:

[`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b)

The pretraining corpus is a mixed-source English dataset containing factual text, educational text, math reasoning data, code data, conversation-style data, and clean web text.

The pretraining corpus scan found:

| Metric | Value |
|---|---:|
| Rows | 3,013,308 |
| Characters | 7,767,447,861 |
| Words | 1,249,832,587 |
| Approx. tokens | ~1.81B tokenizer-independent estimate |

The training run used an estimated **~2.1B training tokens**. Token counts may differ depending on tokenizer, packing, filtering, and training pipeline details.

## Dataset Mix

The pretraining corpus includes these broad categories:

| Category | Rows | Percent |
|---|---:|---:|
| `factual` | 773,492 | 25.67% |
| `educational` | 752,625 | 24.98% |
| `math_reasoning` | 633,341 | 21.02% |
| `code` | 326,019 | 10.82% |
| `conversation` | 296,728 | 9.85% |
| `clean_web` | 231,103 | 7.67% |

## Relationship to TinyBrain

TinyBrain is a small LLM project focused on compact datasets, small base models, and instruction-tuned models.

| Stage | Repository | Purpose |
|---|---|---|
| Pretraining corpus | [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) | Base language model training data |
| Base model | [`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base) | Small causal LM trained from scratch |
| SFT dataset | [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k) | Instruction/chat fine-tuning data |
| Instruct model | [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct) | Chat/instruct model fine-tuned from the base model |

Pipeline:

```text
TinyBrain Pretrain Corpus 2B
        ↓
TinyBrain-100M Base
        ↓
TinyBrain Instruct 200K
        ↓
TinyBrain-100M Instruct
```

## Evaluation

A quick WikiText-2 evaluation was run on the base model.

| Metric | Value |
|---|---:|
| Eval tokens | 38,138 |
| Eval text chars | 159,791 |
| Loss | 3.7440 |
| Perplexity | 42.27 |

This is a lightweight evaluation, not a full benchmark suite. Results may vary depending on evaluation script, tokenizer settings, context length, and dataset preprocessing.

## Base Model Behavior

TinyBrain-100M Base is a raw pretrained model. It can complete text, but it is not tuned to follow instructions.

Example base prompt:

```text
Photosynthesis is the process by which plants
```

The base model may continue with partially useful text, but it can also repeat, drift, hallucinate, or produce broken completions. This is expected for a small base model and is one reason instruction tuning is needed.

For better chat behavior, use:

[`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct)

## Recommended Generation Settings

For raw base-model text completion:

```python
temperature = 0.8
top_p = 0.9
max_new_tokens = 80
repetition_penalty = 1.08
```

For more stable completions:

```python
temperature = 0.5
top_p = 0.85
max_new_tokens = 80
repetition_penalty = 1.1
```

For deterministic testing:

```python
do_sample = False
max_new_tokens = 80
```

## Example: Text Completion

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "exnivo/tinybrain-100m-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "Gravity is the force that"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.9,
    repetition_penalty=1.08,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Example: Fine-Tuning Starting Point

TinyBrain-100M Base can be fine-tuned on the TinyBrain SFT dataset:

```python
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import SFTTrainer, SFTConfig

base_model = "exnivo/tinybrain-100m-base"
dataset_id = "exnivo/tinybrain-instruct-sft-200k"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(base_model)

ds = load_dataset(dataset_id, split="train")

def format_example(example):
    text = ""

    for message in example["messages"]:
        role = message["role"]
        content = message["content"].strip()

        if role == "user":
            text += f"User: {content}\n"
        elif role == "assistant":
            text += f"Assistant: {content}\n"

    return {"text": text.strip()}

ds = ds.map(format_example)

config = SFTConfig(
    output_dir="tinybrain-100m-instruct-sft",
    dataset_text_field="text",
    max_seq_length=512,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    num_train_epochs=1,
    logging_steps=20,
    save_steps=500,
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=ds,
    args=config,
)

trainer.train()
```

## Training

TinyBrain-100M Base was trained from scratch on the TinyBrain pretraining corpus.

Training details:

| Item | Value |
|---|---|
| Training type | From-scratch causal language modeling |
| Dataset | TinyBrain Pretrain Corpus 2B |
| Approx. training tokens | ~2.1B |
| Reported best validation loss | 2.6779 |
| Training precision | bf16 |
| Hardware | NVIDIA RTX PRO 6000 Blackwell Server Edition |

## Strengths

TinyBrain-100M Base is useful because it is:

- small and lightweight
- trained from scratch
- easy to inspect
- easy to fine-tune
- based on an open TinyBrain data pipeline
- trained on a compact mixed-source corpus
- suitable for small-model experiments
- useful as a base checkpoint for SFT

## Limitations

TinyBrain-100M Base has important limitations.

The model may:

- hallucinate facts
- produce broken or repetitive text
- fail at math
- fail at instruction following
- misunderstand prompts
- generate incomplete code
- produce outdated or incorrect information
- drift off-topic
- repeat web/data artifacts

This is expected for a small **base** model. It has not been tuned to reliably follow user instructions.

For chat and assistant behavior, use the instruction-tuned model instead.

## Suggested Evaluation

Recommended checks:

- validation loss / perplexity
- text completion quality
- repetition behavior
- short factual completions
- simple math completions
- code completion sanity checks
- hallucination checks
- before/after SFT comparison
- downstream instruction-following after fine-tuning

Example base-model prompts:

```text
Paris is the capital city of
```

```text
The Netherlands is a country in
```

```text
A cat is an animal that
```

```text
One plus one equals
```

```text
Photosynthesis is the process by which plants
```

## Citation

If you use this model, you can cite it as:

```bibtex
@misc{tinybrain_100m_base,
  title = {TinyBrain-100M Base},
  author = {exnivo},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/exnivo/tinybrain-100m-base}}
}
```

## Related Repositories

- Pretraining corpus: [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b)
- Base model: [`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base)
- SFT dataset: [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k)
- Instruct model: [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct)

## License

This model is released under the Apache 2.0 license.

The training dataset is mixed-source and currently listed under `license: other`. Users should review the upstream dataset licenses and source metadata before commercial use of models trained or fine-tuned from this checkpoint.

## Disclaimer

TinyBrain-100M Base is an experimental small base language model. It may produce incorrect, biased, unsafe, nonsensical, or misleading outputs.

Do not use it for high-stakes applications without additional training, filtering, evaluation, and safeguards.