tinybrain-100m-instruct/README.md

---
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - llama
  - text-generation
  - causal-lm
  - instruct
  - chat
  - sft
  - tinybrain
  - 100m
  - small-language-model
  - tiny-llm
  - english
base_model:
  - exnivo/tinybrain-100m-base
datasets:
  - exnivo/tinybrain-instruct-sft-200k
---

<p align="center">
  <img
    src="https://huggingface.co/exnivo/tinybrain-100m-instruct/resolve/main/assets/tinybrain-100m-instruct-banner.png"
    alt="TinyBrain-100M Instruct — Instruction-tuned model for small LLMs"
    width="100%"
  />
</p>

# TinyBrain-100M Instruct

**A 103M parameter experimental chat/instruct model fine-tuned from TinyBrain-100M Base.**

TinyBrain-100M Instruct is a small instruction-tuned causal language model fine-tuned from [`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base) using [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k).

This is a very small instruct model. It can answer simple prompts, explain basic ideas, give short plans, and sometimes show uncertainty behavior, but it is not a reliable general assistant. It may hallucinate, repeat text, fail at math, produce broken completions, or misunderstand prompts.

TinyBrain-100M Instruct was fine-tuned with a simple **User/Assistant** style format and **no system prompt**.

## Quick Start

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "exnivo/tinybrain-100m-instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "User: Explain photosynthesis in simple words.\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=False,
    repetition_penalty=1.15,
    no_repeat_ngram_size=3,
    pad_token_id=tokenizer.eos_token_id,
)

text = tokenizer.decode(outputs[0], skip_special_tokens=True)

answer = text.split("Assistant:", 1)[-1].strip()
answer = answer.split("\nUser:", 1)[0].strip()

print(answer)
```

## At a Glance

| Item | Details |
|---|---|
| Model type | Instruction-tuned causal language model |
| Parameters | 103,385,856 |
| Approx. size | 103.4M |
| Architecture | LLaMA-style causal transformer |
| Language | English |
| Context length | 2048 tokens |
| Vocabulary size | 24,000 |
| Tokenizer | Custom TinyBrain tokenizer |
| Base model | [`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base) |
| SFT dataset | [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k) |
| Prompt style | `User:` / `Assistant:` |
| System prompt | Not used during fine-tuning |

## Model Details

| Item | Value |
|---|---|
| Parameters | 103.4M |
| Architecture | `llama` / `LlamaForCausalLM` |
| Vocabulary size | 24,000 |
| Context length | 2048 tokens |
| Hidden size | 768 |
| Intermediate size | 2048 |
| Layers | 12 |
| Attention heads | 12 |
| Key/value heads | 12 |
| Activation | SiLU |
| RMS norm epsilon | `1e-05` |
| Tied embeddings | true |
| BOS token | `<|bos|>` |
| EOS token | `<|eos|>` |
| PAD token | `<|pad|>` |
| Base model | [`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base) |
| SFT dataset | [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k) |

## Prompt Format

TinyBrain-100M Instruct was fine-tuned without a system prompt.

Use this simple format:

```text
User: Your message here
Assistant:
```

Example:

```text
User: Explain photosynthesis in simple words.
Assistant:
```

For best results:

- keep prompts short and direct
- do not use a system prompt
- use short generation lengths
- prefer greedy or low-temperature generation
- stop/cut the output if it starts a new `User:` turn

## Recommended Generation Settings

For stable short answers:

```python
outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=False,
    repetition_penalty=1.15,
    no_repeat_ngram_size=3,
    pad_token_id=tokenizer.eos_token_id,
)
```

For slightly more varied answers:

```python
outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.5,
    top_p=0.85,
    repetition_penalty=1.15,
    no_repeat_ngram_size=3,
    pad_token_id=tokenizer.eos_token_id,
)
```

For a very small model like this, long generations often become repetitive or unstable. Short completions usually work better.

## Example Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "exnivo/tinybrain-100m-instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

def ask(message, max_new_tokens=80):
    prompt = f"User: {message}\nAssistant:"
    inputs = tokenizer(prompt, return_tensors="pt")

    outputs = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        do_sample=False,
        repetition_penalty=1.15,
        no_repeat_ngram_size=3,
        pad_token_id=tokenizer.eos_token_id,
    )

    text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    answer = text.split("Assistant:", 1)[-1].strip()
    answer = answer.split("\nUser:", 1)[0].strip()
    return answer

print(ask("Explain gravity in simple words."))
```

## Training Data

TinyBrain-100M Instruct was fine-tuned on:

[`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k)

The SFT dataset contains **196,668 rows** of English instruction/chat examples focused on short, learnable assistant behavior.

Dataset categories include:

| Category | Rows | Percent |
|---|---:|---:|
| `source_grounded_education_factual` | 49,882 | 25.36% |
| `math_reasoning` | 37,611 | 19.12% |
| `clean_conversation` | 34,257 | 17.42% |
| `messy_idea_to_plan` | 29,978 | 15.24% |
| `simplify_explain` | 19,990 | 10.16% |
| `honesty_uncertainty` | 14,957 | 7.61% |
| `simple_coding` | 9,993 | 5.08% |

The dataset was designed for small models and uses short assistant responses across education, basic math, planning, simplification, simple coding, clean conversation, and uncertainty behavior.

## Relationship to TinyBrain

TinyBrain is a small LLM project focused on compact datasets, small base models, and instruction-tuned models.

| Stage | Repository | Purpose |
|---|---|---|
| Pretraining corpus | [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) | Base language model training data |
| Base model | [`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base) | Small causal LM trained from scratch |
| SFT dataset | [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k) | Instruction/chat fine-tuning data |
| Instruct model | [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct) | Chat/instruct model fine-tuned from the base model |

Pipeline:

```text
TinyBrain Pretrain Corpus 2B
        ↓
TinyBrain-100M Base
        ↓
TinyBrain Instruct 200K
        ↓
TinyBrain-100M Instruct
```

## Expected Behavior

TinyBrain-100M Instruct can sometimes handle:

- simple explanations
- short educational questions
- basic planning
- simple rewriting/simplification
- simple coding prompts
- some uncertainty/refusal-style prompts
- casual assistant-like responses

Example prompt:

```text
User: Explain photosynthesis in simple words.
Assistant:
```

Possible output style:

```text
Photosynthesis is how plants make their own food using sunlight, water, and air. They turn these into sugar and oxygen.
```

Because this is a very small model, outputs may be unstable. It can repeat, drift, or produce broken text, especially with long generations or higher sampling temperatures.

## Evaluation Notes

A lightweight local report confirmed the model architecture and parameter count:

| Metric | Value |
|---|---:|
| Total parameters | 103,385,856 |
| Trainable parameters | 103,385,856 |
| Approx. parameters | 103.4M |

Manual prompt tests showed that the model behaves better with the plain `User:` / `Assistant:` format than with custom chat-special-token formatting.

The model can produce useful short answers for some prompts, but it still performs poorly on reliable math, longer reasoning, and some instruction-following tasks.

This should be treated as an experimental small-model checkpoint, not a benchmark-grade assistant.

## Intended Use

TinyBrain-100M Instruct is intended for:

- small-model experiments
- local lightweight assistant tests
- instruction-tuning research
- comparing base vs instruct behavior
- educational model experiments
- studying tiny LLM limitations
- continued fine-tuning
- dataset/model pipeline demos

This model is useful for exploring how much instruction-following behavior can be added to a small 100M-parameter model.

## Not Intended For

Do not rely on this model for:

- medical advice
- legal advice
- financial advice
- emergency decisions
- safety-critical systems
- factual authority
- current news or live information
- advanced math
- advanced coding
- long-form reasoning
- production assistant use without further training and evaluation

This is an experimental model and should not be used as a source of truth.

## Strengths

TinyBrain-100M Instruct is useful because it is:

- small
- lightweight
- easy to run locally
- fine-tuned from a matching TinyBrain base model
- trained on a public TinyBrain SFT dataset
- designed for short assistant-style responses
- useful for base-vs-instruct comparison
- good for studying tiny model behavior

## Limitations

TinyBrain-100M Instruct has major limitations.

The model may:

- hallucinate facts
- fail simple math
- repeat words or phrases
- produce broken text
- drift off-topic
- answer too briefly
- misunderstand prompts
- generate unreliable code
- fail at longer reasoning
- fail refusal or safety behavior
- continue into fake new user turns

For best results, keep prompts short and use short generation lengths.

## Known Weaknesses

Based on local testing, this model is especially weak at:

- reliable arithmetic
- robust coding
- long answers
- multi-step reasoning
- clean formatting
- high-temperature sampling
- long context use

It may answer simple educational prompts better than math or code prompts.

## Suggested Evaluation

Recommended checks:

- short factual prompts
- simple explanation prompts
- basic math prompts
- correction prompts
- refusal/uncertainty prompts
- repetition tests
- prompt-format tests
- base vs instruct comparison
- SFT dataset overfitting checks
- generation temperature sensitivity

Example prompts:

```text
User: Explain gravity in simple words.
Assistant:
```

```text
User: What is 17 + 25?
Assistant:
```

```text
User: What will the weather be tomorrow in my city?
Assistant:
```

```text
User: Give me 3 quick tips to keep my room tidy.
Assistant:
```

```text
User: Write a simple Python function that reverses a string.
Assistant:
```

## Training

TinyBrain-100M Instruct was fine-tuned from:

[`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base)

using:

[`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k)

The base model was trained from scratch on [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b).

## Citation

If you use this model, you can cite it as:

```bibtex
@misc{tinybrain_100m_instruct,
  title = {TinyBrain-100M Instruct},
  author = {exnivo},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/exnivo/tinybrain-100m-instruct}}
}
```

## Related Repositories

- Pretraining corpus: [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b)
- Base model: [`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base)
- SFT dataset: [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k)
- Instruct model: [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct)

## License

This model is released under the Apache 2.0 license.

The SFT dataset and pretraining corpus are mixed-source datasets and may have their own licensing considerations. Users should review the dataset cards and upstream source metadata before commercial use.

## Disclaimer

TinyBrain-100M Instruct is an experimental tiny instruction-tuned language model. It may produce incorrect, biased, unsafe, nonsensical, or misleading outputs.

Do not use this model for high-stakes decisions or as a reliable source of factual information.