---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-edu
tags:
- llama
- causal-lm
- fineweb-edu
- small-language-model
- text-generation
widget:
- text: "A computer is a machine that"
- text: "I like traveling by train because"
- text: "Once upon a time,"
---

# MiniBananaMind-V1

![MiniBananaMind-V1 preview](assets/minibananamind-v1.png)

MiniBananaMind-V1 is a compact LLaMA-style causal language model from BananaMind.
It is trained from scratch for next-token text generation on streamed
FineWeb-Edu data and is intended as a small, inspectable base model for
experiments, demos, and lightweight research workflows.

This is a base language model, not an instruction-tuned assistant. It is best
used for continuation-style generation and experimentation rather than factual
question answering or chat.

## Model Details

- **Developer:** BananaMind
- **Model type:** LLaMA-style causal language model
- **Library:** Transformers
- **Task:** Text generation
- **Training data:** FineWeb-Edu
- **Checkpoint:** MiniBananaMind-V1 uploaded training checkpoint
- **License:** Apache 2.0

## Architecture

| Setting | Value |
| --- | ---: |
| Layers | 6 |
| Hidden size | 256 |
| Attention heads | 8 |
| KV heads | 8 |
| Intermediate size | 768 |
| Context length | 512 tokens |
| Vocabulary size | 32,000 |
| Parameters | ~21.5M |
| Precision | float32 checkpoint |

## Intended Use

MiniBananaMind-V1 is suitable for:

- Small-scale language-model experiments
- Educational demos of decoder-only generation
- Testing tokenization, generation settings, and inference pipelines
- Research prototypes where a very small causal LM is useful

It is not recommended for production assistants, safety-critical use, or tasks
that require reliable factual knowledge.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo_id = "BananaMind/MiniBananaMind-V1"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype=torch.float32,
    device_map="auto",
)

prompt = "A computer is a machine that"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.2,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Generation Notes

Because this is a small base model, output quality depends heavily on prompt
style and sampling settings. A `temperature` of `0.2` is recommended for more
stable continuations. For more varied text, increase `temperature` or `top_p`.

## Limitations

- The model may hallucinate facts, names, citations, and dates.
- It has not been instruction tuned or aligned for chat behavior.
- It may reproduce biases or unsafe patterns present in web-scale training data.
- The short 512-token context length limits long-document use.
- Small model size means weaker reasoning and factual recall than larger LMs.

## Training Data

MiniBananaMind-V1 was trained on streamed FineWeb-Edu text. FineWeb-Edu is a
large educational-quality web corpus, so users should expect broad web-language
coverage as well as the usual limitations of internet-scale data.

Training data attribution: this model was trained on
[FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), a
dataset released by Hugging Face as part of the FineWeb family.

## Citation

If you use this model in a project, cite the Hugging Face repository and
attribute the FineWeb-Edu training data:

```bibtex
@misc{minibananamindv1,
  title = {MiniBananaMind-V1},
  author = {BananaMind},
  year = {2026},
  howpublished = {\url{https://huggingface.co/BananaMind/MiniBananaMind-V1}}
}
```

Dataset: [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)