---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- llama
- causal-lm
- chat
- tiny-language-model
- neo50m
---

# Neo50M

Neo50M is a tiny decoder-only chat language model trained from scratch. It is designed for toy/local assistant use, educational experiments, lightweight generation, and testing training pipelines.

## Model Details

- **Type:** decoder-only causal language model, Llama-compatible architecture
- **Parameters:** approximately 52.6M
- **Context length target:** 16k tokens
- **Training target:** about 15B pretraining tokens plus chat/instruction tuning
- **Hardware:** 8x NVIDIA RTX 5090 cloud GPUs
- **Tokenizer:** TinyLlama/Llama-style 32k tokenizer with a Neo50M chat template

## Intended Uses

- toy/local assistant experiments
- educational training and inference demos
- lightweight generation
- testing HF, GGUF, ONNX, and distributed training pipelines

## Limitations

Neo50M is very small. It is not reliable for factual accuracy, has limited reasoning ability, may hallucinate, and should not be used for safety-critical decisions or high-stakes advice.

## Transformers Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "KookiesXy/Neo50M"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, device_map="auto")

messages = [{"role": "user", "content": "Write a short thank-you note."}]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=120, temperature=0.7, top_p=0.9)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
```

## GGUF Usage

After downloading a GGUF file:

```bash
llama-cli -m neo50m-q4_k_m.gguf -p "User: Write a haiku about GPUs.\nAssistant:"
```

## ONNX Usage

The ONNX export is intended for forward-pass validation and integration experiments. Use ONNX Runtime to load `onnx/model.onnx` and feed integer `input_ids` plus `attention_mask`.

## Dataset Summary

The training pipeline streams a configurable mixture of FineWeb-Edu, Cosmopedia, Wikipedia-like text, TinyStories, and a small permissive code component. SFT uses OpenHermes-style, UltraChat-style, Alpaca-style, and small refusal/helpfulness examples when available. Dataset availability can change; the exact configs are included with the upload.

## Eval Results

Eval artifacts, when present, are uploaded under `evals/`.