---
license: apache-2.0
language:
- en
library_name: transformers
tags:
- transformers
- pytorch
- safetensors
- veyra
- causal-lm
- base-model
- small-language-model
- pretraining
pipeline_tag: text-generation
datasets:
- HuggingFaceTB/cosmopedia-v2
parameters: 31988224
new_version: veyra-ai/veyra-30m-base-5b-tokens
---

# Veyra 30M Base 2.5B Checkpoint !! 5B CHECKPOINT OUT NOW !!

This is an early **Veyra-30M base checkpoint** trained for approximately **2.5B pretraining tokens**.

It is **not instruction tuned** and should not be evaluated like a finished chat assistant. It is expected to hallucinate, repeat, fail simple factual/math prompts, and continue text in odd ways. This checkpoint is uploaded for transparency, reproducibility, and milestone tracking before further continuation training.

## Training summary

Approximate training stages:

- **1B tokens**: Cosmopedia v2 bootstrap pretraining.
- **+1.5B tokens**: mixed continuation using Cosmopedia-v2 repository configs including `cosmopedia-v2`, `fineweb-edu-dedup`, and `python-edu`.
- **Total**: about **2.5B pretraining tokens**.

## Architecture

Veyra-30M is a small attention-sparse decoder-only language model.

Key details:

- Exact parameters: **31,988,224** / **31.99M**
- Vocabulary: 8,192 tokens
- Hidden size: 512
- Layers: 8
- Attention heads: 8 query heads, 2 KV heads
- MLP intermediate size: 2048
- Activation: SwiGLU
- Normalization: RMSNorm
- Position encoding: RoPE
- Tied token embeddings / LM head
- Context in this checkpoint: 512 tokens

## Loading

This repository uses custom Transformers code.

Minimal usage:

    from transformers import AutoTokenizer, AutoModelForCausalLM
    import torch

    repo = "veyra-ai/veyra-30m-base-2.5b-tokens"

    tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
    model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32)
    model.eval()

    prompt = "Photosynthesis is the process by which"
    input_ids = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")

    with torch.no_grad():
        out = model.generate(
            input_ids,
            do_sample=True,
            temperature=0.5,
            top_k=30,
            repetition_penalty=1.15,
            no_repeat_ngram_size=2,
            max_new_tokens=80,
        )

    print(tokenizer.decode(out[0], skip_special_tokens=True))

For raw completion prompts, use `add_special_tokens=False`.

## Optimizer

Training used:

- **CosineGatedAdam / CGA-v0** on 2D projection matrices
- **AdamW** on embeddings, norms, tied head, and auxiliary parameters

## Intended use

This checkpoint is primarily for:

- continued pretraining
- research / ablations
- tracking Veyra training milestones
- testing tiny model behavior

It is not intended for production use or reliable factual answering.

## Known limitations

This model can:

- hallucinate confidently
- repeat phrases
- fail arithmetic
- fail simple factual questions
- produce fake code
- continue in textbook-like or tutorial-like styles

Further continuation pretraining and post-training are planned.