---
language:
- en
license: apache-2.0
pipeline_tag: text-generation
tags:
- llama
- causal-lm
- pretrained
- chytrej
- base
library_name: transformers
datasets:
- HuggingFaceFW/fineweb-edu
---

# Chytrej2-90M-Base

A fully custom pretrained language model built from scratch on the LLaMA architecture trained on [FineWeb Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset.

Built by [PingVortex Labs](https://github.com/PingVortexLabs).

[![Discord](https://img.shields.io/badge/Discord-5865F2?logo=discord&logoColor=white)](https://discord.gg/5SzkjVJBs2)

---

## Model Details

+ **Parameters:** 90M
+ **Context length:** 8,192 tokens
+ **Language:** English only
+ **Format:** Base model
+ **Architecture:** LLaMA
+ **License:** Apache 2.0

---

## Benchmarks

+ No benchmarks - at this scale the benchmarks are more random than based on what the model has learned - we got a checkpoint 55k get better arc easy score than the final model.

---

## Usage

```python
from transformers import LlamaForCausalLM, PreTrainedTokenizerFast

model = LlamaForCausalLM.from_pretrained("pvlabs/Chytrej2-90M-Base")
tokenizer = PreTrainedTokenizerFast.from_pretrained("pvlabs/Chytrej2-90M-Base")

prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, repetition_penalty=1.3)
print(tokenizer.decode(outputs[0]))
```

---

*Made by [PingVortex](https://pingvortex.com).*