---
license: apache-2.0
language:
- en
datasets:
- Skylion007/openwebtext
- Salesforce/wikitext
pipeline_tag: text-generation
model_size: 33M
---
# CLover T1 0.03B (Base)

CLover T1 (or Chat Lover Transformer v1) is a small-scale causal language model designed for efficient text generation and experimentation with compact transformer architectures.

This model represents the base pretraining stage and is intended as a foundation for further instruction tuning and architectural exploration.

---

## Model Overview

- Model size: ~33M parameters
- Architecture: GPT-style decoder-only transformer
- Context length: 512 tokens
- Vocabulary: BPE Tokenizer

This version uses a traditional transformer architecture without modern enhancements such as SwiGLU feed-forward networks or rotary positional embeddings.

---

## Intended Use

This model is suitable for:

- Text continuation
- Basic conversational experiments
- Research on small language model behavior
- Prototyping instruction-tuning pipelines

It is not instruction-tuned and does not consistently follow user commands in a chat-like manner.

---

## Limitations

Because this is a base pretrained model:

- It may produce vague or incomplete answers
- It does not reliably follow instructions
- It can lose coherence in longer outputs
- It may generate repetitive or inconsistent text

These limitations are expected for a model at this scale and training stage.

---

## How to Use

You can load and run the model using Hugging Face Transformers:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "rkaluzny/CLoverT1-0.03B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "What is machine learning?"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_k=50,
    top_p=0.9,
    repetition_penalty=1.2,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))