CLoverT1-0.03B-Base/README.md

license, language, datasets, pipeline_tag, model_size

license	language	datasets	pipeline_tag	model_size
apache-2.0	en	Skylion007/openwebtext Salesforce/wikitext	text-generation	33M

CLover T1 0.03B (Base)

CLover T1 (or Chat Lover Transformer v1) is a small-scale causal language model designed for efficient text generation and experimentation with compact transformer architectures.

This model represents the base pretraining stage and is intended as a foundation for further instruction tuning and architectural exploration.

---

Model Overview

• Model size: ~33M parameters
• Architecture: GPT-style decoder-only transformer
• Context length: 512 tokens
• Vocabulary: BPE Tokenizer

This version uses a traditional transformer architecture without modern enhancements such as SwiGLU feed-forward networks or rotary positional embeddings.

---

Intended Use

This model is suitable for:

• Text continuation
• Basic conversational experiments
• Research on small language model behavior
• Prototyping instruction-tuning pipelines

It is not instruction-tuned and does not consistently follow user commands in a chat-like manner.

---

Limitations

Because this is a base pretrained model:

• It may produce vague or incomplete answers
• It does not reliably follow instructions
• It can lose coherence in longer outputs
• It may generate repetitive or inconsistent text

These limitations are expected for a model at this scale and training stage.

---

How to Use

You can load and run the model using Hugging Face Transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "rkaluzny/CLoverT1-0.03B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "What is machine learning?"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_k=50,
    top_p=0.9,
    repetition_penalty=1.2,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))