82 lines
2.0 KiB
Markdown
82 lines
2.0 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
datasets:
|
|
- Skylion007/openwebtext
|
|
- Salesforce/wikitext
|
|
pipeline_tag: text-generation
|
|
model_size: 33M
|
|
---
|
|
# CLover T1 0.03B (Base)
|
|
|
|
CLover T1 (or Chat Lover Transformer v1) is a small-scale causal language model designed for efficient text generation and experimentation with compact transformer architectures.
|
|
|
|
This model represents the base pretraining stage and is intended as a foundation for further instruction tuning and architectural exploration.
|
|
|
|
---
|
|
|
|
## Model Overview
|
|
|
|
- Model size: ~33M parameters
|
|
- Architecture: GPT-style decoder-only transformer
|
|
- Context length: 512 tokens
|
|
- Vocabulary: BPE Tokenizer
|
|
|
|
This version uses a traditional transformer architecture without modern enhancements such as SwiGLU feed-forward networks or rotary positional embeddings.
|
|
|
|
---
|
|
|
|
## Intended Use
|
|
|
|
This model is suitable for:
|
|
|
|
- Text continuation
|
|
- Basic conversational experiments
|
|
- Research on small language model behavior
|
|
- Prototyping instruction-tuning pipelines
|
|
|
|
It is not instruction-tuned and does not consistently follow user commands in a chat-like manner.
|
|
|
|
---
|
|
|
|
## Limitations
|
|
|
|
Because this is a base pretrained model:
|
|
|
|
- It may produce vague or incomplete answers
|
|
- It does not reliably follow instructions
|
|
- It can lose coherence in longer outputs
|
|
- It may generate repetitive or inconsistent text
|
|
|
|
These limitations are expected for a model at this scale and training stage.
|
|
|
|
---
|
|
|
|
## How to Use
|
|
|
|
You can load and run the model using Hugging Face Transformers:
|
|
|
|
```python
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
import torch
|
|
|
|
model_name = "rkaluzny/CLoverT1-0.03B-Base"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
model = AutoModelForCausalLM.from_pretrained(model_name)
|
|
|
|
prompt = "What is machine learning?"
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
|
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=256,
|
|
temperature=0.7,
|
|
top_k=50,
|
|
top_p=0.9,
|
|
repetition_penalty=1.2,
|
|
do_sample=True,
|
|
)
|
|
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |