---
license: mit
language:
- tr
pipeline_tag: text-generation
tags:
- transformer
- gpt2
- turkish
- baseline
links:
  - label: "Experiment Paper"
    url: "https://huggingface.co/oguzatas/mamba-tr-project-transformer/resolve/main/OguzhanAtasHW%20state-space-models%20on%20Morphologically%20Rich%20Languages.pdf"
---
---

[![Paper](https://img.shields.io/badge/PDF-Technical_Paper-red?style=for-the-badge&logo=adobeacrobatreader&logoColor=white)](https://huggingface.co/oguzatas/mamba-tr-project-transformer/resolve/main/OguzhanAtasHW%20state-space-models%20on%20Morphologically%20Rich%20Languages.pdf)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1UbkR-i3P6X2cFVWXlntySjki_WBdg0ac?usp=sharing)
# Transformer Baseline (GPT-2 Style) - Turkish

This model serves as the **baseline** for a comparative study between Transformer and Mamba architectures on agglutinative languages (specifically Turkish). It is a decoder-only Transformer model (~111M parameters) trained on the Turkish Wikipedia dataset.

## Model Description
- **Architecture:** GPT-2 (Small)
- **Parameters:** ~111 Million
- **Context Length:** 1024
- **Training Data:** Turkish Wikipedia (Nov 2023)
- **Purpose:** To provide a performance benchmark for the Mamba architecture.

## Usage

```python
from transformers import GPT2LMHeadModel, PreTrainedTokenizerFast

model_id = "oguzatas/mamba-tr-project-transformer"
tokenizer_id = "oguzatas/mamba-tr-project-tokenizer"

tokenizer = PreTrainedTokenizerFast.from_pretrained(tokenizer_id)
model = GPT2LMHeadModel.from_pretrained(model_id)

text = "Türkiye'nin başkenti"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=20)

print(tokenizer.decode(outputs[0]))