gpt2-citations/README.md

---
library_name: transformers
tags:
- text-generation
- gpt2
- fine-tuned
- citations
- causal-lm
---

# GPT‑2 Fine‑tuned on English Quotes

## Model Description

This model is a fine‑tuned version of [GPT‑2 small](https://huggingface.co/gpt2) (124M parameters) on the [Abirate/english_quotes](https://huggingface.co/datasets/Abirate/english_quotes) dataset.  
The goal is to generate text in the style of philosophical or literary quotes, including the author’s name.

**⚠️ This model was created for educational and research purposes only. It is not intended for production use.**  
It demonstrates full fine‑tuning of a causal language model on a small dataset and the improvements in generation quality compared to the base model.

**Base model**: `gpt2`  
**Task**: Causal language modelling (text generation)  
**Fine‑tuning type**: Full fine‑tuning (all parameters updated)

## Intended Uses & Limitations

### Direct Use (Research / Experimentation)

You can use this model to generate short quotes given a prompt. The model expects prompts to start with the special token `<|startoftext|>` and will learn to produce a quote followed by an author and the `<|endoftext|>` token.

**Example**:
```python
from transformers import pipeline

generator = pipeline("text-generation", model="lorcannrauzduel/gpt2-citations")
output = generator("<|startoftext|> The secret to", max_new_tokens=50, do_sample=True)
print(output[0]['generated_text'])
```

### Limitations

- The model is small (124M) and was trained on only ~2,500 quotes. It may sometimes produce repetitive or nonsensical outputs.
- It only generates English text.
- It does not have factual knowledge about the authors; it merely mimics the style of the training quotes.
- **Not suitable for any commercial or critical application.**

## Training Details

### Training Data

- **Dataset**: [Abirate/english_quotes](https://huggingface.co/datasets/Abirate/english_quotes) – 2,508 quotes, each with a `quote` and an `author` field.
- **Preprocessing**: Each example was formatted as:
  ```
  <|startoftext|> "quote" — author <|endoftext|>
  ```
  The special tokens help the model learn where a quote starts and ends.

### Training Procedure

The model was trained for 5 epochs using the Hugging Face `Trainer` with the following hyperparameters:

| Hyperparameter          | Value |
|------------------------|-------|
| Learning rate          | 5e-5  |
| Batch size (per device)| 8     |
| Gradient accumulation  | 2     |
| Effective batch size   | 16    |
| Warmup steps           | 100   |
| Weight decay           | 0.01  |
| Optimizer              | AdamW |
| Precision              | fp16  |
| Max sequence length    | 128   |
| Training steps         | 1410  |

**Hardware**: NVIDIA Tesla T4 (15 GB VRAM) on Google Colab / Kaggle.  
**Training time**: ~5 minutes.

### Evaluation Results

The final training loss was **2.506**, corresponding to a perplexity of **12.26**.  
Validation loss stagnated around 2.30, indicating a slight overfitting after 3‑4 epochs – acceptable for a small generative model.

## How to Use the Model

### With 🤗 Transformers

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("lorcannrauzduel/gpt2-citations")
model = AutoModelForCausalLM.from_pretrained("lorcannrauzduel/gpt2-citations")

prompt = "<|startoftext|> Life is"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=False))
```

### With Pipeline

```python
from transformers import pipeline

pipe = pipeline("text-generation", model="lorcannrauzduel/gpt2-citations")
print(pipe("<|startoftext|> You can never", max_new_tokens=50)[0]['generated_text'])
```

### With vLLM (for high‑throughput inference)

```bash
pip install vllm
vllm serve "lorcannrauzduel/gpt2-citations"
```

Then query with curl:

```bash
curl -X POST "http://localhost:8000/v1/completions" \
     -H "Content-Type: application/json" \
     --data '{
         "model": "lorcannrauzduel/gpt2-citations",
         "prompt": "<|startoftext|> The secret to",
         "max_tokens": 50,
         "temperature": 0.8
     }'
```

### With Ollama (local deployment after GGUF conversion)

1. Download the GGUF version from the repository (if available) or convert it yourself using `llama.cpp`.
2. Create a `Modelfile`:
   ```
   FROM ./gpt2-citations-q4km.gguf
   SYSTEM "You are a quote generator."
   PARAMETER temperature 0.8
   PARAMETER stop "<|endoftext|>"
   ```
3. Import and run:
   ```bash
   ollama create gpt2-citations -f Modelfile
   ollama run gpt2-citations "<|startoftext|> Life is"
   ```

## Model Comparison (Base vs Fine‑tuned)

| Prompt | GPT‑2 Base (no fine‑tuning) | GPT‑2 Fine‑tuned |
|--------|-----------------------------|------------------|
| `<|startoftext|> The secret to` | "The secret to making the most of your life... [[...]]" | "The secret to happiness is to trust your instincts rather than your brain.” — Albert Einstein" |
| `<|startoftext|> Life is` | "Life is precious and life is precious..." | "Life is full of opportunities, but few opportunities are worth the time..." — Jodi Picoult |
| `<|startoftext|> You can never` | "You can never get around to finding out what you want..." | "You can never lose your way because you are still thinking about the things you've done..." |

The fine‑tuned model consistently produces coherent quotes with an author attribution, while the base model generates irrelevant or repetitive text.

## Environmental Impact

Training was performed on a cloud GPU (Tesla T4) for about 5 minutes. Estimated CO₂ emissions are negligible (< 0.01 kg CO₂eq).

## Acknowledgements

- The [Hugging Face](https://huggingface.co) team for `transformers` and `datasets`.
- The original GPT‑2 paper by Radford et al. (2019).
- Dataset provided by [Abirate](https://huggingface.co/Abirate).

## License

This model is released under the MIT license (same as the original GPT‑2 small).

---

**Model card created by [lorcannrauzduel](https://huggingface.co/lorcannrauzduel) for research and experimentation purposes.**