gpt2-screenplay-generator/README.md

---
library_name: transformers
tags:
- gpt2
- causal-lm
- text-generation
- screenplay
- scriptwriting
- fine-tuned
language:
- en
license: mit
pipeline_tag: text-generation
datasets:
- raghavnimbalkar/movies-screenplays-tokenized-dataset
base_model:
- openai-community/gpt2
---

# GPT-2 Small — Screenplay Scriptwriting Model

> **Study Context:** This is the first model in a dual-architecture comparative study on screenplay generation using GPT-2 Small. This model is a full-parameter fine-tune executed on a cloud NVIDIA T4 GPU. The [second model](https://huggingface.co/raghavnimbalkar1/screenplay-gpt2-lora) is a LoRA adapter trained entirely on consumer edge hardware — an Apple Silicon MacBook Air — using PEFT to operate within the hard constraints of a fanless, unified-memory device.

> A fully fine-tuned GPT-2 Small (124M) causal language model trained end-to-end on ~94M tokens of professional screenplay corpora, with stateful MLOps checkpoint recovery from a mid-run hardware preemption event.

---

## Model Description

This model is a **full-parameter fine-tune** of OpenAI's [GPT-2 Small](https://huggingface.co/openai-community/gpt2) (124M parameters) for the task of **Causal Language Modeling** with a specialization in **screenplay and script generation**. Every one of the 124,439,808 parameters was unfrozen and updated during training — this is **not** a LoRA, adapter, or PEFT-based model. All weights have been fully overwritten from the base GPT-2 checkpoint.

The model has internalized the highly structured formatting conventions of professional screenplays: scene slugs (`INT./EXT.`), character action lines, dialogue blocks, parentheticals, and production draft metadata — making it capable of generating coherent, industry-formatted script content from open-ended prompts.

| Property              | Value                                      |
|-----------------------|--------------------------------------------|
| **Base Model**        | GPT-2 Small (`openai-community/gpt2`)      |
| **Parameter Count**   | 124,439,808 (100% updated)                 |
| **Architecture**      | Decoder-only Transformer (GPT-2)           |
| **Fine-tune Method**  | Full-Parameter Overwrite (no PEFT/LoRA)    |
| **Task**              | Causal Language Modeling / Script Generation |
| **Context Window**    | 512 tokens (contiguous)                    |
| **Language**          | English                                    |

---

## Training Data

The model was trained on a corpus of approximately **94 million tokens** of raw, professionally formatted screenplay text files. The dataset consists of:

- Standard industry-formatted `.fountain` / plain-text screenplay sources
- Scene slugline notation (`INT. LOCATION - DAY/NIGHT`)
- Character cues, action blocks, parentheticals, and dialogue
- Production draft metadata headers and transition markers

No dataset card is available at this time. The corpus was not filtered for content rating or genre — the model reflects the full stylistic and tonal range of the training material.

---

## Training Procedure & Infrastructure

### Compute Infrastructure

| Component              | Specification                              |
|------------------------|--------------------------------------------|
| **Accelerator**        | NVIDIA T4 Cloud GPU                        |
| **CUDA Backend**       | Enabled                                    |
| **Precision Strategy** | FP16 Mixed Precision (`torch.cuda.amp` via HF Accelerate) |

### Hyperparameters

| Hyperparameter                      | Value                       |
|-------------------------------------|-----------------------------|
| **Optimizer**                       | AdamW                       |
| **Learning Rate**                   | `5e-5` (linear decay)       |
| **per_device_train_batch_size**     | 4                           |
| **gradient_accumulation_steps**     | 4                           |
| **Effective Global Batch Size**     | 16                          |
| **Total Optimization Steps**        | 9,272 (1 full epoch)        |
| **Total FLOs**                      | 3.876 × 10¹⁶               |

---

## MLOps Resiliency & Checkpoint Recovery

A defining characteristic of this training run is its **stateful recovery from a mid-training hardware preemption event**. The full timeline is documented below as an engineering reference.

### Timeline

```
[00:00:00] → Training initiated on primary cloud instance (T4 GPU).
                Checkpoints configured to persist every 200 global steps.

[04:43:00] → HARDWARE PREEMPTION at global Step 5,600 (60.4% complete).
                Primary compute container abruptly disconnected.
                Checkpoint preserved: model.safetensors, optimizer.pt, scheduler.pt

[04:43:xx] → Hot-resume initiated on secondary cloud instance from Step 5,601.
                Full optimizer state (momentum buffers, variance estimates),
                learning rate scheduler, and gradient context fully restored.

[07:43:30] → Training complete at global Step 9,272.
                Zero loss discontinuity detected across the resume boundary.
```

**Total aggregate compute time:** 7 hours, 43 minutes, 30 seconds across both instances.

The pre-crash and post-resume loss values at Steps 5,600 and 5,800 (see convergence table below) confirm **perfect gradient and loss continuity** with no regression caused by the preemption event. This demonstrates that HuggingFace's `Trainer`-native checkpoint serialization — saving full optimizer and scheduler state — is sufficient for lossless mid-run recovery on stateless cloud infrastructure.

---

## Training Metrics & Convergence

The model shows clear **asymptotic convergence** on screenplay formatting conventions and domain vocabulary across the full 9,272-step run.

| Global Step | Training Phase            | Validation Loss | Notes                                  |
|-------------|---------------------------|-----------------|----------------------------------------|
| **200**     | Baseline (early)          | 1.4586          | Initial domain vocabulary acquisition  |
| **2,000**   | Formatting alignment      | 1.3653          | Scene/dialogue structure stabilizing   |
| **5,600**   | Pre-crash state           | 1.3305          | Checkpoint preserved at preemption     |
| **5,800**   | Post-resume stability     | 1.3276          | Confirmed loss continuity after resume |
| **9,272**   | Final (absolute termination) | **1.3194**   | Convergence plateau reached            |

**Total loss reduction:** −0.1392 across the full run (−9.5% relative improvement from baseline).

The negligible delta between Steps 5,600 and 5,800 (−0.0029) confirms that the optimizer state was fully restored and training resumed without gradient shock or instability.

---

## Usage & Inference

### Loading the Model

```python
from transformers import GPT2LMHeadModel, GPT2Tokenizer

model_id = "raghavnimbalkar/gpt2-screenplay-generator"

tokenizer = GPT2Tokenizer.from_pretrained(model_id)
model = GPT2LMHeadModel.from_pretrained(model_id)
model.eval()
```

### Recommended Inference Parameters

The following nucleus sampling configuration is recommended to produce high-fidelity, coherent screenplay output while avoiding repetitive boilerplate:

| Parameter            | Recommended Value         | Notes                                             |
|----------------------|---------------------------|---------------------------------------------------|
| `max_length`         | Up to `512`               | Hard context window limit                         |
| `temperature`        | `0.75` – `0.85`           | Lower = sharper dialogue; higher = creative variance |
| `top_k`              | `40` or `50`              | Limits vocabulary sampling pool                   |
| `top_p`              | `0.92` – `0.95`           | Nucleus sampling threshold                        |
| `repetition_penalty` | `1.12` – `1.15`           | **Critical** — prevents screenplay boilerplate loops |

### Inference Example

```python
import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

model_id = "raghavnimbalkar/gpt2-screenplay-generator"  

tokenizer = GPT2Tokenizer.from_pretrained(model_id)
model = GPT2LMHeadModel.from_pretrained(model_id)
model.eval()

prompt = "INT. POLICE PRECINCT - NIGHT\n\nDetective HARRIS slams a folder on the table."

inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_length=512,
        temperature=0.80,
        top_k=50,
        top_p=0.92,
        repetition_penalty=1.13,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )

generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)
```

> **Tip:** A `repetition_penalty` in the `1.12–1.15` range is especially important for this model. Screenplay corpora contain many repeated structural tokens (`INT.`, `EXT.`, `CUT TO:`, character cues) that, without penalty, the model will loop aggressively during unconstrained generation.

---

## Comparison with LoRA Adapter Model

This model is one half of an ongoing comparative study. The table below contrasts both trained models across architecture, compute, and convergence dimensions.

| Property                 | Full-Parameter (This Model)                  | LoRA Adapter (Local)                    |
|--------------------------|----------------------------------------------|-----------------------------------------|
| **Hardware**             | NVIDIA T4 (Cloud GPU)                        | Apple Silicon MacBook Air (MPS)         |
| **Fine-tune Method**     | Full-parameter overwrite                     | LoRA / PEFT (`c_attn` only)             |
| **Trainable Parameters** | 124,439,808 (100%)                           | 294,912 (0.24%)                         |
| **Epoch Coverage**       | 1.0 (full corpus)                            | 0.51 (half corpus)                      |
| **Total Steps**          | 9,272                                        | 4,700                                   |
| **Training Time**        | 7h 43m 30s                                   | 7h 51m 02s                              |
| **Final Eval Loss**      | **1.3194**                                   | 2.4017                                  |
| **Step Throughput**      | ~3.0s/step                                   | ~6.01s/step                             |
| **MLOps Event**          | Hardware preemption + stateful hot-resume    | 17× speedup via LoRA over full-param attempt |

Both models spent approximately the same wall-clock time training (~7h 45m). The divergence in final evaluation loss is a direct consequence of full-parameter depth and full corpus coverage versus adapter-based efficiency on constrained hardware — not a difference in compute investment. The LoRA adapter represents a deliberate trade-off: edge-feasibility over convergence depth.

---

## Intended Use

**Intended uses:**
- Screenplay drafting assistance and creative ideation
- Automated scene/dialogue continuation from a provided slug or action line
- Style transfer and scriptwriting research
- Educational exploration of domain-adaptive fine-tuning on structured text

**Out-of-scope uses:**
- Factual question answering (this is a generative, not retrieval, model)
- Production-ready script generation without human editorial review
- Any use case requiring truthfulness, citation, or factual accuracy

---

## Bias, Risks, and Limitations

- The model was trained on an unfiltered corpus spanning multiple genres and tones; it may generate content reflecting biases, stereotypes, or mature themes present in its training data.
- As a 124M parameter model, outputs are prone to incoherence over long sequences and may not maintain narrative or character consistency beyond a few exchanges.
- The model has no instruction-following capability; it is a raw next-token predictor conditioned on screenplay-formatted text.
- Users should apply content moderation filters appropriate for their deployment context.

---

## Environmental Impact

Carbon emissions were estimated using the [Machine Learning Impact Calculator](https://mlco2.github.io/impact#compute).

| Property            | Value                         |
|---------------------|-------------------------------|
| **Hardware Type**   | NVIDIA T4 (Cloud GPU)         |
| **Hours Used**      | ~7.72 hours (across 2 instances) |
| **Cloud Provider**  | *(Not disclosed)*             |
| **Compute Region**  | *(Not disclosed)*             |
| **Carbon Emitted**  | *0.31 kg*                     |

---

## Citation

If you reference this model or its training methodology in research, please cite the base model:

```bibtex
@article{radford2019language,
  title   = {Language Models are Unsupervised Multitask Learners},
  author  = {Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},
  year    = {2019}
}
```

---

## Model Card Contact

For questions about this fine-tune's training methodology, dataset, or inference behavior, please open an issue in this repository.