265 lines
13 KiB
Markdown
265 lines
13 KiB
Markdown
---
|
||
library_name: transformers
|
||
tags:
|
||
- gpt2
|
||
- causal-lm
|
||
- text-generation
|
||
- screenplay
|
||
- scriptwriting
|
||
- fine-tuned
|
||
language:
|
||
- en
|
||
license: mit
|
||
pipeline_tag: text-generation
|
||
datasets:
|
||
- raghavnimbalkar/movies-screenplays-tokenized-dataset
|
||
base_model:
|
||
- openai-community/gpt2
|
||
---
|
||
|
||
# GPT-2 Small — Screenplay Scriptwriting Model
|
||
|
||
> **Study Context:** This is the first model in a dual-architecture comparative study on screenplay generation using GPT-2 Small. This model is a full-parameter fine-tune executed on a cloud NVIDIA T4 GPU. The [second model](https://huggingface.co/raghavnimbalkar1/screenplay-gpt2-lora) is a LoRA adapter trained entirely on consumer edge hardware — an Apple Silicon MacBook Air — using PEFT to operate within the hard constraints of a fanless, unified-memory device.
|
||
|
||
> A fully fine-tuned GPT-2 Small (124M) causal language model trained end-to-end on ~94M tokens of professional screenplay corpora, with stateful MLOps checkpoint recovery from a mid-run hardware preemption event.
|
||
|
||
---
|
||
|
||
## Model Description
|
||
|
||
This model is a **full-parameter fine-tune** of OpenAI's [GPT-2 Small](https://huggingface.co/openai-community/gpt2) (124M parameters) for the task of **Causal Language Modeling** with a specialization in **screenplay and script generation**. Every one of the 124,439,808 parameters was unfrozen and updated during training — this is **not** a LoRA, adapter, or PEFT-based model. All weights have been fully overwritten from the base GPT-2 checkpoint.
|
||
|
||
The model has internalized the highly structured formatting conventions of professional screenplays: scene slugs (`INT./EXT.`), character action lines, dialogue blocks, parentheticals, and production draft metadata — making it capable of generating coherent, industry-formatted script content from open-ended prompts.
|
||
|
||
| Property | Value |
|
||
|-----------------------|--------------------------------------------|
|
||
| **Base Model** | GPT-2 Small (`openai-community/gpt2`) |
|
||
| **Parameter Count** | 124,439,808 (100% updated) |
|
||
| **Architecture** | Decoder-only Transformer (GPT-2) |
|
||
| **Fine-tune Method** | Full-Parameter Overwrite (no PEFT/LoRA) |
|
||
| **Task** | Causal Language Modeling / Script Generation |
|
||
| **Context Window** | 512 tokens (contiguous) |
|
||
| **Language** | English |
|
||
|
||
---
|
||
|
||
## Training Data
|
||
|
||
The model was trained on a corpus of approximately **94 million tokens** of raw, professionally formatted screenplay text files. The dataset consists of:
|
||
|
||
- Standard industry-formatted `.fountain` / plain-text screenplay sources
|
||
- Scene slugline notation (`INT. LOCATION - DAY/NIGHT`)
|
||
- Character cues, action blocks, parentheticals, and dialogue
|
||
- Production draft metadata headers and transition markers
|
||
|
||
No dataset card is available at this time. The corpus was not filtered for content rating or genre — the model reflects the full stylistic and tonal range of the training material.
|
||
|
||
---
|
||
|
||
## Training Procedure & Infrastructure
|
||
|
||
### Compute Infrastructure
|
||
|
||
| Component | Specification |
|
||
|------------------------|--------------------------------------------|
|
||
| **Accelerator** | NVIDIA T4 Cloud GPU |
|
||
| **CUDA Backend** | Enabled |
|
||
| **Precision Strategy** | FP16 Mixed Precision (`torch.cuda.amp` via HF Accelerate) |
|
||
|
||
### Hyperparameters
|
||
|
||
| Hyperparameter | Value |
|
||
|-------------------------------------|-----------------------------|
|
||
| **Optimizer** | AdamW |
|
||
| **Learning Rate** | `5e-5` (linear decay) |
|
||
| **per_device_train_batch_size** | 4 |
|
||
| **gradient_accumulation_steps** | 4 |
|
||
| **Effective Global Batch Size** | 16 |
|
||
| **Total Optimization Steps** | 9,272 (1 full epoch) |
|
||
| **Total FLOs** | 3.876 × 10¹⁶ |
|
||
|
||
---
|
||
|
||
## MLOps Resiliency & Checkpoint Recovery
|
||
|
||
A defining characteristic of this training run is its **stateful recovery from a mid-training hardware preemption event**. The full timeline is documented below as an engineering reference.
|
||
|
||
### Timeline
|
||
|
||
```
|
||
[00:00:00] → Training initiated on primary cloud instance (T4 GPU).
|
||
Checkpoints configured to persist every 200 global steps.
|
||
|
||
[04:43:00] → HARDWARE PREEMPTION at global Step 5,600 (60.4% complete).
|
||
Primary compute container abruptly disconnected.
|
||
Checkpoint preserved: model.safetensors, optimizer.pt, scheduler.pt
|
||
|
||
[04:43:xx] → Hot-resume initiated on secondary cloud instance from Step 5,601.
|
||
Full optimizer state (momentum buffers, variance estimates),
|
||
learning rate scheduler, and gradient context fully restored.
|
||
|
||
[07:43:30] → Training complete at global Step 9,272.
|
||
Zero loss discontinuity detected across the resume boundary.
|
||
```
|
||
|
||
**Total aggregate compute time:** 7 hours, 43 minutes, 30 seconds across both instances.
|
||
|
||
The pre-crash and post-resume loss values at Steps 5,600 and 5,800 (see convergence table below) confirm **perfect gradient and loss continuity** with no regression caused by the preemption event. This demonstrates that HuggingFace's `Trainer`-native checkpoint serialization — saving full optimizer and scheduler state — is sufficient for lossless mid-run recovery on stateless cloud infrastructure.
|
||
|
||
---
|
||
|
||
## Training Metrics & Convergence
|
||
|
||
The model shows clear **asymptotic convergence** on screenplay formatting conventions and domain vocabulary across the full 9,272-step run.
|
||
|
||
| Global Step | Training Phase | Validation Loss | Notes |
|
||
|-------------|---------------------------|-----------------|----------------------------------------|
|
||
| **200** | Baseline (early) | 1.4586 | Initial domain vocabulary acquisition |
|
||
| **2,000** | Formatting alignment | 1.3653 | Scene/dialogue structure stabilizing |
|
||
| **5,600** | Pre-crash state | 1.3305 | Checkpoint preserved at preemption |
|
||
| **5,800** | Post-resume stability | 1.3276 | Confirmed loss continuity after resume |
|
||
| **9,272** | Final (absolute termination) | **1.3194** | Convergence plateau reached |
|
||
|
||
**Total loss reduction:** −0.1392 across the full run (−9.5% relative improvement from baseline).
|
||
|
||
The negligible delta between Steps 5,600 and 5,800 (−0.0029) confirms that the optimizer state was fully restored and training resumed without gradient shock or instability.
|
||
|
||
---
|
||
|
||
## Usage & Inference
|
||
|
||
### Loading the Model
|
||
|
||
```python
|
||
from transformers import GPT2LMHeadModel, GPT2Tokenizer
|
||
|
||
model_id = "raghavnimbalkar/gpt2-screenplay-generator"
|
||
|
||
tokenizer = GPT2Tokenizer.from_pretrained(model_id)
|
||
model = GPT2LMHeadModel.from_pretrained(model_id)
|
||
model.eval()
|
||
```
|
||
|
||
### Recommended Inference Parameters
|
||
|
||
The following nucleus sampling configuration is recommended to produce high-fidelity, coherent screenplay output while avoiding repetitive boilerplate:
|
||
|
||
| Parameter | Recommended Value | Notes |
|
||
|----------------------|---------------------------|---------------------------------------------------|
|
||
| `max_length` | Up to `512` | Hard context window limit |
|
||
| `temperature` | `0.75` – `0.85` | Lower = sharper dialogue; higher = creative variance |
|
||
| `top_k` | `40` or `50` | Limits vocabulary sampling pool |
|
||
| `top_p` | `0.92` – `0.95` | Nucleus sampling threshold |
|
||
| `repetition_penalty` | `1.12` – `1.15` | **Critical** — prevents screenplay boilerplate loops |
|
||
|
||
### Inference Example
|
||
|
||
```python
|
||
import torch
|
||
from transformers import GPT2LMHeadModel, GPT2Tokenizer
|
||
|
||
model_id = "raghavnimbalkar/gpt2-screenplay-generator"
|
||
|
||
tokenizer = GPT2Tokenizer.from_pretrained(model_id)
|
||
model = GPT2LMHeadModel.from_pretrained(model_id)
|
||
model.eval()
|
||
|
||
prompt = "INT. POLICE PRECINCT - NIGHT\n\nDetective HARRIS slams a folder on the table."
|
||
|
||
inputs = tokenizer(prompt, return_tensors="pt")
|
||
|
||
with torch.no_grad():
|
||
output = model.generate(
|
||
**inputs,
|
||
max_length=512,
|
||
temperature=0.80,
|
||
top_k=50,
|
||
top_p=0.92,
|
||
repetition_penalty=1.13,
|
||
do_sample=True,
|
||
pad_token_id=tokenizer.eos_token_id,
|
||
)
|
||
|
||
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
|
||
print(generated_text)
|
||
```
|
||
|
||
> **Tip:** A `repetition_penalty` in the `1.12–1.15` range is especially important for this model. Screenplay corpora contain many repeated structural tokens (`INT.`, `EXT.`, `CUT TO:`, character cues) that, without penalty, the model will loop aggressively during unconstrained generation.
|
||
|
||
---
|
||
|
||
## Comparison with LoRA Adapter Model
|
||
|
||
This model is one half of an ongoing comparative study. The table below contrasts both trained models across architecture, compute, and convergence dimensions.
|
||
|
||
| Property | Full-Parameter (This Model) | LoRA Adapter (Local) |
|
||
|--------------------------|----------------------------------------------|-----------------------------------------|
|
||
| **Hardware** | NVIDIA T4 (Cloud GPU) | Apple Silicon MacBook Air (MPS) |
|
||
| **Fine-tune Method** | Full-parameter overwrite | LoRA / PEFT (`c_attn` only) |
|
||
| **Trainable Parameters** | 124,439,808 (100%) | 294,912 (0.24%) |
|
||
| **Epoch Coverage** | 1.0 (full corpus) | 0.51 (half corpus) |
|
||
| **Total Steps** | 9,272 | 4,700 |
|
||
| **Training Time** | 7h 43m 30s | 7h 51m 02s |
|
||
| **Final Eval Loss** | **1.3194** | 2.4017 |
|
||
| **Step Throughput** | ~3.0s/step | ~6.01s/step |
|
||
| **MLOps Event** | Hardware preemption + stateful hot-resume | 17× speedup via LoRA over full-param attempt |
|
||
|
||
Both models spent approximately the same wall-clock time training (~7h 45m). The divergence in final evaluation loss is a direct consequence of full-parameter depth and full corpus coverage versus adapter-based efficiency on constrained hardware — not a difference in compute investment. The LoRA adapter represents a deliberate trade-off: edge-feasibility over convergence depth.
|
||
|
||
---
|
||
|
||
## Intended Use
|
||
|
||
**Intended uses:**
|
||
- Screenplay drafting assistance and creative ideation
|
||
- Automated scene/dialogue continuation from a provided slug or action line
|
||
- Style transfer and scriptwriting research
|
||
- Educational exploration of domain-adaptive fine-tuning on structured text
|
||
|
||
**Out-of-scope uses:**
|
||
- Factual question answering (this is a generative, not retrieval, model)
|
||
- Production-ready script generation without human editorial review
|
||
- Any use case requiring truthfulness, citation, or factual accuracy
|
||
|
||
---
|
||
|
||
## Bias, Risks, and Limitations
|
||
|
||
- The model was trained on an unfiltered corpus spanning multiple genres and tones; it may generate content reflecting biases, stereotypes, or mature themes present in its training data.
|
||
- As a 124M parameter model, outputs are prone to incoherence over long sequences and may not maintain narrative or character consistency beyond a few exchanges.
|
||
- The model has no instruction-following capability; it is a raw next-token predictor conditioned on screenplay-formatted text.
|
||
- Users should apply content moderation filters appropriate for their deployment context.
|
||
|
||
---
|
||
|
||
## Environmental Impact
|
||
|
||
Carbon emissions were estimated using the [Machine Learning Impact Calculator](https://mlco2.github.io/impact#compute).
|
||
|
||
| Property | Value |
|
||
|---------------------|-------------------------------|
|
||
| **Hardware Type** | NVIDIA T4 (Cloud GPU) |
|
||
| **Hours Used** | ~7.72 hours (across 2 instances) |
|
||
| **Cloud Provider** | *(Not disclosed)* |
|
||
| **Compute Region** | *(Not disclosed)* |
|
||
| **Carbon Emitted** | *0.31 kg* |
|
||
|
||
---
|
||
|
||
## Citation
|
||
|
||
If you reference this model or its training methodology in research, please cite the base model:
|
||
|
||
```bibtex
|
||
@article{radford2019language,
|
||
title = {Language Models are Unsupervised Multitask Learners},
|
||
author = {Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},
|
||
year = {2019}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Model Card Contact
|
||
|
||
For questions about this fine-tune's training methodology, dataset, or inference behavior, please open an issue in this repository. |