112 lines
3.4 KiB
Markdown
112 lines
3.4 KiB
Markdown
---
|
||
license: mit
|
||
tags:
|
||
- generated_from_trainer
|
||
model-index:
|
||
- name: gpt2-shakespeare
|
||
results: []
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
||
should probably proofread and complete it, then remove this comment. -->
|
||
|
||
# gpt2-shakespeare
|
||
|
||
This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on [datasets](https://github.com/sadia-sust/dataset-finetune-gpt2) containing Shakespeare Books.
|
||
It achieves the following results on the evaluation set:
|
||
- Loss: 2.5738
|
||
|
||
## Model description
|
||
|
||
GPT-2 model is finetuned with text corpus.
|
||
|
||
## Intended uses & limitations
|
||
|
||
Intended use for this model is to write novel in Shakespeare Style. It has limitations to write in other writer's style.
|
||
|
||
## Datasets Description
|
||
|
||
Text corpus is developed for fine-tuning gpt-2 model. Books are downloaded from [Project Gutenberg](http://www.gutenberg.org/) as plain text files.
|
||
A large text corpus were needed to train the model to be abled to write in Shakespeare style.
|
||
|
||
|
||
The following books are used to develop text corpus:
|
||
|
||
- Macbeth, word count: 38197
|
||
- THE TRAGEDY OF TITUS ANDRONICUS, word count: 40413
|
||
- King Richard II, word count: 48423
|
||
- Shakespeare's Tragedy of Romeo and Juliet, word count: 144935
|
||
- A MIDSUMMER NIGHT’S DREAM, word count: 36597
|
||
- ALL’S WELL THAT ENDS WELL, word count: 49363
|
||
- THE TRAGEDY OF HAMLET, PRINCE OF DENMARK, word count: 57471
|
||
- THE TRAGEDY OF JULIUS CAESAR, word count: 37391
|
||
- THE TRAGEDY OF KING LEAR, word count: 54101
|
||
- THE LIFE AND DEATH OF KING RICHARD III, word count: 55985
|
||
- Romeo and Juliet, word count: 51417
|
||
- Measure for Measure, word count: 62703
|
||
- Much Ado about Nothing, word count: 45577
|
||
- Othello, the Moor of Venice, word count: 53967
|
||
- THE WINTER’S TALE, word count: 52911
|
||
- The Comedy of Errors, word count: 43179
|
||
- The Merchant of Venice, word count: 45903
|
||
- The Taming of the Shrew, word count: 44777
|
||
- The Tempest, word count: 32323
|
||
- TWELFTH NIGHT: OR, WHAT YOU WILL, word count: 42907
|
||
- The Sonnets, word count: 39849
|
||
|
||
Corpus has total 1078389 word tokens.
|
||
|
||
## Datasets Preprocessing
|
||
|
||
- Header text are removed manually.
|
||
- Using sent_tokenize() function from NLTK python library, extra spaces and new-lines were removed programmatically.
|
||
|
||
|
||
## Training and evaluation data
|
||
|
||
Training dataset has 880447 word tokens and test dataset has 197913 word tokens.
|
||
|
||
## Training procedure
|
||
|
||
To train the model, training api from Transformer class is used.
|
||
|
||
### Training hyperparameters
|
||
|
||
The following hyperparameters were used during training:
|
||
- learning_rate: 5e-05
|
||
- train_batch_size: 32
|
||
- eval_batch_size: 64
|
||
- seed: 42
|
||
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
|
||
- lr_scheduler_type: linear
|
||
- lr_scheduler_warmup_steps: 350
|
||
- num_epochs: 3
|
||
|
||
### Training results
|
||
|
||
| Training Loss | Epoch | Step | Validation Loss |
|
||
|:-------------:|:-----:|:----:|:---------------:|
|
||
| No log | 0.63 | 250 | 2.7133 |
|
||
| 2.8492 | 1.25 | 500 | 2.6239 |
|
||
| 2.8492 | 1.88 | 750 | 2.5851 |
|
||
| 2.3842 | 2.51 | 1000 | 2.5738 |
|
||
|
||
|
||
## Sample Code Using Transformers Pipeline
|
||
|
||
```
|
||
from transformers import pipeline
|
||
|
||
story = pipeline('text-generation',model='./gpt2-shakespeare', tokenizer='gpt2', max_length = 300)
|
||
story("how art thou")
|
||
|
||
```
|
||
|
||
|
||
### Framework versions
|
||
|
||
- Transformers 4.26.1
|
||
- Pytorch 1.13.1+cu116
|
||
- Datasets 2.10.0
|
||
- Tokenizers 0.13.2 |