70 lines
1.5 KiB
Markdown
70 lines
1.5 KiB
Markdown
|
|
||
|
|
---
|
||
|
|
language: en
|
||
|
|
tags:
|
||
|
|
- gpt2
|
||
|
|
- text-generation
|
||
|
|
- children-stories
|
||
|
|
- tinystories
|
||
|
|
license: mit
|
||
|
|
---
|
||
|
|
|
||
|
|
# TinyStories GPT2 124M
|
||
|
|
|
||
|
|
A GPT2 model trained from scratch on the
|
||
|
|
TinyStories dataset to generate children's stories.
|
||
|
|
|
||
|
|
## Training Details
|
||
|
|
- **Base Architecture:** GPT2 (124M parameters)
|
||
|
|
- **Dataset:** karpathy/tinystories-gpt4-clean
|
||
|
|
- **Training Steps:** 100,000
|
||
|
|
- **Best Val Loss:** 1.1295
|
||
|
|
- **Hardware:** NVIDIA RTX PRO 6000 (G4)
|
||
|
|
|
||
|
|
## How To Use
|
||
|
|
```python
|
||
|
|
from transformers import GPT2LMHeadModel
|
||
|
|
from transformers import GPT2TokenizerFast
|
||
|
|
import torch
|
||
|
|
|
||
|
|
model = GPT2LMHeadModel.from_pretrained(
|
||
|
|
"{HF_USERNAME}/{MODEL_NAME}"
|
||
|
|
)
|
||
|
|
tokenizer = GPT2TokenizerFast.from_pretrained(
|
||
|
|
"{HF_USERNAME}/{MODEL_NAME}"
|
||
|
|
)
|
||
|
|
|
||
|
|
prompt = "Once upon a time there was a little cat"
|
||
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
||
|
|
|
||
|
|
outputs = model.generate(
|
||
|
|
inputs["input_ids"],
|
||
|
|
max_new_tokens = 200,
|
||
|
|
temperature = 0.8,
|
||
|
|
top_p = 0.9,
|
||
|
|
do_sample = True,
|
||
|
|
repetition_penalty = 1.2,
|
||
|
|
pad_token_id = tokenizer.eos_token_id
|
||
|
|
)
|
||
|
|
|
||
|
|
story = tokenizer.decode(
|
||
|
|
outputs[0],
|
||
|
|
skip_special_tokens = True
|
||
|
|
)
|
||
|
|
print(story)
|
||
|
|
```
|
||
|
|
|
||
|
|
## Example Output
|
||
|
|
|
||
|
|
"Once upon a time there was a little cat called
|
||
|
|
Mimi. She loved to play with her toys, but one
|
||
|
|
day she got very sad because she couldn't find
|
||
|
|
her favorite toy. They searched everywhere and
|
||
|
|
finally found it under the bed! Mimi was so
|
||
|
|
happy and hugged her mom tight."
|
||
|
|
|
||
|
|
## Limitations
|
||
|
|
- Generates children's stories only
|
||
|
|
- Works best with story-style prompts
|
||
|
|
- 512 token context window
|