68 lines
2.0 KiB
Markdown
68 lines
2.0 KiB
Markdown
|
# TinyTimV1: Fine-tuning TinyLlama on Finnegan's Wake
|
||
|
|
|
||
|
|
A project exploring the fine-tuning of TinyLlama-1.1B on James Joyce's *Finnegan's Wake* to generate Joyce-inspired text.
|
||
|
|
|
||
|
|
## Overview
|
||
|
|
|
||
|
|
This project fine-tunes the TinyLlama-1.1B-Chat model on the complete text of James Joyce's *Finnegan's Wake*, creating a language model capable of generating text in Joyce's distinctive experimental style. The model learns to replicate the complex wordplay, neologisms, and stream-of-consciousness narrative techniques characteristic of Joyce's final work.
|
||
|
|
|
||
|
|
## Files
|
||
|
|
|
||
|
|
- `process_wake.py` - Preprocesses the raw text, removes page numbers, and splits into manageable chunks
|
||
|
|
- `fine_tune_joyce.py` - Main training script using HuggingFace Transformers
|
||
|
|
- `text_gen.py` - Text generation script for the fine-tuned model
|
||
|
|
- `finn_wake.txt` - Complete text of Finnegan's Wake (1.51 MB)
|
||
|
|
- `finn_wake.csv` - Processed dataset in CSV format
|
||
|
|
- `finn_wake_dataset/` - Tokenized dataset directory
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
### 1. Data Preprocessing
|
||
|
|
```bash
|
||
|
|
python process_wake.py
|
||
|
|
```
|
||
|
|
This removes page numbers and splits the text into 100-word chunks for training.
|
||
|
|
2. Fine-tuning
|
||
|
|
|
||
|
|
```bash
|
||
|
|
python fine_tune_joyce.py
|
||
|
|
```
|
||
|
|
Fine-tunes TinyLlama on the processed dataset for 3 epochs with CPU training.
|
||
|
|
3. Text Generation
|
||
|
|
```bash
|
||
|
|
python text_gen.py
|
||
|
|
```
|
||
|
|
Generates Joyce-inspired text using the fine-tuned model.
|
||
|
|
|
||
|
|
Model Details
|
||
|
|
|
||
|
|
Base Model: TinyLlama-1.1B-Chat-v1.0
|
||
|
|
Training Data: Finnegan's Wake (~1.5MB text)
|
||
|
|
Training Parameters:
|
||
|
|
|
||
|
|
3 epochs
|
||
|
|
Batch size: 1
|
||
|
|
Max sequence length: 128 tokens
|
||
|
|
Temperature: 0.7
|
||
|
|
Top-k: 50, Top-p: 0.95
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
Example Output
|
||
|
|
Input: "ae left to go to ireland and found a fairy"
|
||
|
|
The model generates text continuing in Joyce's experimental style with invented words, Irish references, and complex linguistic play.
|
||
|
|
Requirements
|
||
|
|
transformers
|
||
|
|
datasets
|
||
|
|
pandas
|
||
|
|
torch
|
||
|
|
Installation
|
||
|
|
bashpip install transformers datasets pandas torch
|
||
|
|
Notes
|
||
|
|
|
||
|
|
Training was performed on CPU due to resource constraints
|
||
|
|
Model checkpoints saved every 500 steps
|
||
|
|
Resume training supported from checkpoints
|
||
|
|
|
||
|
|
|