gpt2-irish-folk-tune-generator/README.md

---
language:
- en
license: mit
library_name: transformers
base_model: openai-community/gpt2
datasets:
- sander-wood/irishman
tags:
- text-generation
- music-generation
- abc-notation
- irish-music
- gpt2
- full-finetune
pipeline_tag: text-generation
---

Try the live demo Space: https://huggingface.co/spaces/Vedant3907/gpt2-irish-folk-tune-generator-space


<video controls width="100%">
  <source src="https://huggingface.co/Vedant3907/gpt2-irish-folk-tune-generator/resolve/main/assets/irish-gpt2-model.mp4" type="video/mp4">
</video>

Weekend project demo: an original dataset tune is played first, followed by a GPT-2 generated tune from similar ABC notation context.


# GPT-2 Irish ABC Tune Generator

This model is a full fine-tune of `openai-community/gpt2` on the
`sander-wood/irishman` dataset, a collection of Irish folk tunes represented in
ABC notation.

The model generates symbolic music text, not audio. Generated output can be
pasted into an ABC player such as [abc.rectanglered.com](https://abc.rectanglered.com/)
or [ABCjs Editor](https://editor.drawthedots.com/) to hear the tune.

## Model Details

- **Base model:** `openai-community/gpt2`
- **Training method:** Full fine-tuning
- **Dataset:** `sander-wood/irishman`
- **Task:** Causal language modeling / ABC notation continuation
- **Max sequence length:** 512 tokens
- **Training hardware:** Google Colab T4 GPU
- **Training duration:** Approximately one epoch
- **Validation loss:** `0.9962592720985413`

## What The Model Learns

The training text was formatted as:

```text
<control code>
<ABC notation>
```

So a prompt can start with a control code, and the model will continue by
generating ABC notation.

Example prompt:

```text
S:2 B:8 E:6 B:8
```

The output should look like ABC music notation with headers and note sequences.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Vedant3907/gpt2-irish-folk-tune-generator"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()

def generate_tune(prompt="S:2 B:8 E:6 B:8\n", max_new_tokens=400):
    inputs = tokenizer(prompt, return_tensors="pt").to(device)
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=0.9,
            top_k=40,
            top_p=0.95,
            pad_token_id=tokenizer.eos_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(output[0], skip_special_tokens=True)

print(generate_tune())
```

## Example Prompt

```text
S:2 B:8 E:6 B:8
```

If the generated output does not include a complete ABC header, add one manually
before playing it:

```text
X:1
M:4/4
L:1/8
K:G
```

Then paste the full ABC text into:

- <https://abc.rectanglered.com/>
- <https://editor.drawthedots.com/>
- <https://thesession.org/tunes/convert>

## Evaluation

The model was evaluated on the dataset validation split.

```text
eval_loss: 0.9962592720985413
```

The final logged training loss was around `1.02`, and the validation loss was
close to that value, suggesting the model did not obviously overfit during this
run.



## Training Loss Table

Compact view of the training loss curve. The full logged loss table is available in `training_loss_curve.csv`.

| Step | Training Loss |
|---:|---:|
| 50 | 3.143394 |
| 500 | 1.461160 |
| 1000 | 1.315162 |
| 1500 | 1.255494 |
| 2000 | 1.233580 |
| 2500 | 1.160884 |
| 3000 | 1.140212 |
| 3500 | 1.128867 |
| 4000 | 1.115015 |
| 4500 | 1.089873 |
| 5000 | 1.078019 |
| 5500 | 1.059447 |
| 6000 | 1.082058 |
| 6500 | 1.072979 |
| 7000 | 1.060261 |
| 7500 | 1.066326 |
| 8000 | 1.051891 |
| 8500 | 1.054574 |
| 9000 | 1.058629 |
| 9500 | 1.041122 |
| 10000 | 1.046246 |
| 10500 | 1.023033 |
| 11000 | 1.031550 |
| 11500 | 1.030448 |
| 12000 | 1.031115 |
| 12500 | 1.029919 |
| 12800 | 1.029599 |

## Limitations

- The model generates ABC notation, not direct audio.
- Some generations may be syntactically invalid ABC.
- Some outputs may need manual cleanup before playback.
- The model may reproduce fragments or patterns from the training data.
- Musical quality varies; sampling multiple outputs and selecting the best one
is recommended.

## Intended Use

This model is intended for experimentation with Irish folk tune generation,
symbolic music modeling, and ABC notation text generation.

It is not intended for claims of originality or commercial music production
without additional review for memorization and licensing concerns.

## Training Summary

This was trained as a practical free-GPU fine-tuning experiment after the
original Karpathy `autoresearch` training setup proved unsuitable for Google
Colab's free T4 GPU. Instead of training from scratch, this model uses GPT-2 as
a pretrained base and adapts it to Irish ABC notation.