---
language:
- en
license: apache-2.0
tags:
- text-generation
- pytorch
- causal-lm
- occult
- supernatural
- magic
- coding
- nsfw
base_model: distilgpt2
datasets:
- WithinUsAI/Supernatural_25k
- WithinUsAI/high_priest_occult_25k
- WithinUsAI/high_priest_supernatural_magic_FACT_BASED_1M
- WithinUsAI/gods_universe_codex_distill_god_seed_25k
- jjmachan/NSFW-reddit
---

# distilgpt2-supernatural-occult-coder

## Model Details

### Model Description
This model is a full fine-tuned version of **DistilGPT2**, specialized in generating a unique blend of text covering the occult, supernatural magic, coding/programming, and mature internet discourse. It was trained comprehensively on a massive merged dataset of over 1.5 million rows to synthesize these distinct themes into a single generative framework.

As a lightweight causal language model (82M parameters), it is optimized for extremely fast text generation across varied esoteric and technical domains.

- **Developed by:** GODsStrongestSoldier
- **Model type:** Causal Language Model (Transformer Decoder)
- **Language:** English
- **License:** Apache 2.0
- **Finetuned from model:** `distilgpt2`

---

## Datasets Used for Fine-Tuning
This model was trained on a concatenated corpus consisting of the following datasets:
- [WithinUsAI/high_priest_occult_25k](https://huggingface.co/datasets/WithinUsAI/high_priest_occult_25k)
- [WithinUsAI/gods_universe_codex_distill_god_seed_25k](https://huggingface.co/datasets/WithinUsAI/gods_universe_codex_distill_god_seed_25k)
- [WithinUsAI/Supernatural_25k](https://huggingface.co/datasets/WithinUsAI/Supernatural_25k)
- [WithinUsAI/high_priest_supernatural_magic_FACT_BASED_1M](https://huggingface.co/datasets/WithinUsAI/high_priest_supernatural_magic_FACT_BASED_1M)
- [acheong08/nsfw_reddit](https://huggingface.co/datasets/acheong08/nsfw_reddit)

---

## Training Details

### Training Procedure
The model underwent **full fine-tuning** (no LoRA or adapters). All layers of the base model were globally updated. Datasets were dynamically loaded, stripped of extraneous columns, converted entirely to text, concatenated, and shuffled with a fixed seed to ensure an even distribution of themes throughout the training process. 

Texts were grouped into continuous sequences of 512 tokens.

#### Hardware
- **Environment:** Kaggle
- **Accelerators:** Dual NVIDIA T4 GPUs (15GB VRAM each)

#### Hyperparameters
- **Epochs:** 1 (Due to the massive 1.5M+ row dataset size)
- **Per-Device Batch Size:** 8
- **Gradient Accumulation Steps:** 8
- **Effective Global Batch Size:** 128
- **Learning Rate:** 5e-05
- **Optimizer:** Fused AdamW (`adamw_torch_fused`)
- **Mixed Precision:** fp16
- **Gradient Checkpointing:** Enabled