69 lines
2.7 KiB
Markdown
69 lines
2.7 KiB
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
license: apache-2.0
|
||
|
|
tags:
|
||
|
|
- text-generation
|
||
|
|
- pytorch
|
||
|
|
- causal-lm
|
||
|
|
- occult
|
||
|
|
- supernatural
|
||
|
|
- magic
|
||
|
|
- coding
|
||
|
|
- nsfw
|
||
|
|
base_model: distilgpt2
|
||
|
|
datasets:
|
||
|
|
- WithinUsAI/Supernatural_25k
|
||
|
|
- WithinUsAI/high_priest_occult_25k
|
||
|
|
- WithinUsAI/high_priest_supernatural_magic_FACT_BASED_1M
|
||
|
|
- WithinUsAI/gods_universe_codex_distill_god_seed_25k
|
||
|
|
- jjmachan/NSFW-reddit
|
||
|
|
---
|
||
|
|
|
||
|
|
# distilgpt2-supernatural-occult-coder
|
||
|
|
|
||
|
|
## Model Details
|
||
|
|
|
||
|
|
### Model Description
|
||
|
|
This model is a full fine-tuned version of **DistilGPT2**, specialized in generating a unique blend of text covering the occult, supernatural magic, coding/programming, and mature internet discourse. It was trained comprehensively on a massive merged dataset of over 1.5 million rows to synthesize these distinct themes into a single generative framework.
|
||
|
|
|
||
|
|
As a lightweight causal language model (82M parameters), it is optimized for extremely fast text generation across varied esoteric and technical domains.
|
||
|
|
|
||
|
|
- **Developed by:** GODsStrongestSoldier
|
||
|
|
- **Model type:** Causal Language Model (Transformer Decoder)
|
||
|
|
- **Language:** English
|
||
|
|
- **License:** Apache 2.0
|
||
|
|
- **Finetuned from model:** `distilgpt2`
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Datasets Used for Fine-Tuning
|
||
|
|
This model was trained on a concatenated corpus consisting of the following datasets:
|
||
|
|
- [WithinUsAI/high_priest_occult_25k](https://huggingface.co/datasets/WithinUsAI/high_priest_occult_25k)
|
||
|
|
- [WithinUsAI/gods_universe_codex_distill_god_seed_25k](https://huggingface.co/datasets/WithinUsAI/gods_universe_codex_distill_god_seed_25k)
|
||
|
|
- [WithinUsAI/Supernatural_25k](https://huggingface.co/datasets/WithinUsAI/Supernatural_25k)
|
||
|
|
- [WithinUsAI/high_priest_supernatural_magic_FACT_BASED_1M](https://huggingface.co/datasets/WithinUsAI/high_priest_supernatural_magic_FACT_BASED_1M)
|
||
|
|
- [acheong08/nsfw_reddit](https://huggingface.co/datasets/acheong08/nsfw_reddit)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Training Details
|
||
|
|
|
||
|
|
### Training Procedure
|
||
|
|
The model underwent **full fine-tuning** (no LoRA or adapters). All layers of the base model were globally updated. Datasets were dynamically loaded, stripped of extraneous columns, converted entirely to text, concatenated, and shuffled with a fixed seed to ensure an even distribution of themes throughout the training process.
|
||
|
|
|
||
|
|
Texts were grouped into continuous sequences of 512 tokens.
|
||
|
|
|
||
|
|
#### Hardware
|
||
|
|
- **Environment:** Kaggle
|
||
|
|
- **Accelerators:** Dual NVIDIA T4 GPUs (15GB VRAM each)
|
||
|
|
|
||
|
|
#### Hyperparameters
|
||
|
|
- **Epochs:** 1 (Due to the massive 1.5M+ row dataset size)
|
||
|
|
- **Per-Device Batch Size:** 8
|
||
|
|
- **Gradient Accumulation Steps:** 8
|
||
|
|
- **Effective Global Batch Size:** 128
|
||
|
|
- **Learning Rate:** 5e-05
|
||
|
|
- **Optimizer:** Fused AdamW (`adamw_torch_fused`)
|
||
|
|
- **Mixed Precision:** fp16
|
||
|
|
- **Gradient Checkpointing:** Enabled
|