856c482bd3
Model: GODsStrongestSoldier/distilgpt2-supernatural-occult-coder Source: Original Platform
2.7 KiB
language, license, tags, base_model, datasets
| language | license | tags | base_model | datasets | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
apache-2.0 |
|
distilgpt2 |
|
distilgpt2-supernatural-occult-coder
Model Details
Model Description
This model is a full fine-tuned version of DistilGPT2, specialized in generating a unique blend of text covering the occult, supernatural magic, coding/programming, and mature internet discourse. It was trained comprehensively on a massive merged dataset of over 1.5 million rows to synthesize these distinct themes into a single generative framework.
As a lightweight causal language model (82M parameters), it is optimized for extremely fast text generation across varied esoteric and technical domains.
- Developed by: GODsStrongestSoldier
- Model type: Causal Language Model (Transformer Decoder)
- Language: English
- License: Apache 2.0
- Finetuned from model:
distilgpt2
Datasets Used for Fine-Tuning
This model was trained on a concatenated corpus consisting of the following datasets:
- WithinUsAI/high_priest_occult_25k
- WithinUsAI/gods_universe_codex_distill_god_seed_25k
- WithinUsAI/Supernatural_25k
- WithinUsAI/high_priest_supernatural_magic_FACT_BASED_1M
- acheong08/nsfw_reddit
Training Details
Training Procedure
The model underwent full fine-tuning (no LoRA or adapters). All layers of the base model were globally updated. Datasets were dynamically loaded, stripped of extraneous columns, converted entirely to text, concatenated, and shuffled with a fixed seed to ensure an even distribution of themes throughout the training process.
Texts were grouped into continuous sequences of 512 tokens.
Hardware
- Environment: Kaggle
- Accelerators: Dual NVIDIA T4 GPUs (15GB VRAM each)
Hyperparameters
- Epochs: 1 (Due to the massive 1.5M+ row dataset size)
- Per-Device Batch Size: 8
- Gradient Accumulation Steps: 8
- Effective Global Batch Size: 128
- Learning Rate: 5e-05
- Optimizer: Fused AdamW (
adamw_torch_fused) - Mixed Precision: fp16
- Gradient Checkpointing: Enabled