49 lines
1.6 KiB
Markdown
49 lines
1.6 KiB
Markdown
---
|
|
base_model: 0xA50C1A1/Llama-3.3-8B-Instruct-128K-Heretic
|
|
tags:
|
|
- text-generation-inference
|
|
- transformers
|
|
- unsloth
|
|
- llama
|
|
- heretic
|
|
- uncensored
|
|
- decensored
|
|
- abliterated
|
|
- storytelling
|
|
- creative-writing
|
|
license: llama3.3
|
|
language:
|
|
- en
|
|
datasets:
|
|
- Gryphe/Opus-WritingPrompts
|
|
- Gryphe/ChatGPT-4o-Writing-Prompts
|
|
---
|
|
|
|
# Llama-3.3-8B-Instruct-OmniWriter
|
|
|
|
An experiment in creating a creative, uncensored storyteller based on the "smart" Llama 3.3 8B Instruct base.
|
|
|
|
### Training Specs
|
|
|
|
* **Training Method:** LoRA (16-bit)
|
|
* **LoRA Rank (r):** 32
|
|
* **LoRA Alpha:** 16
|
|
* **Scaling Type:** Rank-Stabilized LoRA (RS-LoRA)
|
|
* **Batch Size:** 32
|
|
* **Gradient Accumulation:** 2
|
|
* **Epochs:** 1
|
|
* **Learning Rate:** 2e-5
|
|
* **Optimizer:** AdamW (Fused)
|
|
* **LR Scheduler:** Cosine
|
|
* **Noise Level:** NEFTune (alpha=5)
|
|
|
|
|
|
### Special Thanks
|
|
|
|
This fine-tune wouldn't be possible without the incredible work of the community:
|
|
|
|
* **[p-e-w](https://huggingface.co/p-e-w)** for developing **[Heretic](https://github.com/p-e-w/heretic)** - an essential tool for censorship removal.
|
|
* **[allura-forge](https://huggingface.co/allura-forge)** and **[shb777](https://huggingface.co/shb777)** for providing access to the **Llama 3.3 8B** weights.
|
|
* **[Gryphe](https://huggingface.co/Gryphe)** for the meticulously curated **Writing Prompts** datasets (Opus and ChatGPT-4o versions).
|
|
* **[unsloth](https://github.com/unslothai/unsloth)** for their training acceleration framework that makes local fine-tuning fast and memory-efficient.
|
|
* **[AMD](https://oneclickamd.ai/)** for their Instinct™ MI300X GPU. |