68 lines
2.2 KiB
Markdown
68 lines
2.2 KiB
Markdown
|
---
|
||
|
|
base_model: 0xA50C1A1/Qwen3-4B-Instruct-2507-SOM-MPOA
|
||
|
|
library_name: transformers
|
||
|
|
model_name: Qwen3-4B-Nymphaea-RP
|
||
|
|
tags:
|
||
|
|
- uncensored
|
||
|
|
- roleplay
|
||
|
|
- trl
|
||
|
|
- rp
|
||
|
|
- creative-writing
|
||
|
|
license: apache-2.0
|
||
|
|
---
|
||
|
|
|
||
|
|
# Qwen3-4B-Nymphaea-RP
|
||
|
|
|
||
|
|
A fine-tune of Qwen3-4B-Instruct-2507 for roleplay and creative writing.
|
||
|
|
|
||
|
|
Suitable for mobile roleplay: tested on Nothing Phone 2 in Q4_K_M quantization (7-8 t/s)
|
||
|
|
|
||
|
|
> [!Tip]
|
||
|
|
> The SillyTavern preset is available [here](https://huggingface.co/0xA50C1A1/Qwen3-4B-Nymphaea-RP/blob/main/ST-Preset.json).
|
||
|
|
> For custom presets, please use the **ChatML** instruct template.
|
||
|
|
|
||
|
|
<details>
|
||
|
|
<summary><b>Chat Example</b></summary><p>
|
||
|
|
|
||
|
|
Tested at Q8_0 quantization.
|
||
|
|
|
||
|
|
|
||
|
|
</p></details>
|
||
|
|
|
||
|
|
## Training Notes
|
||
|
|
|
||
|
|
Trained on the latest iteration of my Darkmere dataset. This version features expanded genre variety, built upon a mix of manually curated synthetics and human-written stories.
|
||
|
|
|
||
|
|
> [!IMPORTANT]
|
||
|
|
> The base weights are abliterated via [Heretic](https://github.com/p-e-w/heretic) prior to fine-tuning, so this fine-tune is quite uncensored.
|
||
|
|
|
||
|
|
<details>
|
||
|
|
<summary><b>Training Specs</b></summary><p>
|
||
|
|
|
||
|
|
**Method:**
|
||
|
|
|
||
|
|
* **Training Method:** DoRA (Weight-Decomposed LoRA)
|
||
|
|
* **Target Modules** `all-linear`
|
||
|
|
* **LoRA Rank:** 32
|
||
|
|
* **LoRA Alpha:** 32
|
||
|
|
* **LoRA Dropout:** 0.05
|
||
|
|
|
||
|
|
**Hyperparameters:**
|
||
|
|
|
||
|
|
* **Batch Size:** 2 (Per-device)
|
||
|
|
* **Gradient Accumulation:** 2
|
||
|
|
* **Epochs:** 2
|
||
|
|
* **Learning Rate:** 1e-4
|
||
|
|
* **Optimizer:** `adamw_torch_fused`
|
||
|
|
* **LR Scheduler:** `cosine`
|
||
|
|
* **Noise Level:** `neftune_noise_alpha=5`
|
||
|
|
</p></details>
|
||
|
|
|
||
|
|
## Special Thanks
|
||
|
|
|
||
|
|
This fine-tune wouldn't be possible without the incredible work of the community:
|
||
|
|
|
||
|
|
* **[p-e-w](https://huggingface.co/p-e-w)** for developing **[Heretic](https://github.com/p-e-w/heretic)** - an essential tool for censorship removal.
|
||
|
|
* **[SicariusSicariiStuff](https://huggingface.co/SicariusSicariiStuff)** for developing **[SLOP_Detector](https://github.com/SicariusSicariiStuff/SLOP_Detector)** script.
|
||
|
|
* **[Qwen Team](https://huggingface.co/Qwen)** for their **[Qwen3](https://huggingface.co/collections/Qwen/qwen3)** weights.
|
||
|
|
* **[AMD](https://oneclickamd.ai/)** for their Instinct™ MI300X GPU.
|