82 lines
2.9 KiB
Markdown
82 lines
2.9 KiB
Markdown
---
|
|
license: llama3.3
|
|
base_model:
|
|
- shb777/Llama-3.3-8B-Instruct-128K
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- heretic
|
|
- uncensored
|
|
- decensored
|
|
- abliterated
|
|
---
|
|
# This is a decensored version of [shb777/Llama-3.3-8B-Instruct-128K](https://huggingface.co/shb777/Llama-3.3-8B-Instruct-128K), made using [Heretic](https://github.com/p-e-w/heretic) v1.2.0
|
|
|
|
## Abliteration parameters
|
|
|
|
| Parameter | Value |
|
|
| :-------- | :---: |
|
|
| **direction_index** | 15.07 |
|
|
| **attn.o_proj.max_weights.0** | 0: 1.40 |
|
|
| **attn.o_proj.max_weights.1** | 1: 0.93 |
|
|
| **attn.o_proj.max_weights.2** | 2: 1.06 |
|
|
| **attn.o_proj.max_weights.3** | 3: 1.45 |
|
|
| **attn.o_proj.max_weights.4** | 4: 1.41 |
|
|
| **attn.o_proj.max_weights.5** | 5: 1.45 |
|
|
| **attn.o_proj.max_weights.6** | 6: 1.25 |
|
|
| **attn.o_proj.max_weights.7** | 7: 1.38 |
|
|
| **attn.o_proj.max_weight_position** | 28.70 |
|
|
| **attn.o_proj.min_weights.0** | 0: 0.71 |
|
|
| **attn.o_proj.min_weights.1** | 1: 0.21 |
|
|
| **attn.o_proj.min_weights.2** | 2: 0.94 |
|
|
| **attn.o_proj.min_weights.3** | 3: 1.36 |
|
|
| **attn.o_proj.min_weights.4** | 4: 0.19 |
|
|
| **attn.o_proj.min_weights.5** | 5: 1.09 |
|
|
| **attn.o_proj.min_weights.6** | 6: 0.87 |
|
|
| **attn.o_proj.min_weights.7** | 7: 1.14 |
|
|
| **attn.o_proj.min_weight_distance** | 15.91 |
|
|
| **mlp.down_proj.max_weights.0** | 0: 1.38 |
|
|
| **mlp.down_proj.max_weights.1** | 1: 1.26 |
|
|
| **mlp.down_proj.max_weights.2** | 2: 1.10 |
|
|
| **mlp.down_proj.max_weights.3** | 3: 0.85 |
|
|
| **mlp.down_proj.max_weights.4** | 4: 1.12 |
|
|
| **mlp.down_proj.max_weights.5** | 5: 1.31 |
|
|
| **mlp.down_proj.max_weights.6** | 6: 1.25 |
|
|
| **mlp.down_proj.max_weights.7** | 7: 0.94 |
|
|
| **mlp.down_proj.max_weight_position** | 30.75 |
|
|
| **mlp.down_proj.min_weights.0** | 0: 0.27 |
|
|
| **mlp.down_proj.min_weights.1** | 1: 0.56 |
|
|
| **mlp.down_proj.min_weights.2** | 2: 0.77 |
|
|
| **mlp.down_proj.min_weights.3** | 3: 0.47 |
|
|
| **mlp.down_proj.min_weights.4** | 4: 0.93 |
|
|
| **mlp.down_proj.min_weights.5** | 5: 0.67 |
|
|
| **mlp.down_proj.min_weights.6** | 6: 1.19 |
|
|
| **mlp.down_proj.min_weights.7** | 7: 0.01 |
|
|
| **mlp.down_proj.min_weight_distance** | 6.58 |
|
|
|
|
## Performance
|
|
|
|
| Metric | This model | Original model ([shb777/Llama-3.3-8B-Instruct-128K](https://huggingface.co/shb777/Llama-3.3-8B-Instruct-128K)) |
|
|
| :----- | :--------: | :---------------------------: |
|
|
| **KL divergence** | 0.0563 | 0 *(by definition)* |
|
|
| **Refusals** | 3/100 | 93/100 |
|
|
|
|
-----
|
|
|
|
|
|
# Llama 3.3 8B 128K Instruct (Fixed)
|
|
|
|
> [!IMPORTANT]
|
|
> Original [allura-forge/Llama-3.3-8B-Instruct](https://huggingface.co/allura-forge/Llama-3.3-8B-Instruct), Thanks!
|
|
|
|
> [!TIP]
|
|
> [imatrix GGUF's by mradermacher (Recommended)](https://huggingface.co/mradermacher/Llama-3.3-8B-Instruct-128K-i1-GGUF)
|
|
>
|
|
> [static GGUF's](https://huggingface.co/shb777/Llama-3.3-8B-Instruct-128K-GGUF)
|
|
>
|
|
> [Evals](https://huggingface.co/datasets/shb777/Llama-3.3-8B-Instruct-128K-Evals)
|
|
|
|
Additional Fixes:
|
|
- Added `rope_scaling`
|
|
- Added chat template (Unsloth) in tokenizer config
|
|
- Updated generation config
|
|
- Enabled full context length |