199 lines
7.2 KiB
Markdown
199 lines
7.2 KiB
Markdown
---
|
|
license: apache-2.0
|
|
tags:
|
|
- merge
|
|
- mergekit
|
|
base_model:
|
|
- failspy/Llama-3-8B-Instruct-abliterated
|
|
model-index:
|
|
- name: Aura-Llama-Abliterated
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: AI2 Reasoning Challenge (25-Shot)
|
|
type: ai2_arc
|
|
config: ARC-Challenge
|
|
split: test
|
|
args:
|
|
num_few_shot: 25
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 49.23
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: HellaSwag (10-Shot)
|
|
type: hellaswag
|
|
split: validation
|
|
args:
|
|
num_few_shot: 10
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 72.27
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MMLU (5-Shot)
|
|
type: cais/mmlu
|
|
config: all
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 55.71
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: TruthfulQA (0-shot)
|
|
type: truthful_qa
|
|
config: multiple_choice
|
|
split: validation
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: mc2
|
|
value: 46.63
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: Winogrande (5-shot)
|
|
type: winogrande
|
|
config: winogrande_xl
|
|
split: validation
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 69.3
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: GSM8k (5-shot)
|
|
type: gsm8k
|
|
config: main
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 27.6
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
|
|
name: Open LLM Leaderboard
|
|
---
|
|
|
|
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<title>Aura-llama-3 Data Card</title>
|
|
<link href="https://fonts.googleapis.com/css2?family=Quicksand:wght@400;500;600&display=swap" rel="stylesheet">
|
|
<style> body { font-family: 'Quicksand', sans-serif; background: linear-gradient(135deg, #2E3440 0%, #1A202C 100%); color: #D8DEE9; margin: 0; padding: 0; font-size: 16px; }
|
|
.container { width: 100%; max-width: 1440px; margin: 20px auto; background-color: rgba(255, 255, 255, 0.02); padding: 20px; border-radius: 12px; box-shadow: 0 4px 10px rgba(0, 0, 0, 0.2); backdrop-filter: blur(10px); border: 1px solid rgba(255, 255, 255, 0.1); }
|
|
.header h1 { font-size: 28px; color: #ECEFF4; margin: 0 0 20px 0; text-shadow: 2px 2px 4px rgba(0, 0, 0, 0.3); }
|
|
.update-section { margin-top: 30px; } .update-section h2 { font-size: 24px; color: #88C0D0; }
|
|
.update-section p { font-size: 16px; line-height: 1.6; color: #ECEFF4; }
|
|
.info img { width: 100%; border-radius: 10px; margin-bottom: 15px; }
|
|
a { color: #88C0D0; text-decoration: none; }
|
|
a:hover { color: #A3BE8C; }
|
|
pre { background-color: rgba(255, 255, 255, 0.05); padding: 10px; border-radius: 5px; overflow-x: auto; }
|
|
code { font-family: 'Courier New', monospace; color: #A3BE8C; } </style> </head> <body> <div class="container">
|
|
<div class="header">
|
|
<h1>Aura-llama-3-Abliterated</h1> </div> <div class="info">
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/64545af5ec40bbbd01242ca6/AwLNDVB-GIY7k0wnVV_TX.png" alt="Aura-llama-Abliterated Image">
|
|
<p>Now that the cute anime girl has your attention.</p>
|
|
<p>UPDATE: Model is now using the abliterated version of meta llama 3 8b </p>
|
|
<p>Aura-llama is using the methodology presented by SOLAR for scaling LLMs called depth up-scaling (DUS), which encompasses architectural modifications with continued pretraining. Using the solar paper as a base, I integrated Llama-3 weights into the upscaled layers, and In the future plan to continue training the model.</p>
|
|
<p>Aura-llama is a merge of the following models to create a base model to work from:</p>
|
|
<ul>
|
|
<li><a href="https://huggingface.co/failspy/Llama-3-8B-Instruct-abliterated">failspy/Llama-3-8B-Instruct-abliterated</a></li>
|
|
<li><a href="https://huggingface.co/failspy/Llama-3-8B-Instruct-abliterated">failspy/Llama-3-8B-Instruct-abliterated</a></li>
|
|
</ul>
|
|
</div>
|
|
<div class="update-section">
|
|
<h2>Abliterated Merged Evals (Has Not Been Finetuned):</h2>
|
|
<p>Aura-llama-Abliterated</p>
|
|
<ul>
|
|
<li>Avg: ?</li>
|
|
<li>ARC: ?</li>
|
|
<li>HellaSwag: ?</li>
|
|
<li>MMLU: ?</li>
|
|
<li>T-QA: ?</li>
|
|
<li>Winogrande: ?</li>
|
|
<li>GSM8K: ?</li>
|
|
</ul>
|
|
<h2>Non Abliterated Merged Evals (Has Not Been Finetuned):</h2>
|
|
<p>Aura-llama-Original</p>
|
|
<ul>
|
|
<li>Avg: 63.13</li>
|
|
<li>ARC: 58.02</li>
|
|
<li>HellaSwag: 77.82</li>
|
|
<li>MMLU: 65.61</li>
|
|
<li>T-QA: 51.94</li>
|
|
<li>Winogrande: 73.40</li>
|
|
<li>GSM8K: 52.01</li>
|
|
</ul>
|
|
</div>
|
|
<div class="update-section">
|
|
<h2>🧩 Configuration</h2>
|
|
<pre><code>
|
|
dtype: bfloat16
|
|
merge_method: passthrough
|
|
slices:
|
|
- sources:
|
|
- layer_range: [0, 12]
|
|
model: failspy/Llama-3-8B-Instruct-abliterated
|
|
- sources:
|
|
- layer_range: [8, 20]
|
|
model: failspy/Llama-3-8B-Instruct-abliterated
|
|
- sources:
|
|
- layer_range: [16, 28]
|
|
model: failspy/Llama-3-8B-Instruct-abliterated
|
|
- sources:
|
|
- layer_range: [24, 32]
|
|
model: failspy/Llama-3-8B-Instruct-abliterated
|
|
</code></pre>
|
|
</div>
|
|
</div>
|
|
</body>
|
|
</html>
|
|
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_TheSkullery__Aura-Llama-Abliterated)
|
|
|
|
| Metric |Value|
|
|
|---------------------------------|----:|
|
|
|Avg. |53.46|
|
|
|AI2 Reasoning Challenge (25-Shot)|49.23|
|
|
|HellaSwag (10-Shot) |72.27|
|
|
|MMLU (5-Shot) |55.71|
|
|
|TruthfulQA (0-shot) |46.63|
|
|
|Winogrande (5-shot) |69.30|
|
|
|GSM8k (5-shot) |27.60|
|
|
|