234 lines
7.8 KiB
Markdown
234 lines
7.8 KiB
Markdown
---
|
||
license: cc-by-nc-nd-4.0
|
||
library_name: transformers
|
||
tags:
|
||
- UNA
|
||
- juanako
|
||
- cybertron
|
||
- xaberius
|
||
datasets:
|
||
- fblgit/tree-of-knowledge
|
||
- garage-bAInd/Open-Platypus
|
||
- allenai/ultrafeedback_binarized_cleaned
|
||
- Open-Orca/OpenOrca
|
||
model-index:
|
||
- name: una-xaberius-34b-v1beta
|
||
results:
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: AI2 Reasoning Challenge (25-Shot)
|
||
type: ai2_arc
|
||
config: ARC-Challenge
|
||
split: test
|
||
args:
|
||
num_few_shot: 25
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 70.39
|
||
name: normalized accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-xaberius-34b-v1beta
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: HellaSwag (10-Shot)
|
||
type: hellaswag
|
||
split: validation
|
||
args:
|
||
num_few_shot: 10
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 86.77
|
||
name: normalized accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-xaberius-34b-v1beta
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MMLU (5-Shot)
|
||
type: cais/mmlu
|
||
config: all
|
||
split: test
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 78.15
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-xaberius-34b-v1beta
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: TruthfulQA (0-shot)
|
||
type: truthful_qa
|
||
config: multiple_choice
|
||
split: validation
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: mc2
|
||
value: 61.45
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-xaberius-34b-v1beta
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: Winogrande (5-shot)
|
||
type: winogrande
|
||
config: winogrande_xl
|
||
split: validation
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 84.93
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-xaberius-34b-v1beta
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: GSM8k (5-shot)
|
||
type: gsm8k
|
||
config: main
|
||
split: test
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 63.38
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-xaberius-34b-v1beta
|
||
name: Open LLM Leaderboard
|
||
---
|
||
|
||
# Model Card for una-xaberius-34b-v1-beta (UNA: Uniform Neural Alignment)
|
||
|
||
**This is another King-Breed from Juanako.AI**
|
||
|
||
**We have Identified some Problems with regular Quants** [use these models to play with Xaberius-34B and harness its power in full](https://huggingface.co/models?search=xaberius%20lonestriker).
|
||
**Unfortunately we were not able to use any of TheBloke models, seems there is some undesired results out of it.**
|
||
|
||
|
||
Introducing THE MODEL: **XABERIUS 34B v1-BETA** an *experimental* 34B LLaMa-Yi-34B based model, best on it's series. Trained on SFT, DPO and UNA (Unified Neural Alignment) on multiple datasets.
|
||
|
||
Timeline:
|
||
* 05-Dec-2023 **v1-beta released**
|
||
* 08-Dec-2023 **Evaluation been "RUNNING" for 2 days.. no results yet**
|
||
* 09-Dec-2023 **Evaluation been "FINISHED", confirming #1 spot** outperforming the contaminated-disqualified tigerbot :)
|
||
|
||
[Results Here](https://huggingface.co/datasets/open-llm-leaderboard/details_fblgit__una-xaberius-34b-v1beta/blob/main/results_2023-12-09T11-16-37.904970.json)
|
||
|
||
Sidenote: Tests took 19H to run, wonder what happened in the 48H that HF held this one.. interim releasing manually other results??..
|
||
|
||
| Model | Average | ARC (25-s) | HellaSwag (10-s) | MMLU (5-s) | TruthfulQA (MC) (0-s) | Winogrande (5-s) | GSM8K (5-s) |
|
||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||
| [fblgit/una-cybertron-7b-v1-fp16](https://huggingface.co/fblgit/una-cybertron-7b-v1-fp16) | **69.49** | **68.43** | **85.85** | 63.34 | **63.28** | **80.90** | **55.12** |
|
||
| [fblgit/una-cybertron-7b-v2-bf16](https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16) | **69.67** | **68.26** | **85.?4** | 63.23 | **64.63** | **81.37** | **55.04** |
|
||
| [fblgit/una-xaberius-34b-v1beta](https://huggingface.co/fblgit/una-xaberius-34b-v1beta) | **74.18** | **70.39** | **86.77** | **78.15** | **61.45** | **84.93** | **63.38** |
|
||
|
||
## Evaluations
|
||
|
||
- Scores **74.21** Outperforming former leader tigerbot-70b-chat and landing on #1 position of HuggingFace LeaderBoard: 08 December 2023.
|
||
- Scores **79.13** in MMLU, setting a new record not just for 34B but also for all OpenSource LLM's :)
|
||
|
||
SideNote: MMLU was a very solid 79+ .. weird, we'll dive further on this for irregularities :)
|
||
|
||
## Model Details
|
||
|
||
Adiestrated with UNA: Uniform Neural Alignment technique (paper going out soon).
|
||
* What is **NOT** UNA? Its not a merged layers model. Is not SLERP or SLURP or similar.
|
||
* What **is** UNA? A formula & A technique to *TAME* models
|
||
* When will be released the code and paper? When have time, contribute and it'll be faster.
|
||
|
||
### Model Description
|
||
|
||
- **Developed by:** [juanako.ai](https://juanako.ai)
|
||
- **Author:** [Xavier M.](xavi@juanako.ai)
|
||
- **Investors** [CONTACT HERE](billing@juanako.ai)
|
||
- **Model type:** LLaMa YI-34B
|
||
- **Funded by Cybertron's H100's** with few hours training.
|
||
|
||
### Prompt
|
||
The model is very good, works well on almost any prompt but ChatML format and Alpaca System gets the best
|
||
```
|
||
<|im_start|>system
|
||
- You are a helpful assistant chatbot trained by MosaicML.
|
||
- You answer questions.
|
||
- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.
|
||
- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.<|im_end|>
|
||
<|im_start|>user
|
||
Explain QKV<|im_end|>
|
||
<|im_start|>assistant
|
||
```
|
||
```
|
||
### Assistant: I am StableVicuna, a large language model created by CarperAI. I am here to chat!
|
||
|
||
### Human: Explain QKV
|
||
### Assistant:
|
||
```
|
||
```
|
||
[Round <|round|>]
|
||
问：Explain QKV
|
||
答：
|
||
```
|
||
```
|
||
[Round <|round|>]
|
||
Question：Explain QKV
|
||
Answer：
|
||
```
|
||
```
|
||
Question：Explain QKV
|
||
Answer：
|
||
```
|
||
|
||
### Framework versions
|
||
|
||
- Transformers 4.35.2-UNA
|
||
- Pytorch 2.1.0
|
||
- Datasets 2.14.6
|
||
- Tokenizers 0.14.1
|
||
|
||
### Citations
|
||
If you find Xaberius, Cybertron, Juanako or any of our models useful, specially if you use it for your big brand or you cloning/merge/SLERP my modelsm, cite please:
|
||
```
|
||
@misc{unaxaberius34b,
|
||
title={Xaberius 34B: Uniform Neural Alignment},
|
||
author={Xavier Murias},
|
||
year={2023},
|
||
publisher = {HuggingFace},
|
||
journal = {HuggingFace repository},
|
||
howpublished = {\url{https://huggingface.co/fblgit/una-xaberius-34b-v1beta}},
|
||
}
|
||
```
|
||
|
||
**Thanks to LoneStriker for his ExLLama2 models of high quality that works properly.**
|
||
**Enormous Ku2 to Yi-34b Team for the outstanding model, UNA is only as good as its pre-trained model** THANK YOU!
|
||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_fblgit__una-xaberius-34b-v1beta)
|
||
|
||
| Metric |Value|
|
||
|---------------------------------|----:|
|
||
|Avg. |74.18|
|
||
|AI2 Reasoning Challenge (25-Shot)|70.39|
|
||
|HellaSwag (10-Shot) |86.77|
|
||
|MMLU (5-Shot) |78.15|
|
||
|TruthfulQA (0-shot) |61.45|
|
||
|Winogrande (5-shot) |84.93|
|
||
|GSM8k (5-shot) |63.38|
|
||
|