228 lines
7.0 KiB
Markdown
228 lines
7.0 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
tags:
|
||
- juanako
|
||
- UNA
|
||
datasets:
|
||
- fblgit/tree-of-knowledge
|
||
- Open-Orca/SlimOrca-Dedup
|
||
- HuggingFaceH4/ultrafeedback_binarized
|
||
model-index:
|
||
- name: una-cybertron-7b-v1-fp16
|
||
results:
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: AI2 Reasoning Challenge (25-Shot)
|
||
type: ai2_arc
|
||
config: ARC-Challenge
|
||
split: test
|
||
args:
|
||
num_few_shot: 25
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 68.43
|
||
name: normalized accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: HellaSwag (10-Shot)
|
||
type: hellaswag
|
||
split: validation
|
||
args:
|
||
num_few_shot: 10
|
||
metrics:
|
||
- type: acc_norm
|
||
value: 85.42
|
||
name: normalized accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: MMLU (5-Shot)
|
||
type: cais/mmlu
|
||
config: all
|
||
split: test
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 63.34
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: TruthfulQA (0-shot)
|
||
type: truthful_qa
|
||
config: multiple_choice
|
||
split: validation
|
||
args:
|
||
num_few_shot: 0
|
||
metrics:
|
||
- type: mc2
|
||
value: 63.28
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: Winogrande (5-shot)
|
||
type: winogrande
|
||
config: winogrande_xl
|
||
split: validation
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 81.37
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
|
||
name: Open LLM Leaderboard
|
||
- task:
|
||
type: text-generation
|
||
name: Text Generation
|
||
dataset:
|
||
name: GSM8k (5-shot)
|
||
type: gsm8k
|
||
config: main
|
||
split: test
|
||
args:
|
||
num_few_shot: 5
|
||
metrics:
|
||
- type: acc
|
||
value: 55.12
|
||
name: accuracy
|
||
source:
|
||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
|
||
name: Open LLM Leaderboard
|
||
---
|
||
# Model Card for una-cybertron-7b-v1 (UNA: Uniform Neural Alignment)
|
||
|
||
We strike back, introducing **Cybertron 7B v1** a 7B MistralAI based model, best on it's series. Trained on SFT, DPO and UNA (Unified Neural Alignment) on multiple datasets.
|
||
He scores **64.60**+ on HF LeaderTests (without DROP for now).
|
||
|
||
Scoring **#1** at 2 December 2023:
|
||
|
||
| Model | Average | ARC (25-s) | HellaSwag (10-s) | MMLU (5-s) | TruthfulQA (MC) (0-s) | Winogrande (5-s) | GSM8K (5-s) |
|
||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||
| [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 60.97 | 59.98 | 83.31 | 64.16 | 42.15 | 78.37 | 37.83 |
|
||
| [perlthoughts/Chupacabra-7B-v2](https://huggingface.co/perlthoughts/Chupacabra-7B-v2) | 63.54 | 66.47 | 85.17 | 64.49 | 57.6 | 79.16 | 28.35 |
|
||
| [fblgit/una-cybertron-7b-v1](https://huggingface.co/fblgit/una-cybertron-7b-v1) | **64.60** | **68.17** | 85.14 | 62.07 | **63.98** | **80.9** | 27.34 |
|
||
|
||
The model excels in mathematics, logic, reasoning, overall very smart.
|
||
|
||
## Model Details
|
||
|
||
Adiestrated with UNA: Uniform Neural Alignment technique (paper going out soon).
|
||
|
||
|
||
### Model Description
|
||
|
||
- **Developed by:** [juanako.ai](https://juanako.ai)
|
||
- **Author:** [Xavier M.](xavi@juanako.ai)
|
||
- **Model type:** MistralAI 7B
|
||
- **Funded by Cybertron's H100's**
|
||
|
||
### Prompt
|
||
The model is very good, works well on almost any prompt but ChatML format and Alpaca System gets the best
|
||
```
|
||
<|im_start|>system
|
||
- You are a helpful assistant chatbot trained by MosaicML.
|
||
- You answer questions.
|
||
- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.
|
||
- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.<|im_end|>
|
||
<|im_start|>user
|
||
Explain QKV<|im_end|>
|
||
<|im_start|>assistant
|
||
```
|
||
```
|
||
### Assistant: I am StableVicuna, a large language model created by CarperAI. I am here to chat!
|
||
|
||
### Human: Explain QKV
|
||
### Assistant:
|
||
```
|
||
```
|
||
[Round <|round|>]
|
||
问：Explain QKV
|
||
答：
|
||
```
|
||
```
|
||
[Round <|round|>]
|
||
Question：Explain QKV
|
||
Answer：
|
||
```
|
||
```
|
||
Question：Explain QKV
|
||
Answer：
|
||
```
|
||
|
||
## Evaluation
|
||
```
|
||
| Tasks |Version|Shots | Metric |Value | |Stderr|
|
||
|--------------|-------|------|--------|-----:|---|-----:|
|
||
|arc_challenge | | 25 |acc_norm|0.6817|± |0.0136|
|
||
|truthfulqa_mc2| | 0 |acc |0.6398|± |0.0151|
|
||
|hellaswag | | 10 |acc_norm|0.8492|± |0.0036|
|
||
|winogrande | | 0 |acc |0.809 |± |0.011 |
|
||
|gsm8k | | 5 |acc |0.2733|± |0.0137|
|
||
|mmlu | | 5 |acc |0.6207|± |0.1230|
|
||
| |average| |acc |0.6456| | |
|
||
|
||
| Groups |Version|Filter|n-shot|Metric|Value | |Stderr|
|
||
|------------------|-------|------|-----:|------|-----:|---|-----:|
|
||
|mmlu |N/A |none | 0|acc |0.6207|_ |0.1230|
|
||
| - humanities |N/A |none | 5|acc |0.5675|_ |0.1125|
|
||
| - other |N/A |none | 5|acc |0.6933|_ |0.1108|
|
||
| - social_sciences|N/A |none | 5|acc |0.7270|_ |0.0666|
|
||
| - stem |N/A |none | 5|acc |0.5249|_ |0.1311|
|
||
```
|
||
|
||
### Framework versions
|
||
|
||
- Transformers 4.35.0-UNA
|
||
- Pytorch 2.1.0
|
||
- Datasets 2.14.6
|
||
- Tokenizers 0.14.1
|
||
|
||
### Citations
|
||
If you find Cybertron, Juanako or any of our models useful, specially if you use it for your big brand.. cite please:
|
||
```
|
||
@misc{unacybertron7a,
|
||
title={Cybertron: Uniform Neural Alignment},
|
||
author={Xavier Murias},
|
||
year={2023},
|
||
publisher = {HuggingFace},
|
||
journal = {HuggingFace repository},
|
||
howpublished = {\url{https://huggingface.co/fblgit/una-cybertron-7b-v1}},
|
||
}
|
||
```
|
||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_fblgit__una-cybertron-7b-v1-fp16)
|
||
|
||
| Metric |Value|
|
||
|---------------------------------|----:|
|
||
|Avg. |69.49|
|
||
|AI2 Reasoning Challenge (25-Shot)|68.43|
|
||
|HellaSwag (10-Shot) |85.42|
|
||
|MMLU (5-Shot) |63.34|
|
||
|TruthfulQA (0-shot) |63.28|
|
||
|Winogrande (5-shot) |81.37|
|
||
|GSM8k (5-shot) |55.12|
|
||
|