439 lines
13 KiB
Markdown
439 lines
13 KiB
Markdown
---
|
|
language:
|
|
- en
|
|
license: llama2
|
|
library_name: transformers
|
|
datasets:
|
|
- garage-bAInd/Open-Platypus
|
|
- ehartford/dolphin
|
|
- psmathur/orca_mini_v1_dataset
|
|
- psmathur/WizardLM_Orca
|
|
- psmathur/alpaca_orca
|
|
- psmathur/dolly-v2_orca
|
|
- tatsu-lab/alpaca
|
|
- databricks/databricks-dolly-15k
|
|
- WizardLM/WizardLM_evol_instruct_V2_196k
|
|
pipeline_tag: text-generation
|
|
model-index:
|
|
- name: model_007_13b_v2
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: AI2 Reasoning Challenge (25-Shot)
|
|
type: ai2_arc
|
|
config: ARC-Challenge
|
|
split: test
|
|
args:
|
|
num_few_shot: 25
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 61.95
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: HellaSwag (10-Shot)
|
|
type: hellaswag
|
|
split: validation
|
|
args:
|
|
num_few_shot: 10
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 82.48
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MMLU (5-Shot)
|
|
type: cais/mmlu
|
|
config: all
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 57.32
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: TruthfulQA (0-shot)
|
|
type: truthful_qa
|
|
config: multiple_choice
|
|
split: validation
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: mc2
|
|
value: 53.5
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: Winogrande (5-shot)
|
|
type: winogrande
|
|
config: winogrande_xl
|
|
split: validation
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 75.85
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: GSM8k (5-shot)
|
|
type: gsm8k
|
|
config: main
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 1.36
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: IFEval (0-Shot)
|
|
type: HuggingFaceH4/ifeval
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: inst_level_strict_acc and prompt_level_strict_acc
|
|
value: 30.56
|
|
name: strict accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=pankajmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: BBH (3-Shot)
|
|
type: BBH
|
|
args:
|
|
num_few_shot: 3
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 25.45
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=pankajmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MATH Lvl 5 (4-Shot)
|
|
type: hendrycks/competition_math
|
|
args:
|
|
num_few_shot: 4
|
|
metrics:
|
|
- type: exact_match
|
|
value: 1.21
|
|
name: exact match
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=pankajmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: GPQA (0-shot)
|
|
type: Idavidrein/gpqa
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 4.47
|
|
name: acc_norm
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=pankajmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MuSR (0-shot)
|
|
type: TAUR-Lab/MuSR
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 17.2
|
|
name: acc_norm
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=pankajmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MMLU-PRO (5-shot)
|
|
type: TIGER-Lab/MMLU-Pro
|
|
config: main
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 16.23
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=pankajmathur/model_007_13b_v2
|
|
name: Open LLM Leaderboard
|
|
---
|
|
|
|
# model_007_13b_v2
|
|
|
|
A hybrid (explain + instruct) style Llama2-13b model, Pleae check examples below for both style prompts, Here is the list of datasets used:
|
|
|
|
* Open-Platypus
|
|
* Alpaca
|
|
* WizardLM
|
|
* Dolly-V2
|
|
* Dolphin Samples (~200K)
|
|
* Orca_minis_v1
|
|
* Alpaca_orca
|
|
* WizardLM_orca
|
|
* Dolly-V2_orca
|
|
|
|
|
|
<br>
|
|
|
|
<strong>
|
|
Passionate about Generative AI? I help companies to privately train and deploy custom LLM/MLLM affordably. For startups, I can even assist with securing GPU grants to get you started. Let's chat!
|
|
|
|
<a href="https://www.linkedin.com/in/pankajam" target="_blank">https://www.linkedin.com/in/pankajam</a> Looking forward to connecting!
|
|
</strong>
|
|
|
|
<br>
|
|
|
|
|
|
### quantized versions
|
|
|
|
|
|
<br>
|
|
|
|
#### license disclaimer:
|
|
|
|
This model is bound by the license & usage restrictions of the original Llama-2 model. And comes with no warranty or gurantees of any kind.
|
|
|
|
<br>
|
|
|
|
## Evaluation
|
|
|
|
We evaluated model_007_13b_v2 on a wide range of tasks using [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) from EleutherAI.
|
|
|
|
Here are the results on metrics used by [HuggingFaceH4 Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
|
|
|
|||||
|
|
|:------:|:--------:|:-------:|:--------:|
|
|
|**Task**|**Metric**|**Value**|**Stderr**|
|
|
|*arc_challenge*|acc_norm|0.6314|0.0141|
|
|
|*hellaswag*|acc_norm|0.8242|0.0038|
|
|
|*mmlu*|acc_norm|0.5637|0.0351|
|
|
|*truthfulqa_mc*|mc2|0.5127|0.0157|
|
|
|**Total Average**|-|**0.6329877193**||
|
|
|
|
|
|
<br>
|
|
|
|
## Example Usage
|
|
|
|
Here is the Orca prompt format
|
|
|
|
```
|
|
### System:
|
|
You are an AI assistant that follows instruction extremely well. Help as much as you can.
|
|
|
|
### User:
|
|
Tell me about Orcas.
|
|
|
|
### Assistant:
|
|
|
|
```
|
|
|
|
Below shows a code example on how to use this model
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("psmathur/model_007_13b_v2")
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
"psmathur/model_007_13b_v2",
|
|
torch_dtype=torch.float16,
|
|
load_in_8bit=True,
|
|
low_cpu_mem_usage=True,
|
|
device_map="auto"
|
|
)
|
|
system_prompt = "### System:\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\n\n"
|
|
|
|
#generate text steps
|
|
instruction = "Tell me about Orcas."
|
|
prompt = f"{system_prompt}### User: {instruction}\n\n### Assistant:\n"
|
|
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
|
|
output = model.generate(**inputs, do_sample=True, top_p=0.95, top_k=0, max_new_tokens=4096)
|
|
|
|
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
|
|
|
```
|
|
|
|
|
|
Here is the Alpaca prompt format
|
|
|
|
```
|
|
|
|
### User:
|
|
Tell me about Alpacas.
|
|
|
|
### Assistant:
|
|
|
|
```
|
|
|
|
Below shows a code example on how to use this model
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("psmathur/model_007_13b_v2")
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
"psmathur/model_007_13b_v2",
|
|
torch_dtype=torch.float16,
|
|
load_in_8bit=True,
|
|
low_cpu_mem_usage=True,
|
|
device_map="auto"
|
|
)
|
|
#generate text steps
|
|
instruction = "Tell me about Alpacas."
|
|
prompt = f"### User: {instruction}\n\n### Assistant:\n"
|
|
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
|
|
output = model.generate(**inputs, do_sample=True, top_p=0.95, top_k=0, max_new_tokens=4096)
|
|
|
|
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
|
|
|
```
|
|
|
|
<br>
|
|
|
|
#### Limitations & Biases:
|
|
|
|
While this model aims for accuracy, it can occasionally produce inaccurate or misleading results.
|
|
|
|
Despite diligent efforts in refining the pretraining data, there remains a possibility for the generation of inappropriate, biased, or offensive content.
|
|
|
|
Exercise caution and cross-check information when necessary.
|
|
|
|
|
|
<br>
|
|
|
|
### Citiation:
|
|
|
|
Please kindly cite using the following BibTeX:
|
|
|
|
```
|
|
@misc{model_007_13b_v2,
|
|
author = {Pankaj Mathur},
|
|
title = {model_007_13b_v2: A hybrid (explain + instruct) style Llama2-70b model},
|
|
year = {2023},
|
|
publisher = {HuggingFace},
|
|
journal = {HuggingFace repository},
|
|
howpublished = {\url{https://https://huggingface.co/psmathur/model_007_13b_v2},
|
|
}
|
|
```
|
|
|
|
```
|
|
@misc{mukherjee2023orca,
|
|
title={Orca: Progressive Learning from Complex Explanation Traces of GPT-4},
|
|
author={Subhabrata Mukherjee and Arindam Mitra and Ganesh Jawahar and Sahaj Agarwal and Hamid Palangi and Ahmed Awadallah},
|
|
year={2023},
|
|
eprint={2306.02707},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CL}
|
|
}
|
|
```
|
|
|
|
```
|
|
@software{touvron2023llama2,
|
|
title={Llama 2: Open Foundation and Fine-Tuned Chat Models},
|
|
author={Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava,
|
|
Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,
|
|
Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez Madian Khabsa, Isabel Kloumann,
|
|
Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov,
|
|
Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith,
|
|
Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu , Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan,
|
|
Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom},
|
|
year={2023}
|
|
}
|
|
```
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_psmathur__model_007_13b_v2)
|
|
|
|
| Metric | Value |
|
|
|-----------------------|---------------------------|
|
|
| Avg. | 53.78 |
|
|
| ARC (25-shot) | 61.95 |
|
|
| HellaSwag (10-shot) | 82.48 |
|
|
| MMLU (5-shot) | 57.32 |
|
|
| TruthfulQA (0-shot) | 53.5 |
|
|
| Winogrande (5-shot) | 75.85 |
|
|
| GSM8K (5-shot) | 1.36 |
|
|
| DROP (3-shot) | 43.97 |
|
|
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_psmathur__model_007_13b_v2)
|
|
|
|
| Metric |Value|
|
|
|---------------------------------|----:|
|
|
|Avg. |55.41|
|
|
|AI2 Reasoning Challenge (25-Shot)|61.95|
|
|
|HellaSwag (10-Shot) |82.48|
|
|
|MMLU (5-Shot) |57.32|
|
|
|TruthfulQA (0-shot) |53.50|
|
|
|Winogrande (5-shot) |75.85|
|
|
|GSM8k (5-shot) | 1.36|
|
|
|
|
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_pankajmathur__model_007_13b_v2)
|
|
|
|
| Metric |Value|
|
|
|-------------------|----:|
|
|
|Avg. |15.86|
|
|
|IFEval (0-Shot) |30.56|
|
|
|BBH (3-Shot) |25.45|
|
|
|MATH Lvl 5 (4-Shot)| 1.21|
|
|
|GPQA (0-shot) | 4.47|
|
|
|MuSR (0-shot) |17.20|
|
|
|MMLU-PRO (5-shot) |16.23|
|
|
|