260 lines
7.9 KiB
Markdown
260 lines
7.9 KiB
Markdown
---
|
|
language:
|
|
- en
|
|
license: llama2
|
|
library_name: transformers
|
|
datasets:
|
|
- psmathur/lima_unchained_v1
|
|
model-index:
|
|
- name: test_42_70b
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: AI2 Reasoning Challenge (25-Shot)
|
|
type: ai2_arc
|
|
config: ARC-Challenge
|
|
split: test
|
|
args:
|
|
num_few_shot: 25
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 68.26
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: HellaSwag (10-Shot)
|
|
type: hellaswag
|
|
split: validation
|
|
args:
|
|
num_few_shot: 10
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 87.65
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MMLU (5-Shot)
|
|
type: cais/mmlu
|
|
config: all
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 70.0
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: TruthfulQA (0-shot)
|
|
type: truthful_qa
|
|
config: multiple_choice
|
|
split: validation
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: mc2
|
|
value: 48.76
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: Winogrande (5-shot)
|
|
type: winogrande
|
|
config: winogrande_xl
|
|
split: validation
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 83.66
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: GSM8k (5-shot)
|
|
type: gsm8k
|
|
config: main
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 45.94
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
|
name: Open LLM Leaderboard
|
|
---
|
|
|
|
# Lima_Unchained_70b
|
|
|
|
A Llama2-70b model fine-tuned using QLora on all the linear layers with carefully selected ~900 conversations from the [Lima](https://arxiv.org/pdf/2305.11206.pdf)
|
|
|
|
<br>
|
|
|
|
|
|
**P.S. If you're interested to collaborate, please connect with me at www.linkedin.com/in/pankajam.**
|
|
|
|
## Evaluation
|
|
|
|
We evaluated Lima_Unchained_70b on a wide range of tasks using [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) from EleutherAI.
|
|
|
|
Here are the results on metrics used by [HuggingFaceH4 Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
|
|
|
|||||
|
|
|:------:|:--------:|:-------:|:--------:|
|
|
|**Task**|**Metric**|**Value**|**Stderr**|
|
|
|*arc_challenge*|acc_norm|0.6826|0.0141|
|
|
|*hellaswag*|acc_norm|0.8765|0.0038|
|
|
|*mmlu*|acc_norm|0.70|0.0351|
|
|
|*truthfulqa_mc*|mc2|0.4876|0.0157|
|
|
|**Total Average**|-|**0.6867**||
|
|
|
|
|
|
<br>
|
|
|
|
## Example Usage
|
|
|
|
Here is the prompt format
|
|
|
|
```
|
|
|
|
### User:
|
|
Write a stand-up skit in the style of George Carlin that ridicules Pacific Gas and Electric.
|
|
|
|
### Assistant:
|
|
|
|
```
|
|
|
|
Below shows a code example on how to use this model
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
|
|
|
|
model_path="pankajmathur/Lima_Unchained_70b"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_path)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_path,
|
|
torch_dtype=torch.float16,
|
|
load_in_8bit=True,
|
|
low_cpu_mem_usage=True,
|
|
device_map="auto"
|
|
)
|
|
|
|
#generate text steps
|
|
instruction = "Write a stand-up skit in the style of George Carlin that ridicules Pacific Gas and Electric."
|
|
prompt = f"### User: {instruction}\n\n### Assistant:\n"
|
|
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
|
|
output = model.generate(**inputs, do_sample=True, top_p=0.95, top_k=0, max_new_tokens=4096)
|
|
|
|
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
|
|
|
```
|
|
|
|
<br>
|
|
|
|
#### Limitations & Biases:
|
|
|
|
While this model aims for accuracy, it can occasionally produce inaccurate or misleading results.
|
|
|
|
Despite diligent efforts in refining the pretraining data, there remains a possibility for the generation of inappropriate, biased, or offensive content.
|
|
|
|
Exercise caution and cross-check information when necessary.
|
|
|
|
|
|
<br>
|
|
|
|
### Citiation:
|
|
|
|
Please kindly cite using the following BibTeX:
|
|
|
|
```
|
|
@misc{Lima_Unchained_70b,
|
|
author = {Pankaj Mathur},
|
|
title = {Lima_Unchained_70b: A LIMA style Llama2-70b model},
|
|
year = {2023},
|
|
publisher = {HuggingFace},
|
|
journal = {HuggingFace repository},
|
|
howpublished = {\url{https://https://huggingface.co/psmathur/model_42_70b},
|
|
}
|
|
```
|
|
|
|
```
|
|
@misc{ChuntingZhou,
|
|
title={LIMA: Less Is More for Alignment},
|
|
author={Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu,
|
|
Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy},
|
|
year={2023},
|
|
eprint={2305.11206},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CL}
|
|
}
|
|
```
|
|
|
|
```
|
|
@software{touvron2023llama2,
|
|
title={Llama 2: Open Foundation and Fine-Tuned Chat Models},
|
|
author={Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava,
|
|
Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,
|
|
Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez Madian Khabsa, Isabel Kloumann,
|
|
Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov,
|
|
Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith,
|
|
Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu , Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan,
|
|
Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom},
|
|
year={2023}
|
|
}
|
|
```
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_psmathur__model_42_70b)
|
|
|
|
| Metric | Value |
|
|
|-----------------------|---------------------------|
|
|
| Avg. | 58.2 |
|
|
| ARC (25-shot) | 68.26 |
|
|
| HellaSwag (10-shot) | 87.65 |
|
|
| MMLU (5-shot) | 70.0 |
|
|
| TruthfulQA (0-shot) | 48.76 |
|
|
| Winogrande (5-shot) | 83.66 |
|
|
| GSM8K (5-shot) | 34.72 |
|
|
| DROP (3-shot) | 14.37 |
|
|
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_psmathur__test_42_70b)
|
|
|
|
| Metric |Value|
|
|
|---------------------------------|----:|
|
|
|Avg. |67.38|
|
|
|AI2 Reasoning Challenge (25-Shot)|68.26|
|
|
|HellaSwag (10-Shot) |87.65|
|
|
|MMLU (5-Shot) |70.00|
|
|
|TruthfulQA (0-shot) |48.76|
|
|
|Winogrande (5-shot) |83.66|
|
|
|GSM8k (5-shot) |45.94|
|
|
|