初始化项目，由ModelHub XC社区提供模型
Model: pankajmathur/Lima_Unchained_70b Source: Original Platform
This commit is contained in:
ModelHub XC
259
README.md
Normal file
259
README.md
Normal file
@@ -0,0 +1,259 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: llama2
|
||||
library_name: transformers
|
||||
datasets:
|
||||
- psmathur/lima_unchained_v1
|
||||
model-index:
|
||||
- name: test_42_70b
|
||||
results:
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: AI2 Reasoning Challenge (25-Shot)
|
||||
type: ai2_arc
|
||||
config: ARC-Challenge
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 25
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: 68.26
|
||||
name: normalized accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: HellaSwag (10-Shot)
|
||||
type: hellaswag
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 10
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: 87.65
|
||||
name: normalized accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: MMLU (5-Shot)
|
||||
type: cais/mmlu
|
||||
config: all
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 70.0
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: TruthfulQA (0-shot)
|
||||
type: truthful_qa
|
||||
config: multiple_choice
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 0
|
||||
metrics:
|
||||
- type: mc2
|
||||
value: 48.76
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: Winogrande (5-shot)
|
||||
type: winogrande
|
||||
config: winogrande_xl
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 83.66
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: GSM8k (5-shot)
|
||||
type: gsm8k
|
||||
config: main
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 45.94
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/test_42_70b
|
||||
name: Open LLM Leaderboard
|
||||
---
|
||||
|
||||
# Lima_Unchained_70b
|
||||
|
||||
A Llama2-70b model fine-tuned using QLora on all the linear layers with carefully selected ~900 conversations from the [Lima](https://arxiv.org/pdf/2305.11206.pdf)
|
||||
|
||||
<br>
|
||||
|
||||
|
||||
**P.S. If you're interested to collaborate, please connect with me at www.linkedin.com/in/pankajam.**
|
||||
|
||||
## Evaluation
|
||||
|
||||
We evaluated Lima_Unchained_70b on a wide range of tasks using [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) from EleutherAI.
|
||||
|
||||
Here are the results on metrics used by [HuggingFaceH4 Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
||||
|
||||
|||||
|
||||
|:------:|:--------:|:-------:|:--------:|
|
||||
|**Task**|**Metric**|**Value**|**Stderr**|
|
||||
|*arc_challenge*|acc_norm|0.6826|0.0141|
|
||||
|*hellaswag*|acc_norm|0.8765|0.0038|
|
||||
|*mmlu*|acc_norm|0.70|0.0351|
|
||||
|*truthfulqa_mc*|mc2|0.4876|0.0157|
|
||||
|**Total Average**|-|**0.6867**||
|
||||
|
||||
|
||||
<br>
|
||||
|
||||
## Example Usage
|
||||
|
||||
Here is the prompt format
|
||||
|
||||
```
|
||||
|
||||
### User:
|
||||
Write a stand-up skit in the style of George Carlin that ridicules Pacific Gas and Electric.
|
||||
|
||||
### Assistant:
|
||||
|
||||
```
|
||||
|
||||
Below shows a code example on how to use this model
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
|
||||
|
||||
model_path="pankajmathur/Lima_Unchained_70b"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_path)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_path,
|
||||
torch_dtype=torch.float16,
|
||||
load_in_8bit=True,
|
||||
low_cpu_mem_usage=True,
|
||||
device_map="auto"
|
||||
)
|
||||
|
||||
#generate text steps
|
||||
instruction = "Write a stand-up skit in the style of George Carlin that ridicules Pacific Gas and Electric."
|
||||
prompt = f"### User: {instruction}\n\n### Assistant:\n"
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
|
||||
output = model.generate(**inputs, do_sample=True, top_p=0.95, top_k=0, max_new_tokens=4096)
|
||||
|
||||
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
||||
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
#### Limitations & Biases:
|
||||
|
||||
While this model aims for accuracy, it can occasionally produce inaccurate or misleading results.
|
||||
|
||||
Despite diligent efforts in refining the pretraining data, there remains a possibility for the generation of inappropriate, biased, or offensive content.
|
||||
|
||||
Exercise caution and cross-check information when necessary.
|
||||
|
||||
|
||||
<br>
|
||||
|
||||
### Citiation:
|
||||
|
||||
Please kindly cite using the following BibTeX:
|
||||
|
||||
```
|
||||
@misc{Lima_Unchained_70b,
|
||||
author = {Pankaj Mathur},
|
||||
title = {Lima_Unchained_70b: A LIMA style Llama2-70b model},
|
||||
year = {2023},
|
||||
publisher = {HuggingFace},
|
||||
journal = {HuggingFace repository},
|
||||
howpublished = {\url{https://https://huggingface.co/psmathur/model_42_70b},
|
||||
}
|
||||
```
|
||||
|
||||
```
|
||||
@misc{ChuntingZhou,
|
||||
title={LIMA: Less Is More for Alignment},
|
||||
author={Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu,
|
||||
Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy},
|
||||
year={2023},
|
||||
eprint={2305.11206},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CL}
|
||||
}
|
||||
```
|
||||
|
||||
```
|
||||
@software{touvron2023llama2,
|
||||
title={Llama 2: Open Foundation and Fine-Tuned Chat Models},
|
||||
author={Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava,
|
||||
Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,
|
||||
Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez Madian Khabsa, Isabel Kloumann,
|
||||
Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov,
|
||||
Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith,
|
||||
Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu , Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan,
|
||||
Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom},
|
||||
year={2023}
|
||||
}
|
||||
```
|
||||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
||||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_psmathur__model_42_70b)
|
||||
|
||||
| Metric | Value |
|
||||
|-----------------------|---------------------------|
|
||||
| Avg. | 58.2 |
|
||||
| ARC (25-shot) | 68.26 |
|
||||
| HellaSwag (10-shot) | 87.65 |
|
||||
| MMLU (5-shot) | 70.0 |
|
||||
| TruthfulQA (0-shot) | 48.76 |
|
||||
| Winogrande (5-shot) | 83.66 |
|
||||
| GSM8K (5-shot) | 34.72 |
|
||||
| DROP (3-shot) | 14.37 |
|
||||
|
||||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
||||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_psmathur__test_42_70b)
|
||||
|
||||
| Metric |Value|
|
||||
|---------------------------------|----:|
|
||||
|Avg. |67.38|
|
||||
|AI2 Reasoning Challenge (25-Shot)|68.26|
|
||||
|HellaSwag (10-Shot) |87.65|
|
||||
|MMLU (5-Shot) |70.00|
|
||||
|TruthfulQA (0-shot) |48.76|
|
||||
|Winogrande (5-shot) |83.66|
|
||||
|GSM8k (5-shot) |45.94|
|
||||
|
||||
Reference in New Issue
Block a user