231 lines
9.2 KiB
Markdown
231 lines
9.2 KiB
Markdown
---
|
|
license: llama2
|
|
language:
|
|
- en
|
|
tags:
|
|
- summary
|
|
|
|
---
|
|
# Bubo Bubo 13B
|
|
|
|
|
|
|
|
# Prompting
|
|
|
|
## Prompt Template for alpaca style
|
|
|
|
```
|
|
### Instruction:
|
|
|
|
<prompt> (without the <>)
|
|
|
|
### Response:
|
|
```
|
|
|
|
## Sample Code
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
torch.set_default_device("cuda")
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("ibivibiv/bubo-bubo-13b", torch_dtype="auto", device_config='auto')
|
|
tokenizer = AutoTokenizer.from_pretrained("ibivibiv/bubo-bubo-13b")
|
|
|
|
inputs = tokenizer("### Instruction: Summarize this email chain : <email chain stuff here>.\n### Response:\n", return_tensors="pt", return_attention_mask=False)
|
|
|
|
outputs = model.generate(**inputs, max_length=200)
|
|
text = tokenizer.batch_decode(outputs)[0]
|
|
print(text)
|
|
```
|
|
|
|
# Model Details
|
|
* **Trained by**: [ibivibiv](https://huggingface.co/ibivibiv)
|
|
* **Library**: [HuggingFace Transformers](https://github.com/huggingface/transformers)
|
|
* **Model type:** **bubo-bubo-13b** is an auto-regressive language model fine tuned on the Llama 2 transformer architecture.
|
|
* **Language(s)**: English
|
|
* **Purpose**: Has specific training for summary tasks. This model is targeted towards summarizing communication chains specifically.
|
|
|
|
# Benchmark Scores
|
|
|
|
I ran the benchmark harness, for curiousity, but this model is completely geared towards summarizing.
|
|
|
|
| Test Name | Accuracy |
|
|
|------------------------------------------------------|----------------------|
|
|
| all | 0.579149139810157 |
|
|
| arc:challenge | 0.5631399317406144 |
|
|
| hellaswag | 0.6317466640111532 |
|
|
| hendrycksTest-abstract_algebra | 0.32 |
|
|
| hendrycksTest-anatomy | 0.5481481481481482 |
|
|
| hendrycksTest-astronomy | 0.5657894736842105 |
|
|
| hendrycksTest-business_ethics | 0.55 |
|
|
| hendrycksTest-clinical_knowledge | 0.6 |
|
|
| hendrycksTest-college_biology | 0.6388888888888888 |
|
|
| hendrycksTest-college_chemistry | 0.38 |
|
|
| hendrycksTest-college_computer_science | 0.43 |
|
|
| hendrycksTest-college_mathematics | 0.34 |
|
|
| hendrycksTest-college_medicine | 0.5260115606936416 |
|
|
| hendrycksTest-college_physics | 0.3431372549019608 |
|
|
| hendrycksTest-computer_security | 0.71 |
|
|
| hendrycksTest-conceptual_physics | 0.49361702127659574 |
|
|
| hendrycksTest-econometrics | 0.35964912280701755 |
|
|
| hendrycksTest-electrical_engineering | 0.5586206896551724 |
|
|
| hendrycksTest-elementary_mathematics | 0.3439153439153439 |
|
|
| hendrycksTest-formal_logic | 0.3333333333333333 |
|
|
| hendrycksTest-global_facts | 0.42 |
|
|
| hendrycksTest-high_school_biology | 0.6903225806451613 |
|
|
| hendrycksTest-high_school_chemistry | 0.45320197044334976 |
|
|
| hendrycksTest-high_school_computer_science | 0.58 |
|
|
| hendrycksTest-high_school_european_history | 0.6787878787878788 |
|
|
| hendrycksTest-high_school_geography | 0.7424242424242424 |
|
|
| hendrycksTest-high_school_government_and_politics | 0.8341968911917098 |
|
|
| hendrycksTest-high_school_macroeconomics | 0.558974358974359 |
|
|
| hendrycksTest-high_school_mathematics | 0.3 |
|
|
| hendrycksTest-high_school_microeconomics | 0.5672268907563025 |
|
|
| hendrycksTest-high_school_physics | 0.33112582781456956 |
|
|
| hendrycksTest-high_school_psychology | 0.7577981651376147 |
|
|
| hendrycksTest-high_school_statistics | 0.4212962962962963 |
|
|
| hendrycksTest-high_school_us_history | 0.8186274509803921 |
|
|
| hendrycksTest-high_school_world_history | 0.759493670886076 |
|
|
| hendrycksTest-human_aging | 0.6547085201793722 |
|
|
| hendrycksTest-human_sexuality | 0.6412213740458015 |
|
|
| hendrycksTest-international_law | 0.6776859504132231 |
|
|
| hendrycksTest-jurisprudence | 0.75 |
|
|
| hendrycksTest-logical_fallacies | 0.6993865030674846 |
|
|
| hendrycksTest-machine_learning | 0.41964285714285715 |
|
|
| hendrycksTest-management | 0.7281553398058253 |
|
|
| hendrycksTest-marketing | 0.8504273504273504 |
|
|
| hendrycksTest-medical_genetics | 0.6 |
|
|
| hendrycksTest-miscellaneous | 0.7624521072796935 |
|
|
| hendrycksTest-moral_disputes | 0.6560693641618497 |
|
|
| hendrycksTest-moral_scenarios | 0.4346368715083799 |
|
|
| hendrycksTest-nutrition | 0.673202614379085 |
|
|
| hendrycksTest-philosophy | 0.7009646302250804 |
|
|
| hendrycksTest-prehistory | 0.7067901234567902 |
|
|
| hendrycksTest-professional_accounting | 0.4645390070921986 |
|
|
| hendrycksTest-professional_law | 0.45697522816166886 |
|
|
| hendrycksTest-professional_medicine | 0.5514705882352942 |
|
|
| hendrycksTest-professional_psychology | 0.6013071895424836 |
|
|
| hendrycksTest-public_relations | 0.6636363636363637 |
|
|
| hendrycksTest-security_studies | 0.6448979591836734 |
|
|
| hendrycksTest-sociology | 0.7611940298507462 |
|
|
| hendrycksTest-us_foreign_policy | 0.84 |
|
|
| hendrycksTest-virology | 0.4819277108433735 |
|
|
| hendrycksTest-world_religions | 0.7894736842105263 |
|
|
| truthfulqa:mc | 0.4762440289139372 |
|
|
| winogrande | 0.7616416732438832 |
|
|
| gsm8k | 0.20621683093252463 |
|
|
|
|
|
|
## Citations
|
|
|
|
```
|
|
@misc{open-llm-leaderboard,
|
|
author = {Edward Beeching and Clémentine Fourrier and Nathan Habib and Sheon Han and Nathan Lambert and Nazneen Rajani and Omar Sanseviero and Lewis Tunstall and Thomas Wolf},
|
|
title = {Open LLM Leaderboard},
|
|
year = {2023},
|
|
publisher = {Hugging Face},
|
|
howpublished = "\url{https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard}"
|
|
}
|
|
```
|
|
```
|
|
@software{eval-harness,
|
|
author = {Gao, Leo and
|
|
Tow, Jonathan and
|
|
Biderman, Stella and
|
|
Black, Sid and
|
|
DiPofi, Anthony and
|
|
Foster, Charles and
|
|
Golding, Laurence and
|
|
Hsu, Jeffrey and
|
|
McDonell, Kyle and
|
|
Muennighoff, Niklas and
|
|
Phang, Jason and
|
|
Reynolds, Laria and
|
|
Tang, Eric and
|
|
Thite, Anish and
|
|
Wang, Ben and
|
|
Wang, Kevin and
|
|
Zou, Andy},
|
|
title = {A framework for few-shot language model evaluation},
|
|
month = sep,
|
|
year = 2021,
|
|
publisher = {Zenodo},
|
|
version = {v0.0.1},
|
|
doi = {10.5281/zenodo.5371628},
|
|
url = {https://doi.org/10.5281/zenodo.5371628}
|
|
}
|
|
```
|
|
```
|
|
@misc{clark2018think,
|
|
title={Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge},
|
|
author={Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord},
|
|
year={2018},
|
|
eprint={1803.05457},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.AI}
|
|
}
|
|
```
|
|
```
|
|
@misc{zellers2019hellaswag,
|
|
title={HellaSwag: Can a Machine Really Finish Your Sentence?},
|
|
author={Rowan Zellers and Ari Holtzman and Yonatan Bisk and Ali Farhadi and Yejin Choi},
|
|
year={2019},
|
|
eprint={1905.07830},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CL}
|
|
}
|
|
```
|
|
```
|
|
@misc{hendrycks2021measuring,
|
|
title={Measuring Massive Multitask Language Understanding},
|
|
author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},
|
|
year={2021},
|
|
eprint={2009.03300},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CY}
|
|
}
|
|
```
|
|
```
|
|
@misc{lin2022truthfulqa,
|
|
title={TruthfulQA: Measuring How Models Mimic Human Falsehoods},
|
|
author={Stephanie Lin and Jacob Hilton and Owain Evans},
|
|
year={2022},
|
|
eprint={2109.07958},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CL}
|
|
}
|
|
```
|
|
```
|
|
@misc{DBLP:journals/corr/abs-1907-10641,
|
|
title={{WINOGRANDE:} An Adversarial Winograd Schema Challenge at Scale},
|
|
author={Keisuke Sakaguchi and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi},
|
|
year={2019},
|
|
eprint={1907.10641},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CL}
|
|
}
|
|
```
|
|
```
|
|
@misc{DBLP:journals/corr/abs-2110-14168,
|
|
title={Training Verifiers to Solve Math Word Problems},
|
|
author={Karl Cobbe and
|
|
Vineet Kosaraju and
|
|
Mohammad Bavarian and
|
|
Mark Chen and
|
|
Heewoo Jun and
|
|
Lukasz Kaiser and
|
|
Matthias Plappert and
|
|
Jerry Tworek and
|
|
Jacob Hilton and
|
|
Reiichiro Nakano and
|
|
Christopher Hesse and
|
|
John Schulman},
|
|
year={2021},
|
|
eprint={2110.14168},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CL}
|
|
}
|
|
```
|