56 lines
1.8 KiB
Markdown
56 lines
1.8 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
datasets:
|
||
|
|
- stanfordnlp/SHP
|
||
|
|
- Anthropic/hh-rlhf
|
||
|
|
- OpenAssistant/oasst1
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
metrics:
|
||
|
|
- accuracy
|
||
|
|
tags:
|
||
|
|
- human feedback
|
||
|
|
- rlhf
|
||
|
|
- preferences
|
||
|
|
- alignment
|
||
|
|
- HALO
|
||
|
|
- halos
|
||
|
|
- dpo
|
||
|
|
- rl
|
||
|
|
---
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
This repo contains the model checkpoints for:
|
||
|
|
- model family <b>EleutherAI/pythia-2.8b</b>
|
||
|
|
- optimized with the loss <b>PPO</b>
|
||
|
|
- aligned using the SHP, Anthropic HH and Open Assistant datasets.
|
||
|
|
|
||
|
|
To prompt Archangel models, ensure that the format is consistent with that of TuluV2.
|
||
|
|
For example, a prompt should be formatted as follows, where `<|user|>` corresponds to the human's role and `<|assistant|>` corresponds to the LLM's role.
|
||
|
|
The human should speak first:
|
||
|
|
```
|
||
|
|
<|user|>
|
||
|
|
Hi! I'm looking for a cake recipe.
|
||
|
|
<|assistant|>
|
||
|
|
What kind of cake?
|
||
|
|
<|user|>
|
||
|
|
Chocolate cake.
|
||
|
|
<|assistant|>
|
||
|
|
```
|
||
|
|
Note that a beginning-of-sequence (BOS) token is automatically added by all Archangel models during tokenization and does not have to be added by you. No end-of-sequence (EOS) token is added to the prompt.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
Please refer to our [code repository](https://github.com/ContextualAI/HALOs) or [blog](https://contextual.ai/better-cheaper-faster-llm-alignment-with-kto/) which contains intructions for training your own HALOs and links to our model cards.
|
||
|
|
|
||
|
|
If you find this repo or the technical paper useful in your research, please feel free to cite [our work](https://github.com/ContextualAI/HALOs/blob/main/assets/report.pdf):
|
||
|
|
```
|
||
|
|
@techreport{ethayarajh2023halos,
|
||
|
|
author = {Ethayarajh, Kawin and Xu, Winnie, and Jurafsky, Dan and Kiela, Douwe},
|
||
|
|
title = {Human-Centered Loss Functions (HALOs)},
|
||
|
|
institution = {Contextual AI},
|
||
|
|
note = {https://github.com/ContextualAI/HALOs/blob/main/assets/report.pdf},
|
||
|
|
year = {2023},
|
||
|
|
}
|
||
|
|
```
|