58 lines
1.7 KiB
Markdown
58 lines
1.7 KiB
Markdown
|
---
|
||
|
|
license: cc-by-nc-4.0
|
||
|
|
datasets:
|
||
|
|
- pankajmathur/orca_mini_v1_dataset
|
||
|
|
- openai/summarize_from_feedback
|
||
|
|
- PygmalionAI/PIPPA
|
||
|
|
- chargoddard/rpguild
|
||
|
|
- lemonilia/LimaRP
|
||
|
|
- PKU-Alignment/PKU-SafeRLHF
|
||
|
|
- Intel/orca_dpo_pairs
|
||
|
|
- allenai/ultrafeedback_binarized_cleaned
|
||
|
|
tags:
|
||
|
|
- merge
|
||
|
|
- mergekit
|
||
|
|
---
|
||
|
|
|
||
|
|
Another experiment in the line of [loyal-piano-m7](https://huggingface.co/chargoddard/loyal-piano-m7).
|
||
|
|
|
||
|
|
Steps taken to produce this model:
|
||
|
|
|
||
|
|
* Train loyal-piano-m7
|
||
|
|
* cDPO with HuggingFaceH4/ultrafeedback_binarized to produce loyal-piano-m7-cdpo
|
||
|
|
* Train another model with different sampling of the same source datasets as loyal-piano, let's call it servile-harpsichord
|
||
|
|
* cDPO servile-harpsichord with allenai/ultrafeedback_binarized_cleaned, Intel/orca_dpo_pairs, and a helpfulness-only version of PKU-Alignment/PKU-SafeRLHF
|
||
|
|
* TIES merge several checkpoints of servile-harpsichord-cdpo with loyal-piano-m7-cdpo
|
||
|
|
|
||
|
|
Local benchmarks show the result to be better than any of the individual components. Let's see if that holds up!
|
||
|
|
|
||
|
|
Trained using the Alpaca prompt format.
|
||
|
|
|
||
|
|
|
||
|
|
Configuration for final merge:
|
||
|
|
```yml
|
||
|
|
models:
|
||
|
|
- model: chargoddard/loyal-piano-m7-cdpo
|
||
|
|
parameters:
|
||
|
|
density: 1.0
|
||
|
|
weight: 1.0
|
||
|
|
- model: /home/ubuntu/servile-harpsichord-cdpo/checkpoint-4186
|
||
|
|
parameters:
|
||
|
|
weight: 0.1
|
||
|
|
- model: /home/ubuntu/servile-harpsichord-cdpo/checkpoint-5796
|
||
|
|
parameters:
|
||
|
|
weight: 0.2
|
||
|
|
- model: /home/ubuntu/servile-harpsichord-cdpo/checkpoint-6118
|
||
|
|
parameters:
|
||
|
|
weight: 0.3
|
||
|
|
- model: /home/ubuntu/servile-harpsichord-cdpo/final
|
||
|
|
parameters:
|
||
|
|
weight: 0.4
|
||
|
|
merge_method: ties
|
||
|
|
base_model: mistralai/Mistral-7B-v0.1
|
||
|
|
dtype: bfloat16
|
||
|
|
parameters:
|
||
|
|
density: 0.4
|
||
|
|
normalize: true
|
||
|
|
int8_mask: true
|
||
|
|
```
|