76 lines
1.6 KiB
Markdown
76 lines
1.6 KiB
Markdown
|
---
|
||
|
|
library_name: transformers
|
||
|
|
tags:
|
||
|
|
- dpo
|
||
|
|
- mistral
|
||
|
|
- finetune
|
||
|
|
license: apache-2.0
|
||
|
|
datasets:
|
||
|
|
- Intel/orca_dpo_pairs
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
---
|
||
|
|
|
||
|
|
# NeuralHermes 2.5 - Mistral 7B
|
||
|
|
|
||
|
|
NeuralHermes-2.5 was created by fine-tuning OpenHermes-2.5 using a RLHF-like technique: Direct Preference Optimization (DPO) using the `Intel/orca_dpo_pairs` dataset.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
You can also run this model using the following code:
|
||
|
|
|
||
|
|
```python
|
||
|
|
import transformers
|
||
|
|
from transformers import AutoTokenizer
|
||
|
|
|
||
|
|
# Format prompt
|
||
|
|
message = [
|
||
|
|
{"role": "system", "content": "You are a helpful assistant chatbot."},
|
||
|
|
{"role": "user", "content": "What is a Large Language Model?"}
|
||
|
|
]
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(new_model)
|
||
|
|
prompt = tokenizer.apply_chat_template(message, add_generation_prompt=True, tokenize=False)
|
||
|
|
|
||
|
|
# Create pipeline
|
||
|
|
pipeline = transformers.pipeline(
|
||
|
|
"text-generation",
|
||
|
|
model=new_model,
|
||
|
|
tokenizer=tokenizer
|
||
|
|
)
|
||
|
|
|
||
|
|
# Generate text
|
||
|
|
sequences = pipeline(
|
||
|
|
prompt,
|
||
|
|
do_sample=True,
|
||
|
|
temperature=0.7,
|
||
|
|
top_p=0.9,
|
||
|
|
num_return_sequences=1,
|
||
|
|
max_length=200,
|
||
|
|
)
|
||
|
|
print(sequences[0]['generated_text'])
|
||
|
|
```
|
||
|
|
|
||
|
|
## Training hyperparameters
|
||
|
|
|
||
|
|
**LoRA**:
|
||
|
|
* r=16
|
||
|
|
* lora_alpha=16
|
||
|
|
* lora_dropout=0.05
|
||
|
|
* bias="none"
|
||
|
|
* task_type="CAUSAL_LM"
|
||
|
|
* target_modules=['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']
|
||
|
|
|
||
|
|
**Training arguments**:
|
||
|
|
* per_device_train_batch_size=4
|
||
|
|
* gradient_accumulation_steps=4
|
||
|
|
* gradient_checkpointing=True
|
||
|
|
* learning_rate=5e-5
|
||
|
|
* lr_scheduler_type="cosine"
|
||
|
|
* max_steps=200
|
||
|
|
* optim="paged_adamw_32bit"
|
||
|
|
* warmup_steps=100
|
||
|
|
|
||
|
|
**DPOTrainer**:
|
||
|
|
* beta=0.1
|
||
|
|
* max_prompt_length=1024
|
||
|
|
* max_length=1536
|