43 lines
1.2 KiB
Markdown
43 lines
1.2 KiB
Markdown
|
---
|
||
|
|
license: cc-by-4.0
|
||
|
|
datasets:
|
||
|
|
- Photolens/alpaca-cleaned-airoboros-2.1-no-code-oasst1-en-merged
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
---
|
||
|
|
|
||
|
|
## Model overview
|
||
|
|
This model is finetuned on *[a merged dataset of: oasst1-en, alpaca-cleaned and airoboros-2.1-no-code](https://huggingface.co/datasets/Photolens/alpaca-cleaned-airoboros-2.1-no-code-oasst1-en-merged)* on a base model: *[Marx-3b-V2](https://huggingface.co/acrastt/Marx-3B-V2)*
|
||
|
|
- License: "`Creative-Commons-Attribution-4.0`"
|
||
|
|
- Language: "`en`"
|
||
|
|
- Size: "`3.43b params`"
|
||
|
|
|
||
|
|
## Prompt template
|
||
|
|
Prompt template:
|
||
|
|
```
|
||
|
|
### SYSTEM:
|
||
|
|
<system_prompt_here>
|
||
|
|
|
||
|
|
### HUMAN:
|
||
|
|
<prompter_message_here>
|
||
|
|
|
||
|
|
### INPUT:
|
||
|
|
<input_text_here>
|
||
|
|
|
||
|
|
### RESPONSE:
|
||
|
|
<leave_a_blank_line_here>
|
||
|
|
```
|
||
|
|
*Note: If you dont have a system or input text, do not include the tokens in the prompt.*
|
||
|
|
|
||
|
|
## Training Details
|
||
|
|
This model took `2:40:54` to train in LoRA on a single `A100 40gb` GPU.<br>
|
||
|
|
- *epochs*: `1`
|
||
|
|
- *train batch size*: `8`
|
||
|
|
- *eval batch size*: `8`
|
||
|
|
- *gradient accumulation steps*: `1`
|
||
|
|
- *maximum gradient normal*: `0.3`
|
||
|
|
- *learning rate*: `2e-4`
|
||
|
|
- *weight decay*: `0.001`
|
||
|
|
- *optimizer*: `paged_adamw_32bit`
|
||
|
|
- *learning rate schedule*: `cosine`
|
||
|
|
- *warmup ratio (linear)*: `0.03`
|