72 lines
3.1 KiB
Markdown
72 lines
3.1 KiB
Markdown
---
|
|
license: other
|
|
language:
|
|
- en
|
|
pipeline_tag: text2text-generation
|
|
tags:
|
|
- alpaca
|
|
- llama
|
|
- chat
|
|
- gpt4
|
|
---
|
|
<!-- header start -->
|
|
<div style="width: 100%;">
|
|
<img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
|
|
</div>
|
|
<div style="display: flex; justify-content: space-between; width: 100%;">
|
|
<div style="display: flex; flex-direction: column; align-items: flex-start;">
|
|
<p><a href="https://discord.gg/Jq4vkcDakD">Chat & support: my new Discord server</a></p>
|
|
</div>
|
|
<div style="display: flex; flex-direction: column; align-items: flex-end;">
|
|
<p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
|
|
</div>
|
|
</div>
|
|
<!-- header end -->
|
|
|
|
This is the HF format merged model for [chansung's gpt4-alpaca-lora-13b](https://huggingface.co/chansung/gpt4-alpaca-lora-13b).
|
|
|
|
<!-- footer start -->
|
|
## Discord
|
|
|
|
For further support, and discussions on these models and AI in general, join us at:
|
|
|
|
[TheBloke AI's Discord server](https://discord.gg/Jq4vkcDakD)
|
|
|
|
## Thanks, and how to contribute.
|
|
|
|
Thanks to the [chirper.ai](https://chirper.ai) team!
|
|
|
|
I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
|
|
|
|
If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.
|
|
|
|
Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.
|
|
|
|
* Patreon: https://patreon.com/TheBlokeAI
|
|
* Ko-Fi: https://ko-fi.com/TheBlokeAI
|
|
|
|
**Patreon special mentions**: Aemon Algiz, Dmitriy Samsonov, Nathan LeClaire, Trenton Dambrowitz, Mano Prime, David Flickinger, vamX, Nikolai Manek, senxiiz, Khalefa Al-Ahmad, Illia Dulskyi, Jonathan Leane, Talal Aujan, V. Lukas, Joseph William Delisle, Pyrater, Oscar Rangel, Lone Striker, Luke Pendergrass, Eugene Pentland, Sebastain Graf, Johann-Peter Hartman.
|
|
|
|
Thank you to all my generous patrons and donaters!
|
|
<!-- footer end -->
|
|
# Original model card
|
|
|
|
This repository comes with LoRA checkpoint to make LLaMA into a chatbot like language model. The checkpoint is the output of instruction following fine-tuning process with the following settings on 8xA100(40G) DGX system.
|
|
- Training script: borrowed from the official [Alpaca-LoRA](https://github.com/tloen/alpaca-lora) implementation
|
|
- Training script:
|
|
```shell
|
|
python finetune.py \
|
|
--base_model='decapoda-research/llama-30b-hf' \
|
|
--data_path='alpaca_data_gpt4.json' \
|
|
--num_epochs=10 \
|
|
--cutoff_len=512 \
|
|
--group_by_length \
|
|
--output_dir='./gpt4-alpaca-lora-30b' \
|
|
--lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \
|
|
--lora_r=16 \
|
|
--batch_size=... \
|
|
--micro_batch_size=...
|
|
```
|
|
|
|
You can find how the training went from W&B report [here](https://wandb.ai/chansung18/gpt4_alpaca_lora/runs/w3syd157?workspace=user-chansung18).
|