104 lines
6.4 KiB
Markdown
104 lines
6.4 KiB
Markdown
---
|
|
inference: false
|
|
license: other
|
|
---
|
|
|
|
<!-- header start -->
|
|
<div style="width: 100%;">
|
|
<img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
|
|
</div>
|
|
<div style="display: flex; justify-content: space-between; width: 100%;">
|
|
<div style="display: flex; flex-direction: column; align-items: flex-start;">
|
|
<p><a href="https://discord.gg/theblokeai">Chat & support: my new Discord server</a></p>
|
|
</div>
|
|
<div style="display: flex; flex-direction: column; align-items: flex-end;">
|
|
<p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
|
|
</div>
|
|
</div>
|
|
<!-- header end -->
|
|
|
|
# Camel AI's CAMEL 13B Combined Data fp16
|
|
|
|
This is fp16 pytorch format model files for [Camel AI's CAMEL 13B Combined Data](https://huggingface.co/camel-ai/CAMEL-13B-Combined-Data) merged with [Kaio Ken's SuperHOT 8K](https://huggingface.co/kaiokendev/superhot-13b-8k-no-rlhf-test).
|
|
|
|
[Kaio Ken's SuperHOT 13b LoRA](https://huggingface.co/kaiokendev/superhot-13b-8k-no-rlhf-test) is merged on to the base model, and then 8K context can be achieved during inference by using `trust_remote_code=True`.
|
|
|
|
Note that `config.json` has been set to a sequence length of 8192. This can be modified to 4096 if you want to try with a smaller sequence length.
|
|
|
|
## Repositories available
|
|
|
|
* [4-bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/CAMEL-13B-Combined-Data-SuperHOT-8K-GPTQ)
|
|
* [Unquantised SuperHOT fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/TheBloke/CAMEL-13B-Combined-Data-SuperHOT-8K-fp16)
|
|
* [Unquantised base fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/camel-ai/CAMEL-13B-Combined-Data)
|
|
|
|
<!-- footer start -->
|
|
## Discord
|
|
|
|
For further support, and discussions on these models and AI in general, join us at:
|
|
|
|
[TheBloke AI's Discord server](https://discord.gg/theblokeai)
|
|
|
|
## Thanks, and how to contribute.
|
|
|
|
Thanks to the [chirper.ai](https://chirper.ai) team!
|
|
|
|
I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
|
|
|
|
If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.
|
|
|
|
Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.
|
|
|
|
* Patreon: https://patreon.com/TheBlokeAI
|
|
* Ko-Fi: https://ko-fi.com/TheBlokeAI
|
|
|
|
**Special thanks to**: Luke from CarbonQuill, Aemon Algiz, Dmitriy Samsonov.
|
|
|
|
**Patreon special mentions**: Pyrater, WelcomeToTheClub, Kalila, Mano Prime, Trenton Dambrowitz, Spiking Neurons AB, Pierre Kircher, Fen Risland, Kevin Schuppel, Luke, Rainer Wilmers, vamX, Gabriel Puliatti, Alex , Karl Bernard, Ajan Kanaga, Talal Aujan, Space Cruiser, ya boyyy, biorpg, Johann-Peter Hartmann, Asp the Wyvern, Ai Maven, Ghost , Preetika Verma, Nikolai Manek, trip7s trip, John Detwiler, Fred von Graf, Artur Olbinski, subjectnull, John Villwock, Junyu Yang, Rod A, Lone Striker, Chris McCloskey, Iucharbius , Matthew Berman, Illia Dulskyi, Khalefa Al-Ahmad, Imad Khwaja, chris gileta, Willem Michiel, Greatston Gnanesh, Derek Yates, K, Alps Aficionado, Oscar Rangel, David Flickinger, Luke Pendergrass, Deep Realms, Eugene Pentland, Cory Kujawski, terasurfer , Jonathan Leane, senxiiz, Joseph William Delisle, Sean Connelly, webtim, zynix , Nathan LeClaire.
|
|
|
|
Thank you to all my generous patrons and donaters!
|
|
|
|
<!-- footer end -->
|
|
|
|
# Original model card: Kaio Ken's SuperHOT 8K
|
|
|
|
### SuperHOT Prototype 2 w/ 8K Context
|
|
|
|
This is a second prototype of SuperHOT, this time 30B with 8K context and no RLHF, using the same technique described in [the github blog](https://kaiokendev.github.io/til#extending-context-to-8k).
|
|
Tests have shown that the model does indeed leverage the extended context at 8K.
|
|
|
|
You will need to **use either the monkeypatch** or, if you are already using the monkeypatch, **change the scaling factor to 0.25 and the maximum sequence length to 8192**
|
|
|
|
#### Looking for Merged & Quantized Models?
|
|
- 30B 4-bit CUDA: [tmpupload/superhot-30b-8k-4bit-safetensors](https://huggingface.co/tmpupload/superhot-30b-8k-4bit-safetensors)
|
|
- 30B 4-bit CUDA 128g: [tmpupload/superhot-30b-8k-4bit-128g-safetensors](https://huggingface.co/tmpupload/superhot-30b-8k-4bit-128g-safetensors)
|
|
|
|
|
|
#### Training Details
|
|
I trained the LoRA with the following configuration:
|
|
- 1200 samples (~400 samples over 2048 sequence length)
|
|
- learning rate of 3e-4
|
|
- 3 epochs
|
|
- The exported modules are:
|
|
- q_proj
|
|
- k_proj
|
|
- v_proj
|
|
- o_proj
|
|
- no bias
|
|
- Rank = 4
|
|
- Alpha = 8
|
|
- no dropout
|
|
- weight decay of 0.1
|
|
- AdamW beta1 of 0.9 and beta2 0.99, epsilon of 1e-5
|
|
- Trained on 4-bit base model
|
|
|
|
# Original model card: Camel AI's CAMEL 13B Combined Data
|
|
|
|
CAMEL-13B-Combined-Data is a chat large language model obtained by finetuning LLaMA-13B model on a total of 229K conversations collected through our [CAMEL](https://arxiv.org/abs/2303.17760) framework, 100K English public conversations from ShareGPT that can be found [here](https://github.com/lm-sys/FastChat/issues/90#issuecomment-1493250773), and 52K instructions from Alpaca dataset that can be found [here](https://github.com/tatsu-lab/stanford_alpaca/blob/761dc5bfbdeeffa89b8bff5d038781a4055f796a/alpaca_data.json). We evaluate our model offline using EleutherAI's language model evaluation harness used by Huggingface's Open LLM Benchmark. CAMEL<sup>*</sup>-13B scores an average of 58.9.
|
|
|
|
| Model | size | ARC-C (25 shots, acc_norm) | HellaSwag (10 shots, acc_norm) | MMLU (5 shots, acc_norm) | TruthfulQA (0 shot, mc2) | Average | Delta |
|
|
|-------------|:----:|:---------------------------:|:-------------------------------:|:-------------------------:|:-------------------------:|:-------:|-------|
|
|
| LLaMA | 13B | 56.3 | 80.9 | 46.7 | 39.9 | 56.0 | - |
|
|
| Vicuna | 13B | 52.8 | 80.1 | 50.5 | 51.8 | 58.8 | 2.8 |
|
|
| CAMEL<sup>*</sup> | 13B | 56.1 | 79.9 | 50.5 | 49.0 | 58.9 | 2.9 |
|
|
|